Human body action intelligent coaching method, storage medium and electronic equipment
Through the monocular acquisition device and neural network model combined with the alignment processing of inertial sensors, the problem of incomplete motion information in motion capture is solved, and the precise capture of high-precision wrist joint movements and the complete collection of motion information is achieved, reducing costs.
Patent Information
- Application Number
- CN202411487053.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2024-12-20
AI Technical Summary
The prior art is difficult to obtain complete motion information during the motion capture process, especially when high-precision wrist joint movements and handheld prop movements are required, it is difficult to achieve accurate motion capture.
The whole-body motion video data is collected through the monocular acquisition device, the subject joint motion information is identified using the neural network model, and the wrist joint motion information is obtained by combining the inertial sensors in the handheld device, and the alignment process is performed to generate target motion information.
It realizes complete capture of movements, especially when high-precision required movements and handheld props, accurate wrist joint movement information can be obtained, reducing costs, and achieving action collection under popular conditions.
Smart Images
Figure CN119169703A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of wearable intelligent equipment, and in particular, to a method and device for intelligent coaching of human movements, a storage medium, and an electronic device. Background Art
[0002] People often need to learn movements: sports, technology, fitness, rehabilitation, etc. However, everyone has different abilities to learn movements, and some progress faster than others. The coach’s time is precious, and he cannot provide movement guidance all the time. Therefore, it is a feasible solution to design an AI-based movement coach to provide targeted, full-time guidance and movement suggestions for each person’s learning status. Even for some innovative movements that no coach can guide, a virtual coach can guide them on a virtual platform and then guide students to learn. Summary of the invention
[0003] The present disclosure provides a motion capture method, device, electronic device and storage medium to at least solve the problem in the related art that it is difficult to obtain complete motion information during the motion capture process. The technical solution of the present disclosure is as follows: According to a first aspect of an embodiment of the present disclosure, a motion capture method is provided, comprising: Collect full-body motion video data of the target object through a monocular acquisition device; The neural network model is used to identify the whole-body motion video data of the target object and obtain the subject's joint motion information. The main body joints include all joints in the human body except the wrist joint; receiving wrist joint motion information output by a handheld device on the target object, wherein the handheld device is provided with an inertial sensor; The main body joint motion information and the wrist joint motion information are aligned to obtain the target action information of the target object.
[0004] In one embodiment, the body joint motion information and the wrist joint motion information are aligned to obtain the target motion information of the target object, including: Obtaining subject joint motion information corresponding to each frame of video image in the whole-body motion video data; For each frame of video image in the whole-body motion video data, obtaining wrist joint motion information that is closest in time to each frame of video image as the wrist joint motion information corresponding to each frame of video image; According to the main body joint motion information and wrist joint motion information corresponding to each frame of video image, a frame of target action information is generated.
[0005] In one embodiment, the main body joint motion information includes the rotation angles of multiple joints, and the wrist joint motion information includes the wrist joint direction; according to the main body joint motion information and the wrist joint motion information corresponding to each frame of video image, a frame of target action information is generated, including: Obtaining the order of multiple pre-configured joints, wherein the multiple joints include an elbow joint, and the order of the elbow joint is the last one among the multiple joints; According to the order of multiple joints, each joint is rotated in turn according to the rotation angle of each joint, until the elbow joint is rotated to obtain the direction of the elbow joint as the arm direction; Obtain the rotation angle between the wrist joint direction and the arm direction as the rotation angle of the wrist joint; The rotation angles of multiple joints and the rotation angle of the wrist joint corresponding to each frame of video image are used as one frame of target action information.
[0006] In one embodiment, the handheld device includes a left-hand handheld device and a right-hand handheld device; the body joint motion information and the wrist joint motion information are aligned to obtain target motion information of the target object, including: The main body joint motion information, the wrist joint motion information corresponding to the left hand holding device and the wrist joint motion information corresponding to the right hand holding device are aligned to obtain the target action information of the target object.
[0007] In one embodiment, before receiving the wrist joint motion information output by the handheld device on the target object, the method further includes: receiving a calibration request from a handheld device; In response to the calibration request, the rotation direction of the handheld device in the world coordinate system is zeroed and calibrated, and the world coordinate system is the coordinate system of the monocular acquisition device.
[0008] In one embodiment, in response to a calibration request, performing zero calibration on the rotation direction of the handheld device in the world coordinate system includes: In response to the calibration request, the rotation Euler angles of the handheld device are reset to zero, wherein, when performing the zeroing calibration, the handheld device is vertically pointed to the monocular acquisition device, and the screen of the handheld device faces directly upward.
[0009] In one embodiment, the inertial sensor is a gyroscope.
[0010] According to a second aspect of an embodiment of the present disclosure, a motion capture device is provided, characterized in that it includes: A collection module is configured to collect full-body motion video data of a target object through a monocular collection device; The recognition module is configured to perform full body motion video data recognition of the target object through a neural network model, Obtaining the subject joint motion information, where the subject joints include other joints in the human body except the wrist joint; The receiving module is configured to receive wrist joint motion information output by a handheld device on the target object, The handheld device is provided with an inertial sensor; The motion information generating module is configured to perform alignment processing on the body joint motion information and the wrist joint motion information to obtain the target motion information of the target object.
[0011] In one embodiment, the action information generating module includes: An acquisition module is configured to execute to acquire the body joint motion information corresponding to each frame of video image in the whole body action video data; an alignment module is configured to execute to acquire the wrist joint motion information closest in time to each frame of video image in the whole body action video data as the wrist joint motion information corresponding to each frame of video image; A frame action information generation module is configured to generate a frame of target action information according to the main body joint motion information and wrist joint motion information corresponding to each frame of video image.
[0012] In one embodiment, the body joint motion information includes the rotation angles of multiple joints, and the wrist joint motion information includes the wrist joint direction; a frame motion information generation module includes: An acquisition unit is configured to acquire a pre-configured sequence of multiple joints, wherein the multiple joints include an elbow joint, and the order of the elbow joint is the last one among the multiple joints; The arm direction generating unit is configured to perform rotation processing on each joint in sequence according to the order of the plurality of joints, until the elbow joint is rotated to obtain the elbow joint direction as the arm direction; A wrist joint rotation angle generating unit is configured to obtain a rotation angle between a wrist joint direction and an arm direction as a rotation angle of the wrist joint; A frame action information generating unit is configured to execute the rotation angles of multiple joints and the rotation angle of the wrist joint corresponding to each frame of video image as a frame of target action information.
[0013] In one embodiment, the handheld device includes a left-hand handheld device and a right-handheld device; the action information generation module is configured to perform alignment processing on the main body joint motion information, the wrist joint motion information corresponding to the left-hand handheld device, and the wrist joint motion information corresponding to the right-hand handheld device to obtain the target action information of the target object.
[0014] In one embodiment, the receiving module is further configured to receive a calibration request sent by the handheld device; The device also includes a calibration module configured to execute a zero calibration on the rotation direction of the handheld device in a world coordinate system in response to a calibration request, where the world coordinate system is a coordinate system of the monocular acquisition device.
[0015] In one embodiment, the calibration module is configured to execute zeroing of the rotation Euler angles of the handheld device in response to a calibration request, wherein, when performing the zeroing calibration, the handheld device is vertically pointed to the monocular acquisition device and the screen of the handheld device is facing upward.
[0016] In one embodiment, the inertial sensor is a gyroscope.
[0017] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the motion capture method as described in any one of the embodiments of the first aspect.
[0018] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the motion capture method as described in any one of the embodiments of the first aspect.
[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device executes the motion capture method described in any one of the embodiments of the first aspect.
[0020] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: The whole-body motion video data of the target object is collected by a monocular acquisition device; the whole-body motion video data is recognized by a neural network model to obtain the main body joint motion information; the wrist joint motion information is collected by a handheld device equipped with an inertial sensor; the main body joint motion information and the wrist joint motion information are aligned to obtain the target motion information. According to the scheme disclosed in the present invention, on the one hand, a real-time motion capture system based on multimodal input is used to respectively collect the main body joint motion information and the wrist joint motion information. When the performer makes actions with high-precision requirements on the wrist direction (such as saluting, clapping), and holds props (such as swinging a sword, raising a glass), etc., the target motion information containing accurate wrist joint motion information can be obtained, thereby achieving complete motion capture. On the other hand, the above technical scheme performs motion capture based on a pure visual method, which realizes complete motion acquisition at a civilian cost.
[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 is a structural diagram of the disclosed human movement intelligent coach; Figure 2 It is a schematic diagram of the process of the disclosed human movement intelligent coach; Figure 3 is a structural block diagram of the data acquisition module of the present disclosure; Figure 4 It is a structural block diagram of the data analysis unit of the present disclosure.
Claims
1. A method for intelligent coaching of human movements, characterized in that: include: Acquire motion data of raw human motions collected by an inertial sensor array; Generating motion images from the original human motion data, including student motion images generated from the original human motion data of students, and coach motion images generated from the original human motion data of coaches; combining the student's action image and the coach's action image into a combined action image; Inputting the combined action image into a trained convolutional neural network; The convolutional neural network output controls the action motor to guide the student to perform the correct action.
2. The method according to claim 1, characterized in that The inertial sensor array comprises: More than three inertial sensors, forming an inertial sensor array, are installed at synonymous points of the human body; The inertial sensor communicates with the data transceiver unit via a wireless method; Each sensor in the inertial sensor array is mounted in a fixed orientation.
3. The method according to claim 1, characterized in that The original human body motions collected by the inertial sensor array include: Each sensor in the inertial sensor array has unified timing and the same data collection frequency; When the inertial sensor uploads data to the data collection unit, each frame of data is timestamped; The inertial sensor array collects raw human body motions at multiple locations in the same time period; Divide the time period of the action equally to generate a standard acquisition time series; Based on the data from each inertial sensor, resampled data is generated at the time points of the standard acquisition time series.
4. The method according to claim 1, characterized in that: The motion image generated by the original human motion data has three dimensions: (sensor serial number, sensor data, time series).
5. The method according to claim 4, characterized in that The three-dimensional action image includes: The data of the inertial sensor array at each time point is stitched into two-dimensional data; Combine the two-dimensional motion data into three-dimensional data according to the standard acquisition time sequence; The three-dimensional data is used as a human motion image.
6. The method according to claim 4, characterized in that The motion image is normalized in the sensor data dimension.
7. The method according to claim 1, characterized in that The convolutional neural network output corresponds to an IO port, the IO port is connected to a vibration motor array, each dimension of the convolutional neural network output corresponds to each motor of the vibration motor array, and the convolutional neural network guides students to perform correct movements by controlling the vibration of the vibration motor.
8. The method according to claim 7, characterized in that Each motor of the vibration motor array is installed at a synonymous point of the human body.
9. The method according to claim 1, characterized in that: The convolutional neural network is trained by using a combined action image of a coach's action and a student's action as input, and the coach's guidance of the student's action as a label to form an input-label pair.
10. A method and device for intelligent human movement coaching, a storage medium, and an electronic device, characterized in that: include: A data acquisition module acquires the original human motion data collected by the inertial sensor array; The data transceiver unit collects the motion data of the inertial sensor array to generate a motion image, receives the output of the convolutional neural network, and sends each data in the output to the corresponding vibration motor; A data analysis unit combines the student action image and the coach action image into a combined action image, inputs the combined action image into a convolutional neural network, and transmits the output of the convolutional neural network to a data transceiver unit; The action guidance unit controls the vibration motor according to the output of the convolutional neural network to guide students to perform correct actions; Data storage unit, which stores action images and the output of convolutional neural network.