Limb motion disassembling capturing system based on multi-modal analysis

Abstract

The application discloses a limb action disassembly capturing system based on multi-modal analysis and relates to the technical field of limb action disassembly capturing, and solves the technical problem that in the prior art, the limb action disassembly capturing cannot be selected through multi-modal acquisition and analysis to improve the capturing efficiency of the limb action, specifically, a multi-modal analysis module records the whole limb action and selects the action through multi-modal analysis to obtain a multi-modal analysis period; after the multi-modal analysis period is obtained, an action disassembly recognition unit analyzes the action execution subject of the multi-modal analysis period; an action execution track is obtained according to the analysis; after the action track is set, the action track is sent to an action track distinguishing unit; after receiving, the action track distinguishing unit distinguishes the action track of the action execution subject.

Description

Technical Field

[0001] This invention relates to the field of limb motion capture technology, specifically a limb motion capture system based on multimodal analysis. Background Technology

[0002] Multimodal analysis is an information processing technology that integrates multiple data modalities (such as text, images, audio, video, sensor data, etc.). By collaboratively analyzing the complementary and correlated features of different modal data, it achieves a more comprehensive and accurate cognitive understanding. The limb movement decomposition and capture system is an intelligent system that integrates multiple sensor data (such as vision, inertial measurement unit IMU, pressure sensor, etc.) and achieves high-precision movement decomposition and real-time capture through deep learning and sensor fusion technology. Its core objective is to accurately analyze the spatiotemporal characteristics of complex movements through the collaborative analysis of multi-dimensional data, providing scientific basis and technical support for fields such as sports training, medical rehabilitation, and virtual reality.

[0003] However, in existing technologies, when capturing body movements, it is impossible to select movements through multimodal acquisition and analysis to improve the efficiency of body movement capture. Furthermore, it is impossible to analyze and divide execution points according to the multimodal analysis period, which cannot guarantee the accuracy of movement capture. In addition, it is impossible to distinguish and process movement trajectories, which cannot ensure the efficiency of movement processing and cannot reduce the workload of capturing unnecessary movements.

[0004] To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0005] The purpose of this invention is to solve the problems mentioned above by proposing a limb motion decomposition and capture system based on multimodal analysis.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] The limb motion decomposition and capture system based on multimodal analysis includes a motion capture platform, wherein the motion capture platform is communicatively connected to a multimodal analysis module, and the multimodal analysis platform is connected to a motion decomposition and recognition unit and a motion trajectory differentiation unit.

[0008] The multimodal analysis module records the entire limb movement and performs movement segmentation through multimodal analysis to obtain the multimodal analysis time segment;

[0009] After obtaining the multimodal analysis time period, the action decomposition and recognition unit performs action execution subject analysis on the multimodal analysis time period; and obtains the action execution trajectory based on the analysis;

[0010] After the motion trajectory is set, it is sent to the motion trajectory differentiation unit; after receiving it, the motion trajectory differentiation unit differentiates the motion trajectory of the motion execution subject.

[0011] In a preferred embodiment of the present invention, the process of the multimodal analysis module is as follows:

[0012] The action execution subject is identified, and action video of the action execution subject is captured using multimodal technology. Based on the analysis of the captured video, the action execution subject is segmented and the segmented video is used as the action decomposition and capture video. The captured video is obtained, and the location of the action execution subject and the background of the captured video are identified and analyzed. Based on the action execution subject, the action needs to be decomposed and captured to obtain the scene required for action execution.

[0013] First, determine the execution location of the action subject based on the scenario required for the action execution, and then determine the corresponding point of the action subject, ensuring a one-to-one correspondence.

[0014] As a preferred embodiment of the present invention, the overlapping time of the execution position and the action execution subject point is obtained, and the cumulative value of the number of overlaps of the execution position and the action execution subject point corresponding to the overlapping time is obtained. At the same time, the floating frequency of the number of points that maintain the overlapping state at adjacent overlapping times after the execution position and the action execution subject point overlap is obtained.

[0015] If the cumulative value of the number of overlapping points of the execution position and the main body of the action at the overlapping time exceeds the cumulative value threshold, and the floating frequency of the number of points that remain overlapping at adjacent overlapping times after the execution position and the main body of the action overlap does not exceed the floating frequency threshold, then the current overlapping time is marked as the preset selected time.

[0016] If the cumulative value of the number of overlapping points of the execution position and the main body of the action at the overlapping time does not exceed the cumulative value threshold, or if the frequency of the number of points that remain overlapping at adjacent overlapping times after the execution position and the main body of the action overlap exceeds the frequency of the fluctuation, then the current overlapping time will be marked as a non-preset selected time.

[0017] In a preferred embodiment of the present invention, after obtaining the preset selected time, the overlapping points of adjacent times after the preset selected time are monitored. If the overlapping points continue to increase and the adjacent times of the already overlapping points are always overlapping, the current preset node time is marked as the set node time. If the overlapping points do not continue to increase or the adjacent times of the already overlapping points are always overlapping, the time analysis continues.

[0018] In a preferred embodiment of the present invention, after obtaining the set node time, the background of the scene where the action execution subject point is located corresponding to the overlapping time is compared. If the background parameters are consistent with the background parameters of the action execution subject point, the video segment is collected at the current set node time, and the video segment ends when the number of overlapping points between the action execution subject point and the execution position rapidly decreases, thus obtaining the multimodal analysis period.

[0019] In a preferred embodiment of the present invention, the process of the action disassembly and identification unit is as follows:

[0020] Based on the multimodal analysis period, point trajectory analysis is performed on the action execution subject. In the point trajectory, the points where the position moves as a whole are marked as main joint points, and the points where the angle of the corresponding main body point changes are marked as secondary joint points.

[0021] The system acquires the orientation change status of the main joint point's movement position and the relative position during the multimodal analysis period. If the orientation change status of the main joint point's movement position and the relative position is a non-fixed orientation angle change, it infers that the main joint point type setting is incorrect, generates a main joint point setting error signal and sends it to the motion capture platform, and prevents the current point from being set as a main joint point. If the orientation change status of the main joint point's movement position and the relative position is a non-fixed orientation angle change, it infers that the main joint point type setting is normal.

[0022] In a preferred embodiment of the present invention, the number of non-intersecting angles between the real-time active angle range of the subjoint point and the active angle range of the corresponding type of subjoint point during the multimodal analysis period is obtained. If the number of non-intersecting angles exceeds a set threshold, it is inferred that there is a deviation in the setting of the current subjoint point, a subjoint point setting deviation signal is generated and sent to the motion capture platform, and the current point is not set as a subjoint point; if the number of non-intersecting angles does not exceed the set threshold, it is inferred that the setting of the current subjoint point is normal.

[0023] In a preferred embodiment of the present invention, a motion model of the action execution subject is constructed based on the distribution of the main joint points and the secondary joint points; and the action execution is constructed by various types of points of the action execution subject within the motion model, the point movement trajectory of the main joint points is set as the position movement trajectory of the action execution subject, and the point movement trajectory of the secondary joint points is set as the action execution trajectory of each point of the action execution subject.

[0024] In a preferred embodiment of the present invention, the process of the motion trajectory differentiation unit is as follows:

[0025] The movement trajectory of the action executor is monitored and analyzed. Data is collected at various locations within the movement trajectory, and the points are periodically monitored based on the dwell time at each location. The execution cycle of the current periodic action is determined based on the current action performed by the action executor. If an occasional point shift occurs within the execution cycle, and the corresponding point at an adjacent time periodically repeats, it is inferred that the overall change in the action executor caused by the point shift at the time of the occasional point shift is an unnecessary action. The current action is then marked, and the corresponding point of the action executor is monitored to prevent its recurrence.

[0026] In a preferred embodiment of the present invention, unnecessary actions are removed from the motion trajectory, and the remaining actions are gradually decomposed. That is, the location of each point of the action execution subject and the changes in position are combined and analyzed to form a slow motion of the motion trajectory execution, and the motion execution subject trajectory between the points is decomposed and sent to the motion capture platform.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] 1. In this invention, the entire limb movement is recorded and the movement is segmented through multimodal analysis, which facilitates more accurate motion capture and avoids the simultaneous statistics of redundant movements, which would increase the data processing volume of the entire capture platform and make it impossible to capture movements in a timely and accurate manner. Multimodal analysis can capture movements more accurately.

[0029] 2. In this invention, the action execution subject is analyzed during the multimodal analysis period to obtain the execution point of the action execution subject. Based on the type of each point, the action is accurately decomposed to accurately infer the execution action of the current action execution subject, thereby improving the accuracy and timeliness of the motion capture platform.

[0030] 3. In this invention, the action trajector is distinguished by action trajectory, and the action of the action executor is excluded by action trajectory screening, so as to improve the accuracy of action decomposition and capture, avoid synchronous monitoring of unnecessary actions of the action executor, avoid increasing the complexity of action capture, and avoid making it impossible to accurately evaluate the action of the action executor. Attached Figure Description

[0031] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0032] Figure 1 This is a system principle block diagram of the present invention;

[0033] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] Please see Figures 1-2 As shown, the limb motion decomposition and capture system based on multimodal analysis includes a motion capture platform, wherein the motion capture platform is communicatively connected to a multimodal analysis module, and the multimodal analysis platform is connected to a motion decomposition and recognition unit and a motion trajectory differentiation unit.

[0037] After receiving the capture signal, the motion capture platform generates a multimodal analysis command and sends it to the multimodal analysis module. After receiving the command, the multimodal analysis module records the entire limb movement and performs motion segmentation through multimodal analysis, which facilitates more accurate motion capture and avoids the simultaneous statistics of redundant movements, which would increase the data processing load of the entire capture platform and prevent timely and accurate motion capture. Multimodal analysis enables more accurate motion capture.

[0038] It should be explained that multimodal data acquisition systems typically integrate multiple sensors:

[0039] Visual sensors: such as depth cameras (e.g., Kinect) or multi-camera arrays, using optical markers (e.g., reflective spheres) or markerless technologies (e.g., OpenPose, AlphaPose) to capture human joint coordinates and skeletal structure. Inertial measurement units (IMUs): worn on limbs, head, etc., to collect acceleration and angular velocity data in real time, combined with algorithms such as Kalman filtering to eliminate drift and achieve dynamic posture tracking. Pressure sensors: embedded in the ground or wearable devices to capture plantar pressure distribution, aiding in the analysis of movement stability and biomechanical characteristics. Other modalities: such as electromyography (EMG) signals and eye-tracking data, which can further supplement information on muscle activity and attention allocation.

[0040] Identify the subject performing the action, and capture the action video of the subject using multimodal technology. Analyze the captured video to select segments of the subject performing the action, and use these segments as action decomposition and capture videos.

[0041] The system acquires the captured video and identifies and analyzes the points and background of the action subject in the captured video. Based on the action subject, the captured action is broken down to obtain the scene required for the action execution, such as moving or jumping in a specific area. The scene of the specific area is set as the execution background, such as the position around the execution, the brightness points of the specific position, etc.

[0042] First, determine the execution location of the action execution subject based on the scenario required for action execution, and then determine the corresponding point of the action execution subject, ensuring a one-to-one correspondence.

[0043] Obtain the overlap time between the execution position and the main body point of the action execution, and obtain the cumulative value of the overlap between the execution position and the main body point of the action execution at the overlap time. At the same time, obtain the floating frequency of the number of points that remain in the overlap state at adjacent overlap times after the execution position and the main body point of the action execution overlap.

[0044] If the cumulative value of the number of overlapping points of the execution position and the main body of the action at the overlapping time exceeds the cumulative value threshold, and the floating frequency of the number of points that remain overlapping at adjacent overlapping times after the execution position and the main body of the action overlap does not exceed the floating frequency threshold, it is inferred that the execution position and the main body of the action have a high degree of fit at the current overlapping time, and the current overlapping time is marked as the preset selection time, where the floating frequency of the number of points is represented by the decreasing frequency of the number of points;

[0045] If the cumulative value of the number of overlapping points of the execution position and the main body of the action at the overlapping time does not exceed the cumulative value threshold, or if the frequency of the number of points that remain overlapping at adjacent overlapping times after the execution position and the main body of the action overlap exceeds the frequency of the fluctuation, it is inferred that the degree of fit between the execution position and the main body of the action is low at the current overlapping time, and the current overlapping time is marked as a non-preset selection time.

[0046] After obtaining the preset selected time, the overlapping points of adjacent time after the preset selected time are monitored. If the overlapping points continue to increase and the adjacent time of the already overlapping points is always overlapping, the current preset node time is marked as the set node time; if the overlapping points do not continue to increase or the adjacent time of the already overlapping points is always overlapping, the time analysis continues.

[0047] After obtaining the set node time, the background of the scene where the action execution subject point is located is compared with the background of the overlapping time. If the background parameters are consistent with the background parameters of the action execution subject point, the video segment is collected at the current set node time, and the video segment ends when the number of overlapping points between the action execution subject point and the execution position rapidly decreases, thus obtaining the multimodal analysis period.

[0048] After obtaining the multimodal analysis time period, the multimodal analysis module generates an action disassembly and identification signal and sends it to the action disassembly and identification unit;

[0049] After receiving the data, the motion decomposition and recognition unit analyzes the motion execution subject during the multimodal analysis period to obtain the execution point of the motion execution subject. Based on the type of each point, it performs accurate motion decomposition to accurately infer the current motion execution subject's execution action, thereby improving the accuracy and timeliness of the motion capture platform.

[0050] Based on the multimodal analysis period, point trajectory analysis is performed on the action execution subject. In the point trajectory, the points where the position moves as a whole are marked as main joint points, and the points where the angle of the corresponding main body point changes are marked as secondary joint points.

[0051] The system acquires the orientation change status of the main joint point's movement position and the relative point position during the multimodal analysis period. If the orientation change status of the main joint point's movement position and the relative point position is a non-fixed orientation angle change, it infers a main joint point type setting deviation, generates a main joint point setting deviation signal and sends it to the motion capture platform, and then does not set the current point as a main joint point. It should be explained that the main joint point and the corresponding relative point are mutually corresponding. That is, when the main joint point and the relative point move, the corresponding relative orientation will not move. That is, when the point is horizontal, it is impossible for the corresponding relative point to have multiple non-fixed orientations such as horizontal and vertical when moving under the same action.

[0052] If the orientation change of the main joint point position and the relative position position is a non-fixed orientation angle change, it is inferred that the main joint point type setting is normal.

[0053] The number of non-intersecting angles between the real-time active angle range of the subjoint point and the active angle range of the corresponding type of subjoint point during the multimodal analysis period is obtained. If the number of non-intersecting angles exceeds the set threshold, it is inferred that there is a deviation in the setting of the current subjoint point. A subjoint point setting deviation signal is generated and sent to the motion capture platform, and the current point is not set as a subjoint point.

[0054] If the number of non-intersecting angles does not exceed the set threshold for the number of non-intersecting angles, it is inferred that the current subjoint point setting is normal.

[0055] A motion model of the action execution subject is constructed based on the distribution of the main joint points and the secondary joint points; and the action execution is constructed for each type of point of the action execution subject within the motion model. The point movement trajectory of the main joint points is set as the position movement trajectory of the action execution subject, and the point movement trajectory of the secondary joint points is set as the action execution trajectory of each point of the action execution subject.

[0056] After completing the motion trajectory setting, it is sent to the motion trajectory differentiation unit;

[0057] After receiving the motion trajectory differentiation unit, the motion trajectory of the motion execution subject is differentiated. The motion trajectory is screened to exclude the motion of the motion execution subject, so as to improve the accuracy of motion decomposition and capture, avoid synchronous monitoring of unnecessary motion of the motion execution subject, avoid increasing the complexity of motion capture, and avoid making it impossible to accurately evaluate the motion of the motion execution subject.

[0058] The system monitors and analyzes the movement trajectory of the subject performing the action. It collects data at various points along the trajectory and periodically checks these points based on their dwell time. The execution cycle of the current periodic action is determined based on the subject's current action. If an occasional point shift occurs within the execution cycle, and the corresponding point at an adjacent time periodically repeats, it is inferred that the overall change in the subject's action caused by the occasional point shift is an unnecessary action. This action is then marked, and the corresponding point of the subject is monitored to prevent recurrence. For example, if the subject is moving a limb laterally and the landing point at a certain point shifts, all subsequent points will also show shifts. This indicates that the limb missed a step during the current action but was corrected in time. Therefore, this type of action is not captured, and occasional actions do not affect the overall execution efficiency.

[0059] Unnecessary actions are removed from the motion trajectory, and the remaining actions are broken down step by step. This involves combining and analyzing the location of each point of the action execution subject and the changes in location to form a slow motion of the motion trajectory execution. The motion execution subject trajectory between points is then broken down and sent to the motion capture platform.

[0060] In use, the multimodal analysis module records the entire limb movement and performs movement segmentation through multimodal analysis to obtain a multimodal analysis time period. After obtaining the multimodal analysis time period, the movement decomposition and recognition unit performs movement execution subject analysis on the multimodal analysis time period. The movement execution trajectory is obtained based on the analysis. After completing the movement trajectory setting, it is sent to the movement trajectory differentiation unit. After receiving the movement trajectory, the movement trajectory differentiation unit differentiates the movement trajectory of the movement execution subject.

[0061] Thresholds, preset values, preset ranges, etc. are set for result comparison and analysis to determine whether they are good or bad. The value of these thresholds is determined by a combination of large-scale model analysis of sample data and human experience. They can also be adjusted appropriately based on seasonal or common-sense influences.

[0062] Furthermore, the settings for weighting ratios, influence factors, etc., are based on the magnitude of each parameter's influence on the results. The specific values ​​are allocated to ultimately reflect the impact on the results. The settings for input and storage are also determined by a combination of large-scale model analysis of sample data and human experience. Appropriate adjustments can also be made based on seasonal or rational influence conditions.

[0063] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A limb motion decomposition and capture system based on multimodal analysis, characterized in that, It includes a motion capture platform, which is connected to a multimodal analysis module and is connected to a motion decomposition and recognition unit and a motion trajectory differentiation unit. The multimodal analysis module records the entire limb movement and performs movement segmentation through multimodal analysis to obtain the multimodal analysis time segment; the process of the multimodal analysis module is as follows: Identify the subject performing the action, and capture the action video of the subject using multimodal technology. Analyze the captured video to select segments of the subject performing the action, and use these segments as action decomposition and capture videos. The video is acquired, and the location of the subject performing the action and the background in the video are identified and analyzed. Based on the subject performing the action, the captured action is disassembled to obtain the scene required for the action to be performed. First, determine the execution location of the action execution subject based on the scenario required for action execution, and then determine the corresponding point of the action execution subject, ensuring a one-to-one correspondence. Obtain the overlap time between the execution position and the main body point of the action execution, and obtain the cumulative value of the overlap between the execution position and the main body point of the action execution at the overlap time. At the same time, obtain the floating frequency of the number of points that remain in the overlap state at adjacent overlap times after the execution position and the main body point of the action execution overlap. If the cumulative value of the number of overlapping points of the execution position and the main body of the action at the overlapping time exceeds the cumulative value threshold, and the floating frequency of the number of points that remain overlapping at adjacent overlapping times after the execution position and the main body of the action overlap does not exceed the floating frequency threshold, then the current overlapping time is marked as the preset selected time. If the cumulative value of the number of overlapping points of the execution position and the main body of the action at the overlapping time does not exceed the cumulative value threshold, or if the frequency of the number of points that remain overlapping at adjacent overlapping times after the execution position and the main body of the action overlap exceeds the frequency of the fluctuation, then the current overlapping time will be marked as a non-preset selected time. After obtaining the preset selected time, the overlapping points of adjacent time after the preset selected time are monitored. If the overlapping points continue to increase and the adjacent time of the already overlapping points is always overlapping, the current preset node time is marked as the set node time; if the overlapping points do not continue to increase or the adjacent time of the already overlapping points is always overlapping, the time analysis continues. After obtaining the set node time, the background of the scene where the action execution subject point is located is compared with the background of the overlapping time. If the background parameters are consistent with the background parameters of the action execution subject point, the video segment is collected at the current set node time, and the video segment ends when the number of overlapping points between the action execution subject point and the execution position rapidly decreases, thus obtaining the multimodal analysis period. After obtaining the multimodal analysis time period, the action decomposition and recognition unit performs action execution subject analysis on the multimodal analysis time period; and obtains the action execution trajectory based on the analysis; After the motion trajectory is set, it is sent to the motion trajectory differentiation unit; after receiving it, the motion trajectory differentiation unit differentiates the motion trajectory of the motion execution subject.

2. The limb motion decomposition and capture system based on multimodal analysis according to claim 1, characterized in that, The process of the action decomposition and recognition unit is as follows: Based on the multimodal analysis period, point trajectory analysis is performed on the action execution subject. In the point trajectory, the points where the position moves as a whole are marked as main joint points, and the points where the angle of the corresponding main body point changes are marked as secondary joint points. The system acquires the orientation change status of the main joint point's movement position and the relative position during the multimodal analysis period. If the orientation change status of the main joint point's movement position and the relative position is a non-fixed orientation angle change, it infers that the main joint point type setting is incorrect, generates a main joint point setting error signal and sends it to the motion capture platform, and prevents the current point from being set as a main joint point. If the orientation change status of the main joint point's movement position and the relative position is a non-fixed orientation angle change, it infers that the main joint point type setting is normal.

3. The limb motion decomposition and capture system based on multimodal analysis according to claim 2, characterized in that, The system obtains the number of non-intersecting angles between the real-time active angle range of the subjoint points and the active angle range of the corresponding type of subjoint points during the multimodal analysis period. If the number of non-intersecting angles exceeds the set threshold, it is inferred that there is a deviation in the setting of the current subjoint point. A subjoint point setting deviation signal is generated and sent to the motion capture platform, and the current point is not set as a subjoint point. If the number of non-intersecting angles does not exceed the set threshold, it is inferred that the setting of the current subjoint point is normal.

4. The limb motion decomposition and capture system based on multimodal analysis according to claim 3, characterized in that, A motion model of the action execution subject is constructed based on the distribution of the main joint points and the secondary joint points. The motion model is used to construct the execution actions of each type of point of the action execution subject. The point movement trajectory of the main joint points is set as the position movement trajectory of the action execution subject, and the point movement trajectory of the secondary joint points is set as the action execution trajectory of each point of the action execution subject.

5. The limb motion decomposition and capture system based on multimodal analysis according to claim 4, characterized in that, The process of distinguishing units based on motion trajectories is as follows: The movement trajectory of the action executor is monitored and analyzed. Data is collected at various locations within the movement trajectory, and the points are periodically monitored based on the dwell time at each location. The execution cycle of the current periodic action is determined based on the current action performed by the action executor. If an occasional point shift occurs within the execution cycle, and the corresponding point at an adjacent time periodically repeats, it is inferred that the overall change in the action executor caused by the point shift at the time of the occasional point shift is an unnecessary action. The current action is then marked, and the corresponding point of the action executor is monitored to prevent its recurrence.

6. The limb motion decomposition and capture system based on multimodal analysis according to claim 5, characterized in that, Unnecessary actions are removed from the motion trajectory, and the remaining actions are broken down step by step. This involves combining and analyzing the location of each point of the action execution subject and the changes in location to form a slow motion of the motion trajectory execution. The motion execution subject trajectory between points is then broken down and sent to the motion capture platform.

Images