Wearing-free motion capture and evaluation method based on computer vision

Abstract

The invention discloses a wearing-free motion capture and evaluation method based on computer vision. The method comprises the steps that motion videos are collected, a human body area is detected, key points are recognized, and a standardized three-dimensional skeleton sequence is reconstructed; constructing a skeleton topology adjacency matrix, extracting joint structure features, and generating an action topology space-time tensor; identifying a topology mode and a phase state, detecting a phase change point and generating an action topology phase change map; matching and aligning topological phases of the user and the standard action by using a phase change graph; executing topological folding and unfolding mapping, judging reversibility, and identifying structural error parts; and constructing a topological behavior imprint map, matching a standard imprint, and completing action deviation recognition and evaluation. According to the method, structural error recognition and interpretable quality evaluation of human body actions under the wearing-free condition are realized by constructing an action topological structure expression and imprinting analysis system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a computer vision-based non-wearable motion capture and evaluation method. BACKGROUND

[0002] Existing non-wearable motion capture and evaluation technologies mainly rely on video input, and obtain two-dimensional or three-dimensional skeleton representations of user motions through human body detection, pose estimation and key point tracking. In most application scenarios, such technologies usually generate key point coordinates by using deep learning pose estimation algorithms, and then complete motion analysis through geometric distances between key points, angle changes or time sequence alignment of motions. However, the core dependency of these methods is the geometric relationship of key points, and the essence is a “point-level” motion description method, which lacks deep modeling capability for internal structural relationships of motions. Therefore, these methods are often unstable in the face of changes in viewing angles, inconsistent motion speeds, mixed multi-motion stages or abnormal joint structures, and the reliability of motion analysis results is obviously limited.

[0003] In the prior art, motion evaluation usually relies on preset angle thresholds, trajectory alignment, dynamic time warping or deep feature comparison to determine whether the motion is standard, does not start from the motion structure itself, cannot represent the dynamic changes of the skeletal topological structure over time, and cannot depict the structural mutation phenomenon in the motion execution process. When the user performs motions with complex situations such as joint coordination errors, discontinuous motion paths and fuzzy motion stages, the prior art can only output numerical errors or classification results, and it is difficult to locate the specific structural position and corresponding motion stage where the problem occurs. Existing motion analysis methods usually lack explainability and do not have the ability to structurally describe the motion execution process, and cannot provide clear and traceable basis for motion correction.

[0004] Based on the above deficiencies of the prior art, the motion structural errors cannot be effectively identified through traditional key point geometric relationships, the explainable phase information cannot be obtained from the topological structure changes of the motion execution process, and the motion quality cannot be evaluated through complete topological features and behavior traces.

[0005] Therefore, how to provide a computer vision-based non-wearable motion capture and evaluation method is a problem that those skilled in the art need to solve. SUMMARY

[0006] One object of the present application is to provide a computer vision-based non-wearable motion capture and evaluation method. The present application is based on three-dimensional skeleton data obtained from video, constructs a motion topology space-time tensor, generates a motion topology phase transition atlas, performs a topology folding-unfolding reversibility determination, and forms a motion topology behavior imprint atlas. The comprehensive features of motion in topology structure, phase change, and behavior imprint are used to realize motion structure analysis and motion quality evaluation. The present application breaks through the traditional shallow analysis mode relying on key point geometric data through topology structure modeling, phase change path analysis, and behavior imprint expression, and has the advantages of structure error localization, motion stage identification, evaluation result interpretation, and robustness to changes in viewing angle and motion speed.

[0007] According to an embodiment of the present application, a computer vision-based non-wearable motion capture and evaluation method comprises: Collecting motion video data of a user, performing human region detection, human key point identification, and monocular three-dimensional pose reconstruction, smoothing the obtained three-dimensional skeleton data, compensating for missing data, and performing time alignment processing to generate a standardized three-dimensional skeleton sequence; Constructing a skeleton topology adjacency matrix according to the connection relationship of human joints, calculating joint relative structure features and topology invariants based on the standardized three-dimensional skeleton sequence, and constructing a motion topology space-time tensor using a time sequence unfolding method; Performing topology pattern recognition on the motion topology space-time tensor, determining the topology phase state of the motion in each time period, detecting topology phase change points occurring during motion execution, constructing a motion topology phase transition atlas according to the topology phase state sequence, and dividing the motion execution process into multiple topology phase segments; Corresponding matching the topology phases of the user's motion and the standard motion using the phase segments of the motion topology phase transition atlas, and establishing an alignment relationship at the motion phase level; Based on the motion topology space-time tensor, performing topology folding mapping to generate a folded topology structure, performing topology unfolding mapping to generate an unfolded topology structure, determining the reversibility of the folded topology structure and the unfolded topology structure, identifying irreversible topology deformation and determining the corresponding structural error positions; Constructing a motion topology behavior imprint surface according to the motion topology space-time tensor, extracting topology texture imprint, folded imprint, impact imprint, and structural hollow imprint, forming a motion topology behavior imprint atlas, and matching the imprint signature of the standard motion to complete motion execution deviation identification and motion quality evaluation.

[0008] Optionally, the motion video data includes a sequence of continuous video frames for recording the user's motion execution process, and the sequence of continuous video frames contains human region information, human motion trajectory information, and joint attitude change information.

[0009] Optionally, the human body region detection, human key point identification and monocular three-dimensional pose reconstruction include: performing human body region detection on the action video frames to determine the spatial position and the circumscribed region of the human body in the video; performing human key point identification on each video frame in the circumscribed region to obtain two-dimensional key point coordinates; calculating corresponding three-dimensional joint positions based on the two-dimensional key point coordinates to form three-dimensional skeleton data.

[0010] Optionally, the action topological space-time tensor is constructed in a time sequence unfolding manner, including: taking the standardized three-dimensional skeleton sequence as input, determining a skeleton frame corresponding to each time, and labeling the number of each joint and the hierarchical attribution in the skeleton in each skeleton frame; determining the joint connection relationship according to the human anatomy structure, recording the joint pairs having direct skeletal connection relationship as a skeletal topological adjacency list, and assigning a unique topological connection identifier to each joint pair in the skeletal topological adjacency list; for each time frame in the standardized three-dimensional skeleton sequence, traversing all joint pairs according to the skeletal topological adjacency list, calculating the relative structural feature information of each joint pair at the time frame, and taking the relative structural feature information as the topological feature description of the joint pair at the time frame; performing a time sequence unfolding operation on the time axis of the standardized three-dimensional skeleton sequence, arranging the topological feature descriptions in the time sequence according to the skeletal topological adjacency list, establishing a unique time sequence index identifier for each time frame and each joint pair, binding the time sequence index identifier and the corresponding topological feature description, and forming a one-to-one reversible mapping relationship between the time sequence unfolding and the original time sequence structure; based on the topological feature description bound with the time sequence index identifier after time sequence unfolding, organizing the topological feature of all joint pairs corresponding to each time frame in time sequence and topological connection sequence into a multi-dimensional data structure having a time dimension and a topological structure dimension, and defining the multi-dimensional data structure as an action topological space-time tensor.

[0011] Optionally, the action topological phase transition map is constructed according to the topological phase state sequence, and the action execution process is divided into multiple topological phase sections, including: obtaining the action topological space-time tensor, performing normalization and noise suppression processing on the topological features corresponding to each time frame, and organizing the topological features into a topological feature sequence for phase analysis in time sequence; performing mode division on the topological feature sequence, grouping time frames having similar topological features into the same topological phase category, assigning a unique topological phase state identifier to each time frame, and forming a topological phase state sequence arranged in time; The topology phase state sequence is traversed along a time axis, time frames with adjacent topology phase state identifiers are merged into a topology phase section, a start time, an end time and a corresponding topology phase state identifier of each topology phase section are recorded, and a plurality of topology phase sections covering the whole action execution process are obtained; Each topology phase section is taken as a phase change node, a transition connection between adjacent topology phase sections is established according to the order and adjacent relationship of the topology phase sections on the time axis, each transition connection is marked as a directed relationship from a start topology phase section to a target topology phase section, and an action phase change link structure is formed; On the basis of the action phase change link structure, a phase duration and a phase internal topology stability degree attribute information are added to each topology phase section, a phase switching frequency, a phase combination legality identifier and an abnormal phase change mark attribute information are added to each transition connection, and the topology phase sections and the transition connections carrying the attribute information are organized into a multi-layer action topology phase change graph.

[0012] Optionally, the establishing of the alignment relationship of the action phase levels comprises: An action topology phase change graph generated for a user action and a standard action topology phase change graph pre-generated and stored for a standard action are obtained, and topology phase sections, topology phase identifiers, start times and end times corresponding to each topology phase section are read from the two types of topology phase change graphs respectively; Based on the topology phase identifiers, each topology phase section of the user action and each topology phase section of the standard action are preliminarily corresponded, a candidate topology phase corresponding relationship set is constructed, and the order position of the user topology phase section and the standard topology phase section on the time axis is recorded for each candidate corresponding relationship; In the candidate topology phase corresponding relationship set, a phase matching degree value is calculated for each candidate corresponding relationship based on the topology phase identifier consistency, the topology phase section duration closeness and the adjacent topology phase section transition mode consistency, and the phase matching degree value is stored in association with the corresponding candidate corresponding relationship; According to the order of the user action topology phase sections on the time axis, the phase matching degree values of the candidate corresponding relationships and the time sequence constraint are comprehensively selected, a corresponding relationship combination satisfying the topology phase order continuity and having a relatively high overall phase matching degree value is selected, and the corresponding relationship combination is determined as the final corresponding matching result between the user action topology phase sections and the standard action topology phase sections. The final corresponding matching result is organized as an alignment relationship table of the action phase levels, and the standard topology phase section identifier, the start time, the end time and the phase matching degree value corresponding to each user topology phase section are recorded in the alignment relationship table.

[0013] Optionally, the reversibility of the folded topology and the unfolded topology is determined, irreversible topological deformation is identified, and corresponding structural error sites are determined, including: The action topology space-time tensor, the time index identifier, and the topology connection identifier are read to determine the positioning information of each time frame and each joint connection unit in the action topology space-time tensor; According to the preset folding rule, folding mapping is performed in the time dimension and the topology structure dimension, adjacent connection units with similar topology characteristics are aggregated to form a folded topology structure, and a one-to-one correspondence relationship with the time index identifier and the topology connection identifier is maintained in the folded topology structure; According to the unfolding rule corresponding to the folding rule, unfolding mapping is performed to restore the folded topology structure to an unfolded topology structure with time dimension and topology structure dimension, and the mapping relationship with the time index identifier and the topology connection identifier is restored; The reversibility of the folded topology and the unfolded topology is determined: First, the index backtracking consistency check is performed to check whether each connection unit after unfolding can be accurately traced back to the original time index identifier and topology connection identifier; Second, the neighborhood consistency check is performed to check whether the relative order and relative adjacency relationship of each connection unit in the time neighborhood and the topology neighborhood after unfolding are consistent with those before folding; Third, the topology fingerprint check is performed, the fingerprint information is generated based on the connection order, the branch number, the joint number, and the connection connectivity, and the fingerprint information of the corresponding positions before folding, after folding, and after unfolding is compared for consistency; Fourth, the path delay detection is performed to record the mapping path difference between folding and unfolding. If there is a path that cannot be merged or multiple value back mapping occurs, it is determined to be irreversible; For any time index position or topology connection position that does not satisfy the consistency of the check, it is marked as an irreversible topological deformation position; According to the time index identifier and the topology connection identifier corresponding to the marked position, the specific time section and the specific joint connection site in the original three-dimensional skeleton sequence are located, the time section and the joint connection site are determined as a structural error site, and the corresponding annotation information is output.

[0014] Optionally, the action execution deviation identification and the action quality evaluation are completed according to the imprint structure difference, including: The action topology space-time tensor and its time index identifier and topology connection identifier are obtained, and the behavior imprint surface is unfolded in the time dimension and the topology structure dimension under the premise of maintaining the one-to-one correspondence of the index, and a backtracking mapping relationship from the time index identifier, the topology connection identifier to the behavior imprint surface coordinates is established; Multi-scale patching and rasterization encoding are performed on the behavior imprint surface, and imprint features are extracted for each scale of the raster unit, and the imprint features of each raster unit are bound with time index identification and topological connection identification; The imprint features extracted by multi-scale patching and bound are organized into a multi-layer structure of action topological behavior imprint atlas, and the multi-layer structure comprises a bottom layer recording basic imprint features, a suture layer recording adjacent imprint transition relationship, a confidence mask layer recording feature stability and a hollow mounting layer recording missing segment placeholder information; Fingerprint encoding is performed on the action topological behavior imprint atlas to generate imprint fingerprint encoding for cross-sample alignment and rapid comparison, the imprint fingerprint encoding is associated with phase alignment relationship, and the imprint atlas alignment result under phase alignment constraint is obtained; The imprint signature of the standard action is called, corresponding matching is performed on the imprint atlas under phase alignment constraint, the time section and joint connection position where the difference is located are positioned according to the difference between the imprint fingerprint encoding and the multi-layer structure, and the action execution deviation recognition result and the action quality evaluation result are output.

[0015] The beneficial effects of the present application are:

[0016] The present application constructs an action topological space-time tensor, essentially improves the traditional action description method which only depends on the geometric relationship of key points, so that the core structural features of the action can be expressed in the topological dimension. By integrating joint connection relationship, relative structural change and time sequence information into the topological coding framework, the present application realizes stable representation of structural changes in action execution process, significantly enhances the robustness of motion capture under different angles, different speeds and complex background conditions. Compared with the prior art, the present application can maintain the consistency of structural expression under the conditions of skeleton data fluctuation, key point partial loss or action execution inconsistency, thereby improving the basic accuracy of action analysis.

[0017] The present application further proposes an action topological phase transition atlas and a topological folding-unfolding reversibility determination mechanism, so that structural errors in action execution process can be accurately identified from the topological change path. Relying on phase section, phase transition relationship and reversibility check, the present application can locate the specific time position and corresponding joint connection position where the action error occurs, and realize the error revealing ability of action structure level which cannot be achieved by the prior art. Especially when the action stage is mixed, the rhythm is unstable or the action execution has sudden structural change, the present application can still provide stable and traceable error identification basis through topological phase transition and reversibility analysis, greatly improving the reliability of action evaluation under complex motion mode.

[0018] In the final evaluation stage, the application expresses the topological feature change in the action execution process in the form of a trace by constructing an action topological behavior trace atlas, thereby giving the action analysis result stronger explainability. The behavior trace atlas can intuitively present the action texture change, folding trace, impact feature and structural cavity, so that the action deviation analysis no longer relies on the numerical result, but can have the support of multi-dimensional information such as spatial distribution, structural morphology and execution trajectory. Therefore, the application can provide fine-grained and more understandable action quality feedback, provide clear error correction basis and targeted adjustment direction for trainers, rehabilitation personnel or motion analysis systems, and overall improve the application value and actual effect of motion capture and evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application, and are used to explain the application together with the embodiments of the application, and do not constitute a limitation on the application.

[0020] In the drawings: Figure 1 A flowchart of a computer vision-based non-wearable motion capture and evaluation method proposed by the application; Figure 2 A schematic diagram of a topological folding-unfolding reversibility determination process of a computer vision-based non-wearable motion capture and evaluation method proposed by the application. DETAILED DESCRIPTION

[0021] The application will now be described in further detail with reference to the drawings. These drawings are all simplified schematic diagrams, and only illustrate the basic structure of the application in a schematic manner, and therefore only show the components related to the application.

[0022] Reference Figure 1 and Figure 2 A computer vision-based non-wearable motion capture and evaluation method, comprising: Collecting action video data of a user, performing human region detection, human key point recognition and monocular three-dimensional pose reconstruction, performing smoothing processing, missing compensation processing and time alignment processing on the obtained three-dimensional skeleton data, and generating a standardized three-dimensional skeleton sequence; Constructing a skeletal topological adjacency matrix according to the human joint connection relationship, calculating the joint relative structural features and topological invariants based on the standardized three-dimensional skeleton sequence, and constructing an action topological space-time tensor in a time sequence unfolding manner; Performing topological pattern recognition on the action topological space-time tensor, determining the topological phase state of the action in each time period, detecting the topological phase change points appearing in the action execution process, constructing an action topological phase transition atlas according to the topological phase state sequence, and dividing the action execution process into multiple topological phase sections; The user action is matched with the topology phase of the standard action by using the phase section of the action topology phase diagram, and an alignment relationship of the action phase level is established. Based on the action topology space-time tensor, a folded topology structure is generated by performing topology folding mapping, an unfolded topology structure is generated by performing topology unfolding mapping, reversibility of the folded topology structure and the unfolded topology structure is determined, irreversible topology deformation is identified, and corresponding structural error positions are determined; According to the action topology space-time tensor, an action topology behavior imprint surface is constructed, topology texture imprints, folded imprints, impact imprints, and structural hollow imprints are extracted, an action topology behavior imprint diagram is formed, and imprint signatures of standard actions are matched, so as to complete action execution deviation identification and action quality evaluation according to imprint structure differences.

[0023] In this embodiment, the action video data includes a sequence of continuous video frames for recording the execution process of the user action, and the sequence of continuous video frames contains human body region information, human body motion trajectory information, and joint posture change information.

[0024] In this embodiment, the human body region detection, human body key point identification, and monocular three-dimensional posture reconstruction include: The human body region detection is performed on the action video frame, and the spatial position and the circumscribed region of the human body in the video are determined; The human body key point identification is performed on each video frame in the circumscribed region, and two-dimensional key point coordinates are obtained; Based on the two-dimensional key point coordinates, corresponding three-dimensional joint positions are calculated, and three-dimensional skeleton data are formed.

[0025] In this embodiment, the action topology space-time tensor is constructed in a time sequence unfolding manner, including: The standardized three-dimensional skeleton sequence is taken as input, the skeleton frame corresponding to each time instant is determined, and the number of each joint and the hierarchical attribution in the skeleton are labeled in each skeleton frame; According to the human anatomy structure, the joint connection relationship is determined in advance, the joint pairs having a direct skeletal connection relationship are recorded as a skeletal topology adjacency table, and a unique topology connection identifier is assigned to each joint pair in the skeletal topology adjacency table; For each time frame in the standardized three-dimensional skeleton sequence, all joint pairs are traversed according to the skeletal topology adjacency table, the relative structural feature information of each joint pair at the time frame is calculated, the relative structural feature information is taken as the topology feature description of the joint pair at the time frame, and the calculation of the relative structural feature information of each joint pair at the time frame is specifically: The three-dimensional position data of the two joints in the joint pair at the time frame are obtained, and the spatial difference information for representing the connection direction is generated according to the coordinate difference of the two joints in space; Calculate a spatial separation degree between two joints based on the three-dimensional position data, and take the spatial separation degree as length feature information representing a connection state of the joint pair in the time frame; In combination with a connection relationship of the joint pair in the skeleton, calculate a relative orientation change of adjacent joint pairs in the time frame, and generate structure change feature information for describing a local topological morphology; Perform a time sequence unfolding operation on a time axis of the standardized three-dimensional skeleton sequence, arrange topological feature descriptions in each time frame in a time sequence according to a topological adjacency list of the skeleton topology, establish a unique time sequence index for each time frame and each joint pair, bind the time sequence index and the corresponding topological feature description, and form a one-to-one reversible mapping relationship between the time sequence unfolding and the original time sequence structure; Based on the topological feature descriptions that are unfolded in the time sequence and bound with the time sequence index, organize all joint pair topological features corresponding to each time frame in a time sequence and a topological connection sequence into a multi-dimensional data structure with a time dimension and a topological structure dimension, and define the multi-dimensional data structure as a motion topological space-time tensor.

[0026] In this embodiment, the motion topological phase change map is constructed according to the topological phase state sequence, and the motion execution process is divided into multiple topological phase sections, including: Obtain the motion topological space-time tensor, normalize and suppress noise of the topological features corresponding to each time frame, and organize the topological features into a topological feature sequence for phase analysis in a time sequence; Divide the topological feature sequence into patterns, classify time frames with similar topological features into the same topological phase category, assign a unique topological phase state identifier to each time frame, and form a topological phase state sequence arranged in time; Traverse the topological phase state sequence along the time axis, merge adjacent time frames with the same topological phase state identifier into a topological phase section, record the start time, end time and corresponding topological phase state identifier of each topological phase section, and obtain multiple topological phase sections covering the entire motion execution process; Take each topological phase section as a phase change node, establish a transfer connection between adjacent topological phase sections according to the order and adjacent relationship of the topological phase sections on the time axis, mark each transfer connection as a directed relationship from a start topological phase section to a target topological phase section, and form a motion phase change link structure; On the basis of the motion phase change link structure, add phase duration and phase internal topological stability attribute information to each topological phase section, and add phase switching frequency, phase combination legality identifier and abnormal phase change marker attribute information to each transfer connection, and organize the topological phase sections and transfer connections carrying attribute information into a multi-layer motion topological phase change map.

[0027] In this embodiment, the alignment relationship of the action phase level is established, including: The action topology phase change atlas generated for the user action and the standard action topology phase change atlas pre-generated and stored for the standard action are obtained, and the respective topology phase segments, the topology phase identifiers corresponding to each topology phase segment, the start time and the end time information are read from the two types of topology phase change atlases; Based on the topology phase identifier, each topology phase segment of the user action is preliminarily corresponded to each topology phase segment of the standard action, and a candidate topology phase corresponding relationship set is constructed, and the order positions of the user topology phase segment and the standard topology phase segment on the time axis are recorded for each candidate corresponding relationship; In the candidate topology phase corresponding relationship set, based on the consistency of the topology phase identifier, the closeness of the topology phase segment duration and the consistency of the adjacent topology phase segment transition mode, the phase matching degree value is calculated for each candidate corresponding relationship, and the phase matching degree value is associated and stored with the corresponding candidate corresponding relationship. The phase matching degree value is calculated for each candidate corresponding relationship, specifically: The phase identifiers of the user topology phase segment and the standard topology phase segment in the candidate corresponding relationship are verified for consistency, and the identifier matching result when the phase identifiers are consistent is recorded; The durations of the user topology phase segment and the standard topology phase segment in the candidate corresponding relationship are compared, and a duration matching result is generated according to the closeness of the durations; The transition order between adjacent segments of the user topology phase segment and the standard topology phase segment in the candidate corresponding relationship is compared, and a transition mode matching result is generated according to the similarity of the transition order; According to the order of the user action topology phase segment on the time axis, the phase matching degree values and the time sequence constraints of each candidate corresponding relationship are integrated, a corresponding relationship combination that satisfies the topology phase sequence continuity and has a relatively high overall phase matching degree value is selected, and the corresponding relationship combination is determined as the final corresponding matching result between the user action topology phase segment and the standard action topology phase segment. The final corresponding matching result is organized as an action phase level alignment relationship table, and the standard topology phase segment identifier, the start time, the end time and the phase matching degree value corresponding to each user topology phase segment are recorded in the alignment relationship table.

[0028] In this embodiment, the reversibility of the folded topology structure and the unfolded topology structure is determined, and the irreversible topology deformation is identified and the corresponding structural error site is determined, including: The action topology space-time tensor and the time index identifier and the topology connection identifier are read to determine the positioning information of each time frame and each joint connection unit in the action topology space-time tensor; According to the preset folding rules, folding mapping is performed in the time dimension and topology dimension, and adjacent connection units with similar topological features are aggregated to form a folded topology, and a one-to-one correspondence between the time index identifier and the topology connection identifier is retained in the folded topology. Perform unfolding mapping according to the unfolding rules corresponding to the folding rules, restore the folded topology to an unfolded topology with time and topology dimensions, and restore the mapping relationship with time index identifiers and topology connection identifiers; Determine the reversibility of folded and unfolded topologies: First, there is an index backtracking consistency check, which checks whether each connection unit can accurately backtrack to the original time index identifier and topology connection identifier after expansion. Second, neighborhood consistency verification is performed to check whether the relative order and relative adjacency of each connected unit in the temporal and topological neighborhoods after unfolding are consistent with those before folding. Third, topological fingerprint verification is performed. Fingerprint information is generated based on connection order, number of branches, joint degree and connection connectivity. The fingerprint information at the corresponding positions before folding, after folding and after unfolding is compared for consistency. Fourth, path lag detection records the differences in the mapping paths between folding and unfolding. If there are cases where paths cannot be merged or multiple values ​​are reflected, it is determined to be irreversible. For any time index position or topology connection position where the verification fails to meet consistency, mark it as an irreversible topology deformation position; Based on the time index and topological connection identifiers corresponding to the marked positions, the specific time segments and joint connection points in the original 3D skeleton sequence are located, and the time segments and joint connection points are identified as structural error locations and the corresponding annotation information is output.

[0029] In this embodiment, the step of identifying action execution deviations and assessing action quality based on differences in imprint structure includes: Obtain the action topology spatiotemporal tensor and its time index identifier and topology connection identifier. While maintaining a one-to-one correspondence between the indices, expand it into a behavior imprint surface according to the time dimension and the topology structure dimension, and establish a backtrackable mapping relationship from the time index identifier, topology connection identifier to the coordinates of the behavior imprint surface. Multi-scale segmentation and rasterization encoding are performed on the behavioral imprint surface. Imprint features are extracted for each scale raster unit, and the imprint features of each raster unit are bound with time index identifier and topology connection identifier. The print feature organizations extracted by multi-scale block and completed binding are organized into a multi-layer structure of action topological behavior print atlas, the multi-layer structure including a bottom layer recording basic print features, a stitching layer recording adjacent print transition relations, a confidence mask layer recording feature stability, and a hollow mounting layer recording missing segment placeholder information; The action topological behavior print atlas is fingerprinted and encoded to generate print fingerprint encoding for cross-sample alignment and rapid comparison, the print fingerprint encoding is associated with phase alignment relations to obtain print atlas alignment results under phase alignment constraints; The print signature of a standard action is called to perform corresponding matching on the print atlas under phase alignment constraints, and the print fingerprint encoding and the difference marker in the multi-layer structure are used to locate the time segment and joint connection position of the difference, and the action execution deviation recognition result and the action quality evaluation result are output.

[0030] Embodiment 1 In order to verify the feasibility of the present application in implementation, the present application is applied to a rehabilitation medical center in a certain province, which has long provided lower limb motor function recovery training for postoperative rehabilitation patients of knee joints and groups with decreased balance ability of the elderly. However, the traditional video action evaluation system used by the center previously mainly relies on two-dimensional key point error and single-dimensional trajectory analysis, the action stage division is not accurate, phase misjudgment caused by action speed change frequently occurs, and structural action errors such as knee joint inward buckling, trunk compensation, and unbalanced weight bearing cannot be identified from the action structure level. It is difficult for rehabilitation physicians to make effective judgments based on the results provided by such a system, which seriously affects the training effect and intervention plan development.

[0031] In this embodiment, the rehabilitation center selected 20 postoperative recovery period patients of knee joints as subjects, the training content included deep squatting action, squatting and standing up action and single leg support action. The training place was a rehabilitation training room, the shooting device was a common 1080p camera, the distance from the subject was about 2.3 meters, and no sensor was needed. The method of the present application first performed human region detection, key point identification and three-dimensional pose reconstruction on the collected video to form a standardized three-dimensional skeleton sequence, providing basic data for subsequent topological modeling. Compared with the traditional system, the present application smoothes, time aligns and compensates for missing in the pre-processing stage, making the three-dimensional skeleton sequence more stable and reducing misjudgment caused by shaking.

[0032] Then, the system constructs the skeleton topology adjacency structure according to the joint connection relationship, and calculates the relative structural features of each time frame based on the standardized three-dimensional skeleton sequence, to complete the construction of the action topology space-time tensor. The topology space-time tensor can completely express the topology relationship and structure mode of the action changing with time, so that the action description originally relying on coordinates is upgraded to action expression based on topology structure. The system performs pattern recognition on the topology features to form a topology phase state sequence, and constructs an action topology phase transition map according to the phase state change. Through the map, the system can automatically divide the action process of the subject into multiple topology phases. For example, in deep squat training, the system can stably identify four phases: standing preparation, descent transition, lowest point stability and rising recovery, even if there is a significant difference in the action speed of the subject, the phase division is still accurate and consistent.

[0033] Subsequently, the system matches the phase transition map with the phase transition map of the standard action to realize the alignment of the action stage level. The traditional system usually relies on time alignment, and once the speed of the subject changes too much, it will be misaligned, while the present application relies on topology phase transition for alignment, which significantly improves the robustness of stage recognition and action comparison in this test.

[0034] In terms of structural error recognition, the present application constructs folded topology structure and unfolded topology structure through topology folding mapping and topology unfolding mapping, and determines the reversibility thereof to accurately locate the position where irreversible topology deformation occurs. For example, a 62-year-old female patient had irreversible topology deformation in the left knee during the lowest point stage of the 8th deep squat, which was automatically labeled by the system as a structural error position. After on-site examination, the rehabilitation doctor confirmed that there was indeed a knee inward buckling problem in the action, which was completely consistent with the system determination.

[0035] The present application constructs an action topology behavior imprint surface according to the action topology space-time tensor, generates an action topology behavior imprint map by extracting texture imprint, folding imprint, impact imprint and hollow imprint, and matches it with the standard action imprint signature. In this process, the system not only identifies whether the action is standard, but also clearly presents which force output is unstable, which joint has sudden compensation at a certain stage, or which action stage is not fully executed in the subject's action. For example, a 55-year-old male patient had dense impact imprints in the right ankle area in the single-leg support action, reflecting the problem of unstable balance and sudden shift of center of gravity to the right side.

[0036] Table 1 Comparison of the performance of the present application and the traditional action evaluation system in real rehabilitation scenarios: Evaluation index Traditional system result Invention result Data source explanation Structural error recognition accuracy 54.2% 88.7% Statistical comparison of structural problems in 400 actions of 20 patients Phase division consistency 71.5% 94.3% Statistical comparison of action phase consistency with artificial labeling Stability under different speed conditions (standard / slow down) Fluctuation about 18% Fluctuation about 4% Comparison of the same subject training with two rhythms repeatedly Different camera position error (front / 30° / 45°) 12.1% 3.2% Phase and structural error consistency evaluation under different shooting angles Interpretability feedback satisfaction (physician score) 3.1 / 5 4.7 / 5 Rehabilitation physician score on the understandability of the evaluation report From the data in Table 1, it can be seen that the present application has significant advantages compared with the traditional system. In the test of 20 patients with 400 actions, the traditional system can only identify about half of the structural abnormalities, with an accuracy of 54.2%, while the present application can capture the irreversible deformation in the action structure through the topological folding-unfolding reversibility determination, and the recognition accuracy is improved to 88.7%. This shows that the present application is more reliable in detecting structural level problems such as knee buckling, joint compensation, and center of gravity shift, and effectively makes up for the shortcomings of the traditional system in structural interpretation ability.

[0037] In terms of consistency between action phase division and action execution, the present application relies on action topological phase diagram for action segmentation, so that phase division is no longer affected by factors such as speed change and rhythm difference. Experiments show that the traditional system is only 71.5% consistent with manual annotation in phase recognition, while the present application reaches 94.3%, and the phase division is more accurate. At the same time, under different rhythms and different shooting angles, the recognition error of the present application is lower than that of the traditional system, the fluctuation caused by speed change is reduced from about 18% of the traditional system to about 4%, and the error caused by angle change is reduced from 12.1% of the traditional system to 3.2%, showing high stability and environmental adaptability.

[0038] In terms of explainability, the present application visualizes the topological changes, structural problems and abnormal stages of the action execution process in the form of imprint by constructing the action topological behavior imprint atlas, making the content of the evaluation report more intuitive and clear. The understanding score of the rehabilitation doctor on the feedback results of the system is improved from 3.1 of the traditional system to 4.7, improving the readability and clinical usability of the evaluation results. The present application is significantly superior to the prior art in structural analysis, phase recognition, environmental adaptability and explainability, and shows higher evaluation accuracy and application value.

[0039] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A wearable motion capture and evaluation method based on computer vision, characterized in that, include: Collect user motion video data, perform human region detection, human key point recognition and monocular 3D pose reconstruction, and perform smoothing, missing point compensation and time alignment on the obtained 3D skeleton data to generate a standardized 3D skeleton sequence. A skeletal topological adjacency matrix is ​​constructed based on the connection relationship of human joints. The relative structural features and topological invariants of joints are calculated based on standardized three-dimensional skeletal sequences. The action topological spatiotemporal tensor is constructed using a temporal unfolding method. Topological pattern recognition is performed on the action topological spatiotemporal tensor to determine the topological phase state of the action in each time period, detect the topological phase change points that occur during the action execution, construct the action topological phase transition map based on the topological phase state sequence, and divide the action execution process into multiple topological phase segments. By utilizing the phase segments of the action topology phase transition map, the topology phases of user actions and standard actions are matched to establish an alignment relationship at the action phase level. Based on the action topology spacetime tensor, a topology folding mapping is performed to generate a folded topology structure, and a topology unfolding mapping is performed to generate an unfolded topology structure. The reversibility of the folded and unfolded topologies is determined, and irreversible topological deformations are identified and the corresponding structural error locations are determined. Based on the action topology spatiotemporal tensor, an action topology behavior imprint surface is constructed. Topological texture imprints, folding imprints, impact imprints, and structural void imprints are extracted to form an action topology behavior imprint map. This map is then matched with the imprint signature of a standard action. Based on the differences in imprint structure, action execution deviation identification and action quality assessment are completed.

2. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The motion video data includes a sequence of continuous video frames used to record the user's motion execution process. The sequence of continuous video frames contains information on human body regions, human motion trajectory, and joint posture changes.

3. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The process of performing human region detection, human key point recognition, and monocular 3D pose reconstruction includes: Perform human region detection on action video frames to determine the spatial position and bounding area of ​​the human body in the video; Human key point recognition is performed on each video frame within the outer region to obtain two-dimensional key point coordinates; The corresponding three-dimensional joint positions are calculated based on the coordinates of two-dimensional key points to form three-dimensional skeleton data.

4. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The method of constructing the action topology spatiotemporal tensor using temporal expansion includes: Using a standardized 3D skeleton sequence as input, determine the skeleton frame corresponding to each time point, and mark the number of each joint and its hierarchical affiliation in the skeleton in each skeleton frame. Based on the human anatomical structure, the joint connection relationship is determined in advance. Joint pairs with direct skeletal connection relationship are recorded as a skeletal topology adjacency table. A unique topology connection identifier is assigned to each joint pair in the skeletal topology adjacency table. For each time frame in the standardized 3D skeleton sequence, all joint pairs are traversed according to the bone topology adjacency list. The relative structural feature information of each joint pair in the time frame is calculated, and the relative structural feature information is used as the topological feature description of the joint pair in the time frame. A temporal unfolding operation is performed on the time axis of the standardized 3D skeleton sequence. The topological feature descriptions sorted according to the bone topological adjacency list in each time frame are arranged in chronological order. A unique temporal index identifier is established for each time frame and each joint pair. The temporal index identifier is bound to the corresponding topological feature description, forming a one-to-one reversible mapping relationship between the temporal unfolding and the original temporal structure. Based on the topological feature description that has been expanded temporally and bound with temporal index identifiers, the topological features of all joint pairs corresponding to each time frame are organized into a multidimensional data structure with temporal and topological structure dimensions according to the temporal order and topological connection order. The multidimensional data structure is defined as the action topology spatiotemporal tensor.

5. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The action topological phase transition map is constructed based on the topological phase state sequence, dividing the action execution process into multiple topological phase segments, including: The action topology spatiotemporal tensor is obtained, and the topological features corresponding to each time frame are normalized and noise suppressed. They are then organized in chronological order into a topological feature sequence for phase analysis. The topological feature sequence is divided into patterns, and time frames with similar topological features are grouped into the same topological phase category. A unique topological phase state identifier is assigned to each time frame, forming a topological phase state sequence arranged by time. Traverse the topological phase state sequence along the time axis, group adjacent time frames with the same topological phase state identifier into a topological phase segment, record the start time, end time and corresponding topological phase state identifier of each topological phase segment, and obtain multiple topological phase segments covering the entire action execution process. Using each topological phase segment as a phase transition node, and based on the order and adjacency of each topological phase segment on the time axis, a transfer connection is established between adjacent topological phase segments. Each transfer connection is marked as a directed relationship from the starting topological phase segment to the target topological phase segment, forming an action phase transition link structure. Based on the action phase transition link structure, each topological phase segment is given additional attributes of phase duration and internal topological stability. Each transfer connection is given additional attributes of phase switching frequency, phase combination legality identifier, and abnormal phase transition marker. The topological phase segments and transfer connections carrying attribute information are organized into a multi-layer action topological phase transition map.

6. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The establishment of alignment relationships at the action phase level includes: Obtain the topological phase transition map of the action generated for the user action and the topological phase transition map of the standard action pre-generated and stored for the standard action. Read the topological phase segment of each of the two types of topological phase transition maps and the topological phase identifier, start time and end time information corresponding to each topological phase segment. Based on the topology phase identifier, each topology phase segment of the user action is initially mapped to each topology phase segment of the standard action, and a set of candidate topology phase mapping relationships is constructed. For each candidate mapping relationship, the sequential position of the user topology phase segment and the standard topology phase segment on the time axis is recorded. In the candidate topological phase correspondence set, based on the consistency of topological phase identifiers, the similarity of topological phase segment durations, and the consistency of transfer patterns of adjacent topological phase segments, a phase matching degree value is calculated for each candidate correspondence, and the phase matching degree value is associated with and stored with the corresponding candidate correspondence. Based on the chronological order of user action topology phase segments on the time axis, and considering the phase matching degree values ​​and time order constraints of each candidate correspondence, the correspondence combination that satisfies the continuity of topology phase order and has a high overall phase matching degree value is selected. This correspondence combination is determined as the final correspondence matching result between user action topology phase segments and standard action topology phase segments. The final matching results are organized into an alignment table at the action phase level. The alignment table records the standard topology phase segment identifier, start time, end time, and phase matching degree value corresponding to each user topology phase segment.

7. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The determination of the reversibility of folded and unfolded topologies, identifying irreversible topological deformations and determining corresponding structural error locations, includes: Read the action topology spatiotemporal tensor, time index identifier, and topology connection identifier to determine the positioning information of each time frame and each joint connection unit in the action topology spatiotemporal tensor. According to the preset folding rules, folding mapping is performed in the time dimension and topology dimension, and adjacent connection units with similar topological features are aggregated to form a folded topology, and a one-to-one correspondence between the time index identifier and the topology connection identifier is retained in the folded topology. Perform unfolding mapping according to the unfolding rules corresponding to the folding rules, restore the folded topology to an unfolded topology with time and topology dimensions, and restore the mapping relationship with time index identifiers and topology connection identifiers; Determine the reversibility of folded and unfolded topologies: First, there is an index backtracking consistency check, which checks whether each connection unit can accurately backtrack to the original time index identifier and topology connection identifier after expansion. Second, neighborhood consistency verification is performed to check whether the relative order and relative adjacency of each connected unit in the temporal and topological neighborhoods after unfolding are consistent with those before folding. Third, topological fingerprint verification is performed. Fingerprint information is generated based on connection order, number of branches, joint degree and connection connectivity. The fingerprint information at the corresponding positions before folding, after folding and after unfolding is compared for consistency. Fourth, path lag detection records the differences in the mapping paths between folding and unfolding. If there are cases where paths cannot be merged or multiple values ​​are reflected, it is determined to be irreversible. For any time index position or topology connection position where the verification fails to meet consistency, mark it as an irreversible topology deformation position; Based on the time index and topological connection identifiers corresponding to the marked positions, the specific time segments and joint connection points in the original 3D skeleton sequence are located, and the time segments and joint connection points are identified as structural error locations and the corresponding annotation information is output.

8. The wearable motion capture and evaluation method based on computer vision according to claim 1, characterized in that, The process of identifying action execution deviations and assessing action quality based on differences in imprint structure includes: Obtain the action topology spatiotemporal tensor and its time index identifier and topology connection identifier. While maintaining a one-to-one correspondence between the indices, expand it into a behavior imprint surface according to the time dimension and the topology structure dimension, and establish a backtrackable mapping relationship from the time index identifier, topology connection identifier to the coordinates of the behavior imprint surface. Multi-scale segmentation and rasterization encoding are performed on the behavioral imprint surface. Imprint features are extracted for each scale raster unit, and the imprint features of each raster unit are bound with time index identifier and topology connection identifier. The imprint features extracted and bound through multi-scale segmentation are organized into a multi-layered action topology behavior imprint map. The multi-layered structure includes a bottom layer that records basic imprint features, a stitching layer that records the transition relationship between adjacent imprints, a confidence mask layer that records feature stability, and a hole mounting layer that records the space occupied by missing segments. The action topology behavior imprint map is fingerprinted and encoded to generate imprint fingerprint codes for cross-sample alignment and fast comparison. The imprint fingerprint codes are associated with the phase alignment relationship to obtain the imprint map alignment result under the phase alignment constraint. The system calls the imprint signature of the standard action, performs corresponding matching on the imprint map under phase alignment constraints, locates the time segment and joint connection part where the difference is located based on the imprint fingerprint code and the difference mark in the multi-layer structure, and outputs the action execution deviation identification result and action quality evaluation result.

Images