A method and system for obtaining image recognition data

By integrating human kinematics knowledge on machine vision and using multi-hinge kinematics and dynamics models, the lack of image recognition data acquisition in rehabilitation medicine has been solved, and more accurate motion analysis and the formulation of rehabilitation treatment plans have been achieved, which has improved the Personalization and predictive abilities of treatment.

CN113869217BActive Publication Date: 2024-12-06BEIJING PLURAL HEALTH TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202111148127.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-29
Publication Date
2024-12-06
Estimated Expiration
2041-09-29

AI Technical Summary

Technical Problem

The prior art cannot accurately obtain image recognition data in rehabilitation medicine, resulting in limitations in kinematic analysis and electromyography detection, and cannot effectively guide rehabilitation treatment and predict the development of the disease.

Method used

Based on machine vision, human kinematics knowledge is integrated, and data analysis and calibration is carried out through multi-hinge kinematics and dynamics models to generate multimodal simulation results, assisting doctors and therapists in formulating treatment plans and predicting rehabilitation effects.

Benefits of technology

It improves the accuracy and robustness of motion detection, provides an accurate data source, can assist in the explanation and implementation of treatment plans, and the identification of rehabilitation prognosis, and enhances the personalization and effectiveness of treatment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113869217B_ABST
    Figure CN113869217B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for acquiring image recognition data, wherein the method comprises: performing image recognition on dynamic image data using a multi-hinge kinematic model to obtain initial feature point recognition data associated with a target object; customizing and adjusting the initial dynamic model based on attribute information to obtain an initial customized dynamic model associated with the target object; using the initial customized dynamic model as the current customized dynamic model, and using the initial feature point recognition data as the current feature point recognition data; performing motion posture simulation on the target object using the current customized dynamic model, and generating feature point simulation data associated with the target object based on the simulation result of the motion posture simulation; determining the matching degree between the feature point simulation data and the current feature point recognition data, and when the matching degree is greater than or equal to the matching threshold, using the current feature point recognition data as the image recognition data of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and more particularly to a method and system for acquiring image recognition data, as well as a storage medium and an electronic device. Background Art

[0002] Movement is the basis of human and animal life, and it is produced through the interaction of complex nerves, muscles, bones and joint systems. This determines that the study of movement requires the integration of multidisciplinary knowledge, such as medicine, biology, neuroscience, mechanics and robotics. As a basic discipline of rehabilitation medicine, kinesiology is a discipline that uses the concepts and methods of kinematics and kinetics, combined with knowledge of functional anatomy and neuromuscular physiology, to study human movement and movement behavior. Kinesiology takes human movement and movement behavior as the research object, explores the human body's movement function and its changing laws, and reveals the mechanism of human movement generation, change and re-acquisition, so as to provide effective guidance for the rehabilitation of movement function.

[0003] Rehabilitation medicine is an applied medical discipline that uses medical means to promote functional recovery of patients. As one of the most important treatment methods in rehabilitation medicine, exercise therapy uses exercise or force as a therapeutic factor to improve the physical, psychological and social functions of people with functional disabilities through improvement, compensation and substitution methods, so as to achieve the purpose of restoring function, improving quality of life and returning to society. All aspects of it are inseparable from human kinesiology.

[0004] (1) Kinematics and rehabilitation assessment: The most important basic assessment contents such as joint range of motion assessment, muscle strength assessment, and muscle tension assessment all involve the movement of muscles, joints, and bones. Human kinematics introduces the effects of different directions of movement brought about by the contraction of different muscles around the joints, which can help to fully understand that different movements are caused by the changes in the direction of muscle contraction force and the position of the joint center point, as well as the lever effect of muscle contraction.

[0005] (2) Kinematics and exercise therapy: The principles of many methods in exercise therapy come from the biomechanical principles of tissues and systems. For example, ① based on the biomechanical properties of cartilage, we can understand the basic principles and goals of joint loosening and joint traction; ② based on the muscle movement pattern, we can understand that simple muscle strength training often uses open-chain training (such as lying biceps training for patients with spinal cord injury) while functional muscle strength training often uses closed-chain training (such as sitting support training for patients with spinal cord injury), so as to understand how to reasonably choose training methods according to different training purposes.

[0006] (3) Kinesiology and occupational therapy: Occupational therapy requires a high level of creative thinking ability from the therapist. The therapist needs to analyze occupational activities to find out the reasons why the patient is limited in completing daily life activities or entertainment, work and other activities, so as to develop an individualized occupational plan and then conduct corresponding training to improve the patient's daily life activities and social activities. This requires a higher level of movement analysis ability: not only analyzing the movement of a single muscle or a joint, but also analyzing complex daily life activities and work activities.

[0007] (4) Kinematics and prosthetics and orthotics: Fully understand the principles of human kinematics. For example, by analyzing the closed-chain motion of the lower limbs during walking, we can better understand the relationship between the angles of the hip, knee, and ankle joints and walking stability, which can be used to guide the selection and design of orthotics.

[0008] However, in the field of rehabilitation medicine, there are still many limitations in the application of existing technologies to human kinematics:

[0009] 1. Limitations of kinematics applications

[0010] Kinematics is a branch of mechanics that describes and studies the law of the change of the position of an object over time from a geometric perspective. Kinematics does not consider the physical properties of the object itself and the forces applied to the object. It only takes the object as a simplified model of a point mass or a rigid body, and further studies the motion of deformable bodies such as elastic bodies and fluids, including motion trajectory, displacement, velocity, acceleration, rotation trajectory, angle, angular velocity, angular acceleration and other motion characteristics.

[0011] ① It focuses too much on measurement, but cannot assist or guide the design plan, let alone predict the development and make prognosis judgments. Whether it is wearable contact acquisition or visual non-contact acquisition, the existing kinematic analysis mostly stays at the data acquisition level, tracking the dynamic position of the main joints of the body and calculating the relevant spatiotemporal characteristics - such as displacement, velocity, acceleration, rotation trajectory, angle, angular velocity, angular acceleration, etc. - which only solves the measurement problem but cannot provide effective guidance. After the abnormality is obtained from the analysis of the measurement results, the doctor still needs to make prescriptions based on experience, and cannot use the measurement results to assist or guide the design, let alone predict the development of the disease and make prognosis judgments.

[0012] ② Each measurement point is relatively independent, and the structural constraints between the measurement points are not considered. Kinematic measurement only completes the data collection, but the logical relationship between the key points is not considered during the collection process. The human body is a hinge structure with multiple joints and multiple bones in series. Therefore, the human body posture is a complex behavior with multiple dimensions, high flexibility and multiple structural constraints. It is even more impossible to integrate the human musculoskeletal model and consider muscle force, etc.

[0013] ③ The measurement only stays at data collection, and no active research is conducted from the perspective of the human body's own control mechanism. Understanding the human body's own control mechanism can effectively carry out rehabilitation treatment, which is one of the necessary conditions for improving effectiveness. The human body is a complex system composed of hundreds of muscles. With the rapid development of big data and cloud computing technology, it is becoming possible to gradually build a real human muscle and bone model and simulate various human movements.

[0014] 2. Limitations of Dynamics Application

[0015] Kinetics is a branch of mechanics that studies the relationship between the forces acting on an object and its motion. Kinetics mainly considers two types of problems: studying the forces acting on an object through the known motion of the object; studying the motion of an object through the known forces acting on the object. Kinetics applications are mainly concentrated in demyography (dEMG) and surface EMG (sEMG). Demyography (dEMG) is the superposition of motor unit action potentials (MUAPs) in many muscle fibers in time and space. Surface EMG (sEMG) is the combined effect of superficial muscle EMG and electrical activity on nerve trunks on the surface of the skin, which can reflect neuromuscular activity to a certain extent.

[0016] ① Electromyography (dEMG and sEMG) are both invasive or contact examinations, with expensive equipment and time-consuming and laborious operation. Surface electromyography has irremovable systematic errors. Both dEMG and sEMG are contact examinations, which require electrodes to be attached to specific locations on the body surface. The operation is complicated, time-consuming and laborious, especially needle electromyography dEMG, which is an invasive examination that requires professional physicians to operate and has high equipment costs. Although surface electromyography sEMG is non-invasive and simple to operate, surface markers will produce inherent errors. When measuring the movement of skeletal components, there are errors related to the movement of the skin relative to the bones. This can be reduced, but not eliminated.

[0017] ② The acquisition channels and range of electromyography (dEMG and sEMG) are limited, which will limit the free movement area of ​​the subject. And it is impossible to continuously collect for a long time. Electromyography detects the electrophysiological activities of a single or local multiple motor units. Usually, it can only detect the signals of several skeletal muscles of the human body at the same time (usually dEMG can detect 2-4 muscles at the same time, and sEMG can detect 8-16 muscles at the same time). It is usually impossible to detect multiple groups of muscles for macroscopic movements at the same time. In addition, the contact acquisition method will interfere with the tester's own actions, and it is impossible to collect real movement data in a natural state, resulting in deviations in the test results. And contact acquisition is not suitable for long-term continuous detection.

[0018] ③ The EMG data visualization program is low, and can only detect whether the muscles are activated, but cannot determine which muscles cause the activity. Traditional EMG data is often displayed as a one-dimensional waveform signal, with a low degree of visualization. In addition, data analysis often requires supporting professional tools, which is cumbersome to operate and has low report readability. It can only detect whether the muscles are activated, but cannot determine which muscles cause the activity, which is not conducive to understanding and communication between doctors and patients.

[0019] ④ Due to the complexity of the human body structure, some parts of the electromyographic signals cannot be obtained. Due to the complexity of the human body structure, some variables cannot be directly obtained through measurement, such as the force generated by the muscles, and the stretching and recoil of the tendons during exercise.

[0020] ⑤ Electromyography is only a collection process and cannot be effectively simulated and predicted. The few modeling methods are based on simplified demographic data and lack individual patient data, which results in little real guidance for simulation and prediction results and is often only used for demonstration.

[0021] It can be seen from this that since it is impossible to obtain accurate image recognition data, there is a need in the prior art for a technical solution on how to obtain accurate image recognition technology. Summary of the invention

[0022] In view of the above problems, the present invention integrates the professional knowledge of human kinematics (Kinesiology) on the basis of machine vision. From the dimension of kinematics (Kinematics), the multi-dimensional and high-degree-of-freedom human body is abstracted as a hinge structure formed by multiple joints, and the kinematics professional indicators and the structural constraints of the human body are integrated for data analysis, data calibration and data enhancement; from the dimension of kinetics (Kinetics), a mechanical model of human muscle, bone and nerve control is established, and according to the patient's personalized posture data input, the multi-modal simulation results of bone, muscle and nerve control are reversely output, and the relationship between the force, muscle load and posture is visualized, which assists doctors, therapists and patients in communicating about the condition, explaining the cause and formulating the plan. At the same time, the functional adaptive changes of muscles caused by exercise training are predicted, the biomechanical effects are simulated, and the explanation of the auxiliary treatment plan, the execution of exercise training, and the judgment of rehabilitation prognosis are assisted.

[0023] Based on computer vision and computational science, the present invention uses the expertise of human kinesiology to perform corresponding processing:

[0024] 1) From the perspective of kinematics, the traditional machine vision based on the discrete and associative model of the human body is enhanced to identify the key points of the human body in a fragmented way, integrate the hinge-like structural constraints of the human body, calibrate and solve the problem of key point detection failure, improve the accuracy and robustness of motion detection, provide accurate data sources for motion analysis and assessment, and provide strong support for the implementation of non-contact multi-modal vision solutions;

[0025] 2) From the perspective of Kinetics, the traditional abstract posture detection, motion analysis and assessment based on key points of the human body are endowed with mechanical characteristics, and a complete model of nerves, muscles, bones and joints is constructed. In addition, various indicators of individual patients (such as height, weight, age, gender, movements, muscle strength, joint mobility, etc.) are integrated to reversely output multimodal simulation results of bones, muscles and neural control, visualize the relationship between force, muscle load and posture, and assist doctors, therapists and patients in communicating about the condition, explaining the cause of the disease and formulating plans.

[0026] At the same time, it predicts the functional adaptive changes of muscles caused by exercise training, simulates biomechanical effects, assists in the explanation of treatment plans, the implementation of exercise training, and the judgment of rehabilitation prognosis.

[0027] According to one aspect of the present invention, there is provided a method for acquiring image recognition data, the method comprising:

[0028] Step 1, obtaining dynamic image data and original feature point data of the target object, performing image recognition on the dynamic image data using a multi-hinge kinematic model to correct the original feature point data according to the image recognition result, thereby obtaining initial feature point recognition data associated with the target object;

[0029] Step 2, obtaining attribute information and an initial kinetic model of the target object, and customizing the initial kinetic model based on the attribute information, thereby obtaining an initial customized kinetic model associated with the target object;

[0030] Step 3, using the initial customized dynamics model as the current customized dynamics model, and using the initial feature point recognition data as the current feature point recognition data;

[0031] Step 4, using the current customized dynamics model to simulate the motion posture of the target object, and generating feature point simulation data associated with the target object according to the simulation result of the motion posture simulation;

[0032] Step 5: Determine the matching degree between the feature point simulation data and the current feature point recognition data.

[0033] When the matching degree is less than the matching threshold, the current feature point recognition data is calibrated using the feature point simulation data to generate calibrated feature point recognition data, and the current customized dynamics model is adjusted based on the calibrated feature point recognition data to obtain an adjusted customized dynamics model, the adjusted customized dynamics model is used as the current customized dynamics model and the calibrated feature point recognition data is used as the current feature point recognition data, and step 4 is performed;

[0034] When the matching degree is greater than or equal to the matching threshold, proceed to step 6;

[0035] Step 6: Use the current feature point recognition data as the image recognition data of the target object.

[0036] Preferably, obtaining dynamic image data and original feature point data of the target object includes:

[0037] Using a plurality of image acquisition devices to respectively acquire dynamic images of the target object, converting the dynamic images acquired by each image acquisition device into a data file, thereby obtaining a plurality of data files associated with the target object;

[0038] Performing image fusion on a plurality of data files associated with a target object to obtain dynamic image data of the target object;

[0039] Perform machine vision recognition on the dynamic image data of the target object to obtain the original feature point data of the target object.

[0040] Preferably, each of the plurality of image acquisition devices acquires a dynamic image of the target object at a respective reference position and at a respective predetermined shooting angle; or

[0041] Each image acquisition device forms a moving path based on the respective reference position, selects a shooting position on the respective moving path, and respectively acquires a dynamic image of the target object at the shooting position at the selected shooting angle.

[0042] Preferably, the multi-hinge kinematics model is used to perform image recognition on the dynamic image data to correct the original feature point data according to the image recognition result, thereby obtaining initial feature point recognition data associated with the target object, including:

[0043] Construct a multi-joint kinematic model, including:

[0044] Create a standard kinematic model for M knmt = {B i ,J j |i∈(0,H-1],j∈(0,H]}, standard kinematic model M knmt From H feature points Jj and J-1 bone B i Composition; wherein i is a natural number and 1≤i≤H-1, j is a natural number and 1≤j≤H;

[0045] The length of each bone is L bone = {l i |i∈(0,H-1]}, where the hinge relationship in the multi-hinge kinematic model is a series structure;

[0046] Based on the position information of the characteristic points of the serial structure, hinge constraints are added to the standard kinematic model to construct a multi-hinge kinematic model:

[0047] Where u is the two-dimensional position information;

[0048] or

[0049] Where u is the three-dimensional position information;

[0050] Among them, u is the current feature point, POS u Represents the spatial position information of u; when u is two-dimensional position information, the format of u is (x, y); when u is three-dimensional position information, the format of u is (x, y, z);

[0051] parent(u) is the feature point set consisting of all parent feature points in series relationship with the current feature point u under the hinge relationship constraint or the feature point set consisting of all previous feature points;

[0052] Rot(θ v ) is the rotation matrix of each parent feature point or previous feature point, Trans(l v ) is the displacement matrix of each parent feature point or previous feature point; where v is a natural number, 1≤v≤H;

[0053] By rotating the matrix Rot(θ v ) and the displacement matrix Trans(l v ) determines the position information of the current feature point u, so that the multi-hinge kinematic model can perform image recognition on the dynamic image data, and correct the original feature point data according to the image recognition result, so as to obtain the initial feature point recognition data associated with the target object

[0054] The initial feature point recognition data includes a plurality of feature points Wherein i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate for collecting or acquiring the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, wherein k is the serial number of the feature point, k is a positive integer and 0≤k≤K.

[0055] Preferably, the method of obtaining the attribute information and the initial kinetic model of the target object and customizing the initial kinetic model based on the attribute information to obtain an initial customized kinetic model associated with the target object includes:

[0056] Acquire the attribute information of the target object, including basic data, motion capture data, pressure data, surface electromyography (sEMG) data, and image data;

[0057] Constructing an initial dynamic model M for standard human musculoskeletal morphology and neural control kntc ={B,MU,N}, where B represents bones, MU represents muscles, and N represents neural control;

[0058] calculate

[0059]

[0060] Where M is an n*n symmetric positive definite matrix, which is used to represent the internal mass and coordinate matrix of the target object. G is the acceleration constraint Jacobian matrix of m*n dimensions, where m is the constraint term, n is the degree of freedom, and G T is the transposed matrix of G, q is the generalized human body dynamic coordinate position, u and is the velocity information, z is the muscle state, and q, u and z are functions of time t; λ is a vector of m Lagrange multipliers of the constraint force; f(t,q,u,z) is the external force f applied and the internal inertia force f inertial The combined force of

[0061] The initial dynamics model is customized based on the attribute information to obtain an initial customized dynamics model associated with the target object:

[0062] M′ kntc =Modu×{B,MU,N}, where Modu is

[0063] Modu=s(S B ×S MU ×S N )

[0064] Where s is the overall scaling factor and ranges from 0≤s≤1; SB , S MU , S N Representing the scaling matrices for bone, muscle, and neural regulation, respectively.

[0065] Preferably, the step of calibrating the current feature point recognition data using the feature point simulation data to generate calibrated feature point recognition data comprises:

[0066] According to the initial customized kinetic model M′ kntc Perform inverse kinematics calculations Get the simulated data of the feature points of the target object predicted by the model Feature point simulation data includes multiple feature points

[0067] Wherein i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate of acquiring or obtaining the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, wherein k is the serial number of the feature point, k is a positive integer and 0≤k≤K;

[0068] The Loss function is set according to the least squares method, and each feature point in the current feature point recognition data is calibrated from multiple angles of rotation and displacement:

[0069]

[0070] Thus, calibrated feature points integrating intrinsic kinematic structural constraints are obtained, and multiple calibrated feature points are used to form calibrated feature point recognition data, where j is a natural number and 1≤j≤H, H is the number of feature points, is the feature point in the current feature point recognition data, is the feature point in the feature point simulation data.

[0071] Preferably, it also includes:

[0072] Through the inverse kinematics model, the motion posture output results of the customized dynamics model of the target object, that is, the feature point simulation data, are obtained. k∈(0,K])}; is a feature point in the feature point simulation data, where i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate for collecting or acquiring the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, where k is the serial number of the feature point, k is a positive integer and 0≤k≤K,, where j is a natural number and 1≤j≤H, and H is the number of feature points

[0073] Extract angle features, trajectory features, speed features and periodic states to obtain:

[0074] (1) Angle characteristics:

[0075]

[0076] Where i and t represent time, i.e., the i-th frame image or the t-th frame image; a represents the angle to be extracted; after a is selected, three adjacent feature points l, m and k are selected accordingly, and the coordinate information format of each feature point is (X, Y, Z);

[0077] (2) Trajectory characteristics:

[0078]

[0079] There are many variations of trajectory features. The center point trajectory information of the internal structural constraints of the adjacent points is calculated based on the position information of each feature point in three dimensions: X, Y, and Z.

[0080] (3) Speed ​​characteristics:

[0081]

[0082] t∈(0,T*fps)

[0083] exist In the formula, the default value of t is 1, that is, the motion speed of the feature point is solved through two adjacent frames;

[0084] (4) Cycle characteristics:

[0085] Apply periodic operators to the above angle features, trajectory features and speed features to determine the periodic features Period of the angle features, trajectory features and speed features. Ang ,Period trj ,Period vel .

[0086] According to another aspect of the present invention, a system for acquiring image recognition data is provided, the system comprising:

[0087] A recognition device, for acquiring dynamic image data and original feature point data of a target object, performing image recognition on the dynamic image data using a multi-hinge kinematic model to correct the original feature point data according to the image recognition result, thereby obtaining initial feature point recognition data associated with the target object;

[0088] An adjustment device, used to obtain attribute information and an initial dynamic model of a target object, and to perform customized adjustment on the initial dynamic model based on the attribute information, thereby obtaining an initial customized dynamic model associated with the target object;

[0089] A processing device, used to use the initial customized dynamics model as the current customized dynamics model and the initial feature point recognition data as the current feature point recognition data;

[0090] A simulation device, used to simulate the motion posture of the target object using the current customized dynamics model, and generate feature point simulation data associated with the target object according to the simulation result of the motion posture simulation;

[0091] A determination device is used to determine the degree of match between feature point simulation data and current feature point recognition data. When the degree of match is less than a matching threshold, the current feature point recognition data is calibrated using the feature point simulation data to generate calibrated feature point recognition data, and the current customized dynamic model is adjusted based on the calibrated feature point recognition data to obtain an adjusted customized dynamic model, and the adjusted customized dynamic model is used as the current customized dynamic model and the calibrated feature point recognition data is used as the current feature point recognition data; when the degree of match is greater than or equal to the matching threshold, the current feature point recognition data is used as the image recognition data of the target object.

[0092] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute any one of the methods described above.

[0093] According to another aspect of the present invention, there is provided an electronic device, characterized in that the electronic device comprises:

[0094] processor;

[0095] a memory for storing instructions executable by the processor;

[0096] The processor is used to read the executable instructions from the memory and execute the executable instructions to implement any one of the methods described above.

[0097] The key technical improvements of the technical solution of the present invention are mainly:

[0098] (1) A new model for computer vision posture and motion detection that integrates kinematics structural constraints;

[0099] (2) Personalized Kinetics modeling driven by continuous kinematics data collection;

[0100] (3) Based on personalized kinetics modeling and calculation, it can assist in visualizing the causes of the disease, design digital program simulations, and evaluate the prognosis and efficacy of exercise training. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0102] Figure 1 A flowchart of a method for acquiring image recognition data according to an embodiment of the present invention;

[0103] Figure 2 is a schematic diagram of corresponding positions of feature points according to an embodiment of the present invention;

[0104] Figure 3 is a schematic diagram of a two-dimensional kinematic model according to an embodiment of the present invention;

[0105] Figure 4 A schematic diagram of a system architecture from kinematics to dynamics according to an embodiment of the present invention;

[0106] Figure 5 is a schematic diagram of an exemplary angle curve according to an embodiment of the present invention;

[0107] Figure 6 A schematic diagram of a system architecture from dynamics to kinematics according to an embodiment of the present invention;

[0108] Figure 7 Schematic diagram of the structure of a system for acquiring image recognition data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0109] Figure 1 FIG. 1 is a flow chart of a method 100 for acquiring image recognition data according to an embodiment of the present invention. The method 100 starts from step 101 .

[0110] In step 101, the dynamic image data and original feature point data of the target object are obtained, and the dynamic image data are subjected to image recognition using a multi-hinge kinematic model to correct the original feature point data according to the result of the image recognition, thereby obtaining the initial feature point recognition data associated with the target object. The acquisition of the dynamic image data and original feature point data of the target object includes: using multiple image acquisition devices to respectively acquire dynamic images of the target object, converting the dynamic images acquired by each image acquisition device into data files, thereby obtaining multiple data files associated with the target object; performing image fusion on the multiple data files associated with the target object to obtain the dynamic image data of the target object; and performing machine vision recognition on the dynamic image data of the target object to obtain the original feature point data of the target object.

[0111] Among them, each of the multiple image acquisition devices respectively acquires a dynamic image of the target object at its respective reference position and at its respective predetermined shooting angle; or each image acquisition device forms a moving path based on its respective reference position, selects a shooting position on its respective moving path, and respectively acquires a dynamic image of the target object at the shooting position and at the selected shooting angle.

[0112] The multi-hinge kinematics model is used to perform image recognition on the dynamic image data to correct the original feature point data according to the image recognition result, thereby obtaining the initial feature point recognition data associated with the target object, including:

[0113] Construct a multi-joint kinematic model, including:

[0114] Create a standard kinematic model for M knmt = {B i ,J j |i∈(0,H-1],j∈(0,H]}, standard kinematic model M knmt From H feature points J j and J-1 bone B i Composition; wherein i is a natural number and 1≤i≤H-1, j is a natural number and 1≤j≤H;

[0115] The length of each bone is L bone = {l i |i∈(0,H-1]}, where the hinge relationship in the multi-hinge kinematic model is a series structure;

[0116] Based on the position information of the characteristic points of the serial structure, hinge constraints are added to the standard kinematic model to construct a multi-hinge kinematic model:

[0117] Where u is the two-dimensional position information;

[0118] or

[0119] Where u is the three-dimensional position information;

[0120] Among them, u is the current feature point, POS u Represents the spatial position information of u; when u is two-dimensional position information, the format of u is (x, y); when u is three-dimensional position information, the format of u is (x, y, z);

[0121] parent(u) is the feature point set consisting of all parent feature points in series relationship with the current feature point u under the hinge relationship constraint or the feature point set consisting of all previous feature points;

[0122] Rot(θ v ) is the rotation matrix of each parent feature point or previous feature point, Trans(l v ) is the displacement matrix of each parent feature point or previous feature point; where v is a natural number, 1≤v≤H;

[0123] By rotating the matrix Rot(θ v ) and the displacement matrix Trans(l v ) determines the position information of the current feature point u, so that the multi-hinge kinematic model can perform image recognition on the dynamic image data, and correct the original feature point data according to the image recognition result, so as to obtain the initial feature point recognition data associated with the target object

[0124] The initial feature point recognition data includes feature points Wherein i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate for collecting or acquiring the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, wherein k is the serial number of the feature point, k is a positive integer and 0≤k≤K.

[0125] In step 102, the attribute information and the initial kinetic model of the target object are obtained, and the initial kinetic model is customized based on the attribute information, so as to obtain an initial customized kinetic model associated with the target object. The attribute information and the initial kinetic model of the target object are obtained, and the initial kinetic model is customized based on the attribute information, so as to obtain an initial customized kinetic model associated with the target object, including:

[0126] Acquire the attribute information of the target object, including basic data, motion capture data, pressure data, surface electromyography (sEMG) data, and image data;

[0127] Constructing an initial dynamic model M for standard human musculoskeletal morphology and neural control kntc ={B,MU,N}, where B represents bones, MU represents muscles, and N represents neural control;

[0128] calculate

[0129]

[0130] Where M is an n*n symmetric positive definite matrix, which is used to represent the internal mass and coordinate matrix of the target object. G is the acceleration constraint Jacobian matrix of m*n dimensions, where m is the constraint term, n is the degree of freedom, and G T is the transposed matrix of G, q is the generalized human body dynamic coordinate position, u and is the velocity information, z is the muscle state, and q, u and z are functions of time t; λ is a vector of m Lagrange multipliers of the constraint force; f(t,q,u,z) is the external force f applied and the internal inertia force f inertial The combined force of

[0131] The initial dynamics model is customized based on the attribute information to obtain an initial customized dynamics model associated with the target object:

[0132] M′ kntc =Modu×{B,MU,N}, where Modu is

[0133] Modu=s(S B ×S MU ×S N )

[0134] Where s is the overall scaling factor and ranges from 0≤s≤1; S B , S MU , S N Representing the scaling matrices for bone, muscle, and neural regulation, respectively.

[0135] In step 103, the initial customized dynamics model is used as the current customized dynamics model, and the initial feature point recognition data is used as the current feature point recognition data.

[0136] In step 104, motion posture simulation is performed on the target object using the current customized dynamics model, and feature point simulation data associated with the target object is generated according to the simulation result of the motion posture simulation.

[0137] In step 105, the matching degree between the feature point simulation data and the current feature point recognition data is determined.

[0138] In step 106, is it determined whether the degree of match is less than the matching threshold? When the degree of match is less than the matching threshold, step 107 is performed to calibrate the current feature point recognition data using the feature point simulation data to generate calibrated feature point recognition data, and the current customized dynamics model is adjusted based on the calibrated feature point recognition data to obtain an adjusted customized dynamics model, the adjusted customized dynamics model is used as the current customized dynamics model and the calibrated feature point recognition data is used as the current feature point recognition data, and then step 104 is performed.

[0139] The method of calibrating the current feature point recognition data using the feature point simulation data to generate calibrated feature point recognition data includes:

[0140] According to the initial customized kinetic model M′ kntc Perform inverse kinematics calculations Get the simulated data of the feature points of the target object predicted by the model Feature point simulation data includes multiple feature points

[0141] Wherein i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate of acquiring or obtaining the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, wherein k is the serial number of the feature point, k is a positive integer and 0≤k≤K;

[0142] The Loss function is set according to the least squares method, and each feature point in the current feature point recognition data is calibrated from multiple angles of rotation and displacement:

[0143]

[0144] Thus, calibrated feature points integrating intrinsic kinematic structural constraints are obtained, and multiple calibrated feature points are used to form calibrated feature point recognition data, where j is a natural number and 1≤j≤H, H is the number of feature points, is the feature point in the current feature point recognition data, is the feature point in the feature point simulation data.

[0145] When the matching degree is greater than or equal to the matching threshold, proceed to step 108, step 6, using the current feature point recognition data as the image recognition data of the target object.

[0146] It also includes obtaining the motion posture output results of the customized dynamics model of the target object, that is, the feature point simulation data, through the inverse kinematics model. is a feature point in the feature point simulation data, where i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate for collecting or acquiring dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, where k is the serial number of the feature point, k is a positive integer and 0≤k≤K,, where j is a natural number and 1≤j≤H, and H is the number of feature points

[0147] Extract angle features, trajectory features, speed features and periodic states to obtain:

[0148] (1) Angle characteristics:

[0149]

[0150] Where i and t represent time, i.e., the i-th frame image or the t-th frame image; a represents the angle to be extracted; after a is selected, three adjacent feature points l, m and k are selected accordingly, and the coordinate information format of each feature point is (X, Y, Z);

[0151] (2) Trajectory characteristics:

[0152]

[0153] There are many variations of trajectory features. The center point trajectory information of the internal structural constraints of the adjacent points is calculated based on the position information of each feature point in three dimensions: X, Y, and Z.

[0154] (3) Speed ​​characteristics:

[0155]

[0156] t∈(0,T*fps)

[0157] exist In the formula, the default value of t is 1, that is, the motion speed of the feature point is solved through two adjacent frames;

[0158] (4) Cycle characteristics:

[0159] Apply periodic operators to the above angle features, trajectory features and speed features to determine the periodic features Period of the angle features, trajectory features and speed features. Ang ,Period trj ,Period vel .

[0160] The overall framework of the present invention mainly consists of two parts:

[0161] Part 1: From Kinematics to Kinetics

[0162] Figure 4 Schematic diagram of the system architecture from kinematics to dynamics according to an embodiment of the present invention. Figure 4 As shown in the figure, the problem solved in the first part is to enhance the traditional machine vision based on the discrete and associative model of the human body to identify the key points (or feature points) of the human body from the perspective of kinematics, and integrate the hinge-like structural constraints of the human body. At the same time, the standard dynamic model is modulated by the real data of the patient's personal assessment, and the key point data is output in a simulated manner. The adversarial learning is carried out in two aspects to calibrate and solve the problem of key point detection failure, improve the accuracy and robustness of motion detection, provide an accurate data source for motion analysis and assessment, and provide strong support for the implementation of non-contact multi-modal vision solutions.

[0163] 1) Establishment and integration of multi-hinge kinematics model (Kinematics)

[0164] The traditional machine vision solution is to collect frame data

[0165]

[0166] Where i represents the i-th frame data, ranging from 0 to T*fps (T is the video acquisition duration, in seconds; fps is the frame rate, in frames per second, and T*fps is the total number of frames); m represents the data acquisition method or mode, common modes include RGB color mode, RGBD depth map mode,

[0167] gray grayscale mode, infrared infrared imaging mode and so on.

[0168] The feature point (or key point) information obtained is Where K is the total number of key points extracted by the model (here we only take the COCO model as an example, K = 16, that is, 17 key points of the human body are extracted, such as Figure 2 The present invention also supports the output results of other posture extraction models). is the original feature point data.

[0169] In the traditional visual computing process, the overall pixel It is directly used as input for analysis, and low-dimensional and high-dimensional features are gradually extracted through deep neural networks (mainly convolutional neural networks) to achieve key point extraction (i.e. the main joints of the human body).

[0170] Although the pixelated digital representation can restore colors and related local details very well, it loses important internal structural constraints of the human body, such as the kinetic chain, that is, several links of the human body are connected in a certain order with the help of joints, which are divided into open kinematic chain (OKC) and closed kinematic chain (CKC).

[0171] This invention models the structural constraints inherent in the human body and integrates them into the existing data analysis process. Assumptions:

[0172] The human kinematics model is M knmt = {B i ,J j |i∈(0,J-1],j∈(0,J]}, that is, J key points J j , J-1 bone B i The length of each bone is L bone = {l i |i∈(0,J-1]}. Since the human body hinge is a series structure (in the case of parallel connection, such as the hips and shoulders, it is regarded as two independent series connections), the overall key point position constraint

[0173] At this time, u is the 2D position information

[0174] At this time, u is the 3D position information

[0175] Where u is the key point where the spatial position information needs to be calculated. u Represents the spatial position information of u. The format for 2D information is (x, y) and the format for 3D information is (x, y, z). parent(u) is the set of all parent keypoints or previous keypoints connected in series with the current keypoint u under the hinge constraint. Rot(θ v ) is the rotation matrix of each preceding key point, Trans(l v ) is the displacement matrix of each previous key point. Due to the rigidity of the hinge structure, the rotation matrix Rot(θ v ) and the displacement matrix Trans(l v ) can solve the position of the current key point.

[0176] Taking the right lower limb as an example, the two-dimensional kinematic model is as follows Figure 3 shown.

[0177] The above-mentioned intrinsic structural constraints of human kinematics are integrated into the output layer of the neural network and used as prior knowledge to calibrate the results, thereby outputting correct key point information that conforms to the intrinsic structural constraints of the human body.

[0178] New key point information

[0179] It is the initial feature point recognition data or the current feature point recognition data. is the feature point in the current feature point recognition data or the feature point in the initial feature point recognition data, is the feature point in the feature point simulation data.

[0180] 2) Kinematics customizes Kinetics (Scale, model modified)

[0181] Constructing a standard human musculoskeletal morphology and neural control mechanical model M kntc ={B,MU,N}, B represents bones, MU represents muscles, and N represents neural control. Construct a mechanical model.

[0182]

[0183] Where M is an n*n symmetric positive definite matrix, which refers to the internal mass and coordinate matrix of the human body, and G is an m*n dimensional acceleration constraint Jacobian matrix (m is the constraint term, n is the degree of freedom). q is the generalized human body dynamic coordinate position, u and is the velocity information, z is the muscle state, and all three are functions of time t. λ is a vector of m Lagrange multipliers of the constraint force. f(t,q,u,z) is the external force f applied and the internal inertia force f inertial The combined force.

[0184] Based on the standard model, the standard model is personalized according to the patient's own assessment data, including basic information (height, weight), motion capture data, pressure data (plantar pressure), surface electromyography (sEMG) data or other imaging data (such as ultrasound, X-ray, etc.).

[0185] M′ kntc =Modu×{B,MU,N}, where Modu is

[0186] Modu=s(S B ×S MU ×S N )

[0187] Where s is the overall scale adjustment coefficient ranging from 0 to 1.B , S MU , S N Representing the scaling matrices for bone, muscle, and neural regulation, respectively.

[0188] Kinetics result output to calibrate Kinematics: (InverseKinematics)

[0189] According to the above personalized modulation model M′ kntc , to calculate the inverse kinematics problem Get the coordinate information of the key points of the human body predicted by the model

[0190] The Loss function is set according to the least squares method, and the coordinate information of the original detection key points is calibrated from multiple angles such as rotation and displacement.

[0191]

[0192] Thus, accurate key point results are obtained that incorporate the intrinsic kinematic structural constraints of the human body. is the feature point in the current feature point recognition data, is the feature point in the feature point simulation data.

[0193] The overall process of the overall framework is as follows Figure 4 As shown in (the closed loop of ①-②-③-④ represents the process of gradual calibration and tuning). Figure 4 As shown, the key to this part lies in the closed-loop process of ①-②-③-④ above.

[0194] ① As mentioned above, a new dynamic model M′ is obtained by modulating the standard model based on the patient's medical history and basic conditions. kntc .

[0195] ② Through the inverse kinematics model, the motion posture output results of the patient's personalized model can be obtained, that is, the feature point simulation data On this basis, conventional feature extraction is performed, including angle, trajectory, velocity (or acceleration) and period. The calculation method is as follows:

[0196] (1) Angle characteristics:

[0197]

[0198] Where i represents time, i.e. the i-th frame image. a represents the angle to be extracted, such as the knee angle, ankle angle, hip angle, etc. After selecting a, three adjacent key points l, m and k are selected accordingly. The coordinate information format of each key point is (X, Y, Z).

[0199] Figure 5 The angle curves of the left and right hip joints and the left and right knees are shown.

[0200] (2) Trajectory characteristics:

[0201]

[0202] There are many variations of trajectory features. For example, the center point trajectory information of the adjacent points of the internal structural constraints can be calculated based on the X, Y, and Z dimensional position information of each key point.

[0203] (3) Speed ​​characteristics:

[0204]

[0205] t∈(0,T*fps)

[0206] By default, t is equal to 1, which means that the movement speed of the key point is solved by the two frames before and after.

[0207] (4) Cycle characteristics:

[0208] Apply periodic operators (including but not limited to Fourier transform, ACF and PACF operators, etc.) to the above three types of features to find their respective periods Period Ang ,Period trj ,Period vel

[0209] ③ The above calibration new node that integrates kinematic constraints The output of the traditional CV algorithm is optimized by using the inherent structural constraints to filter out obvious recognition errors. However, in the time dimension, the abnormal recognition of dynamic motion is still unable to be effectively corrected, such as the abnormal knee joint of the lower limbs and the abnormal ankle joint speed of Parkinson's patients. Therefore, the characteristics of angle, trajectory and speed obtained by dynamic simulation, as well as their respective periodicity, are compared with the kinematic calibration results with their respective periods as time windows. Compare, slide to find outliers and make corrections.

[0210] ④ The corrected results are used as the patient's new historical data and then imported into the dynamic model M′ as input samples. kntc middle.

[0211] The termination condition of the above loop process iteration is

[0212]

[0213] Or the empirical value of the upper limit of the number of iterations, n it =6 (NVIDIA RTX3090 is used as the reference for recommendation here, and users can make personalized adjustments based on their own hardware computing power)

[0214] The above process draws on the idea of ​​adversarial learning, using dynamic simulation output to assist in identifying errors in machine vision detection, and in turn re-updating the dynamic model through the initially corrected machine vision output, and repeating this process until the simulation calibration and machine vision detection results are consistent. The result at this point is considered to be true, accurate, and conforms to the posture information of human kinematics constraints.

[0215] Part II: From Kinetics to Kinematics

[0216] Figure 6 FIG. 1 is a schematic diagram of a system architecture from dynamics to kinematics according to an embodiment of the present invention. Figure 6 As shown in the figure, part of the problem to be solved is to endow the traditional abstract posture detection, motion analysis and assessment based on key points of the human body with mechanical characteristics from the perspective of kinetics, build a complete model of nerves, muscles, bones and joints, and integrate various indicators of individual patients (such as height, weight, age, gender, movements, muscle strength, joint mobility, etc.), reversely output the multimodal simulation results of bones, muscles and neural control, visualize the relationship between force, muscle load and posture, and assist doctors, therapists and patients in communicating about the condition, explaining the cause and formulating plans. At the same time, predict the functional adaptive changes of muscles caused by exercise training, simulate biomechanical effects, assist in explaining treatment plans, executing exercise training, and judging rehabilitation prognosis.

[0217] According to the human musculoskeletal morphology and neural control dynamics model modulated in the first part,

[0218] M′ kntc =Modu×{B,MU,N}, where Modu is

[0219] Modu=s(S B ×S MU ×S N )

[0220] Where s is the overall scale adjustment coefficient ranging from 0 to 1. B , S MU , S N Representing the scaling matrices for bone, muscle, and neural regulation, respectively.

[0221] First, the inverse operation of the dynamic model is directly performed to obtain the patient's bones corresponding to the current motion posture. muscle strength and neural control And other multi-dimensional information.

[0222]

[0223] It is used to provide a multi-dimensional visual display of the current status, assisting doctors, therapists and patients in communicating about the condition, explaining the cause of the disease and formulating plans.

[0224] Secondly, after the treatment plan is determined, the plan (training frequency E freq , training duration E time 、Training Part E parts , training intensity E force ) is re-entered into the model as a parameter,

[0225]

[0226]

[0227] According to the functional adaptability of muscles, biomechanical effects are simulated, and the simulation effects after each training stage are visualized, namely the bones, muscles, forces and neural control mentioned above.

[0228]

[0229] And the corresponding simulated visualization posture

[0230]

[0231] Based on the above output results, auxiliary treatment plans are explained, exercise training is implemented, and rehabilitation prognosis is determined.

[0232] The main purpose of the first part is to conduct adversarial learning between the kinematic simulation results output by the dynamics simulation model and the kinematic measured results of computer vision monitoring to ensure the accuracy and personalization of the non-contact visual monitoring solution (because the dynamics simulation model is personalized).

[0233] The second part is based on the first part, and it is considered that the output result of computer vision detection is the correct solution. At this time, the final correct data is input into the dynamics simulation model to obtain a visualized simulation solution of muscles, bones, electromyography, etc. This is equivalent to enriching the dimensions of the key point posture detection data that was previously based on machine vision, which is more conducive to doctors' visualized explanation of the condition and communication with patients. The dynamics of this part is the role of traditional dynamics simulation, that is, visualization.

[0234] Figure 7 FIG. 7 is a schematic diagram of a system 700 for acquiring image recognition data according to an embodiment of the present invention. The system 700 includes:

[0235] The recognition device 701 is used to obtain dynamic image data and original feature point data of the target object, and use the multi-hinge kinematic model to perform image recognition on the dynamic image data to correct the original feature point data according to the image recognition result, so as to obtain initial feature point recognition data associated with the target object;

[0236] The adjustment device 702 is used to obtain the attribute information and the initial dynamic model of the target object, and customize the initial dynamic model based on the attribute information to obtain the initial customized dynamic model associated with the target object;

[0237] Processing means 703, for using the initial customized dynamics model as the current customized dynamics model, and using the initial feature point recognition data as the current feature point recognition data;

[0238] The simulation device 704 is used to perform motion posture simulation on the target object using the current customized dynamics model, and generate feature point simulation data associated with the target object according to the simulation result of the motion posture simulation;

[0239] The determination device 705 is used to determine the matching degree between the feature point simulation data and the current feature point recognition data. When the matching degree is less than the matching threshold, the current feature point recognition data is calibrated using the feature point simulation data to generate calibrated feature point recognition data, and the current customized dynamic model is adjusted based on the calibrated feature point recognition data to obtain an adjusted customized dynamic model, and the adjusted customized dynamic model is used as the current customized dynamic model and the calibrated feature point recognition data is used as the current feature point recognition data; when the matching degree is greater than or equal to the matching threshold, the current feature point recognition data is used as the image recognition data of the target object.

Claims

1. A method for obtaining image recognition data, the method comprising: Step 1, obtaining dynamic image data and original feature point data of the target object, performing image recognition on the dynamic image data using a multi-hinge kinematic model to correct the original feature point data according to the image recognition result, thereby obtaining initial feature point recognition data associated with the target object, wherein the multi-hinge kinematic model is constructed based on H feature points and H-1 skeleton components and adding hinge constraints; Step 2, obtaining attribute information of the target object and an initial dynamic model for standard human musculoskeletal morphology and neural control, and performing customized scale adjustment on the initial dynamic model based on the attribute information, thereby obtaining an initial customized dynamic model associated with the target object; Step 3, using the initial customized dynamics model as the current customized dynamics model, and using the initial feature point recognition data as the current feature point recognition data; Step 4, using the current customized dynamics model to simulate the motion posture of the target object, and generating feature point simulation data associated with the target object according to the simulation result of the motion posture simulation; Step 5, determining the degree of match between the feature point simulation data and the current feature point recognition data, and when the degree of match is less than the matching threshold, calibrating the current feature point recognition data using the feature point simulation data to generate calibrated feature point recognition data, and adjusting the current customized dynamics model based on the calibrated feature point recognition data to obtain an adjusted customized dynamics model, using the adjusted customized dynamics model as the current customized dynamics model and using the calibrated feature point recognition data as the current feature point recognition data, and proceeding to step 4; When the matching degree is greater than or equal to the matching threshold, proceed to step 6; Step 6: Use the current feature point recognition data as the image recognition data of the target object.

2. The method according to claim 1, wherein acquiring dynamic image data and original feature point data of the target object comprises: Using a plurality of image acquisition devices to respectively acquire dynamic images of the target object, converting the dynamic images acquired by each image acquisition device into a data file, thereby obtaining a plurality of data files associated with the target object; Performing image fusion on a plurality of data files associated with a target object to obtain dynamic image data of the target object; Perform machine vision recognition on the dynamic image data of the target object to obtain the original feature point data of the target object.

3. The method according to claim 2, wherein: Each of the plurality of image acquisition devices acquires a dynamic image of the target object at a respective reference position and at a respective predetermined shooting angle; Alternatively, each image acquisition device forms a moving path based on the respective reference position, selects a shooting position along the respective moving path, and acquires dynamic images of the target object at the shooting position at the selected shooting angle.

4. The method according to claim 1, wherein the method uses a multi-hinge kinematics model to perform image recognition on the dynamic image data to correct the original feature point data according to the image recognition result, thereby obtaining initial feature point recognition data associated with the target object, comprising: Construct a multi-joint kinematic model, including: Create a standard kinematic model for M knmt = {B i ,J j |i∈(0,H-1],j∈(0,H]}, Standard kinematic model M knmt From H feature points J j and H-1 segment skeleton B i Composition; wherein i is a natural number and 1≤i≤H-1, j is a natural number and 1≤j≤H; The length of each bone is L bone = {l i |i∈(0,H-1]}, The hinge relationship in the multi-hinge kinematic model is a series structure; Based on the position information of the characteristic points of the serial structure, hinge constraints are added to the standard kinematic model to construct a multi-hinge kinematic model: wherein the position information of u is two-dimensional position information; or The position information of u is three-dimensional position information; Among them, u is the current feature point, POS u represents the spatial position information of u; when the position information of u is two-dimensional position information, the format of the position information of u is (x, y); when the position information of u is three-dimensional position information, the format of the position information of u is (x, y, z); parent(u) is the feature point set consisting of all parent feature points in series relationship with the current feature point u under the hinge relationship constraint or the feature point set consisting of all previous feature points; Rot(θ v ) is the rotation matrix of each parent feature point or previous feature point, Trans(l v ) is the displacement matrix of each parent feature point or previous feature point; where v is a natural number, 1≤v≤H; By rotating the matrix Rot(θ v ) and the displacement matrix Trans(l v ) determines the position information of the current feature point u, so that the multi-hinge kinematic model can perform image recognition on the dynamic image data, and correct the original feature point data according to the image recognition result, so as to obtain the initial feature point recognition data associated with the target object The initial feature point recognition data includes a plurality of feature points Wherein i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate for collecting or acquiring the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, where k is the serial number of the feature point, k is a positive integer and 0≤k≤K, and K is the total number of key points extracted by the model.

5. The method according to claim 1, wherein acquiring attribute information of the target object and an initial dynamic model for standard human musculoskeletal morphology and neural control, and performing customized scale adjustment on the initial dynamic model based on the attribute information to obtain an initial customized dynamic model associated with the target object, comprises: Acquire the attribute information of the target object, including basic data, motion capture data, pressure data, surface electromyography (sEMG) data, and image data; Constructing an initial dynamic model M for standard human musculoskeletal morphology and neural control kntc ={B,MU,N}, where B represents bones, MU represents muscles, and N represents neural control; calculate Where M is an n*n symmetric positive definite matrix, which is used to represent the internal mass and coordinate matrix of the target object. G is the acceleration constraint Jacobian matrix of m*n dimensions, where m is the number of constraint terms, n is the number of degrees of freedom, and G is the acceleration constraint Jacobian matrix of m*n dimensions. T is the transposed matrix of G, q is the generalized human body dynamic coordinate position, u and is the velocity information, z is the muscle state, and q, u and z are functions of time t; λ is a vector of m Lagrange multipliers of the constraint force; f(t,q,u,z) is the external force f applied and the internal inertia force f inertial The combined force of The initial dynamics model is customized based on the attribute information to obtain an initial customized dynamics model associated with the target object: M′ kntc =Modu×{B,MU,N}, where Modu is Mode=s(S B ×S MU ×S N ), Where s is the overall scaling factor and ranges from 0≤s≤1; S B , S MU , S N Representing the scaling matrices for bone, muscle, and neural regulation, respectively.

6. The method according to claim 1, wherein calibrating the current feature point recognition data using the feature point simulation data to generate calibrated feature point recognition data comprises: According to the initial customized kinetic model M′ kntc Perform inverse kinematics calculations Get the simulated data of the feature points of the target object predicted by the model Feature point simulation data includes multiple feature points Wherein i is the serial number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate of acquiring or obtaining the dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, wherein k is the serial number of the feature point, k is a positive integer and 0≤k≤K, and K is the total number of key points extracted by the model; The Loss function is set according to the least squares method, and each feature point in the current feature point recognition data is calibrated from multiple angles of rotation and displacement: Thus, calibrated feature points integrating intrinsic kinematic structural constraints are obtained, and multiple calibrated feature points are combined to form calibrated feature point recognition data, where is the feature point in the current feature point recognition data, is the feature point in the feature point simulation data.

7. The method according to claim 1 further comprises obtaining the motion posture output result of the customized dynamics model of the target object, i.e., the feature point simulation data, through an inverse kinematics model. is a feature point in the feature point simulation data, where i is the sequence number of the image frame in the dynamic image data, i is a positive integer and 0≤i≤T*fps, T is the time length of the dynamic image data and the unit is seconds, fps is the data acquisition frame rate for collecting or acquiring dynamic image data and the unit is frames / second, T*fps is the number of image frames included in the dynamic image data, where k is the sequence number of the feature point, k is a positive integer and 0≤k≤K, K is the total number of key points extracted by the model, and the angle features, trajectory features, speed features and periodic states are extracted to obtain: (1) Angle characteristics: Where i and t represent the i-th frame image and the t-th frame image; a represents the angle to be extracted; after a is selected, three adjacent feature points l, m and k are selected accordingly, and the coordinate information format of each feature point is (X, Y, Z), j is a natural number and 1≤j≤H, and H is the number of feature points; (2) Trajectory characteristics: There are many variations of trajectory features. The center point trajectory information of the internal structural constraints of the adjacent points is calculated based on the position information of each feature point in three dimensions: X, Y, and Z. (3) Speed ​​characteristics: t∈(0,T*fps); exist In the formula, the default value of t is 1, that is, the motion speed of the feature point is solved through two adjacent frames; (4) Cycle characteristics: Apply periodic operators to the above angle features, trajectory features and speed features to determine the periodic features Period of the angle features, trajectory features and speed features. Ang ,Period trj ,Period vel .

8. A system for acquiring image recognition data, the system comprising: A recognition device, used to obtain dynamic image data and original feature point data of a target object, and to perform image recognition on the dynamic image data using a multi-hinge kinematics model to correct the original feature point data according to the result of the image recognition, thereby obtaining initial feature point recognition data associated with the target object, wherein the multi-hinge kinematics model is constructed based on H feature points and H-1 skeletons and with hinge constraints added; An adjustment device, for obtaining attribute information of a target object and an initial dynamic model for standard human musculoskeletal morphology and neural control, and performing customized scale adjustment on the initial dynamic model based on the attribute information, thereby obtaining an initial customized dynamic model associated with the target object; A processing device, used to use the initial customized dynamics model as the current customized dynamics model and the initial feature point recognition data as the current feature point recognition data; A simulation device, used to simulate the motion posture of the target object using the current customized dynamics model, and generate feature point simulation data associated with the target object according to the simulation result of the motion posture simulation; A determination device is used to determine the degree of match between feature point simulation data and current feature point recognition data. When the degree of match is less than a matching threshold, the current feature point recognition data is calibrated using the feature point simulation data to generate calibrated feature point recognition data, and the current customized dynamic model is adjusted based on the calibrated feature point recognition data to obtain an adjusted customized dynamic model, and the adjusted customized dynamic model is used as the current customized dynamic model and the calibrated feature point recognition data is used as the current feature point recognition data; when the degree of match is greater than or equal to the matching threshold, the current feature point recognition data is used as the image recognition data of the target object.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Room VR-oriented whole-body three-dimensional posture tracking method

    CN110570455A

  • Moving posture monitoring and guiding method and device based on multiple cameras

    CN111881887A

  • Digital human body modeling method based on action recognition

    CN112184898A

  • Motion model learning device, three-dimensional attitude estimation device, motion model learning method, three-dimensional attitude estimation method and program

    JP2012083955A

  • Three-dimensional posture estimation device, three-dimensional posture estimation method and program

    JP2013020578A