Skeletal muscle force estimation method based on posture recognition and human biomechanics
Through deep learning methods, three-dimensional body posture information is extracted from image or video data, combined with ergonomic model, the complexity and cost problems of traditional musculoskeletal stress analysis is solved, real-time and accurate skeletal muscle stress analysis is achieved, and ergonomics is met. Work attitude requirements of principle.
Patent Information
- Application Number
- CN202010848965.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-08-21
AI Technical Summary
Traditional musculoskeletal stress analysis technology relies on artificial markings, and its operations are complex and expensive, and cannot be widely used in life and work. Two-dimensional body shape recognition cannot truly reflect human behavior in three-dimensional space, and it is difficult to combine mechanical test data for mathematical modeling and analysis.
Deep learning method is used to determine the user's three-dimensional body status information from image or video data, input the human body mechanics model, analyze the muscles, bones, and joint states of the user's body parts, and obtain data in real time through the image data acquisition device, eliminating manual marking operations, combined with The mechanical test data were mathematically modeled and analyzed.
Real-time and accurate skeletal muscle stress analysis is achieved, guiding users to complete their movements scientifically and healthily, meeting the working posture needs of ergonomic principles, and reducing analysis costs and complexity.
Smart Images

Figure CN112069933B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to the technical field of musculoskeletal biomechanics, and more particularly to a skeletal muscle force estimation method based on posture recognition and human biomechanics. Background Art
[0002] In daily life, people coordinate and control muscles, joints, bones and other motor organs to produce various movements to complete various tasks. The human body connects bones together through joints, and controls the body's posture through muscle relaxation and contraction. In order to better guide people to work effectively and live healthily, there are continuous breakthroughs in the research of human motion biomechanics. Its directions include studying the strength, load limit, range of motion and speed of muscles and joints in various parts of the human body, the resistance of human organs and bones to different movements, and the comfort of movements. Musculoskeletal force analysis modeling has been widely used in various industries, especially in aerospace, rehabilitation medicine, automobile transportation, animation modeling and other fields related to human biomechanics.
[0003] However, traditional musculoskeletal force analysis technology relies on manually marking key points of the body and inputting them into the musculoskeletal biomechanical model for force analysis. In order to record human movements for musculoskeletal force analysis, people usually attach spatial positioning sensors to the parts of interest on the monitored target, such as limbs, joints, etc. Then the data transmitted by these sensors is tracked in real time and input into the biomechanical model for analysis. Although this method has high accuracy, it is complicated to operate and expensive, and cannot be widely used in life and work.
[0004] With the development of artificial intelligence, the human body key point recognition technology based on deep learning has gradually been recognized by people and has shown great potential. From human motion understanding, human-computer interaction to augmented reality, animation, fitness and other applications, people have shown their demand for accurate posture recognition.
[0005] However, most of the body posture recognition on the market is currently limited to two-dimensional space; and the recognition points are based on the rough human joint markers provided by the public two-dimensional image video database. These two-dimensional data alone cannot truly reflect the behavior of people in three-dimensional space, and it is even more difficult to combine the mechanical test data in three-dimensional space to carry out corresponding mathematical modeling and simulate the biomechanical loads of human muscles and bones in different postures, so as to analyze whether the skeletal muscle force is reasonable. Therefore, it is necessary to find a skeletal muscle force estimation method based on body posture recognition and human biomechanics that can solve the above technical problems. Summary of the invention
[0006] In view of the above technical problems, the present disclosure proposes a method for estimating the skeletal muscle force based on deep learning and human biomechanics, including: determining the user's three-dimensional body information from image or video data through a deep learning method, inputting the estimated body information and / or motion information in continuous time (such as limb motion trajectory, external force magnitude, etc.) into the established human mechanics model, and accurately analyzing and outputting the current muscle, bone, and joint status of the user's body part of interest. This guides the user to complete the action scientifically and healthily to meet the work posture requirements that conform to ergonomic principles.
[0007] The real image data of the target user is obtained in real time through the image data acquisition device for analysis, eliminating the complex operation of traditional manual marking, and the obtained data can more realistically reflect human behavior than the rough human joint markers in the two-dimensional image video database. The acquired image data will be input into the body key feature point monitoring deep learning neural network, which predicts the body key feature point data based on these data. This process does not require manual identification and marking, which can eliminate complex and accidental manual operations; and can be carried out almost simultaneously with the acquisition of image data, which can be used to monitor and analyze the forces of various parts of the body of the target user during individual movement in real time, and can meet the timeliness of human force analysis in practical applications. The target user's image data and the predicted body key feature point data are processed to obtain estimated three-dimensional human posture information. The three-dimensional posture information can more realistically reflect human behavior and is combined with mechanical test data for corresponding mathematical modeling. Through mathematical modeling, the biomechanical loads of human musculoskeletal muscles in different postures can be simulated and solved, so as to analyze whether the skeletal muscle force is reasonable. The estimated three-dimensional posture information of the human body is input into a pre-established human body force analysis model, and the force state analysis of the part of interest can be obtained based on the correspondence between the three-dimensional posture information and the force state in the model.
[0008] In one aspect of the present disclosure, a skeletal muscle force estimation method based on posture recognition and human biomechanics is provided, comprising the following steps: acquiring image data of at least one target user through an image data acquisition device; inputting the image data of the target user into a body key feature point monitoring deep learning neural network to output predicted body key feature point data, wherein the predicted body key feature points are key feature points required for human biomechanics analysis of a part of interest of the target user; processing the image data of the target user and the predicted body key feature point data to obtain estimated three-dimensional human posture information; and analyzing the force state of the part of interest by inputting the three-dimensional human posture information into a pre-established human force analysis model.
[0009] In a preferred embodiment, the image data acquisition device includes at least one of the following: one or more planar cameras, depth cameras, infrared cameras or thermal imagers, wherein the depth camera includes at least one of: a time-of-flight camera, a structured light camera or a binocular camera.
[0010] In another preferred embodiment, the method further includes preprocessing the image data to improve the accuracy of analyzing the force state, wherein the preprocessing of the image data includes at least one of the following: flipping, translating, cropping or brightness adjustment of the image, or calculating the position of the target user, image optical flow information and image depth information.
[0011] In another preferred embodiment, the output predicted body key feature point data further includes: determining the position coordinates of at least one of the predicted body key feature points in the image; determining category information of at least one of the predicted body key feature points, wherein the category information includes: body feature information of interest, the body feature information of interest includes: key feature points of human body parts required for human body monitoring tasks and human biomechanical model analysis for different applications, wherein the key feature points of human body parts include at least one of the following: bone feature points, joint feature points and / or muscle feature points; and determining state information of at least one of the predicted body key feature points, wherein the state information includes: visible, invisible, and not in the image field of view or unpredictable.
[0012] In another preferred embodiment, the training process of the deep learning neural network for monitoring key body feature points includes: preparing a human body posture image set, wherein the human body posture image set includes human body posture image data, and body key feature point data annotated according to the human body posture image data; and using the human body posture image set to train the deep learning model, updating the parameters of the deep learning neural network for monitoring key body feature points through error back propagation until convergence, and obtaining the fully trained deep learning neural network for monitoring key body feature points.
[0013] In another preferred embodiment, the body key feature point monitoring deep learning neural network is implemented in a top-down manner or a bottom-up manner, wherein the bottom-up implementation method refers to first monitoring all the body key feature points in the image, and then assigning all the body key feature points to different human body instances respectively; and the top-down implementation method refers to first running a human body monitor on the image to find all human body instances, and then dividing all the human body instances into multiple human body instance sub-graphs, and monitoring the body key feature points of each of the human body instance sub-graphs respectively.
[0014] In another preferred embodiment, processing the image data of the target user and the predicted body key feature point data to obtain estimated three-dimensional human body posture information further includes: acquiring shooting parameters of the image data acquisition device, and establishing a three-dimensional space coordinate system according to the shooting parameters, wherein the shooting parameters include at least one of the following: camera orientation, angle, height, focal length and / or viewing angle.
[0015] In another preferred embodiment, processing the image data of the target user and the predicted body key feature point data to obtain estimated three-dimensional human body posture information further includes: in the case of a combination of a planar camera and a depth camera, converting the planar image obtained by the planar camera into a three-dimensional point cloud image by superimposing the planar image obtained by the planar camera with the depth image obtained by the depth camera; obtaining estimated three-dimensional human body posture information by processing the three-dimensional point cloud image and the predicted body key feature point data; or in the case of a combination of a multi-view image data acquisition device, obtaining estimated three-dimensional human body posture information by projecting the predicted body key feature point data into the three-dimensional space coordinate system.
[0016] In another preferred embodiment, the predicted body key feature point data includes characteristics and motion information of the target user, wherein the characteristics of the target user include at least one of the following: key feature points of human body parts, height and weight of the user; the motion information includes at least one of the following: movement direction of the key feature points of human body parts, skeletal joint rotation angle, speed, trajectory and external force magnitude; and the parts of interest include human muscles, bones and joints.
[0017] In another preferred embodiment, by inputting the three-dimensional posture information of the human body into a pre-established human body force analysis model, analyzing the force state of the part of interest further includes: establishing a human skeletal muscle model in a three-dimensional space for the part of interest, wherein the human skeletal muscle model is a three-dimensional model that describes the relationship between the human muscles, bones and joints; matching the pre-established human body force analysis model with the established human skeletal muscle model to analyze the force state of the part of interest, further including: measuring all external forces during the movement process and applying them to the human body force analysis model, wherein the human body force analysis model includes: a simple force model and a complex force model, wherein the simple force model includes: determining the force size of the part of interest by comparing the bone angle at the joint in the human skeletal muscle model with the corresponding initial bone angle in the human body force analysis model and the change of human body torque; and the complex force model includes: calculating the excitation points and forces of the human muscles in the human skeletal muscle model through mathematical modeling according to the current action in the human skeletal muscle model and in combination with all the measured external forces.
[0018] Compared with the prior art, the beneficial effects of the present disclosure are: judging the user's skeletal system from image or video data through a deep learning method, inputting the estimated skeletal system or motion information in continuous time (such as limb motion trajectory, output size, etc.) into the established human body mechanics model, and accurately analyzing and outputting the current muscle, bone, and joint status of the user's body part of interest. Thus, the user is guided to complete the action scientifically and healthily, so as to more scientifically meet the requirements of the working posture in accordance with the ergonomic principle. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The novel features of the present invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following drawings and detailed description of illustrative embodiments utilizing the principles of the present invention in the drawings. The drawings are only used to illustrate the embodiments and should not be considered as limiting the present invention. Also, the same reference numerals are used throughout the drawings to represent the same elements. In the drawings:
[0020] Figure 1 A flow chart showing an embodiment of the present invention is shown;
[0021] Figure 2 A layout of the image acquisition device according to an embodiment of the present invention is shown;
[0022] Figure 3 A deep learning neural network model application flow chart of an embodiment of the present invention is shown;
[0023] Figure 4A training flow chart according to an exemplary embodiment of the present disclosure is shown;
[0024] Figure 5 An example of a body key feature point labeling template is shown;
[0025] Figure 6 A schematic diagram of key body feature points used for sitting and standing posture analysis is shown;
[0026] Figure 7 An example of a model architecture of a bottom-up deep learning body key feature point monitoring neural network is shown;
[0027] Figure 8 An example of a top-down deep learning model architecture for body key feature point monitoring neural network is shown;
[0028] Fig. 9 An example of a human body posture 3D reconstruction process is shown; and
[0029] Fig.10 An example of the biomechanical model calculation process is shown;
[0030] Among them, 201 is the target user, 202 is camera 1, 203 is camera 2, 501 is the center of the neck, 502 is the center of the shoulder, 503 is the left shoulder, 504 is the left elbow, 505 is the left wrist, 506 is the right shoulder, 507 is the right elbow, 508 is the right wrist, 509 is the left hip, 510 is the left knee, 511 is the left ankle, 512 is the right hip, 513 is the right knee, 514 is the right ankle, 515 is the left eye, 516 is the right eye, 517 is the left ear, 518 is the right ear, 601 is the head, 602 is the center of the neck, 603 is the shoulder 604 is the back center, 605 is the waist, 606 is the hip joint, 607 is the knee joint, 608 is the ankle joint, 609 is the sole of the foot, 610 is the sacrum, 611 is the ischium contact point, 701 is the image feature extraction module, 702 is the posture feature extraction module, 703 is the body key feature point judgment module, 704 is the affinity judgment module between the body key feature points, 801 is the human instance monitoring module, 802 is the body key feature point monitoring module, and 803 is the position judgment loss function of the body key feature points. DETAILED DESCRIPTION
[0031] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable those skilled in the art to more thoroughly understand the present disclosure, and to fully convey the scope of the present disclosure to those skilled in the art. Nothing in the following detailed description is intended to indicate that any particular component, feature or step is essential to the present invention. Those skilled in the art will appreciate that various features or steps can be substituted or combined with each other without departing from the scope of the present disclosure.
[0032] In this embodiment, a skeletal muscle force estimation method based on posture recognition and human biomechanics is disclosed. Figure 1 is a flow chart of an embodiment of the present invention, see Figure 1 , including the following steps: S101, acquiring image data of at least one target user through an image data acquisition device; S102, inputting the image data of the target user into a body key feature point monitoring deep learning neural network, and outputting predicted body key feature point data, wherein the predicted body key feature points are key feature points required for human biomechanics analysis of the target user's interested parts; S103, processing the image data of the target user and the predicted body key feature point data to obtain estimated three-dimensional human body posture information; S104, inputting the three-dimensional human body posture information into a pre-established human body force analysis model to analyze the force state of the interested part.
[0033] At present, human body force estimation systems do not combine deep learning human posture analysis and human body mechanics analysis. Most of the existing force estimation methods are based on human-machine coupling models simulated by software or positioning and tracking sensors placed on the human body to analyze and simulate the forces on the parts of the human body of interest. The traditional analysis process is time-consuming and cannot monitor the forces on various parts of the target user's body in real time when the target user moves. Among them, the method based on human-machine coupling model software cannot analyze the forces on various parts of the current real target user's body in real time, and often draws rough force conclusions based on existing templates, which cannot be applied to individuals. Among them, the method of analyzing forces by artificially marking and placing spatial positioning sensors on key feature points of the body is usually time-consuming and expensive, and cannot meet the needs of timely human force analysis in real life.
[0034] It should be made clear that the technology combining human posture analysis and human body mechanics analysis in the present disclosure cannot identify people in images. It only monitors the positions of key body feature points of people in images and videos, and the key body feature points of people do not carry privacy information related to personal identity. Moreover, since the deep learning posture recognition algorithm runs in the terminal chip, no posture data will leave the user's computer. This can effectively protect the user's privacy in terms of the algorithm.
[0035] Different from the existing body recognition, the key feature points of the body in this embodiment are selected specifically according to the needs of human body mechanics analysis, and represent the body parts associated with the human body to complete a certain action, including the contact points between bones and objects, bone joints, the positions of specified muscle bundles, etc., thus having greater tracking freedom than the traditional feature point marking method based on human bone joints.
[0036] Traditional machine learning posture key point recognition algorithms include those based on random forests, etc. These methods all rely on artificially designed features. However, using deep convolutional networks to monitor the process of automatically extracting and identifying key feature points of human posture no longer requires artificially designed features. Among them, the posture detection deep learning model based on force analysis can be constructed using any of the following deep learning frameworks, including but not limited to: Keras, TensorFlow, Pytorch, MXNet, Caffe, etc.
[0037] Currently, there is a lack of methods for fast, cheap, real-time, and automatic analysis of the stress state of the human body's parts of interest in the relevant technical field. Through the above steps, this embodiment enables the image acquisition device to input the image into a well-trained deep learning neural network after acquiring the image containing the target human body in real time, obtain the extracted key feature points of the body, and obtain the three-dimensional human posture through the three-dimensional reconstruction method; then the acquired three-dimensional human posture is input into the established human body force analysis model to obtain the human body force state analysis of the part of interest. The human body force analysis method of this embodiment realizes real-time and fully automatic analysis of human body forces without manual intervention, effectively integrating force analysis into life.
[0038] In this embodiment, the image acquisition device may include but is not limited to: an ordinary color plane camera, a depth camera, an infrared camera or a thermal imager.
[0039] In this embodiment, the depth camera includes but is not limited to: a time of flight camera (TOF), a structured light camera or a binocular camera. The time of flight camera continuously sends light pulses to the target, then uses a sensor to receive the light returned from the object, and then obtains the distance information from the target to the camera by detecting the flight time of the light pulse. The structured light camera scans the object under test by emitting lasers to obtain the distance information from the surface of the object under test to the camera. The binocular camera determines the distance information from the shooting target to the camera by calculating the parallax of the images collected by the two cameras. In this embodiment, the acquired image data is not limited to: the body information of the user collected by the camera device, represented by an image, video, optical flow map or point cloud map.
[0040] In some embodiments, the arrangement of the image acquisition device includes but is not limited to the following arrangements: using a single or multiple planar cameras, or a combination of a planar camera and a depth camera, so that the imaging range meets the needs of user motion monitoring and subsequent force analysis. Figure 2 is an arrangement of the image acquisition device of an embodiment of the present invention, such as Figure 2 As shown, the image acquisition device observes the target user's body information from multiple angles.
[0041] In some embodiments, in order to remove the influence of noise signals, abnormal data, acquisition methods and other factors on deep learning posture monitoring and improve the accuracy of force state analysis, the image data is preprocessed, and the preprocessing includes but is not limited to: scaling, flipping, translating, cropping and brightness adjustment of the image, or calculating the position of the target user, image optical flow information and image depth information. When the image is scaled, when the zoom factor is set to a small value, the reduced image is passed to the neural network, and the calculation speed of the neural network is improved at the expense of accuracy. When the image is flipped, whether the image should be flipped / mirrored horizontally depends on whether the image or video data is horizontally flipped by default, or whether the user wants to return the image or video in a specified direction. In order to achieve real-time transmission of images and improve network transmission speed, the transmitted image needs to be cropped before the image is transmitted to remove useless parts. Image preprocessing also includes image brightness adjustment, so as to control the difference between the object and the real effect after imaging by the optical system, and control the difference as much as possible to improve the working performance of the system. Brightness adjustment does not affect the resolution of imaging.
[0042] In this embodiment, outputting predicted body key feature point data includes: determining the position coordinates of at least one of the predicted body key feature points in the image; determining category information of at least one of the predicted body key feature points, wherein the category information includes: body feature information of interest, and the body feature information of interest includes: key feature points of human body parts required for human body monitoring tasks and human biomechanical model analysis for different applications; and determining state information of at least one of the predicted body key feature points, wherein the state information includes: visible, invisible, and not in the image field of view or unpredictable; body key feature points that are not monitored can be replaced by (0,0,0) in three-dimensional coordinates.
[0043] In some embodiments, the posture detection algorithm based on force analysis is mainly divided into two stages: Figure 3 That is the flow chart of the posture detection algorithm of the embodiment of the present invention, as shown in Figure 3 As shown, the process includes the following steps: S301, training the deep learning network model; S302, using the deep learning network model for reasoning.
[0044] In some embodiments, S301 adopts a supervised learning training method for the deep learning neural network training process, and the training process of the body key feature point monitoring deep learning neural network includes: preparing a human body posture image set, and marking the human body posture image data in the human body posture image set according to the predicted body key feature points; and using the human body posture image set to train the deep learning model, and updating the parameters of the body key feature point monitoring deep learning neural network through error back propagation until convergence, so as to obtain the fully trained body key feature point monitoring deep learning neural network.
[0045] Figure 4 is a training flow chart of a deep learning network model according to an embodiment of the present invention, such as Figure 4 As shown, the flowchart includes: S401 prepares an image set containing a human body, and marks the human body in the human body image based on the key feature points of the body required for force analysis, and the marked feature points are used to distinguish different parts of the human body; S402 builds a neural network model; S403 defines a loss function; S404 model training: through the constructed neural network model, the image data is used as input, the predicted human posture feature points are output, the error between the predicted feature points and the marked feature points is calculated, and the parameters of the neural network are updated through the back propagation of the error until convergence, so as to obtain a fully trained neural network; preferably, a cross-validation method is used to verify the training degree of the deep learning model to enhance its generalization ability and avoid overfitting of the model.
[0046] In the above S401, the posture images in the human posture image set are obtained by manually marking feature point labels. In order to obtain more human posture images for training, the images can be preprocessed in the above steps. In addition, when preparing the posture image database, it can be modified based on a large number of existing labeled human databases, and the human posture in the image can be marked according to the feature points required for biomechanical analysis of the part of interest.
[0047] In this embodiment, popular public data sets can be used, including but not limited to: Human3.6M 3D human pose data set, COCO human pose data set, MPII human pose database, etc. In the MPII data set, each character is marked with 14 key body feature points with different numbers, such as right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, top of the head, and neck; each mark records the coordinate information of the corresponding key body feature point, where the key body feature point mark template is as follows: Figure 5 shown.
[0048] In some embodiments, in order to perform biomechanical analysis to understand the impact of movements on muscles, joints, and bones, a more ergonomically adapted human body feature marking method is proposed, that is, to additionally mark important feature points of the human body in the image according to different human body mechanics analysis requirements. Figure 6 As shown, for sitting and standing posture analysis, the feature points used include but are not limited to the following body features: head, neck center, shoulder center, back center, waist, sacrum, hip joint, femur, knee joint, sciatic contact point, etc.; for standing posture analysis, the feature points used include but are not limited to the following body features: head, neck center, shoulder center, back center, waist, hip joint, knee joint, ankle joint, sole of the foot, etc.
[0049] In this embodiment, in order to further improve the accuracy and stability of the position estimation of the key feature points of the body, a confidence region is generated based on the key feature points of the body marked on the image and the pixel positions on the image as a reference. In this region, the estimated key feature points of the body have a relatively high probability of containing true values. The confidence region can be generated using a confidence region generation method that can be thought of by a person skilled in the art.
[0050] In this embodiment, in S402 and S403, the constructed deep learning neural network model and loss function definition can adopt but not limited to: bottom-up method and / or top-down method for human posture estimation. Among them, the top-down method first runs a human monitor on the image to find all human body instances, then segments each person, and then uses human key point monitoring for each segmented human body instance. This method has a high speed and accuracy in identifying the key feature points of a single person's body, reflecting a high performance, and the time for posture recognition of the entire image increases with the number of people. However, because the key points of the human body cannot be modeled more carefully, the accuracy of identifying complex human postures is often slightly lower. Among them, the bottom-up method first monitors the key points of human body parts in the image, and then distributes the key points of the parts of multiple human bodies in the image to different human body instances. Although this method cannot benefit from the overall human body structure information, when multiple human body instances are contained in the image, the accuracy of detecting and classifying the key feature points of each person's body is high, and the recognition time will not increase significantly with the increase in the number of detected people.
[0051] In the case of multi-person posture extraction, this example preferably uses a bottom-up deep learning body key feature point monitoring neural network architecture model, whose architecture is as follows Figure 7 As shown, the following modules are included.
[0052] Module 701, image feature extraction module, obtains multi-scale features of the image. In this embodiment, multiple convolutional layers are used for feature extraction and mapping. The low convolutional layer extracts some simple image features such as edge, shape and other image information, and the deeper convolutional layer can extract more complex features from the image. The highest recognition accuracy of different key feature points of the body may be on the feature maps obtained at different stages. For example, the arm may be easily recognized on the feature map of the second layer, while the head is easier to recognize on the fifth layer. Therefore, in order to extract human body features of different scales for model learning, and to allow the model to extract image feature maps of different scales from each convolutional layer through continuous convolution and downsampling, the acquired image is input into one or more pre-trained convolutional neural networks for processing to output a series of feature maps. Among them, the pre-trained convolutional neural network model includes but is not limited to VGG-16, VGG-19, ResNet-50, AlexNet or some of the convolutional layers therein, such as the first 5 layers of VGG-16.
[0053] Module 702, posture feature extraction module. In this embodiment, through the positions of key feature points of the body encoded on the image and the direction vectors connecting the key feature points, the neural network can simultaneously learn these two features to estimate the posture of the target user.
[0054] In this embodiment, the positions of the key feature points of the body encoded by module 703 preferably adopt a 5-layer convolutional neural network structure and a convolution kernel of 3*3 size.
[0055] The loss function is an expression related to network parameters and operation results that describes the difference between the true value and the predicted value. The loss function used in this embodiment is preferably a cross entropy loss function or a square error loss function.
[0056] In this embodiment, the position judgment loss function of the body key feature points in module 703 is defined as: the gap between the confidence area of the body key feature points predicted by the neural network calculation and the confidence area of the real key points based on the mark, which can be obtained by square difference calculation.
[0057] In this embodiment, module 704 is a connection affinity judgment module between key feature points of the body. In order to combine the identified key feature points of the body with different human posture instances, it is necessary to encode the connection affinity between the key feature points to determine whether the key points belong to the same human instance. The encoding method includes at least one of the following:
[0058] A) Encode the connection affinity between key feature points of the body based on the position information. After finding an intermediate point located between each pair of key points, check the confidence of whether the intermediate point belongs to the same human instance.
[0059] B) Encode the connection affinity between the key feature points of the body based on both position information and direction information. Define the vector from the key feature point of the body on the image to another key point to represent the direction of the pixel points between the feature points in the image, for example, the vector between the elbow feature point and the wrist feature point. Let Q j1,k and Q j2,k Represents the two feature points and the corresponding labeled pixels of the kth human instance in the image. If the image pixel point P is located on the connecting line between the feature points, The value of represents a unit vector from feature point j1 to feature point j2, otherwise it is set to zero vector. For the pixel position P in the image, the mathematical expression of the connection affinity can be defined as:
[0060]
[0061] in Represents the unit direction vector corresponding to the connection between feature points. According to the acquired image, the real feature point connection affinity vector of each pixel is calculated in turn. The feature point connection affinity vector of each pixel can be taken as the average value of the corresponding feature point connection of all people in the image. The mathematical expression is
[0062]
[0063] where n c (P) is the number of non-zero vectors corresponding to the pixel P of the kth person in the image.
[0064] In this embodiment, the connection affinity judgment loss function between key feature points in module 704 is defined as: calculating the gap between the predicted association graph between key points and the real association graph. The mathematical expression of the predicted loss function of the connection affinity judgment can be defined as:
[0065]
[0066] In other embodiments of the present invention, in order to further improve the accuracy of posture prediction, the posture extraction module 702 is continuously looped. Each loop branch takes the feature result map output by the image feature extraction module 701 and the results of the body key feature point judgment module 703 and the affinity judgment module 704 between feature points of the previous posture detection module as inputs to a new set of body key feature points and the association module between feature points.
[0067] In other embodiments of the present invention, in the case of single-person posture extraction, in order to improve the accuracy of single-person posture detection, a top-down body key feature point recognition method is preferably adopted. In this way, each different human body is recognized, and feature point recognition is performed using the corresponding recognized human body instance. Figure 8 An example of a model architecture of a top-down deep learning body key feature point monitoring neural network is shown, for example, it may include the following modules.
[0068] Module 801, a human instance monitoring module, monitors and extracts all human instances in the image. Its implementation method includes but is not limited to: monitoring and extracting all human instances in the image and extracting the range of each person by training a complete human recognition neural network, such as YOLO and MASK-RCNN.
[0069] Module 802, the body key feature point monitoring module, monitors the body key feature points in the extracted human body instance. Its network architecture model can adopt a downsampling combined with upsampling network structure. Downsampling and upsampling are like encoder and decoder synthesis, responsible for extracting features and generating body key feature point confidence map results.
[0070] In some examples, convolution is combined with a pooling layer in the downsampling stage. The pooling layer is used to increase the scale characteristics of the model, improve the classification recognition of the model, reduce the number of model parameters, and reduce the overfitting problem. The pooling method used in this embodiment is preferably the maximum pooling method.
[0071] In this example, in the upsampling stage, the network restores the input feature map to a higher scale through continuous upsampling and deconvolution, and refits it at the end of the network using a fully connected layer, and classifies and predicts the key feature points of the body at the pixel level for each pixel, and finally generates a confidence map of the predicted key points. The upsampling method preferably uses the nearest neighbor interpolation method.
[0072] In some instances, batch normalization and activation functions are added after the convolutional layers to improve the stability of the network.
[0073] In some embodiments, the position judgment loss function of the key feature points of the body in module 803 is defined as: the loss calculation is used to calculate the square difference between the result predicted by the network and the actual true value.
[0074] The inference using the deep learning network model S302 includes: taking the above image or video data as input, and outputting the estimated coordinates, state and confidence of each key feature point of human body posture on the graph through a well-trained neural network operation. Different user posture monitoring can be obtained by setting different thresholds.
[0075] In this embodiment, the image data of the target user and the predicted body key feature point data are processed to obtain estimated human body three-dimensional posture information S103 as shown in FIG. Fig. 9 As shown, it includes: obtaining shooting parameters of the image data acquisition device, and establishing a three-dimensional space coordinate system according to the shooting parameters, wherein the shooting parameters include at least one of the following: the orientation, angle and viewing angle of the camera.
[0076] S901, obtain the parameters of the image acquisition device, the captured image and the predicted human posture feature points obtained in S302. In the above steps, the parameters of the image acquisition device obtained include: the orientation information and viewing angle of the image acquisition device. Before scanning, the shooting parameters of each camera are obtained, and the spatial coordinate system of the camera is calibrated and adjusted. The camera is placed in one or some orientations of the motion space, and the user action information is collected from multiple optional viewing angles, and three-dimensional reconstruction is performed. Then the position of the camera is calibrated. Each camera can establish a spatial three-dimensional rectangular coordinate system based on the origin of its position. In this example, the preferred definition method is: the coordinate origin is set at the camera lens, the x and y axes are parallel to the two sides of the camera imaging surface, and the z axis is positioned in the direction of the lens shooting optical axis and perpendicular to the image surface.
[0077] In the above steps, the image acquisition device parameters include internal and external parameters. The internal parameters include but are not limited to: camera focal length, distortion parameters, tilt coefficient, image upsampling and downsampling ratios. The internal parameters are used to convert pixel image data into an image of the camera coordinate system. The external parameters include but are not limited to: the position distance parameter of the camera origin relative to the world coordinate origin, the rotation direction parameter of the camera coordinate system relative to the world coordinate system, and the time synchronization parameter of each frame captured by multiple cameras. In this embodiment, the preferred measurement method of the camera parameters is to establish a world coordinate system with the origin of the specified camera as the origin, and measure the external parameters of the camera used for observation. Optionally, the relationship between the cameras is geometrically modeled using a camera marker plate to estimate the internal and external parameters of the camera. The calibration measurement of the parameters affects the accuracy of the prediction results and the stability of the algorithm.
[0078] S902, matching the same person in each view. In the above steps, the method of matching the same person in each view by algorithm includes but is not limited to:
[0079] 1. Determine the location of the target user through location tracking in wearable devices;
[0080] 2. When obtaining the depth image information, determine the human body according to the position of the human body in the three-dimensional space located in the depth view;
[0081] 3. When obtaining planar image information, calculate the similarity of human features across multiple views based on human appearance similarity and geometric compatibility. Preferably, the above-mentioned deep learning algorithm is used to calculate the feature results output by a specific convolutional layer and match the feature results. Optional methods include machine learning or optical flow method.
[0082] S903, reconstructing the three-dimensional posture information of the target according to the matched image data and the key feature point information of the body.
[0083] In some embodiments, processing the image data and predicted body key feature point data of the target user to obtain estimated three-dimensional human body posture information further includes: in the case of a combination of a planar camera and a depth camera, superimposing the planar image obtained by the planar camera with the depth image obtained by the depth camera to convert into a three-dimensional point cloud image; processing the three-dimensional point cloud image with the predicted body key feature point data to obtain estimated three-dimensional human body posture information; or in the case of a combination of a multi-view image data acquisition device, projecting the predicted body key feature point data into a three-dimensional space coordinate system to obtain estimated three-dimensional human body posture information.
[0084] Among them, the three-dimensional estimation method combining plane and depth images includes realizing three-dimensional posture estimation of multi-view plane images by superimposing three-dimensional image information on the plane image.
[0085] Among them, the planar image method of acquiring from multiple perspectives estimates that the captured image can be used as a projection of a human body on different camera planes. After the image coordinate system is converted to the camera coordinate system through the camera internal parameters, the human body diagram can be projected into the three-dimensional space coordinate system according to the positions and relative postures of cameras placed at multiple angles and the marked human body posture data output based on S302 through the measured camera external parameters through the triangulation method, thereby calculating the three-dimensional space coordinates of the feature points to obtain the positions of the target athlete's body skeletal joints in the three-dimensional space.
[0086] In addition, due to the imperfect accuracy of the calibration instrument and human factors, as well as the influence of external conditions, the final calculated estimated key body feature points have errors with the actual situation. The errors include: the error between the position of the key body feature points estimated by the deep learning algorithm and the actual position; the error of the internal and external parameters of the camera calibration; the error of the image time points acquired by different cameras. In order to reduce these errors, the results of 3D posture reconstruction can be constrained by using the prior body key feature point template information, such as arm length and ergonomic parameters.
[0087] The human body mechanics model uses the principles of physics combined with ergonomics to calculate the forces on muscles and bones, thereby guiding people to work and live safely. Bones are connected not only by joints, but also by muscles and ligaments. However, human joints have a certain range of motion, and if they exceed the limit, they will cause damage. For people of different ages, the bone strength indicators of different parts of the human body are also very different. In the human body movement system, bones serve as levers for movement, joints serve as hubs for movement, and muscles provide the power for movement. The three work in unison to complete the movement.
[0088] In some embodiments, the force of the musculoskeletal system is analyzed and calculated by the following steps:
[0089] Step 1: Before scanning, obtain the shooting parameters of each camera (such as shooting angle, focal length, orientation, etc.), calibrate and adjust the spatial coordinate systems of different cameras so that the imaging range meets the user's body monitoring and feedback needs;
[0090] Step 2: using an image acquisition device to acquire one or more video image information, and pre-processing the image by flipping, translating, adjusting brightness, etc., to calculate the object position, optical flow information, and depth information;
[0091] Step 3: According to the information labeling requirements of different key points of the human body required by different human body monitoring tasks and human biomechanics models, the human body images in the database are specifically labeled;
[0092] Step 4: Use deep learning convolutional neural network to calculate the key feature points of the body required for ergonomic analysis;
[0093] Step 5: reconstruct the 3D body posture of the target user through 3D reconstruction technology;
[0094] Step six: Input the annotated features and motion information into the mechanical model of human biomechanics, analyze the stress state of the human muscles, bones, and joints in different postures through dynamic methods, and compare them with the given standard stress conditions to obtain evaluation indicators.
[0095] In some embodiments, the annotated features and motion information, i.e., the predicted body key feature point data, include features and motion information of the target user, wherein the features of the target user include at least one of the following: user height and weight; the motion information includes at least one of the following: direction of movement of body parts, rotation angle of bone joints, speed, trajectory and magnitude of external force; and the parts of interest include human muscles, bones and joints.
[0096] In some embodiments, by inputting the three-dimensional posture information of the human body into a pre-established human body force analysis model, analyzing the force state of the part of interest further includes: establishing a human skeletal muscle model in a three-dimensional space for the part of interest, wherein the human skeletal muscle model is a three-dimensional model that describes the relationship between the human muscles, bones and joints; matching the pre-established human body force analysis model with the established human skeletal muscle model to analyze the force state of the part of interest, further including: measuring all external forces during the movement process and applying them to the human body force analysis model, wherein the human body force analysis model includes: a simple force model and a complex force model, wherein the simple force model includes: determining the force magnitude of the part of interest by comparing the bone angle at the joint in the human skeletal muscle model with the corresponding initial bone angle in the human body force analysis model and the change of human body torque; and the complex force model includes: calculating the excitation points and forces of the human muscles in the human skeletal muscle model through mathematical modeling according to the current action in the human skeletal muscle model and in combination with all the measured external forces.
[0097] In the above steps, the biomechanical model calculation process is as follows: Fig.10 As shown, including but not limited to:
[0098] Establishing a human skeletal muscle model of the part of interest S1001: Based on the predicted human body posture data from deep learning and three-dimensional reconstruction, a human skeletal muscle model is established in three-dimensional space for the part of interest according to the predicted data and the body structure of the target user of interest.
[0099] In some embodiments, a body part of interest is modeled in three-dimensional space, wherein bones are simulated as cylinders, and their lengths are shown in Table 1. Joints are simulated as rotation joints with degrees of freedom, and their active angles, comfortable adjustment ranges, and maximum ranges are shown in Table 2. Muscles are simulated as stretchable springs connected to bones and joints.
[0100] Table 1
[0101] Location Dimensions (mm) Location Dimensions (mm) Location Dimensions (mm) Shoulder height 1385 Sitting knee high 502 Upper arm length 314 waist height 1031 Thick chest 229 Forearm length 237 Hips 933 Hips 933 Calf length 363 Ankle Height 120 Long thighs 492 shoulder width 389
[0102] Table 2
[0103]
[0104] In the above steps, different muscles are stimulated in different postures. For example, in the sitting state, the erector spinae, latissimus dorsi, biceps femoris, gluteus maximus, and gluteus medius are the five muscles that bear the force. In order to facilitate and speed up the calculation, the musculoskeletal model can be simplified by merging muscles. For example, the gluteus maximus and gluteus medius are merged into one muscle for mechanical analysis.
[0105] In the above steps, the human body model can be represented by a planar human body model or a three-dimensional human body model.
[0106] Optimize the human skeletal muscle model S1002.
[0107] In some embodiments, a personalized musculoskeletal model is established, based on the predicted human key point data and user body features such as height, weight data and muscle feature data of deep learning, and the length, proportion and quality of each part of the body of the general model are scaled according to the ratio between the predicted data and the human body marker points in the general model, so as to establish a personalized human musculoskeletal model that meets individual characteristics. In the scaling process, the error between the marker points in the experiment and the theoretical points in the model is controlled by the least squares method. Among them, the factors affecting the selection of the general model include: age, gender, human body size, health, training adaptability, psychology, training time, rhythm, duration, intensity and other factors. In this embodiment, based on the ratio between the predicted human key points of interest of deep learning and three-dimensional reconstruction, the human skeletal muscle model pre-established in S1001 is matched with the body structure of the target user of interest to establish a personalized musculoskeletal model that meets the user. The matching process includes scaling the bone length and quality of the model, and the matching method includes but is not limited to: the gap between the least squares optimization predicted human body posture data and the human body model data. Among them, the human body posture data includes joint angles, key feature point positions, etc. The registered model can thus simulate the user's real posture for analysis.
[0108] In some of these examples, the user's real human musculoskeletal model is affected by different physiological factors, such as human size, body shape, weight, age, gender, etc. In order to reduce the impact of these factors on the analysis results, it is preferred to adjust and optimize the human musculoskeletal model by establishing a regression equation between the musculoskeletal model and various factors.
[0109] Calculate the musculoskeletal forces at the site of interest S1003.
[0110] In some embodiments, a pre-established human body force analysis model is matched with the established human body skeletal muscle model to analyze the force state of the part of interest, including: measuring all external forces during the movement and applying them to the human body force analysis model to obtain the force state of the muscle of interest. The external forces on the user include ground reaction force, gravity and force of external objects, etc.
[0111] The force model includes simple force model and complex force model.
[0112] In some embodiments, S1003 simple force model analysis includes: determining the force magnitude of the part of interest by comparing the bone angle at the joint in the human skeletal muscle model with the corresponding initial bone angle in the human force analysis model and the change of human body torque.
[0113] In some of these instances, the simple model can be defined as a lever model. For example, in a sitting position, the key feature points of the human body include the spine, waist, and hips. The spine is an important load-bearing part. In a sitting position, the force on the human spine is related to many factors. The two most significant factors are the gravity of the trunk and the distance from the position of the trunk to the center of the spine. By calculating the distance from the key feature point of the human body's center of gravity (trunk at the sternocostal joint) to the center of the spine and the waist, the magnitude of the lumbar gravity distance and the resultant distance of the tension of related muscles, ligaments, soft tissues, etc. in different postures are calculated.
[0114] In some of these instances, a simple model can be defined as the bending of the body part of interest. For example, in sitting, the upright sitting posture can be defined as a manifestation of the standard sitting posture. According to the theory of ergonomics and sufficient evidence, the musculoskeletal force and waist bending are positively correlated. For the model parameters of trunk bending, the greater the waist bending angle of the user's body relative to the standard sitting posture, the greater the waist muscle force.
[0115] In some embodiments, S1003 complex muscle force model includes: according to the current action of the user's muscle skeleton model combined with all the measured external forces, the excitation points and forces of the human muscles in the human skeleton muscle model are calculated through mathematical modeling. The motion parameters include the direction trajectory of the movement of the key feature points, acceleration, bone angle, internal and external rotation and extension changes, etc. The muscle force model can change in different situations. The present disclosure provides an optional force analysis model, which is expressed as a mathematical formula:
[0116] F=KS 2 +Ce -|v| +F0
[0117] F represents muscle force, K represents muscle tension coefficient, S represents muscle length, C is the relationship coefficient between muscle force and contraction speed, v is muscle contraction speed, and F0 is the initial force state of the muscle.
[0118] In some embodiments, the complex muscle force model includes: calculating muscle excitation points and muscle force according to the current action of the dynamic musculoskeletal model in combination with the inertia parameters of the human body through proportional differential control and static optimization methods.
[0119] Complex models can be used to analyze the forces on running muscles. This embodiment mainly explores three joints, namely the hip joint, knee joint, and ankle joint. A deep learning algorithm is used to extract the movements of these three joints, including movement characteristics, movement direction trajectory, and load weight. For example, curves of the internal and external rotation and external and internal and external abduction movements of these three joints are collected over a period of time to analyze the changes in bone angles, human body torque, and bone and muscle force at each joint. Including:
[0120] (1) Using the series of curve graphs obtained above, collect relevant data of some users and create corresponding curve graphs;
[0121] (2) Determine whether the patient's gait is normal. If it is not normal, define some corresponding correction plans based on the curve graph. For example, the gait of patients with clubfoot, polio, and Parkinson's disease is different from that of ordinary people. Similar methods are used to evaluate the effects of these gait disorders and corrective orthopedic surgeries.
[0122] In this example, complex muscle force analysis is performed with the help of human motion simulation software, including LifeMOD, AnyBody, ANSYS, etc. For example, the evaluation of sitting posture in this embodiment is performed using Anybody.
[0123] In the specification provided herein, a large number of specific details are described. However, it should be understood that the embodiments of the present disclosure can be practiced without these specific details. In some embodiments, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification.
[0124] Although exemplary embodiments of the present invention have been shown and described herein, it will be readily understood by those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will now appreciate that many modifications, changes and substitutions will occur to them without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in the process of practicing the present invention.
Claims
1. A skeletal muscle force estimation method based on posture recognition and human biomechanics, characterized in that The steps include: Acquire image data of at least one target user through an image data acquisition device; By inputting the image data of the target user into a body key feature point monitoring deep learning neural network, the predicted body key feature point data is output, wherein the predicted body key feature points are key feature points required for performing human biomechanics analysis on the target user's interested parts; Processing the image data of the target user and the predicted body key feature point data to obtain estimated three-dimensional human body posture information; as well as By inputting the three-dimensional posture information of the human body into a pre-established human body force analysis model, the force state of the part of interest is analyzed. The method further includes preprocessing the image data to improve the accuracy of analyzing the stress state, wherein the preprocessing of the image data includes at least one of the following: Flip, translate, crop, or adjust the brightness of the image, or Calculating the position of the target user, image optical flow information and image depth information, The output predicted body key feature point data further includes: Determining the position coordinates of at least one of the predicted key body feature points in the image; and Determine category information of at least one of the predicted key body feature points, wherein the category information includes: body feature information of interest, and the body feature information of interest includes: key feature points of human body parts required for human body monitoring tasks and human biomechanical model analysis for different applications, Wherein, the parts of interest include human muscles, bones and joints, and Wherein, by inputting the three-dimensional posture information of the human body into a pre-established human body force analysis model, analyzing the force state of the part of interest further includes: Establishing a human skeletal muscle model in three-dimensional space for the part of interest, wherein the human skeletal muscle model is a three-dimensional model that describes the relationship between the muscles, bones and joints of the human body; Matching the pre-established human body force analysis model with the established human body skeletal muscle model to analyze the force state of the part of interest further includes: All external forces during the movement are measured and applied to the human body force analysis model, wherein the human body force analysis model includes: a simple force model and a complex force model, wherein The simple force model comprises: determining the force magnitude of the part of interest by comparing the bone angle at the joint in the human skeleton muscle model with the corresponding initial bone angle in the human force analysis model and the change of human body torque; and The complex force model includes: calculating the excitation points and forces of the human muscles in the human skeletal muscle model through mathematical modeling according to the current action in the human skeletal muscle model and in combination with all measured external forces.
2. The method according to claim 1, characterized in that The image data acquisition device includes at least one of the following: one or more planar cameras, depth cameras, infrared cameras or thermal imagers, wherein the depth camera includes at least one of: a time-of-flight camera, a structured light camera or a binocular camera.
3. The method according to claim 1, characterized in that The key feature points of the human body parts include at least one of the following: bone feature points, joint feature points and / or muscle feature points, and The outputting of predicted body key feature point data further includes: determining state information of at least one of the predicted body key feature points, wherein the state information includes: visible, invisible, not within the image field of view or unpredictable.
4. The method according to claim 1, characterized in that The training process of the deep learning neural network for monitoring key body feature points includes: Preparing a human body posture image set, wherein the human body posture image set includes human body posture image data and body key feature point data annotated according to the human body posture image data; and The deep learning model is trained using the human body posture image set, and the parameters of the body key feature point monitoring deep learning neural network are updated through error back propagation until convergence, so as to obtain the fully trained body key feature point monitoring deep learning neural network.
5. The method according to claim 1, characterized in that The body key feature point monitoring deep learning neural network is implemented in a top-down or bottom-up manner, wherein The bottom-up implementation method refers to first monitoring all key body feature points in the image, and then assigning all the key body feature points to different human body instances respectively; and The top-down implementation method refers to first running a human body detector on the image to find all human body instances, then dividing all the human body instances into multiple human body instance sub-graphs, and monitoring key body feature points of each of the human body instance sub-graphs.
6. The method according to claim 1, characterized in that Processing the image data of the target user and the predicted body key feature point data to obtain estimated human body three-dimensional posture information further includes: Acquire shooting parameters of the image data acquisition device, and establish a three-dimensional space coordinate system according to the shooting parameters, wherein the shooting parameters include at least one of the following: the orientation, angle, height, focal length and / or viewing angle of the camera.
7. The method according to claim 6, characterized in that Processing the image data of the target user and the predicted body key feature point data to obtain estimated human body three-dimensional posture information further includes: In the case of a combination of a planar camera and a depth camera, the planar image acquired by the planar camera is superimposed with the depth image acquired by the depth camera to convert into a three-dimensional point cloud image; the three-dimensional point cloud image is processed with the predicted key feature point data of the body to obtain estimated three-dimensional posture information of the human body; or In case of a combination of multi-view image data acquisition devices, the predicted body key feature point data is projected into the three-dimensional space coordinate system to obtain estimated three-dimensional posture information of the human body.
8. The method according to claim 1, characterized in that The predicted body key feature point data includes the characteristics and motion information of the target user, wherein the characteristics of the target user include at least one of the following: the key feature points of the human body parts, height and weight of the user; the motion information includes at least one of the following: the movement direction of the key feature points of the human body parts, the rotation angle of the bone joints, speed, trajectory and external force magnitude.
Citation Information
Patent Citations
Muscle strength model optimizing method for humanoid robot synergic movement
CN103761392A
Three-dimensional human body posture estimation method and device
CN108830150A
Muscle synergy analysis method based on human dynamics modeling and surface EMG signal correction
CN109346176A
Human body action comparison system based on human body posture obtaining system
CN110109532A
Muscle activity estimation method, program, storage medium, muscle activity estimation system and muscle activity database
JP2006110217A
