Visualization method and system for correcting pull-up action

Abstract

The invention discloses a visualization method and system for pull-up motion correction, and belongs to the technical field of computer vision, and the method comprises the steps: obtaining a human body posture image frame of a target performing a pull-up motion; inputting the human body posture image frame into a preset human body posture estimation model to obtain a coordinate data sequence of human body skeleton points when the target moves; processing the coordinate data sequence of the human skeleton points when the target moves according to a preset correction rule to obtain a corrected coordinate data sequence of the human skeleton points; and based on the corrected coordinate data sequence of the human skeleton points, driving the virtual model to execute a pull-up action, and performing visual comparison with an action corresponding to the coordinate data sequence of the human skeleton points when the target moves before correction. According to the scheme, the motion difference before and after correction can be visually displayed, and visual feedback is provided for a practicer to understand the motion deviation and perceive the correction effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a pull-up action correction visualization method and system. BACKGROUND

[0002] As a gold standard movement for evaluating upper limb strength and core stability, the pull-up movement has a high activation effect on the latissimus dorsi muscle, biceps brachii muscle and core muscle group. This movement is very common in life, whether in daily fitness exercises or in various physical tests, we often see it. During exercise, the standardization of the movement will directly affect the training effect and safety, for example, abnormal trunk swinging may cause muscle strain, and incomplete action may lead to low training efficiency. This highlights the importance of correction.

[0003] With the rapid development of artificial intelligence technology and the deepening of sports science research, the cross-fusion of the two is becoming an important driving force for promoting sports training, health management and optimizing competitive performance. In particular, the rapid development of deep learning and computer vision technology provides a new path for sports analysis. Among them, the algorithm based on human posture estimation can extract biomechanical parameters such as joint angle and movement trajectory through video stream in real time, providing quantitative basis for movement standardization evaluation. The cross-fusion of AI and sports science, through the cooperation of data intelligence and human experience, builds a more efficient, accurate and inclusive sports service system. For example, Lu proposed a visual correction strategy based on image reconstruction and trajectory planning, which can identify incorrect movement posture and provide real-time feedback for movements such as deep squats, barbell curls, push-ups, plank supports and running training. Yang et al. fused data vectors to use a multi-modal information processing model to identify leg posture, and used the identification result as the input of the misidentification probability matrix model to realize intelligent correction of incorrect leg posture. Al-Barhamtoshy et al. identified individuals from the best side angle and evaluated the inclination of the neck and torso relative to the specified reference axis. When the individual bends beyond the specified threshold, it tracks the inclination angle. An error posture alarm system is realized using MediaPipe and OpenCV.

[0004] The above-mentioned movement correction methods, although solve the problem of real-time feedback of non-standard movements during the training process of athletes. However, they cannot provide athletes with intuitive visual images. That is, athletes cannot clearly see what effect they will have if they perform movements according to the correction specifications. SUMMARY

[0005] The present application provides a pull-up action correction visualization method and system to solve the technical problem that the prior art cannot provide athletes with intuitive visual images and cannot achieve visual correction.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In one aspect, the present application provides a visualization method for pull-up action correction, comprising: obtaining a human pose image frame of a target performing a pull-up action; inputting the human pose image frame of the target performing the pull-up action into a preset human pose estimation model, and obtaining coordinate data sequence of human skeleton points of the target in motion by using the preset human pose estimation model; processing the coordinate data sequence of human skeleton points of the target in motion according to a preset correction index to obtain coordinate data sequence of human skeleton points after correction; driving a virtual model to perform a pull-up action based on the coordinate data sequence of human skeleton points after correction, and visually comparing the action corresponding to the coordinate data sequence of human skeleton points of the target in motion before correction.

[0007] Further, the processing of the coordinate data sequence of human skeleton points of the target in motion according to the preset correction index to obtain the coordinate data sequence of human skeleton points after correction comprises: calculating a preset angle parameter based on the coordinate data sequence of human skeleton points of the target in motion; wherein the angle parameter includes: the included angle of two arms with the elbow as the vertex, the included angle of two legs with the knee as the vertex, the included angle of the two legs and the hip midpoint, and the included angle of two wrists; generating the coordinate data sequence of human skeleton points after correction according to the preset correction index based on the calculated angle parameter, to ensure that the coordinate data sequence of human skeleton points after correction meets the kinematic constraints of the human body; verifying whether the preset angle parameter corresponding to the coordinate data sequence of human skeleton points after correction meets the preset correction index, and if not, continuing the correction until the preset angle parameter corresponding to the coordinate data sequence of human skeleton points after correction meets the preset correction index.

[0008] Further, the preset correction index comprises: Index 1: At the lowest point, the included angle of the two arms with the elbow as the vertex is close to 180 degrees; Index 2: The included angle of the two legs with the knee as the vertex needs to be close to 180 degrees during the motion process; Index 3: The included angle of the two legs and the hip midpoint cannot exceed a preset angle threshold; Index 4: The included angle of the two wrists is less than 180 degrees, and the vertical coordinate of the lower jaw is higher than the coordinates of the two wrists; Index 5: The time from the start of the pull-up action to the lowest point is less than a preset time threshold.

[0009] Further, based on the calculated angle parameter, a coordinate data sequence of the corrected human body skeletal points is generated according to a preset correction index, so as to ensure that the coordinate data sequence of the corrected human body skeletal points conforms to the human body kinematics constraint, including: If the calculated angle parameter does not meet the requirements of the index one or the index four, the inverse kinematics is first applied to adjust the joint angle after meeting the corresponding index, and then the adjusted trajectory is smoothed by using an interpolation algorithm; If the calculated angle parameter does not meet the requirements of the index two or the index three, the position of the leg is adjusted by rotation or translation while keeping the skeletal length, so that the adjusted joint angle meets the corresponding index. If the time from the start of the upward movement to the lowest point does not meet the index five, the time sequence is compressed by deleting part of the frame or extracting key frames by using interpolation, so as to meet the time requirement.

[0010] Further, based on the coordinate data sequence of the corrected human body skeletal points, a virtual model is driven to perform the pull-up movement, and the movement corresponding to the coordinate data sequence of the human body skeletal points before correction is compared visually. The human body model is constructed in the Unity3D platform, and then the human body model is driven to perform the pull-up movement based on the coordinate data sequence of the corrected human body skeletal points, so as to complete the three-dimensional animation demonstration of the corrected pull-up movement and compare the movement corresponding to the coordinate data sequence of the human body skeletal points before correction visually.

[0011] On the other hand, the present application also provides a visualization system for pull-up movement correction, including: A data acquisition module is configured to acquire human posture image frames of a target performing a pull-up movement. A human posture recognition module is configured to input the human posture image frames of the target performing the pull-up movement acquired by the data acquisition module into a preset human posture estimation model, and obtain a coordinate data sequence of human body skeletal points of the target movement by using the preset human posture estimation model. A correction module is configured to process the coordinate data sequence of the human body skeletal points of the target movement obtained by the human posture recognition module according to a preset correction index, so as to obtain a coordinate data sequence of the human body skeletal points after correction. A visualization comparison module is configured to drive a virtual model to perform the pull-up movement based on the coordinate data sequence of the human body skeletal points after correction by the correction module, and compare the movement corresponding to the coordinate data sequence of the human body skeletal points before correction visually.

[0012] Further, the correction module is specifically configured to: Calculating preset angle parameters based on the coordinate data sequence of the human skeletal points during target motion; wherein the angle parameters include: the angle between the two arms with the elbow as the vertex, the angle between the two legs with the knee as the vertex, the angle between the two legs and the mid-hip point, and the angle between the two wrists; Based on the calculated angle parameters, a coordinate data sequence of the corrected human skeleton points is generated according to a preset correction index to ensure that the coordinate data sequence of the corrected human skeleton points conforms to human kinematic constraints; Verify whether the preset angle parameters corresponding to the coordinate data sequence of the corrected human skeleton points meet the preset correction indicators. If not, continue correction until the preset angle parameters corresponding to the coordinate data sequence of the corrected human skeleton points meet the preset correction indicators.

[0013] Furthermore, the preset correction indicators include: Indicator 1: At the lowest point, the angle between the two arms, with the elbows as the apex, should be close to 180 degrees; Indicator 2: The angle between the two legs with the knee as the apex needs to be close to 180 degrees during the exercise; Indicator 3: The angle between the two legs and the mid-hip point cannot exceed the preset angle threshold; Indicator 4: The angle between the two wrists is less than 180 degrees, and the vertical coordinate of the mandible is higher than the coordinates of the two wrists; Indicator 5: The time from starting to rise by grabbing the pole to lowering to the lowest point should be less than the preset time threshold.

[0014] Furthermore, the coordinate data sequence of the corrected human skeleton points is generated based on the calculated angle parameters according to the preset correction index to ensure that the coordinate data sequence of the corrected human skeleton points conforms to the human kinematic constraints, including: If the calculated angle parameters do not meet the requirements of indicator one or indicator four, inverse kinematics is first applied to adjust the joint angles to meet the corresponding indicators, and then the interpolation algorithm is used to smooth the adjusted trajectory; If the calculated angle parameter does not meet the requirements of indicator 2 or indicator 3, the position of the leg is adjusted by rotation or translation while maintaining the bone length so that the adjusted joint angle meets the corresponding indicator; If the time from the beginning of the ascent by grabbing the pole to the descent to the lowest point does not meet indicator five, the time series is compressed by deleting some frames or extracting key frames using interpolation to make it meet the time requirement.

[0015] Furthermore, the visual comparison module is specifically used to: A human body model is built in the Unity3D platform, and then the human body model is driven to perform a pull-up action based on the coordinate data sequence of the corrected human body skeleton points, so that a three-dimensional animation demonstration of the corrected pull-up movement is completed, and the action corresponding to the coordinate data sequence of the human body skeleton points during the target movement before correction is visualized and compared.

[0016] In yet another aspect, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0017] In yet another aspect, the present application also provides a computer-readable storage medium, which stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0018] The technical solution provided by the present application has at least the following beneficial effects: The pull-up action correction visualization method and system provided by the present application generate skeleton point data sequences conforming to preset correction indicators based on inverse kinematics and cubic spline interpolation algorithm, and then drive a three-dimensional character model to perform a pull-up movement in Unity3D using these data, thereby intuitively displaying the action difference before and after correction, and providing intuitive visual feedback for exercisers to understand action deviation and perceive correction effect. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flowchart of the pull-up action correction visualization method provided by the embodiment of the present application; Figure 2 is a correction data generation flowchart provided by the embodiment of the present application; Figure 3 is a descending position solving model diagram provided by the embodiment of the present application; Figure 4 is an ascending position solving model diagram provided by the embodiment of the present application; Figure 5 is a left arm two-dimensional skeleton chain model diagram provided by the embodiment of the present application; Figure 6 is an angle change curve diagram provided by the embodiment of the present application; Figure 7 is an angle a change curve diagram of the angle Figure 8 is a change curve diagram of the angle provided by the embodiment of the present application a change curve diagram of the angle Figure 9 is a change curve diagram of the nose longitudinal coordinate provided by the embodiment of the present application Figure 10 is a pre-compression coordinate schematic diagram provided by the embodiment of the present application Figure 11 is a post-compression coordinate schematic diagram provided by the embodiment of the present application Figure 12 is a pre-and post-correction comparison diagram provided by the embodiment of the present application; wherein (a) is a pre-correction action; (b) is a post-correction action Figure 13 is a system block diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0022] First of all, it should be noted that in the embodiments of the present application, the words such as "exemplarily", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplarily" is intended to present the concept in a specific way. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0023] First embodiment

[0024] The present embodiment provides a visualization method for pull-up action correction, which can be implemented by an electronic device, which can be a terminal or a server.

[0025] The execution flow of the method is shown in Figure 1 , which includes the following steps: S1, obtaining a human posture image frame of a target performing a pull-up action; S2, inputting the human posture image frame of the target performing the pull-up action into a preset human posture estimation model, and obtaining coordinate data sequence of human skeleton points of the target in motion by using the preset human posture estimation model; The human posture estimation model can be selected from Lightweight OpenPose, AlphaPose, etc.

[0026] S3, processing the coordinate data sequence of the human skeleton points during target motion according to a preset correction index to obtain a corrected coordinate data sequence of the human skeleton points; It should be noted that by using the human posture estimation algorithm, we can obtain the coordinate data sequence of the human skeleton points, and through calculation, we can obtain some angle value sequence data. Then, according to the pull-up movement evaluation standard, the following correction indicators are set: Indicator 1: When reaching the lowest point, the body should be completely lowered, that is, at the lowest point, the angle between the two arms with the elbow as the vertex and They should be close to 180 degrees.

[0027] Indicator 2: During exercise, you cannot use the force of the legs to push forward and backward, that is, the angle between the two legs with the knee as the apex and It needs to be close to 180 degrees during the movement.

[0028] Indicator 3: During exercise, you cannot use your legs to gain momentum, that is, the angle between your legs and the midpoint of your hips. Can't be too big.

[0029] Indicator 4: The chin should be higher than the horizontal bar at the highest point, which can be roughly calculated as the angle between the chin and the two wrists. It is slightly less than 180 degrees, and the mandibular vertical coordinate is higher than the two wrist coordinates.

[0030] Indicator 5: A complete pull-up exercise should not take too long, that is, the time from the beginning of the rise to the lowest point of the pull-up. Not too long.

[0031] Based on the above correction index, the bone point coordinate data can be operated according to the rules in real-time correction, so as to generate a new action sequence meeting the correction rules, and complete the online correction of pull-up movement; wherein, different correction methods are designed for different characteristics of the above different correction indexes. Index one and index four both involve elbow joint angle adjustment, which needs to apply inverse kinematics to adjust the joint angle of the terminal after meeting the correction index, to ensure that the angle meets the requirements. Then, the adjusted trajectory is smoothed using an interpolation algorithm to ensure the coherence of the action. Index two and index three need to limit some angles of the legs and constrain the position of the legs to prevent left and right swinging and leg lifting. And by optimizing the position of the legs, it is ensured that the included angle does not exceed the preset value. During the movement, the included angle between the legs and the hips is checked every frame, and if it exceeds the threshold, the position of the leg needs to be adjusted by rotation or translation while keeping the bone length. Index five involves a time constraint problem, which needs to adjust the speed of the action. If the original action time exceeds the preset value, the time sequence can be compressed by deleting some frames or extracting key frames using interpolation to make it meet the time requirement.

[0032] Specifically, the generation process of the correction data is as shown in Figure 2 , which includes: first inputting the original bone point coordinate data sequence, then calculating the required 6 angle values and judging according to the correction index. For data meeting the correction index, it is skipped; once data not meeting the correction index is found, it enters the corresponding correction module to generate the corrected bone point coordinate data. Then, the global optimization engine module adjusts the bone point coordinates to ensure that the newly generated coordinate data meets the human kinematics constraints. It should be noted that the human kinematics constraints mainly refer to judging whether the two values calculated according to the correction index meet the normal motion state, and discarding one of the values accordingly. For example, when calculating the elbow coordinate, point B does not meet this constraint, so only the value of point A is taken. Finally, it is verified whether the generated sequence meets the angle requirement and the action is coherent and natural.

[0033] For the two correction modules corresponding to index one and index four, the correction index does not meet the requirement due to the incomplete action. Therefore, first, a terminal state position meeting the index requirement is solved, and then the action sequence is completed by using the interpolation method.

[0034] In index one, the terminal state position is solved, that is, the values of and are solved. The body descends to the target height. At this point, the arm can be regarded as a straight line, the wrist coordinates remain unchanged, and the shoulder only moves up and down. It can be abstracted into a mathematical problem to solve: Take the left arm as an example (the right arm is the same), Figure 3 As shown, when the target position that meets the index requirements is reached, the left shoulder coordinates Can be done with the left wrist is the center of the circle, arm length A circle with a radius and the straight line that moves up and down with the left shoulder to confirm.

[0035] The equation of a circle is , the equation of the line is , the ordinate of the intersection of the circle and the straight line is (after the descent, the shoulder ordinate is smaller than the wrist ordinate, so only the smaller value is taken):

[0036] The solved ordinate is the ordinate value of the left shoulder after descending to the target height.

[0037] When solving the terminal state position in indicator 4, it is necessary to solve the The target height to which the body is raised when required. Usually, during a pull-up, the chin only moves up and down, meaning only the vertical coordinate changes. The wrist coordinates remain essentially unchanged during the movement, so this can be abstracted into a mathematical problem to solve. Figure 4 As shown, , , Represent the coordinates of the left wrist, right wrist and chin respectively. , , , , as well as These are all known quantities, and we need to solve , to determine the height of ascent when the requirements of Index 4 are met. Here we can use the geometric method of constructing the circle equation to solve it. From the known conditions, we know that The point must be on a straight line along the longitudinal axis. According to the circular angle theorem, it can be inferred that the point is also on a straight line along the longitudinal axis. For the string, On a circle with a central angle.

[0038] String length: , Chord midpoint: , The unit vector perpendicular to the chord: , Center of circle exist On the perpendicular bisector of The distance is: .

[0039] The coordinates of the center of the circle are , The radius of the circle is , The standard equation of a circle is .

[0040] Substitution , solution about The equation is: , We can get: .

[0041] That is the vertical coordinate value of the chin after rising to the target height.

[0042] After solving for the required height, all coordinates except the elbow coordinate can be determined. In cases where the elbow coordinate is not easily determined directly, this embodiment proposes using an inverse kinematic solution to solve for a state position that meets the indicator requirements. The elbow coordinate solution will be completed below.

[0043] Create a two-dimensional skeleton chain model of the arm, such as Figure 5 shown.

[0044] in, , , Represent the coordinates of the left wrist, left shoulder and left elbow respectively. and Represents the forearm length (elbow to wrist) and upper arm length (shoulder to elbow), Indicates is the center of the circle, The arc made by the radius, Therefore is the center of the circle, The arc made by the radius, and The two points are the intersection of the two arcs.

[0045] During exercise, and The length will not change, so the left elbow is On, and in . Solve for the intersection of the two arcs, and consider that the elbows bend downward during pull-ups. Figure 5 middle The point is the desired position.

[0046] The standard equation of the circle: , The standard equation of the circle: , The intersection of two circles will be solved by using geometric relations as follows: The distance between the centers of the circles: , No solution condition: or , Define the intermediate quantity: , Then the coordinates of the intersection points are:

[0047] After determining the end state that meets the index requirements, the three-spline interpolation method is used to supplement the intermediate frames to make the motion smoother. Suppose the interval that needs to be interpolated is , and according to the historical data, it is estimated that it will take about frames to smooth from the start of interpolation to the end state that meets the index. Then the rate of change is obtained by subtracting the coordinates of the previous frame from the current frame coordinates . A unique cubic polynomial curve is constructed: , which needs to satisfy four constraint conditions:

[0048] That is, the endpoint values and the slopes at the endpoints of the inserted data need to be consistent with the original data. By solving the cubic equation set, the polynomial coefficients are determined, and the coordinates of each frame are calculated. The generated data curve is a parabolic curve, which can reduce the speed from the starting speed to a slow target state, and can achieve a relatively natural motion transition process.

[0049] For the two correction modules corresponding to index two and index three, only the leg coordinates are involved in the correction process, and no data needs to be inserted or deleted. Therefore, by optimizing and adjusting some coordinate positions of the legs, the directions of the two legs are ensured to be vertical downward, so as to ensure that these angles do not exceed the index requirements, that is, the corresponding correction data can be generated. Here, we process the leg (left and right ankle and knee) coordinates in the frames that do not meet the two indicators, so that the horizontal coordinates are the same as those of the previous frame that meets the index requirements, and the vertical coordinates are moved up and down according to the coordinates of the hip midpoint.

[0050] For the correction module corresponding to index five, directly deleting some frames will result in incoherent motion, so the sampling method is considered to compress the data. As mentioned above, we have used the cubic spline interpolation method to complete the data generation, and this method can also be used for data resampling. By using cubic spline to generate a smooth trajectory between key frames, the number of intermediate frames is reduced. For example, the original has frame, want to shorten to frame, , then each coordinate of each joint is resampled to points, so the total time is shortened.

[0051] Further, in order to verify whether the above correction data generation method is effective, the embodiment corrects some non-standard action data according to the above correction data generation method, and obtains experimental results as shown in Figures 6 to 11 . Among them, Figure 6 and Figure 7 are experimental result graphs for correcting leg swing in the movement process; Figure 8 and Figure 9 are experimental result graphs for correcting insufficient height in the movement process, resulting in the chin not passing the horizontal bar; Figure 10 and Figure 11 are experimental result graphs for correcting excessive time.

[0052] Figure 6 and Figure 7 respectively in the chin-up movement process, the leg is kicked for force, and and are comparison graphs of data before and after correction. The horizontal dashed line is the threshold set for the index, and the curve Angle_5 is the data before correction, and some values exceed the threshold line. It can be seen that the leg angle is too large and the leg angle is also too large. The curve new_Angle_5 in the figure is the data generated after correction by using the correction method of the embodiment, which is within the threshold line and can meet the requirements of the correction index.

[0053] Figure 8 indicates the transformation curve when the height of the rise is insufficient , and the horizontal dashed line is the threshold corresponding to index four. The curve Angle_6 is also the data before correction. From this curve, we can see that when the second rise to the highest point is about eighty frames, the minimum point of the curve is not below the threshold line, indicating that the height of the rise is insufficient in the chin-up movement. By using the correction method proposed in the embodiment, the position satisfying the index threshold is calculated first, and then the cubic spline interpolation method is used to generate intermediate data, as shown in the curve new_Angle_5 Figure 8 , a relatively smooth curve is added to the original data, so that the corrected data can meet the requirements of the index. Figure 9 is the transformation of the longitudinal coordinate of the nose after correction.

[0054] For the requirement of index five, suppose there is Figure 10The data shown has 60 frames, and according to the time compression ratio, it needs to be compressed to 0.8 times of the original. After sampling compression according to the above correction method, the obtained data is as shown in the figure Figure 11 Only 48 key data points are retained. Figure 10 And Figure 11 It can be seen that the compressed data curve is basically the same as the curve before compression, and the main feature points are retained.

[0055] S4, based on the coordinate data sequence of the corrected human skeleton points, driving the virtual model to perform pull-up action, and visualizing the action corresponding to the coordinate data sequence of the human skeleton points before correction. It should be noted that although the above real-time correction function can provide timely feedback to the exerciser, the exerciser cannot well feel the effect of the non-standard action at that time if it is done according to the correction prompt. Therefore, in order to help the exerciser more intuitively see the comparison of the exercise effect before and after correction, the embodiment will visualize the generated correction data in Unity3D, generate the action sequence after correction according to the correction index, and visualize the comparison between the action sequence before correction and the action sequence in Unity3D.

[0056] Specifically, in order to demonstrate the exercise effect according to the corrected data, the embodiment will complete the three-dimensional animation demonstration of the pull-up exercise after correction on the Unity3D platform.

[0057] To complete the visualization task, a 3D human model needs to be imported first. We choose to download a suitable model from the Mixamo platform, which is an online 3D character animation platform launched by Adobe Company, with a huge animation and character model library. It supports exporting FBX format to mainstream engine platforms such as Unity and Unreal Engine. We choose Y-Bot model and export it as FBX format recognizable by Unity.

[0058] Drag the downloaded Mixamo model file into the Assets folder of Unity3D to complete the model file import, and then drag the model into the left Scene part, so that we can see the imported character model in the scene. Click the Y-Bot model in Scene to adjust the model initialization position, rotation and scaling in the right inspector. Then, in order to use scripts to control the character model, the binding of the character skeleton points also needs to be completed. In the Avatar window, bind the model's skeleton structure and animation. By selecting the actual bones of the model in the Avatar window to correspond to the Humanoid standard bones in Unity3D.

[0059] After the completion of the 3D human model construction and the skeleton point mapping binding, a control script also needs to be written to drive the motion of the person using the corrected data generated in the previous step. Since the skeleton points identified by the pose estimation algorithm and the skeleton points of the person are not in one-to-one correspondence, the mapping of the corresponding relationship needs to be completed as shown in Table 1.

[0060] Since the three-dimensional person model is driven by two-dimensional data, the depth information, that is, the z-axis coordinate of the person is missing. If the position in the x, y, and z axes is adjusted in the transform to control the motion on the basis of the above-mentioned skeleton point mapping, the limbs will be easily folded and distorted. Therefore, the idea of using joint rotation to control the motion of the person is adopted in the embodiment. Different joints in the person model have different initial rotation amounts, and even the initial coordinate axes of some joints are not the same. Therefore, the initial rotation matrix of all joints needs to be aligned using the initial pose of the person first, so that an intermediate matrix is obtained. When the coordinates are driven subsequently, the intermediate matrix can be used to align all the coordinates. According to the transformation of the coordinates between adjacent frames, the rotation matrix of each joint can be determined, and the angle of rotation of each joint in each frame can be calculated by multiplying the alignment matrix.

[0061] Table 1 Skeleton point mapping relationship table

[0062] The script written is mounted to the person model, and after the program is run, the person can be seen to start the pull-up motion according to the given data.

[0063] Some screenshots in the running process are shown in Figure 12 The left graph is the effect of driving the original uncorrected data, and from the graph we can see that the legs are obviously bent. The right graph is the effect of driving the corrected data, and we can see that the legs are not obviously bent and remain vertical.

[0064] In summary, the embodiment provides a visualization method for pull-up motion correction. First, a corresponding corrected data generation method is proposed based on a correction index. Then, in order to better visualize the motion posture of the corrected person, a three-dimensional person model is imported based on Unity3D, and the skeleton point coordinate data is used for driving control. The visualization display of the pull-up motion correction is completed. Therefore, intuitive visual feedback can be provided for the exerciser to understand the motion deviation and perceive the correction effect. It has great application prospect and market value.

[0065] Second embodiment

[0066] The embodiment provides a pull-up action correction visual system, which comprises the following modules. A data acquisition module is configured to acquire a human posture image of a target performing a pull-up action. A human posture recognition module is configured to input the human posture image of the target performing the pull-up action acquired by the data acquisition module into a preset human posture estimation model, and obtain coordinate data sequences of human skeleton points of the target in motion by using the preset human posture estimation model. A correction module is configured to process the coordinate data sequences of the human skeleton points of the target in motion obtained by the human posture recognition module according to a preset correction index, and obtain coordinate data sequences of human skeleton points after correction. A visual comparison module is configured to drive a virtual model to perform a pull-up action based on the coordinate data sequences of the human skeleton points after correction of the correction module, and perform visual comparison of the action corresponding to the coordinate data sequences of the human skeleton points of the target in motion before correction.

[0067] It should be noted that the pull-up action correction visual system of the embodiment corresponds to the pull-up action correction visual method of the first embodiment; the functions of the functional modules in the pull-up action correction visual system of the embodiment correspond to the flow steps in the pull-up action correction visual method of the first embodiment; and therefore, the functions of the functional modules in the pull-up action correction visual system of the embodiment will not be described herein.

[0068] Third embodiment

[0069] The embodiment provides an electronic device, such as Figure 13 As shown in the figure, the electronic device comprises a processor and a memory; wherein the processor and the memory can be connected through a communication bus; the memory stores at least one instruction, the instruction is loaded and executed by the processor, so as to realize the method of the first embodiment. In addition, the electronic device can further comprise a transceiver, the processor and the transceiver can be connected through a communication bus, and the transceiver is used for communication with other devices.

[0070] Next, the electronic device will be described in detail. Figure 13 The various constituent components of the electronic device will be described in detail. The processor is the control center of the electronic device. The electronic device can include multiple processors. Each of the processors can be a single-CPU or a multi-CPU. The processor can be one processor or a collective term of multiple processing elements. For example, the processor can be one or more central processing units (CPUs), other general purpose processors, application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement an embodiment of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor can be a microprocessor or any conventional processor, or the like. The processor can perform various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0071] In a specific implementation, as an embodiment, the processor can include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 1, of course, this is only an exemplary description. Figure 13

[0072] The memory is used to store software programs for implementing the solution of the present application, and is controlled by the processor to perform. The specific implementation can refer to the method embodiments described above, and will not be described here.

[0073] ​Optionally, the memory may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Figure 13 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0074] The transceiver may include a receiver and a transmitter ( Figure 13 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Figure 13 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0075] In addition, it should be noted that Figure 13 The structure of the electronic device shown in the figure does not constitute a limitation on the device. The actual device may include more or fewer components than shown, or may combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment can refer to the technical effects described in the first embodiment above, and therefore will not be repeated here.

[0076] Fourth embodiment

[0077] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device. The instructions stored therein can be loaded by a processor in a terminal to execute the method described above.

[0078] Moreover, it should be noted that the present application can be provided as a method, an apparatus, or a computer program product. Therefore, the embodiments of the present application can take the form of an entirely or partially hardware embodiment, an entirely or partially software embodiment, or an embodiment combining software and hardware aspects. Furthermore, when implemented in software, the embodiments of the present application can take the form of a computer program product including one or more computer program instructions embodied on one or more computer-usable storage media (media). The computer program instructions can be executed by a computer, a processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart block or blocks or in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart block or blocks or the flowcharts and / or block diagrams.

[0079] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate one or more functions implemented in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The apparatus that implements the functions specified in one or more flows and / or blocks.

[0080] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart block or blocks or the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1the functions specified in the individual block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide processes for implementing the functions specified in the flowchart block(s) or block(s). Figure 1 the functions specified in the individual block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide processes for implementing the functions specified in the flowchart block(s) or block(s). Figure 1 the functions specified in the individual block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide processes for implementing the functions specified in the flowchart block(s) or block(s).

[0081] It should also be noted that, in the present document, the terms such as first and second, etc. are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or terminal device. Without more limitations, the element defined by the statement "including a…", does not exclude the presence of other identical elements in the process, method, article or terminal device including the element. In addition, the term "and / or" is merely a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B together, and the existence of B alone, where A and B can be singular or plural. In addition, the character " / " in the present document generally represents an "or" relationship between the preceding and following associated objects, but it can also represent an "and / or" relationship, which can be understood in the context before and after. "At least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0082] In addition, it can be understood that in various embodiments of the present application, the size of the sequence number of the above processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0083] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0084] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of functional modules / units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms. The units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present, or two or more units can be integrated in one unit.

[0085] If the method is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0086] Finally, it should be noted that the above description is only the preferred embodiment of the application, it should be pointed out that although the preferred embodiment of the application has been described, for those skilled in the art, once the basic creative concept of the application is known, several improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should also be considered as the protection scope of the application. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the application.

Claims

1. A visualization method for correcting pull-up movements, characterized in that: include: Acquire a human body posture image frame of the target performing a pull-up action; Inputting a human body posture image frame of a target performing a pull-up action into a preset human body posture estimation model, and using the preset human body posture estimation model to obtain a coordinate data sequence of human body skeletal points when the target is moving; According to the preset correction index, the coordinate data sequence of the human skeleton points during the target movement is processed to obtain the coordinate data sequence of the human skeleton points after correction; The virtual model is driven to perform pull-ups based on the corrected coordinate data sequence of the human skeleton points, and the movements corresponding to the coordinate data sequence of the human skeleton points during the target movement before correction are visually compared.

2. The visualization method for correcting pull-up movements according to claim 1, wherein: The method of processing the coordinate data sequence of the human skeleton points during target motion according to the preset correction index to obtain the coordinate data sequence of the human skeleton points after correction includes: Calculating preset angle parameters based on the coordinate data sequence of the human skeletal points during target motion; wherein the angle parameters include: the angle between the two arms with the elbow as the vertex, the angle between the two legs with the knee as the vertex, the angle between the two legs and the mid-hip point, and the angle between the two wrists; Based on the calculated angle parameters, a coordinate data sequence of the corrected human skeleton points is generated according to a preset correction index to ensure that the coordinate data sequence of the corrected human skeleton points conforms to human kinematic constraints; Verify whether the preset angle parameters corresponding to the coordinate data sequence of the corrected human skeleton points meet the preset correction indicators. If not, continue correction until the preset angle parameters corresponding to the coordinate data sequence of the corrected human skeleton points meet the preset correction indicators.

3. The visualization method for correcting pull-up movements according to claim 1, wherein: The preset correction indicators include: Indicator 1: At the lowest point, the angle between the two arms, with the elbows as the apex, should be close to 180 degrees; Indicator 2: The angle between the two legs with the knee as the apex needs to be close to 180 degrees during the exercise; Indicator 3: The angle between the two legs and the mid-hip point cannot exceed the preset angle threshold; Indicator 4: The angle between the two wrists is less than 180 degrees, and the vertical coordinate of the mandible is higher than the coordinates of the two wrists; Indicator 5: The time from starting to rise by grabbing the pole to lowering to the lowest point should be less than the preset time threshold.

4. The visualization method for correcting pull-up movements according to claim 2, wherein: The method of generating a coordinate data sequence of the corrected human skeleton points based on the calculated angle parameters and according to preset correction indicators, ensuring that the coordinate data sequence of the corrected human skeleton points conforms to human kinematic constraints, includes: If the calculated angle parameters do not meet the requirements of indicator one or indicator four, inverse kinematics is first applied to adjust the joint angles to meet the corresponding indicators, and then the interpolation algorithm is used to smooth the adjusted trajectory; If the calculated angle parameter does not meet the requirements of indicator 2 or indicator 3, the position of the leg is adjusted by rotation or translation while maintaining the bone length so that the adjusted joint angle meets the corresponding indicator; If the time from the beginning of the ascent by grabbing the pole to the descent to the lowest point does not meet indicator five, the time series is compressed by deleting some frames or extracting key frames using interpolation to make it meet the time requirement.

5. The visualization method for correcting pull-up movements according to claim 1, wherein: The virtual model is driven to perform a pull-up based on the corrected human skeleton point coordinate data sequence, and the corresponding movement is visually compared with the movement corresponding to the human skeleton point coordinate data sequence during the target movement before correction, including: A human body model was constructed in the Unity3D platform. The model was then driven to perform pull-ups based on the corrected coordinate data sequence of the human skeletal points. A 3D animation demonstration of the corrected pull-up movement was completed, and a visual comparison was made with the movement corresponding to the coordinate data sequence of the human skeletal points during the target movement before correction.

6. A visualization system for correcting pull-up movements, characterized in that: include: A data acquisition module is used to acquire image frames of a human body posture of a target performing a pull-up action; a human posture recognition module, configured to input the human posture image frames of the target performing pull-ups acquired by the data acquisition module into a preset human posture estimation model, and obtain a coordinate data sequence of the human skeleton points of the target during the movement using the preset human posture estimation model; a correction module, configured to process the coordinate data sequence of the human skeleton points during the target movement obtained by the human posture recognition module according to a preset correction index, to obtain a corrected coordinate data sequence of the human skeleton points; A visualization comparison module is used to drive the virtual model to perform pull-up movements based on the coordinate data sequence of the human skeleton points corrected by the correction module, and to perform a visualization comparison with the movement corresponding to the coordinate data sequence of the human skeleton points during the target movement before correction.

7. The visualization system for correcting pull-up movements according to claim 6, wherein: The correction module is specifically used for: Calculating preset angle parameters based on the coordinate data sequence of the human skeletal points during target motion; wherein the angle parameters include: the angle between the two arms with the elbow as the vertex, the angle between the two legs with the knee as the vertex, the angle between the two legs and the mid-hip point, and the angle between the two wrists; Based on the calculated angle parameters, a coordinate data sequence of the corrected human skeleton points is generated according to a preset correction index to ensure that the coordinate data sequence of the corrected human skeleton points conforms to human kinematic constraints; Verify whether the preset angle parameters corresponding to the coordinate data sequence of the corrected human skeleton points meet the preset correction indicators. If not, continue correction until the preset angle parameters corresponding to the coordinate data sequence of the corrected human skeleton points meet the preset correction indicators.

8. The visualization system for correcting pull-up movements according to claim 6, wherein: The preset correction indicators include: Indicator 1: At the lowest point, the angle between the two arms, with the elbows as the apex, should be close to 180 degrees; Indicator 2: The angle between the two legs with the knee as the apex needs to be close to 180 degrees during the exercise; Indicator 3: The angle between the two legs and the mid-hip point cannot exceed the preset angle threshold; Indicator 4: The angle between the two wrists is less than 180 degrees, and the vertical coordinate of the mandible is higher than the coordinates of the two wrists; Indicator 5: The time from starting to rise by grabbing the pole to lowering to the lowest point should be less than the preset time threshold.

9. The visualization system for correcting pull-up movements according to claim 7, wherein: The method of generating a coordinate data sequence of the corrected human skeleton points based on the calculated angle parameters and according to preset correction indicators, ensuring that the coordinate data sequence of the corrected human skeleton points conforms to human kinematic constraints, includes: If the calculated angle parameters do not meet the requirements of indicator one or indicator four, inverse kinematics is first applied to adjust the joint angles to meet the corresponding indicators, and then the interpolation algorithm is used to smooth the adjusted trajectory; If the calculated angle parameter does not meet the requirements of indicator 2 or indicator 3, the position of the leg is adjusted by rotation or translation while maintaining the bone length so that the adjusted joint angle meets the corresponding indicator; If the time from the beginning of the ascent by grabbing the pole to the descent to the lowest point does not meet indicator five, the time series is compressed by deleting some frames or extracting key frames using interpolation to make it meet the time requirement.

10. The visualization system for correcting pull-up movements according to claim 6, wherein: The visual comparison module is specifically used for: A human body model was constructed in the Unity3D platform. The model was then driven to perform pull-ups based on the corrected coordinate data sequence of the human skeletal points. A 3D animation demonstration of the corrected pull-up movement was completed, and a visual comparison was made with the movement corresponding to the coordinate data sequence of the human skeletal points during the target movement before correction.

Images