Method for intelligent recognition of postures in a sports process based on image recognition

Abstract

The present application relates to the technical field of motion posture analysis, and particularly to a motion process posture intelligent recognition method based on image recognition. The present application frames the human body joint point coordinates of a motion video, calculates the displacement vectors of the same joint point between adjacent frames and generates displacement segments, uses the reverse time in the displacement segments to cut the joint point sequence into multiple subsegments, so as to avoid the lack of stage division basis in the time dimension of posture analysis. The present application combines the joint points in each subsegment into multiple motion chains according to the limb connection relationship, calculates the included angles between adjacent limb segments in each motion chain and generates an included angle sequence, determines the swing period according to the positive and negative signs of the difference between adjacent included angles in the included angle sequence, so as to quantify the dynamic period change of each limb segment in the swing process.

Description

Technical Field

[0001] This invention relates to the field of motion posture analysis technology, and more specifically to an intelligent posture recognition method for motion processes based on image recognition. Background Technology

[0002] In scenarios such as sports fitness, sports training, and rehabilitation assessment, it is necessary to identify whether the athlete's posture is unstable through sports videos, and to locate the specific joints and movement stages where the instability occurs, so as to provide quantitative basis for posture correction, injury prevention, and training effect evaluation.

[0003] In the prior art, such as Chinese invention patent with publication number CN119314200A, a posture detection method and device are disclosed. The method detects the posture of the acquired image to determine the position data of the human joints of the subject, and determines posture parameters such as forward tilt angle, torso backward bending angle and lateral tilt angle. When the posture parameters determine that the back posture of the subject is an abnormal posture, a prompt message is output.

[0004] For example, Chinese invention with publication number CN117789308A discloses a motion posture recognition method based on deep learning. It collects motion posture images during exercise, focuses the motion posture images on key joints, and extracts the presentation angle of key joints to realize the recognition of strength training exercise postures.

[0005] However, the existing technologies mentioned above still have the following shortcomings: 1. They do not continuously track and analyze the displacement direction of each joint during the movement, especially failing to define the boundary of the movement stage based on the reversal point of the displacement direction, resulting in a lack of time-dimensional stage division basis for posture analysis. 2. They only compare the angles of the whole body or local joints, which cannot quantify the dynamic periodic changes of each limb segment during the swinging process, and therefore cannot locate the specific joint where instability occurs. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent motion process posture recognition method based on image recognition.

[0007] The technical solution adopted by the present invention to solve its technical problem is: a motion process posture intelligent recognition method based on image recognition, including the following steps: locating the coordinates of human joint points in motion video frame by frame to obtain a joint point sequence; calculating the displacement vector of the same joint point between adjacent frames and generating a displacement segment.

[0008] Starting from the beginning of the displacement segment, three displacement vectors are taken sequentially along the time axis as a group for comparison.

[0009] Calculate the angle between the directions of two adjacent displacement vectors in each group of comparison objects to obtain the front angle and the back angle.

[0010] Based on the difference between the front and back angles between adjacent comparison objects, the reversal moment is identified from the consecutive frames corresponding to the comparison objects, and the frame corresponding to the reversal moment is recorded as the switching frame; the key point sequence is divided into multiple sub-segments using the switching frame.

[0011] Based on the limb connection relationship, the joints in each sub-segment are combined into multiple kinematic chains, and the included angles between adjacent limb segments in each kinematic chain are calculated and an included angle sequence is generated.

[0012] The oscillation period, including the rising segment, falling segment, and peak point, is determined based on the sign of the difference between adjacent angles within the angular sequence.

[0013] The number of frames from the start of the rising segment to the peak is recorded as the rising frame number; when the difference in the rising frame number between adjacent swing cycles or the difference in the included angle at the peak exceeds their respective tolerances, the corresponding kinematic chain is recorded as the chain to be verified.

[0014] Extract the coordinates of the joints in the frame range corresponding to the chain to be verified in the joint sequence. Based on the cumulative number of times the displacement direction of the same joint is opposite to the corresponding displacement direction in the displacement segment between adjacent frames, determine the attitude instability joints.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention locates the coordinates of human joints in motion video frame by frame, calculates the displacement vector of the same joint between adjacent frames and generates displacement segments, and uses the reversal time in the displacement segments to divide the joint sequence into multiple sub-segments, thereby avoiding the lack of time dimension stage division basis for attitude analysis.

[0016] 2. This invention combines the joints within each sub-segment into multiple kinematic chains according to the limb connection relationship, calculates the included angle between adjacent limb segments within each kinematic chain and generates an included angle sequence, and determines the swing period based on the sign of the difference between adjacent included angles within the included angle sequence, thereby quantifying the dynamic periodic changes of each limb segment during the swing process.

[0017] 3. This invention selects the chain to be verified by comparing the difference in the number of rising frames and the difference in the angle at the peak point between adjacent swing cycles with their respective tolerances; extracts the coordinates of the joints within the corresponding frame range of the chain to be verified, and counts the cumulative number of times the displacement direction of the same joint between adjacent frames is opposite to the corresponding displacement direction in the non-chain to be verified, thereby determining the attitude instability joints and providing clear location indications for subsequent attitude correction and damage prevention. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the identification method of the present invention;

[0020] Figure 2 This is a schematic diagram illustrating the process of dividing a joint sequence into multiple sub-segments according to the present invention.

[0021] Figure 3 A schematic diagram illustrating the process of determining the oscillation period in this invention;

[0022] Figure 4 This is a schematic diagram illustrating the process of determining the chain to be verified in this invention. Detailed Implementation

[0023] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. Furthermore, it should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale.

[0024] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0025] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] The specific scheme of the motion process posture intelligent recognition method based on image recognition provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0028] Please see Figure 1 The flowchart of the intelligent motion process posture recognition method based on image recognition provided by the present invention is shown, which specifically includes the following steps: Step S1, locate the coordinates of human joint points in motion video frame by frame to obtain the joint point sequence; calculate the displacement vector of the same joint point between adjacent frames and generate displacement segments.

[0029] Motion video refers to a sequence of image frames generated by continuously capturing images of a moving human body using an image acquisition device. Motion video is stored in a compressed format on a storage medium.

[0030] In practice, the motion video is first read frame by frame. The video decoder is called to decode the video file into a sequence of pixel matrices. The frame rate parameter, which represents the number of frames per second, is read from the file header. The pixel matrix data of each frame is read sequentially from the beginning of the video file according to the frame rate parameter, and the read pixel matrix is ​​stored in a memory buffer. During the reading process, the sequence number of the currently read frame is recorded, starting from 1 and incrementing sequentially. At the same time, the timestamp of the current frame is recorded, which is obtained by subtracting 1 from the current frame sequence number and dividing by the frame rate parameter.

[0031] Furthermore, the deep learning-based object detection algorithm takes the pixel matrix of the current frame as input and outputs a bounding rectangle of the human body. The area defined by the rectangle is taken as the upright area of ​​the human body (the specific process is existing technology and will not be described here). A planar coordinate system is established with the row direction of the pixel matrix as the horizontal axis and the column direction of the pixel matrix as the vertical axis. The minimum and maximum coordinate values ​​of all pixels in the upright area of ​​the human body in the current frame are counted in the horizontal axis direction. The arithmetic mean of the minimum and maximum coordinate values ​​is taken as the horizontal center position. A straight line perpendicular to the horizontal axis is drawn through the horizontal center position to obtain the vertical central axis.

[0032] Then, using the vertical center line of the current frame as the dividing line, the pixels of the upright human body region are divided into left and right halves to reconstruct the posture of the limbs on both sides. Within each half, the scan proceeds line by line from the bottom row of the current frame upwards, ensuring that the scan path aligns with the natural top-to-bottom arrangement of the human body structure; within each row, the scan proceeds from left to right, with each row containing all pixels from the left boundary to the right boundary of the pixel matrix. Simultaneously with each row scan, the difference between the next pixel value and the previous adjacent pixel value within that row is calculated.

[0033] Next, pixels located outside the upright area of ​​the human body in the current frame are taken as background pixels; the absolute value of the difference between adjacent pixel values ​​of all background pixels in each row is calculated, and the arithmetic mean of all absolute values ​​in each row is calculated as the background row fluctuation value. The arithmetic mean of the background row fluctuation values ​​of all rows is calculated as the background pixel fluctuation amplitude.

[0034] The left and right boundary points are determined by comparing the absolute value of the difference with the fluctuation range of the background pixels in the current frame and by the sign of the difference.

[0035] The specific rules for determining the boundary point are as follows: Iterate through the differences in each row. If the absolute value of the current difference is greater than the background pixel fluctuation range and the current difference is positive, then the position of the next pixel in the adjacent pixel pair corresponding to the current difference is recorded as the left boundary point. If the absolute value of the current difference is greater than the background pixel fluctuation range and the current difference is negative, then the position of the next pixel in the adjacent pixel pair corresponding to the current difference is recorded as the right boundary point. If the absolute value of the current difference is less than or equal to the background pixel fluctuation range, then no boundary point is recorded.

[0036] If multiple left boundary points are recorded in the same row, the one with the smallest x-coordinate value is taken as the final left boundary point; if multiple right boundary points are recorded in the same row, the one with the largest x-coordinate value is taken as the final right boundary point. The remaining boundary points are discarded.

[0037] After determining the left and right boundary points, pair the left and right boundary points in the same row. Connect the boundary point pairs located in the same half-region using straight line segments in row order (i.e., from bottom to top) to obtain the outer edges of the left and right limbs. The outer edges of the left and right limbs together constitute the outer boundary of the trunk center.

[0038] If no boundary points are found in a row, skip that row, do not generate boundary point pairs, and continue processing the next row. Skipped rows do not generate connections during the process of obtaining the limb outer edge line; the boundary points of the rows above and below them are directly connected.

[0039] Subsequently, for each limb's outer edge line, the ratio of the difference between the ordinates of two adjacent points to the difference between their abscissas is calculated. The arctangent of this ratio is used to obtain the tilt angle. When the difference between the abscissas is zero: if the difference between the ordinates is greater than zero, the tilt angle is 90 degrees; if the difference between the ordinates is less than zero, the tilt angle is -90 degrees; if the difference between the ordinates is zero, it indicates that the two adjacent points coincide, and this pair of points is skipped, and the process continues with the next pair of adjacent points.

[0040] Perform the above calculations on all adjacent pairs of points to obtain the tilt angle sequence of the limb's outer edge. Calculate the difference between two consecutive tilt angles in the tilt angle sequence to obtain the tilt angle change.

[0041] For each limb's outer edge line, if the change in tilt angle remains constant (positive or negative) and non-zero across three consecutive point distances, it indicates that the limb's outer edge line is continuously bending in the same direction within those three consecutive point distances. The position of the middle point is then recorded as the limb's turning point. Otherwise, the line continues to move backward one point distance along the current limb's outer edge line, and the above judgment is repeated for the next three consecutive point distances. The limb turning point corresponds to the position on the limb's outer edge line where the direction of the continuous deflection occurs, i.e., the location of the limb joint. The point distance refers to the interval between two adjacent points on the limb's outer edge line.

[0042] If no turning point of any limb is found on the outer edge of any limb in the current frame, skip the current frame and continue processing the next frame.

[0043] Furthermore, for the first frame, the joint coordinates are determined based on the spatial distribution density of limb turning points. Specifically: the coordinates of all limb turning points in the first frame are projected onto the same plane coordinate system; the Euclidean distance between every two limb turning points in the first frame is calculated, and the sum of all Euclidean distances is divided by the total number of Euclidean distances to obtain the average distance; a circular region is formed with each limb turning point as the center and the average distance as the radius, and the number of limb turning points within the circular region is counted as the spatial distribution density; the abscissas of all limb turning points within the circular region with the highest spatial distribution density are summed and divided by the total number of limb turning points within the circular region to obtain the abscissa of the centroid, and the ordinate of the centroid is calculated similarly. The abscissa and ordinate of the centroid together constitute the centroid coordinates; the centroid coordinates are taken as the joint coordinates.

[0044] For subsequent frames, candidate points are determined based on the spatial distance between the limb turning point and the joint coordinates in the previous frame. This maintains the continuity of the joint trajectory in the time dimension and avoids significant jumps in joint positions between adjacent frames due to independent positioning frame by frame.

[0045] The specific process for determining candidate points is as follows: using each joint point in the previous frame as a reference point, calculate the Euclidean distance between each limb turning point in the current frame and the reference point, and use this distance as the spatial distance; select the limb turning point with the smallest spatial distance as the candidate point for the current frame. If multiple limb turning points have the same and smallest spatial distance from the same reference point, then use the coordinates of the same joint point in the previous frame as the candidate point for the current frame.

[0046] Simultaneously, in the current frame, the pixel area outside the boundaries of the left and right limbs is defined as the background area, and the background area in the previous frame is defined similarly. The difference between the maximum and minimum pixel values ​​in each row of the background area in the current frame is calculated as the pixel value distribution span of the current frame, and the pixel value distribution span of the previous frame is calculated similarly. The pixel value distribution span of each row in the current frame is divided by the pixel value distribution span of the corresponding row in the previous frame to obtain the span ratio for each row. The minimum span ratio of all rows is taken as the lower limit, and the maximum value as the upper limit. The lower limit and the upper limit constitute the range of background pixel span ratios in the current frame.

[0047] Based on this, the line segment between adjacent candidate points on the outer edge of the same limb is taken as the limb segment. The joint coordinates are determined by the length ratio of the limb segment in the current frame to that in the previous frame, combined with the range of background pixel span ratios in the current frame.

[0048] The specific determination rules are as follows: For each candidate point in the current frame, calculate the Euclidean distance between it and adjacent candidate points on the outer edge of the same limb, and use this distance as the line segment length. Record this line segment length as the limb segment length of the current frame. Similarly, calculate the limb segment length of the previous frame. Divide the limb segment length of the current frame by the limb segment length of the previous frame to obtain the length ratio. If the length ratio is less than the lower limit of the background pixel span ratio range, or greater than the upper limit of the background pixel span ratio range, then use the coordinates of the same joint point in the previous frame as the joint point coordinates of the current frame; otherwise, use the candidate point as the joint point coordinates of the current frame.

[0049] Finally, the joint coordinates obtained from each frame are arranged in chronological order to generate a joint sequence. Each item in the joint sequence contains the frame number and the coordinates of all joints within that frame.

[0050] After obtaining the keypoint sequence, based on the coordinates of the same keypoint in two adjacent frames, the coordinates of the earlier keypoint in time sequence are designated as the start point, and the coordinates of the later keypoint in time sequence are designated as the end point.

[0051] The direction pointing to the endpoint is taken as the displacement direction; the Euclidean distance between the endpoint and the starting point is calculated as the linear distance; the linear distance between the two coordinates is taken as the displacement distance, and the displacement direction and displacement distance are combined to form the displacement vector. The change in the direction of the displacement vector reflects the change in the motion direction of the joint.

[0052] Next, all displacement vectors obtained for the same joint point between each two adjacent frames in the joint point sequence are arranged sequentially in time and connected in a broken line manner to generate displacement segments. The directional changes of each displacement vector in the displacement segment reflect the changing pattern of the joint point's motion direction, providing a data foundation for subsequent identification of the moment when the motion direction reverses.

[0053] Please see Figure 2 Step S2: Starting from the beginning of the displacement segment, take three displacement vectors in chronological order as a group of comparison objects, calculate the directional angle between two adjacent displacement vectors in each group of comparison objects, and obtain the front angle and the back angle; based on the difference between the front angle and the back angle between adjacent comparison objects, identify the reversal time from the continuous frames corresponding to the comparison objects, and record the frame corresponding to the reversal time as the switching frame; use the switching frame to divide the joint sequence into multiple sub-segments.

[0054] Because the direction of each displacement vector in a displacement segment reflects the orientation of the joint's motion between adjacent frames, when the motion direction changes in the opposite direction, it indicates that the joint has reached a turning point in its motion trajectory. The frame corresponding to the turning point divides the joint sequence into different motion stages, providing a segmentation basis for subsequent segmented analysis of attitude changes. Therefore, the reversal moment is identified from the consecutive frames corresponding to the comparison object.

[0055] Specifically, starting from the beginning of the displacement segment, three displacement vectors corresponding to every four consecutive frames are taken as a set of comparison objects in chronological order, so as to provide two consecutive included angles. The degree of directional deflection is determined by comparing the changing trends of the two included angles.

[0056] It should be added that the first two displacement vectors of the last three displacement vectors are shared among the comparison groups. For example, the first group uses the three displacement vectors corresponding to frames 1 to 4, and the second group uses the three displacement vectors corresponding to frames 2 to 5. This time-partial overlap setting avoids the omission of subsequent reversal moments due to group boundaries.

[0057] For each pair of comparison objects, the angle between the directions of the two displacement vectors that appear earlier in the time sequence is denoted as the front angle, and the angle between the directions of the two displacement vectors that appear later in the time sequence is denoted as the back angle.

[0058] Taking the calculation of the angle between two displacement vectors that occur first in time sequence as an example: take the direction of the first displacement vector as the starting side and the direction of the second displacement vector as the ending side. Calculate the absolute value of the angle traversed by rotating the starting side counterclockwise to the ending side, and take this angle as the front angle. The value of the angle ranges from 0 degrees to 180 degrees.

[0059] For the first set of comparison objects, since there is no previous set of comparison objects to use as a reference, it does not participate in the determination of the inversion time. Instead, the first set of comparison objects is skipped and the next set of comparison objects is processed.

[0060] For non-first comparison objects, if the product of the difference between the front and back angles in the current comparison object and the difference between the front and back angles in the previous comparison object is less than zero, it indicates that the angle has undergone a reversal change from increasing to decreasing or from decreasing to increasing, that is, the direction of motion has reached the critical point of reversal. At this time, the starting time of the third frame in the four consecutive frames is recorded as the reversal time, and the frame corresponding to the reversal time is recorded as the switching frame.

[0061] If the product of the difference between the front and back angles in the current comparison group and the difference between the front and back angles in the previous comparison group is greater than or equal to zero, it indicates that the angles have not reversed, and the same judgment is made for the next comparison group.

[0062] Then, the earliest chronologically ordered switching frame is used as the starting segmentation point, and the latest chronologically ordered switching frame is used as the ending segmentation point. From the keypoint sequence, the keypoint coordinates corresponding to all frames between any two chronologically adjacent switching frames are extracted as intermediate segments. If there are no other frames between two chronologically adjacent switching frames, no corresponding intermediate segment is generated, and the process continues with the next pair of adjacent switching frames.

[0063] Simultaneously, the coordinates of all key points corresponding to all frames preceding the starting segment are used as the starting sub-segment. If the starting segment is the earliest frame in chronological order, then there are no frames preceding the starting segment, and no starting sub-segment is generated in this case.

[0064] Furthermore, the coordinates of all key points corresponding to all frames after the termination segment are used as the final segment. If the termination segment is the latest frame in the time sequence, then there are no frames after the termination segment, and no final segment is generated in this case.

[0065] If there are no other frames between all adjacent switching frames, all intermediate segments are empty, and neither the starting segment nor the ending segment exists, then it is determined that the subsequent motion chain generation step cannot be executed, the current motion process attitude recognition process ends, and "cannot be recognized" is output as the attitude recognition result.

[0066] If neither the starting segment nor the ending segment exists, but there are intermediate segments, then only all intermediate segments are arranged in chronological order; otherwise, based on the chronological arrangement of all intermediate segments, the starting segment and the ending segment are inserted, with the starting segment placed at the beginning of the sequence and the ending segment placed at the end of the sequence.

[0067] When no reversal moment is obtained after traversing all comparison objects, there is no frame switching. This indicates that the displacement direction of each joint has not reversed during the entire motion process, meaning that there is no divisible stage of directional change in the motion. At this point, the entire joint sequence is directly treated as a complete sub-segment without any segmentation operation. Subsequent kinematic chain and attitude analysis are based on this complete sub-segment.

[0068] Step S3: Combine the joints within each sub-segment into multiple motion chains according to the limb connection relationship, calculate the included angle between adjacent limb segments within each motion chain, and generate an included angle sequence.

[0069] Because the human body has multiple independent limb branches (left and right upper limbs, left and right lower limbs), and the movement patterns of each branch are independent of each other, it is necessary to form corresponding kinetic chains to analyze the movement posture of each limb individually. Therefore, the joints within each sub-segment are combined into multiple kinetic chains according to the limb connection relationship.

[0070] Specifically, based on the inherent, motion-invariant connections between the joints of the human body (common knowledge in the field), i.e., the topology of the human skeleton, all connection records are read from a pre-stored connection mapping table using joint names as indexes. The names of the two joints and the corresponding limb names contained in each connection record are extracted to form limb connection relationships. A partial connection mapping table is shown below, as shown in Table 1.

[0071] Table 1. Human Skeleton Topological Connection Mapping Table

[0072]

[0073] The limb connection relationships include upper limb connection relationships and lower limb connection relationships. Upper limb connection relationships are as follows: the elbow joint connects to the shoulder joint and wrist joint, and the neck joint connects to the shoulder joint and the central trunk joint. Lower limb connection relationships are as follows: the knee joint connects to the hip joint and ankle joint, and the hip joint connects to the central trunk joint.

[0074] Then, for each sub-segment, it is determined whether there are two joints that belong to the same limb and are anatomically directly adjacent. Specifically, if any two joints are not connected to the upper limb or the lower limb, they are determined not to belong to the same limb; otherwise, the connection mapping table is further queried to see if there is a connection record that directly connects the two joints. If it exists, they are determined to be anatomically directly adjacent; otherwise, they are determined to be anatomically not directly adjacent.

[0075] Based on this, two anatomically adjacent joints belonging to the same limb within each sub-segment are connected to form a limb segment. Multiple limb segments sharing the same joint are then sequentially linked end-to-end from the center of the trunk to the extremities, forming multiple kinetic chains. Specifically, starting from the center of the trunk, the shoulder, elbow, and wrist are sequentially linked along the upper limb connection to form an upper limb kinetic chain; and the hip, knee, and ankle are sequentially linked along the lower limb connection to form a lower limb kinetic chain.

[0076] The method for determining the center of the trunk is as follows: add the x-coordinates of the left hip joint and the right hip joint and divide by two to obtain the x-coordinate of the hip center. Similarly, calculate the y-coordinate of the hip center. The hip center is formed by the x-coordinate and y-coordinate of the hip center. Then, add the x-coordinates of the neck joint and the hip center and divide by two to obtain the x-coordinate of the trunk center. Similarly, calculate the y-coordinate of the trunk center. The trunk center is formed by the x-coordinate and y-coordinate of the trunk center.

[0077] Next, for each motion chain, along the path from the center of the torso to the extremities, every two adjacent limb segments are taken as a group of computational objects. The angle between the straight lines containing the two limb segments in each group of computational objects within each frame is calculated, and the frame number of the current frame is recorded. The angle and the frame number are then associated as a recording unit.

[0078] The calculation process for the included angle is as follows: taking the joint point shared by the two limb segments as the vertex, the straight line connecting the other joint point of the first limb segment and the vertex is taken as the first side; the straight line connecting the other joint point of the second limb segment and the vertex is taken as the second side; the angle between the first side and the second side is calculated by dot product operation as the included angle (in degrees). When the angle is greater than 180 degrees, the difference between 360 degrees and the angle is taken as the included angle.

[0079] Then, all recording units of the same motion chain in each frame are arranged in order from the center of the torso to the extremities to generate an angle sequence; each motion chain in each frame corresponds to an angle sequence.

[0080] Please see Figure 3 Step S4: Determine the oscillation period, including the rising segment, falling segment, and peak point, based on the sign of the difference between adjacent angles in the angle sequence.

[0081] Considering that the joint angles of the human body's limbs change periodically over time during movement, in order to provide a time segmentation benchmark for subsequent analysis of the stability and abnormalities of the movement, the swing period is determined based on the sign of the difference between adjacent angles within the angle sequence.

[0082] Specifically, the angles of each motion chain at the same directional position in each frame's angle sequence are extracted in chronological order to obtain the time series of angles at each directional position. Here, the directional position refers to the index of the angle in the angle sequence.

[0083] Then, calculate the difference between the next included angle and the previous adjacent included angle in the time series of the included angles at each direction position to obtain a subsequence; a positive difference indicates that the included angle is increasing, and a negative difference indicates that the included angle is decreasing. Merge the differences at the same time sequence positions in all subsequences in chronological order to obtain a difference sequence. The merging method is as follows: add the k-th difference in each subsequence and divide by the total number of subsequences to obtain the k-th value of the merged difference sequence.

[0084] Next, starting from the beginning of the difference sequence, when the difference changes from positive to negative, it indicates that the angle has changed from continuously increasing to continuously decreasing, that is, the angle has reached a local maximum value. At this time, take the two angles corresponding to the last positive difference before the change of sign, and record the position of the latter of the two selected angles in the corresponding angle time series as the peak point.

[0085] When the difference changes from negative to positive, it indicates that the angle has changed from continuously decreasing to continuously increasing, that is, the angle has reached a local minimum. At this time, take the two angles corresponding to the difference of the last negative sign before the change of sign, and record the position of the latter angle in the corresponding angle time series as the valley point.

[0086] The time series segment of the angle between each peak and the previous adjacent valley is called the rising segment, and the time series segment of the angle between each peak and the next adjacent valley is called the falling segment; the rising segment, the falling segment, and the corresponding valley and peak together constitute an oscillation cycle. The oscillation cycle reflects the complete motion cycle of the joint angle in the kinetic chain from unfolding to closing and then back to unfolding.

[0087] Please see Figure 4 Step S5: Record the number of frames from the start of the rising segment to the peak as the rising frame number; when the difference in the rising frame number between adjacent swing cycles or the difference in the included angle at the peak exceeds their respective tolerances, record the corresponding kinematic chain as the chain to be reviewed.

[0088] To avoid misjudging normal motion fluctuations as abnormal postures and missing motion chains that are truly at risk of instability, the motion chain is reviewed based on the difference in the number of rising frames between adjacent oscillation cycles or the difference in the angle at the peak point.

[0089] In practice, after obtaining the oscillation period including the rising segment, the falling segment, and the corresponding valley and peak points, the frame number corresponding to the beginning of the rising segment is read as the rising segment starting frame number; the frame number corresponding to the peak point is read as the peak point frame number; and the rising segment starting frame number is subtracted from the peak point frame number to obtain the number of rising frames.

[0090] However, when there is no change in sign in the difference sequence, it indicates that the angle is continuously increasing or decreasing, and a complete periodic reciprocating oscillation has not been formed. At this time, the current motion chain has no oscillation period, no chain to be processed is generated, the processing of the current motion chain ends directly, and the next motion chain is processed.

[0091] When the number of oscillation cycles is greater than or equal to two, according to the normal distribution theory, the standard deviation of the rising frame count of all oscillation cycles is multiplied by three to obtain the rising frame count tolerance. The standard deviation of the included angles corresponding to the peaks in all oscillation cycles is multiplied by three to obtain the peak angle tolerance. When the number of oscillation cycles is less than two, there are no adjacent oscillation cycles for comparison, so there is no need to determine the chain to be reviewed; the processing of the current motion chain ends directly, and the next motion chain is processed.

[0092] Then, calculate the absolute value of the difference in the number of rising frames between two adjacent swing cycles, and denot it as the difference in the number of rising frames; calculate the absolute value of the difference in the angles corresponding to the peaks between two adjacent swing cycles, and denot it as the difference in the angles between the peaks.

[0093] When the difference in the number of rising frames is greater than the tolerance for the number of rising frames, or the difference in the angle between peaks is greater than the tolerance for the angle between peaks, it indicates that there is a difference in motion between two adjacent swing cycles, but it is impossible to directly determine whether the anomaly comes from the previous swing cycle or the next swing cycle. Therefore, the motion chains to which the two adjacent swing cycles belong are all recorded as chains to be reviewed.

[0094] When the difference in rising frame counts is less than or equal to the rising frame count tolerance, and the difference in peak angles is less than or equal to the peak angle tolerance, the motion chain to which the two adjacent swing cycles belong is not recorded as the chain to be processed, and the next pair of adjacent swing cycles is processed.

[0095] When no chain to be verified exists after traversing all swing cycles, it is determined that there is no need to determine the unstable joint, the attitude recognition process of this motion process ends, and the attitude is normal is output as the attitude recognition result.

[0096] Step S6: Extract the coordinates of the joints in the frame range corresponding to the chain to be verified in the joint sequence. Based on the cumulative number of times the displacement direction of the same joint is opposite to the corresponding displacement direction in the displacement segment between adjacent frames, determine the attitude instability joints.

[0097] Although the chain to be reviewed indicates that there is an anomaly in the corresponding kinematic chain at the oscillation period level, it does not point out the specific joints that cause the anomaly. Therefore, it is necessary to further identify the instability joints in order to locate the source of the anomaly.

[0098] In practice, the coordinates of the joints in the joint sequence are extracted from the frame range corresponding to the chain to be verified. The reverse cumulative count of each extracted joint is initialized to zero. The coordinates of the currently extracted joint in two adjacent frames are read, and the direction from the coordinate with the earlier time sequence to the coordinate with the later time sequence is taken as the first displacement direction. At the same time, the displacement direction of the currently extracted joint between two adjacent frames is read from the non-chain to be verified in the same time period, and taken as the second displacement direction.

[0099] When the angle between the first displacement direction and the second displacement direction is greater than 90 degrees and less than or equal to 180 degrees, it is determined that the directions are opposite, and the reverse cumulative count of the currently extracted joint point is incremented by one; when the angle between the first displacement direction and the second displacement direction is less than or equal to 90 degrees, it is determined that the directions are not opposite, and the reverse cumulative count of the currently extracted joint point remains unchanged. The reverse cumulative count is maintained independently for each joint point, and each joint point has its own reverse cumulative count.

[0100] After traversing all extracted joints, the joint with the highest back-accumulated count is identified as the unstable joint. The highest back-accumulated count indicates that the motion trajectory of the corresponding joint deviates the most from the expected motion trajectory. If multiple joints have the same highest back-accumulated count, these joints are all identified as unstable joints.

[0101] After identifying the unstable joints, output their names and corresponding coordinates as the attitude recognition result. Once the output is complete, the attitude recognition process for this motion is terminated.

[0102] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product.

[0103] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0104] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0106] Finally, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent posture recognition during motion processes based on image recognition, characterized in that, Includes the following steps: The coordinates of human joints in the motion video are located frame by frame to obtain the joint sequence; the displacement vector of the same joint between adjacent frames is calculated and displacement segments are generated. Starting from the beginning of the displacement segment, three displacement vectors are taken sequentially along the time sequence as a group of comparison objects. The directional angle between two adjacent displacement vectors in each group of comparison objects is calculated to obtain the front angle and the back angle. Based on the difference between the front and back angles between adjacent comparison objects, the reversal moment is identified from the consecutive frames corresponding to the comparison objects, and the frame corresponding to the reversal moment is recorded as the switching frame; the key point sequence is divided into multiple sub-segments using the switching frame; Based on the limb connection relationship, the joints in each sub-segment are combined into multiple kinematic chains, the included angles between adjacent limb segments in each kinematic chain are calculated, and the included angle sequence is generated. The oscillation period, including the rising segment, falling segment, and peak point, is determined based on the sign of the difference between adjacent angles in the angle sequence. The number of frames from the start of the rising segment to the peak is recorded as the rising frame number; when the difference in the rising frame number between adjacent swing cycles or the difference in the angle at the peak exceeds their respective tolerances, the corresponding kinematic chain is recorded as the chain to be reviewed. Extract the coordinates of the joints in the frame range corresponding to the chain to be verified in the joint sequence. Based on the cumulative number of times the displacement direction of the same joint is opposite to the corresponding displacement direction in the displacement segment between adjacent frames, determine the attitude instability joints.

2. The intelligent motion process posture recognition method based on image recognition according to claim 1, characterized in that, The process of obtaining the key point sequence is as follows: Read motion video frame by frame, divide the upright human body area into two halves with the vertical center line of the current frame, and scan line by line from the bottom of the current frame in each half, calculating the difference between adjacent pixel values ​​in each line. The left and right boundary points are determined based on the comparison between the absolute value of the difference and the fluctuation amplitude of the background pixels in the current frame, as well as the sign of the difference. Form boundary point pairs by combining the left and right boundary points in the same row, and connect the boundary point pairs located in the same half area in row order to obtain the outer edge lines of the left and right limbs. Calculate the change in the inclination angle of the line connecting two adjacent points on the outer edge of each limb. When the change in inclination angle remains unchanged in sign and non-zero over three consecutive points, the position of the middle point is recorded as the turning point of the limb. For the first frame, the joint coordinates are determined based on the spatial distribution density of the limb turning points; for subsequent frames, candidate points are determined based on the spatial distance between the limb turning point and the joint coordinates in the previous frame. The line segments between adjacent candidate points on the outer edge of the same limb are taken as limb segments; the joint coordinates are determined based on the length ratio of the limb segment in the current frame to that in the previous frame, combined with the range of background pixel span ratio in the current frame; the joint coordinates obtained in each frame are arranged in chronological order to generate a joint sequence.

3. The intelligent motion posture recognition method based on image recognition according to claim 1, characterized in that, The process of generating the displacement segment is as follows: Based on the coordinates of the same joint in two adjacent frames in the joint sequence, the direction from the coordinate with the earlier time sequence to the coordinate with the later time sequence is taken as the displacement direction; the straight-line distance between the two coordinates is taken as the displacement distance, and the displacement direction and displacement distance are combined into a displacement vector. All displacement vectors obtained between adjacent frames for the same joint in the joint sequence are arranged sequentially in chronological order and connected in a broken line manner to generate displacement segments.

4. The intelligent motion process posture recognition method based on image recognition according to claim 3, characterized in that, The process of identifying the reversal moment is as follows: Each group of comparison objects corresponds to four consecutive frames, and the first two displacement vectors of the last three displacement vectors are shared among the comparison objects. For each pair of comparison objects, the angle between the directions of the two displacement vectors that are earlier in the time sequence is denoted as the front angle, and the angle between the directions of the two displacement vectors that are later in the time sequence is denoted as the back angle. If the product of the difference between the front and back angles in the current comparison group and the difference between the front and back angles in the previous comparison group is less than zero, then the start time of the third frame out of the four consecutive frames is recorded as the reversal time.

5. The intelligent motion process posture recognition method based on image recognition according to claim 1, characterized in that, The process of dividing the keypoint sequence into multiple segments using switching frames is as follows: The earliest switching frame in the time sequence is used as the starting split point, and the latest switching frame in the time sequence is used as the ending split point. From the key point sequence, extract the key point coordinates corresponding to all frames between every two temporally adjacent switching frames, and use them as the middle segment; The keypoint coordinates of all frames before the starting segmentation point are taken as the starting sub-segment, and the keypoint coordinates of all frames after the ending segmentation point are taken as the ending sub-segment. All intermediate segments are arranged in chronological order between the starting segment and the ending segment.

6. The intelligent motion process posture recognition method based on image recognition according to claim 1, characterized in that, The process of obtaining the kinetic chain is as follows: Based on the topology of the human skeleton, the limb connection relationships between various joints are determined, including upper limb connection relationships and lower limb connection relationships; Based on the limb connection relationship, two joints that belong to the same limb and are anatomically directly adjacent in each sub-segment are connected to form a limb segment. Multiple limb segments that share the same joint are connected in series from the center of the trunk to the ends of the limbs to form multiple kinetic chains.

7. The intelligent motion posture recognition method based on image recognition according to claim 6, characterized in that, The process of generating the included angle sequence is as follows: For each kinetic chain, along the direction from the center of the trunk to the extremities, every two adjacent limb segments are taken as a group of calculation objects. Calculate the angle between the straight lines containing the two limb segments in each group of calculation objects within each frame, and record the frame number of the current frame. Associate the angle with the frame number as a recording unit. All recording units of the same motion chain within each frame are arranged in order from the center of the torso to the extremities to generate an angle sequence.

8. The intelligent motion process posture recognition method based on image recognition according to claim 7, characterized in that, The process of determining the oscillation period is as follows: The angles of each motion chain at the same direction position in each frame angle sequence are extracted in chronological order to obtain the time sequence of the angles at each direction position; Calculate the difference between two adjacent angles in each time series to obtain a subsequence; merge the differences at the same time sequence position in all subsequences in chronological order to obtain the difference sequence. Starting from the beginning of the difference sequence, when the difference changes from positive to negative, take the two angles corresponding to the last positive difference before the change, and record the position of the latter of the two angles in the corresponding angle time series as the peak point. When the difference changes from negative to positive, take the two angles corresponding to the last negative difference before the change, and record the position of the latter angle in the time series of the corresponding angle as the valley point. The time series segment of the angle between each peak and the previous adjacent valley is called the rising segment, and the time series segment of the angle between each peak and the next adjacent valley is called the falling segment; the rising segment, the falling segment, and the corresponding valley and peak together constitute an oscillation cycle.

9. The intelligent motion process posture recognition method based on image recognition according to claim 8, characterized in that, The process for determining the chain of checks is as follows: Read the frame number corresponding to the beginning of the rising segment as the starting frame number of the rising segment; read the frame number corresponding to the peak point as the peak point frame number; subtract the starting frame number of the rising segment from the peak point frame number to obtain the number of rising frames. Calculate the absolute value of the difference in the number of rising frames between two adjacent swing cycles, and denot it as the rising frame difference; calculate the absolute value of the difference in the angles corresponding to the peaks between two adjacent swing cycles, and denot it as the peak angle difference. When the difference in the number of rising frames or the difference in the angle between peaks exceeds their respective tolerances, the motion chains belonging to the two adjacent swing cycles are both recorded as chains to be verified.

10. The intelligent motion process posture recognition method based on image recognition according to claim 1, characterized in that, The process of identifying the unstable joint is as follows: Extract the coordinates of the joints in the frame range corresponding to the chain to be verified in the joint sequence, read the coordinates of the currently extracted joint in two adjacent frames, and take the direction from the coordinate with the earlier time sequence to the coordinate with the later time sequence as the first displacement direction; at the same time, read the displacement direction of the currently extracted joint between two adjacent frames from the non-chain to be verified in the same time period as the second displacement direction. Initialize the reverse cumulative count for each extracted joint to zero; When the first displacement direction and the second displacement direction are opposite, increment the reverse cumulative count of the currently extracted joint by one; after traversing all extracted joints, determine the joint with the highest reverse cumulative count as the attitude instability joint.

Images