Astronaut posture recognition method based on target detection

Abstract

An astronaut posture recognition method based on target detection comprises the steps that firstly, a detection model is trained through astronaut influence, feature vectors are calculated based on the features of a space suit, and a target detection result is obtained through estimation of a loss function; and inputting the output detection result into a key point identification model to obtain 2D skeleton data of the astronaut, and performing posture identification according to the data to realize identification of the action of the astronaut. According to the method, image linear transformation and loss function calculation are improved by using the color features of the space suit, the recognition precision and robustness in application are greatly improved, and the method has higher generalization ability.

Description

Technical Field

[0001] The present invention relates to a method for recognizing the posture of an astronaut based on object detection, belonging to the technical field of astronaut operation design. Background Art

[0002] As one of the fastest developing directions in the current intelligent application field, human object detection has realized a variety of human detection applications for outdoor monitoring; while the imaging conditions and environment in the scenario of an astronaut's activities on the surface of a celestial body are relatively complex, and the current widely used YOLO model has a poor detection effect on spacesuits lacking details; and because of less data, the robustness of the detection application after retraining is insufficient; it is necessary to give feedback and interaction according to the actions of the astronaut, so it is also necessary to recognize the posture of the astronaut on the basis of human object detection, but there is a lack of a method in the prior art that can realize the recognition of the motion state and posture of the astronaut. Summary of the Invention

[0003] The technical problem solved by the present invention is: aiming at the blank of the technology for recognizing the motion state and posture of an astronaut in the current prior art, a method for recognizing the posture of an astronaut based on object detection is proposed.

[0004] The present invention solves the above technical problems through the following technical solutions:

[0005] A method for recognizing the posture of an astronaut based on object detection includes:

[0006] Shoot and collect the current scene image data, perform block processing, perform linear projection on the divided scene image blocks, and calculate the mean square deviation value of each scene image block;

[0007] According to the linear projection result, each scene image block is extended until it has the same length as the linear projection result, and after the arrangement of each scene image block, it is connected to the linear projection result to generate a new vector;

[0008] Input the new vector into the SSM structure, and frame the image area where the astronaut is located through convolution processing in the SSM structure;

[0009] Perform human posture recognition and detection on the image data after convolution processing;

[0010] Make a judgment according to the recognition result, and determine the spatial relationship between the key point positions in the recognition result;

[0011] Judge the state of the astronaut according to the spatial relationship between the key point positions, and obtain the posture state of the astronaut at the current moment according to the judgment result.

[0012] The method for performing block processing on the scene image data is:

[0013] Perform an equal division and segmentation operation on the scene image data in the clockwise direction. Segment the input image data in the clockwise direction, and divide the N×N pixel image into hei / N*wid / N regions;

[0014] The method for performing a linear projection on the segmented scene image blocks is as follows:

[0015] Perform a linear projection on the scene image blocks within each region to obtain a linear projection result A;

[0016] The method for calculating the mean square error value of each image block is as follows:

[0017] Var(i) = 1 / N 2 *∑f(x,y)-mean(f) 2

[0018] In the formula, N is the number of pixels in the image block, f(x,y) is the pixel value at the coordinates (x,y), mean is the mean calculation, and i represents the current image block.

[0019] The method for obtaining a new vector is as follows:

[0020] After each scene image block is extended to the same length as the linear projection result A, arrange the extended scene image blocks in the order of the segmentation process to obtain a one-dimensional vector. At the same time, arrange the mean square error of each scene image block in a one-dimensional vector in the clockwise direction. Expand the one-dimensional vector by convolution and connect it with the linear projection result A respectively to obtain a new vector.

[0021] The SSM module processes the new vector based on color features. The processing method is as follows:

[0022] Linearly expand the obtained new vector into an N*N*512-dimensional vector, obtain an image feature block through depth convolution processing, and divide the image feature block into an N*N*512-dimensional vector;

[0023] Obtain a 1*512-dimensional vector through average pooling processing, perform linear compression processing on the obtained vector to obtain a 1*256-dimensional vector, and output the specified area as the framed result of the image area where the astronaut is located after processing through the MLP function and the SiLU function.

[0024] The method for performing human pose recognition and detection on the image data after convolution processing is as follows:

[0025] Input the image data after convolution processing into the Mediapipe human pose recognition model to obtain a recognition result. If there is key point information in the recognition result, the human pose is recognized; if there is no key point information in the recognition result, the human pose is not recognized, and the scene image data of the next frame is collected.

[0026] The method for determining the spatial relationship between key point positions in the recognition result is as follows:

[0027] According to the astronaut's body envelope, label the key point information existing in the recognition result;

[0028] Determine the number of key points on the main trunk of the astronaut's body envelope;

[0029] Calculate the included angle between the connecting lines of points a - b - c;

[0030] Calculate the included angle between the connecting lines of points a + 1 - b + 1 - c + 1;

[0031] Determine the difference range of the two included angle calculation results, and judge the astronaut's state according to the difference range.

[0032] Among them, points a, b, and c are key points on the main trunk, a + 1, b + 1, and c + 1 are the next adjacent points, and the trajectory formed by the key points conforms to the characteristics of the main trunk of the astronaut's body envelope.

[0033] The astronaut's state includes the standing state and the arm state. The method for judging the astronaut's state according to the difference range is as follows:

[0034] If the two difference ranges are within the range of 150° - 180°, then judge that the astronaut is in the standing state; if not, output an abnormal state; if the astronaut is in the standing state, then judge the arm state of the astronaut, and the arm state includes the left - hand state and the right - hand state.

[0035] The method for judging the right - hand state of the astronaut is as follows:

[0036] When the astronaut is in the standing state, calculate the included angle between the connecting lines of d - a and a - b - c. If it is within the range of 0° - 30°, then the right - hand state is natural drop; if it is within the range of 120° - 180°, then the right - hand state is raised; the range of 30° - 120° is not judged. Among them, the selection principle of point d is the same as that of points a, b, and c.

[0037] The method for judging the left - hand state of the astronaut is as follows:

[0038] When the astronaut is in the standing state, calculate the included angle between the connecting lines of d + 1 - a + 1 and a + 1 - b + 1 - c + 1. If it is within the range of 0° - 30°, then the left - hand state is natural drop; if it is within the range of 120° - 180°, then the left - hand state is raised; the range of 30° - 120° is not judged.

[0039] After determining the astronaut's state, calculate the eigenvector of the human joint included angle according to the arm state of the astronaut, and judge the typical posture of the astronaut, including standing with hands down or standing with right hand raised or standing with left hand raised. The judgment method is as follows:

[0040] Calculate the eigenvector of the human joint angle, H = ||f_stand, f_lh, f_rh||

[0041] Wherein, f_stand is the angle of the torso joint, f_lh is the angle value corresponding to the right hand state of the astronaut, and f_rh is the angle value corresponding to the left hand state of the astronaut;

[0042] Determine the typical posture of the astronaut according to the typical posture angle. The calculation method of the typical posture angle is as follows:

[0043]

[0044] Wherein, Hc is the joint eigenvector obtained from the current image, Hj is the joint eigenvector of the typical posture, and the corresponding typical posture of the astronaut is determined according to the relationship between θ c and the corresponding preset value.

[0045] The advantages of the present invention compared with the prior art are as follows:

[0046] An astronaut posture recognition method based on object detection provided by the present invention uses the Mamba architecture as the basic recognition network, improves the model for the application background of astronaut activities to obtain a robust astronaut object detection result, and then connects the Mediapipe model to improve the accuracy of posture recognition. It can use a camera to perform real-time recognition of the astronaut's posture state without additional markings on the spacesuit, and is robust at different relative distances and different shooting angles. By combining the VIM model and the Mediapipe model, the posture is identified on the basis of object detection, thereby judging the current state of the astronaut. The recognition method is novel, and the recognition accuracy is achieved by judging the angle between key points and is very high. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the astronaut posture recognition method provided by the present invention;

[0048] Figure 2 It is a schematic diagram of the block operation processing provided by the present invention;

[0049] Figure 3 It is a schematic diagram of the prediction target process provided by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0050] A method for identifying the posture of astronauts based on object detection. First, use the images of astronauts to train the detection model, calculate the feature vectors based on the characteristics of the spacesuits, and use the loss function to estimate the object detection results. Input the output detection results into the key point recognition model to obtain the 2D skeleton data of the astronauts, and based on this data, perform posture recognition to achieve the identification of the actions of the astronauts. The present invention uses the color features of the spacesuits to improve the image linear transformation and loss function calculation, greatly improving the recognition accuracy and robustness in applications and having a stronger generalization ability.

[0051] The method for identifying the posture of astronauts based on object detection, the overall steps include:

[0052] Shoot and collect the image data of the current scene, and perform block processing. Perform linear projection on the divided scene image blocks and calculate the mean square error value of each scene image block;

[0053] According to the linear projection results, expand each scene image block until it has the same length as the linear projection results. After arranging the scene image blocks, connect them with the linear projection results to generate a new vector;

[0054] Input the new vector into the SSM structure and use the SiLU function for marking;

[0055] Perform convolution processing on the marked new vector to obtain the image data of the framed astronaut position;

[0056] Perform human posture recognition and detection on the image data after convolution processing;

[0057] Make a judgment according to the recognition results to determine the spatial relationship between the key point positions in the recognition results;

[0058] Judge the state of the astronaut according to the spatial relationship between the key point positions, and obtain the posture state of the astronaut at the current moment according to the judgment result.

[0059] The method for performing block processing on the scene image data is:

[0060] Perform equal division block operation on the scene image data in the clockwise direction. Perform block processing on the input image data in the clockwise direction and divide it into multiple regions according to the N×N pixel image;

[0061] The method for performing linear projection on the divided scene image blocks is:

[0062] Perform linear projection on the scene image blocks in each region to obtain the linear projection result A;

[0063] The method for calculating the mean square error value of each image block is:

[0064] Var(i) = 1 / N 2*∑f(x,y) - mean(f) 2

[0065] Where N is the number of pixels in the image block, f(x,y) is the pixel value at coordinates (x,y), mean is the mean calculation, and i represents the current image block.

[0066] The method for obtaining the new vector is as follows:

[0067] After each scene image block is extended to the same length as the linear projection result A, the extended scene image blocks are arranged in the order of block processing to obtain a one-dimensional vector. At the same time, the mean square error of each scene image block is arranged in a clockwise direction into a one-dimensional vector, and they are respectively connected to the linear projection result A to obtain two groups of new vectors.

[0068] The method for performing human pose recognition and detection on the image data after convolution processing is as follows:

[0069] Input the image data after convolution processing into the Mediapipe human pose recognition model to obtain the recognition result. If there is key point information in the recognition result, the human pose is recognized; if there is no key point information in the recognition result, the human pose is not recognized, and the scene image data of the next frame is collected.

[0070] The method for determining the spatial relationship between key point positions in the recognition result is as follows:

[0071] According to the astronaut's human envelope, label the key point information existing in the recognition result;

[0072] Determine the number of key points on the main torso of the astronaut's human envelope;

[0073] Calculate the angle between the lines connecting points a - b - c;

[0074] Calculate the angle between the lines connecting points a + 1 - b + 1 - c + 1;

[0075] Determine the difference range of the two angle calculation results, and judge the astronaut's state according to the difference range.

[0076] The method for determining the spatial relationship between key point positions in the recognition result is as follows:

[0077] According to the astronaut's human envelope, label the key point information existing in the recognition result;

[0078] Determine the number of key points on the main torso of the astronaut's human envelope;

[0079] Calculate the angle between the lines connecting points a - b - c;

[0080] Calculate the angle between the lines connecting points a + 1 - b + 1 - c + 1;

[0081] Determine the difference range of the calculation results of the two included angles, and judge the state of the astronaut according to the difference range.

[0082] The method for judging the state of the astronaut's right hand is as follows:

[0083] When the astronaut is in a standing state, calculate the included angle between d-a and the connecting line between points a-b-c. If it is within the range of 0°-30°, the right hand state is natural drop; if it is within the range of 120°-180°, the right hand state is raised; there is no range within 30°-120°.

[0084] The method for judging the state of the astronaut's left hand is as follows:

[0085] When the astronaut is in a standing state, calculate the included angle between d+1-a+1 and the connecting line between points a+1-b+1-c+1. If it is within the range of 0°-30°, the left hand state is natural drop; if it is within the range of 120°-180°, the left hand state is raised; there is no range within 30°-120°.

[0086] Construct a feature vector from the included angles of the human joints obtained through the above calculations:

[0087] H = ||f_stand, f_lh, f_rh||

[0088] Where f_stand is the included angle of the torso joint, f_lh is the included angle calculated in step 8, and f_rh is the included angle calculated in step 9;

[0089] The typical postures to be concerned about are standing with hands hanging, standing with right hand raised, and standing with left hand raised. The current posture is judged by calculating the included angle:

[0090]

[0091] Where Hc is the joint feature vector obtained from the current image, and Hj is the joint feature vector of the typical posture;

[0092] Judge θ c to determine which typical posture it belongs to by whether its value is less than 0.5.

[0093] The method for recognizing the posture of an astronaut based on object detection is implemented through the following system structure design:

[0094] The astronaut posture recognition system based on object detection includes a backbone network, a detection head, a target box, and a key point detection module. The working process is as follows: Obtain publicly available spacesuit picture data, perform block processing on images taken at different distances, input the images into the backbone network part, output the feature vectors, use the feature vectors as the input of the detection operator, judge to obtain the outer envelope box of the target, and then input the envelope box data into the human posture recognition detection operator to detect the astronaut in the image and discriminate the posture; Train and test the model based on publicly collected spacesuit photos to obtain a trained posture recognition model; The backbone network includes a downsampling module and an SSM module based on color features;

[0095] Among them, the SSM module based on color features includes: a depth convolutional layer, an SSM module, an average pooling layer, linear expansion, and compression processing; Linearly expand the obtained new vector into an N*N*512-dimensional vector, then obtain a new image feature block through the depth convolutional layer, then input it into the SSM layer to obtain an N*N*512-dimensional vector of the same size, then pass through the average pooling layer to obtain a 1*512-dimensional vector, and finally perform linear compression processing to a 1*256-dimensional vector. Finally, the detection result is output after passing through the MLP function and the SiLU layer.

[0096] The following is further described in conjunction with the specification drawings and preferred embodiments:

[0097] In the current embodiment, as Figure 1 、 Figure 3 shown, the method for recognizing the posture of an astronaut based on object detection includes the following steps:

[0098] Step 1: Use a camera to take a current scene image, perform block processing on the current scene image, perform block operations on the input image in a clockwise direction, and divide the N×N pixel image into 9 regions:

[0099] Perform linear projection on the image block, and each block is expanded into one-dimensional data 1D data A, which is (N / 3×N / 3,1); As Figure 2 shown.

[0100] And at the same time calculate the mean square deviation value on the image block;

[0101] Expand each block of data to the same length as A, and arrange and connect them in the block order to obtain a new set of vectors;

[0102] Step 2: Calculate the vectors respectively, input the vectors after linear transformation of the image block into the SSM structure, and use SiLU for marking at the same time;

[0103] Step 3: Perform convolution processing on the two sets of vectors to obtain the detection result of the astronaut;

[0104] Step 4: Input the rectangular image of the detection result into the Mediapipe human pose recognition model to obtain the calculated key point results;

[0105] Step 5: Judge the output key point results:

[0106] Judge the spatial position relationship between the key points: First, calculate whether the included angles between the 11th - 23rd - 27th points and the 12th - 24th - 28th points on the main torso are close to 180°; if the calculation results of both are within the range of 150° - 180°, it is judged that the astronaut is in a standing state; if not, output an abnormal state; if it is in a standing state, continue to calculate the included angle between the 15th point - the 11th point and the 11th - 23rd - 27th points. If it is within the range of 0° - 30°, the right hand naturally drops; if it is within the range of 120° - 180°, the right hand is raised; similarly, calculate the key points on the left side; obtain the posture state of the astronaut.

[0107] In Step 1, the input image data is segmented in a clockwise direction, and the mean square error of each block is arranged into a one - dimensional vector in a clockwise direction;

[0108] In Step 5, judge the posture of the astronaut according to the included angle relationship between the key points; if the included angles between the 11th - 23rd - 27th points and the 12th - 24th - 28th points on the main torso are close to 180°; if the calculation results of both are within the range of 150° - 180°, it is judged that the astronaut is in a standing state; if not, output an abnormal state; if it is in a standing state, continue to calculate the included angle between the 15th point - the 11th point and the 11th - 23rd - 27th points. If it is within the range of 0° - 30°, the right hand naturally drops; if it is within the range of 120° - 180°, the right hand is raised; if it is in a standing state, continue to calculate the included angle between the 16th point - the 12th point and the 12th - 23rd - 28th points. If it is within the range of 0° - 30°, the left hand naturally drops; if it is within the range of 120° - 180°, the left hand is raised.

[0109] The purpose of this embodiment is to fully consider the situation that missing astronaut features may lead to missed detections during the target detection process. Without additional markings on the spacesuit, a camera can be used to perform real - time recognition of the astronaut's posture state, and it has robustness at different relative distances and different shooting angles.

[0110] Although the present invention has been disclosed above in preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and decorations made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

[0111] The content not described in detail in the specification of the present invention belongs to the well-known technology of those skilled in the art.

Claims

1. An astronaut attitude recognition method based on object detection, characterized in that Including: Capture and collect the image data of the current scene, perform block processing, perform linear projection on the divided scene image blocks, and calculate the mean square error value of each scene image block; Expand each scene image block according to the linear projection result until it has the same length as the linear projection result. After arranging the scene image blocks, connect them with the linear projection result to generate a new vector; Input the new vector into the SSM structure, and frame the image area where the astronaut is located through convolution processing in the SSM structure; Perform human pose recognition and detection on the image data after convolution processing; Make a judgment according to the recognition result, and determine the spatial relationship between the key point positions in the recognition result; Judge the astronaut's state according to the spatial relationship between the key point positions, and obtain the astronaut's pose state at the current moment according to the judgment result.

2. The method for recognizing an astronaut's pose based on object detection according to claim 1, characterized in that: The method for performing block processing on the scene image data is: Perform equal division block operation on the scene image data in the clockwise direction. Divide the input image data in the clockwise direction, and divide the N×N pixel image into hei / N*wid / N regions; The method for performing linear projection on the divided scene image blocks is: Perform linear projection on the scene image blocks in each region to obtain the linear projection result A; The method for calculating the mean square error value of each image block is: Var(i) = 1 / N 2 *∑f(x,y) - mean(f) 2 Where N is the number of pixels in the image block, f(x, y) is the pixel value at the coordinate (x, y), mean is the mean calculation, and i represents the current image block.

3. The method for recognizing an astronaut's pose based on object detection according to claim 2, characterized in that: The method for obtaining the new vector is: After each scene image block is expanded to the same length as the linear projection result A, arrange the expanded scene image blocks in the order of block processing to obtain a one-dimensional vector. At the same time, arrange the mean square error of each scene image block in the clockwise direction into a one-dimensional vector, expand the one-dimensional vector by convolution, and connect it with the linear projection result A respectively to obtain a new vector.

4. The method for recognizing an astronaut's pose based on object detection according to claim 2, characterized in that: The SSM module processes the new vector based on color features, and the processing method is: Linearly expand the obtained new vector into an N*N*512-dimensional vector, obtain the image feature block through depth convolution processing, and divide the image feature block into an N*N*512-dimensional vector; Obtain a 1*512-dimensional vector through average pooling processing, perform linear compression processing on the obtained vector to obtain a 1*256-dimensional vector, and output the specified area as the framing result of the image area where the astronaut is located after processing through the MLP function and the SiLU function.

5. The method for recognizing an astronaut's pose based on object detection according to claim 2, characterized in that: The method for performing human pose recognition and detection on the image data after convolution processing is: Input the image data after convolution processing into the Mediapipe human pose recognition model to obtain the recognition result. If there is key point information in the recognition result, the human pose is recognized. If there is no key point information in the recognition result, the human body posture is not recognized, and the scene image data of the next frame is collected.

6. A method for recognizing an astronaut's posture based on object detection according to claim 2, characterized in that: The method for determining the spatial relationship between key point positions in the recognition result is: According to the astronaut's body envelope, label the key point information existing in the recognition result; Determine the number of key points on the main trunk of the astronaut's body envelope; Calculate the included angle between the connecting lines at points a-b-c; Calculate the included angle between the connecting lines at points a+1-b+1-c+1; Determine the difference range of the two included angle calculation results, and judge the astronaut's state according to the difference range; Among them, points a, b, and c are key points on the main trunk, and a+1, b+1, and c+1 are the next adjacent points. The trajectory formed by the key points conforms to the characteristics of the main trunk of the astronaut's body envelope.

7. A method for recognizing an astronaut's posture based on object detection according to claim 6, characterized in that: The astronaut's state includes the standing state and the arm state. The method for judging the astronaut's state according to the difference range is: If the two difference ranges are within the range of 150° to 180°, it is judged that the astronaut is in the standing state; if not, an abnormal state is output; if the astronaut is in the standing state, the arm state of the astronaut is judged, and the arm state includes the left hand state and the right hand state.

8. A method for recognizing an astronaut's posture based on object detection according to claim 7, characterized in that: The method for judging the right hand state of the astronaut is: When the astronaut is in the standing state, calculate the included angle between d-a and the connecting lines at points a-b-c. If it is within the range of 0°-30°, the right hand state is natural drop; if it is within the range of 120°-180°, the right hand state is raised; the range of 30°-120° is not judged; Among them, the selection principle of point d is the same as that of points a, b, and c.

9. A method for recognizing an astronaut's posture based on object detection according to claim 8, characterized in that: The method for judging the left hand state of the astronaut is: When the astronaut is in the standing state, calculate the included angle between d+1-a+1 and the connecting lines at points a+1-b+1-c+1. If it is within the range of 0°-30°, the left hand state is natural drop; if it is within the range of 120°-180°, the left hand state is raised; the range of 30°-120° is not judged.

10. A method for recognizing an astronaut's posture based on object detection according to claim 7, characterized in that: After determining the astronaut's state, calculate the feature vector of the human body joint angle according to the arm state of the astronaut, and judge the typical posture of the astronaut, including standing with hands down or standing with right hand raised or standing with left hand raised. The judgment method is: Calculate the feature vector of the human body joint angle, H = ||f_stand, f_lh, f_rh|| In the formula, f_stand is the included angle of the trunk joint, f_lh is the included angle value corresponding to the right hand state of the astronaut, and f_rh is the included angle value corresponding to the left hand state of the astronaut; Determine the typical posture of the astronaut according to the typical posture included angle. The calculation method of the typical posture included angle is: Where, Hc is the joint feature vector obtained from the current image, and Hj is the joint feature vector of the typical pose. According to the relationship between θ c and the corresponding preset value, the corresponding typical pose of the astronaut is determined.

Images