Bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion

Abstract

The present invention relates to the field of image processing technology and discloses a method for detecting the posture of a bipedal humanoid robot based on temporal multi-scale feature fusion. The method comprises the following steps: obtaining an original image of a target bipedal humanoid robot and extracting multiple feature maps of different scales; iteratively performing temporal multi-scale feature fusion based on the feature map of the current frame at the current scale, the fused feature map of the previous scale of the current frame, and the feature map of the previous scale of the current frame to obtain multiple fused feature maps of the current frame; processing the fused feature map of the minimum scale and the fused feature maps of other scales based on the multiple fused feature maps of the current frame using a preset attention module and a preset skip connection module to obtain an encoding feature of the current frame and multiple skip connection features; and decoding the encoding feature and the multiple skip connection features to obtain an output feature map of the current frame. The present invention improves the accuracy and recall rate of bipedal humanoid robot posture detection through temporal multi-scale feature fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion. Background Art

[0002] Bipedal humanoid robot posture detection involves identifying key points in images captured by cameras using various detection methods. Currently, two common approaches are employed: the first involves directly applying established human posture key point detection methods to bipedal humanoid robot posture detection. However, this approach suffers from the significant differences in shape and size between the human body and bipedal humanoid robots, resulting in low accuracy and recall rates when methods designed for the human body are directly applied to bipedal humanoid robot posture detection. The second approach involves designing specialized network structures based on publicly available bipedal humanoid robot key point datasets to achieve key point detection for bipedal humanoid robots. However, this approach suffers from low accuracy due to its relatively simple network structure.

[0003] The problem with the current method for bipedal humanoid robot posture detection is that the model network structure is relatively simple and does not fully exploit the information in the bipedal humanoid robot image, resulting in low detection accuracy and recall rate. Summary of the Invention

[0004] In view of this, the present invention provides a bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion to solve the problems of low accuracy and recall rate of existing detection methods.

[0005] In a first aspect, the present invention provides a method for posture detection of a bipedal humanoid robot based on temporal multi-scale feature fusion, the method comprising:

[0006] Continuously acquire multiple frames of original images of the target bipedal humanoid robot, and preprocess each original image to extract multiple feature maps of different scales;

[0007] Iteratively perform temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of the current frame;

[0008] Sort the multiple fused feature maps of the current frame by scale to obtain the fused feature map of the smallest scale and multiple fused feature maps of other scales;

[0009] Use the preset attention module to process the minimum-scale fusion feature map to obtain the encoding features of the current frame;

[0010] Use the preset skip connection module to process the fusion feature maps of other scales respectively to obtain multiple skip connection features of the current frame;

[0011] According to the encoding features and multiple skip connection features, decoding is performed to obtain the output feature map of the current frame.

[0012] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided by the present invention increases the robustness of robot recognition at different positions and sizes through temporal multi-scale feature fusion, adds an attention mechanism between encoding and decoding, improves the feature decoding capability of the decoder, utilizes the decoder to enhance the ability to process spatial features, and improves the accuracy and recall rate of bipedal humanoid robot posture detection.

[0013] In an optional embodiment, each original image is preprocessed and then a plurality of feature maps of different scales are extracted, including:

[0014] Normalize the original image to obtain a standard image suitable for deep network input;

[0015] Perform feature extraction of different scales on the standard image to obtain multiple feature maps of different scales.

[0016] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided by the present invention facilitates subsequent deep network processing through normalization processing. By extracting features of different scales, multiple features of the original image are understood from different levels, which is conducive to improving the accuracy of subsequent bipedal humanoid robot posture detection.

[0017] In an optional embodiment, temporal multi-scale feature fusion is iteratively performed based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of the current frame, including:

[0018] The feature map of the minimum scale of the current frame is fused with the feature map of the minimum scale of the previous frame to obtain the fused feature map of the minimum scale of the current frame;

[0019] The fused feature map of the minimum scale of the current frame is used as the fused feature map of the previous scale of the current frame in the first temporal multi-scale feature fusion process. The temporal multi-scale feature fusion is repeated based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of different scales in the current frame.

[0020] In an optional embodiment, performing temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame includes:

[0021] Performing a first preset convolution process on the feature map of the current frame at the current scale to obtain a first feature map, performing a preset upsampling process on the fused feature map of the previous scale of the current frame to obtain a second feature map, and performing a second preset convolution process on the feature map of the previous frame at the current scale to obtain a third feature map;

[0022] Adding the first feature map and the second feature map and then concatenating them with the third feature map to obtain a concatenated feature map;

[0023] Perform depth-wise separable convolution on the spliced ​​feature map to obtain a fused feature map of the current frame and current scale.

[0024] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided by the present invention enhances the robustness of feature expression by fusing the same-scale features of adjacent temporal frames and the different-scale features of the same temporal frame. Deep separable convolution processing can significantly reduce the model parameters and calculation amount after the features are spliced ​​in the channel dimension, thereby improving the model processing speed.

[0025] In an optional embodiment, a preset skip connection module is used to process fusion feature maps of other scales respectively to obtain multiple skip connection features of the current frame, including:

[0026] The fused feature maps of other scales are respectively input into the preset skip connection module, and the first preset convolution processing is performed to obtain the skip connection features of the scale corresponding to the current frame.

[0027] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided by the present invention directly transmits low-level features (such as edges, textures, etc.) in the network to higher-level feature maps by performing jump connections on fused features of other scales, which is beneficial to maintaining spatial information and improving the accuracy of target detection.

[0028] In an optional embodiment, decoding is performed based on the encoding features and multiple skip connection features to obtain an output feature map of the current frame, including:

[0029] Sort multiple skip connection features from small to large scale to obtain a skip connection feature sequence;

[0030] According to the coding features and the skip connection feature sequence, the preset asymmetric convolution processing is iteratively performed to obtain the output coding features of the current frame;

[0031] A second preset convolution process is performed on the output coding features of the current frame to obtain an output feature map of the current frame.

[0032] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided by the present invention uses jump connections and feature addition to achieve more powerful learning capabilities without significantly increasing the number of parameters. Since the amount of data required to be processed is reduced, the computational cost can be reduced and the reasoning speed can be accelerated.

[0033] In an optional embodiment, the method further includes:

[0034] Extract multiple key points and skeleton of the target bipedal humanoid robot based on the output feature map of the current frame;

[0035] The output features are visualized according to the key points and skeleton to obtain the heat map of the target bipedal humanoid robot.

[0036] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided by the present invention facilitates intuitive observation of the specific features captured by the network model at different levels by visualizing the output feature map, and better understands the process of the model extracting information from input data and making predictions, so as to better optimize the model and improve the accuracy of the model in posture detection of bipedal humanoid robots.

[0037] In a second aspect, the present invention provides a bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion, the device comprising:

[0038] A multi-scale feature extraction module is used to continuously acquire multiple frames of original images of the target bipedal humanoid robot and extract multiple feature maps of different scales after preprocessing each original image;

[0039] The temporal multi-scale feature fusion module is used to iteratively fuse temporal multi-scale features based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of the current frame;

[0040] The scale sorting module is used to sort the multiple fused feature maps of the current frame according to the scale size to obtain the fused feature map of the smallest scale and multiple fused feature maps of other scales;

[0041] The attention mechanism processing module is used to process the minimum-scale fusion feature map using the preset attention module to obtain the encoding features of the current frame;

[0042] A skip connection processing module is used to use a preset skip connection module to process fusion feature maps of other scales respectively to obtain multiple skip connection features of the current frame;

[0043] The feature decoding module is used to decode the encoded features and multiple jump connection features to obtain the output feature map of the current frame.

[0044] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 1 is a flow chart of a method for detecting a posture of a bipedal humanoid robot based on temporal multi-scale feature fusion according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the positions of key points and a skeleton of a bipedal humanoid robot in a method for detecting a bipedal humanoid robot posture based on temporal multi-scale feature fusion according to an embodiment of the present invention;

[0049] Figure 3 2 is a schematic diagram of a key point heat map and a skeleton heat map of a bipedal humanoid robot in a bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion according to an embodiment of the present invention;

[0050] Figure 4 is a flow chart of another method for detecting the posture of a bipedal humanoid robot based on temporal multi-scale feature fusion according to an embodiment of the present invention;

[0051] Figure 5 2 is a schematic diagram of a neural network structure executed by a method for detecting a bipedal humanoid robot posture based on temporal multi-scale feature fusion according to an embodiment of the present invention;

[0052] Figure 6 is a structural block diagram of a bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion according to an embodiment of the present invention;

[0053] Figure 7 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0055] The embodiment of the present invention provides a bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion, which improves the accuracy and recall rate of bipedal humanoid robot posture detection by performing temporal multi-scale feature fusion on the collected images.

[0056] According to an embodiment of the present invention, an embodiment of a method for posture detection of a bipedal humanoid robot based on temporal multi-scale feature fusion is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] In this embodiment, a bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion is provided, which can be used in the above-mentioned computer system. Figure 1 FIG. 1 is a flow chart of a method for detecting a bipedal humanoid robot posture based on temporal multi-scale feature fusion according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0058] Step S101 : continuously acquire multiple frames of original images of a target bipedal humanoid robot, and pre-process each original image to extract multiple feature maps of different scales.

[0059] Specifically, a plurality of color images of a target bipedal humanoid robot can be continuously captured by a camera as original images. Since the original image is a continuous image frame, each frame of the original image may contain one or more complete bipedal humanoid robots, one or more parts of a bipedal humanoid robot, or may not contain a bipedal humanoid robot. When the original image does not contain a bipedal humanoid robot, the response of the key points and skeleton feature maps output by the bipedal humanoid robot posture detection method provided in this embodiment will be relatively small and below the threshold, so no robot is detected in the end. For each original image, feature extraction of different scales is performed after pre-processing according to a preset processing method to obtain multiple feature maps of different scales.

[0060] Step S102 , iteratively performing temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame, to obtain multiple fused feature maps of the current frame.

[0061] Specifically, temporal multi-scale feature fusion is achieved by inputting the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame into a preset feature fusion module. If the current scale of the current frame is the minimum scale and there is no fused feature map of the previous scale, then the feature map of the current scale of the current frame and the feature map of the current scale of the previous frame can be directly temporally fused to obtain the fused feature map of the current scale of the current frame. If the current scale of the current frame is not the minimum scale, then the feature map of the current scale of the previous frame has a fused feature map of the previous scale of the current frame, then the preset feature fusion module is used to fuse the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame according to preset rules. Finally, multiple fused feature maps of the current frame are obtained.

[0062] In step S103 , the multiple fused feature maps of the current frame are sorted according to their scales to obtain a fused feature map of the minimum scale and multiple fused feature maps of other scales.

[0063] Specifically, the scale sizes of multiple fused feature maps of the current frame can be 1 / 32, 1 / 16, 1 / 8, 1 / 4, and 1 / 2. They are sorted according to the scale size to obtain the 1 / 32 fused feature map, 1 / 16 fused feature map, 1 / 8 fused feature map, 1 / 4 fused feature map, and 1 / 2 fused feature map of the current frame. Among them, the 1 / 32 fused feature map is the smallest scale fused feature map, which is only used as an example, but is not limited to this.

[0064] Step S104: Use the preset attention module to process the minimum-scale fusion feature map to obtain the coding features of the current frame.

[0065] Specifically, the minimum-scale fused feature map is input into a preset attention module for attention mechanism processing to obtain the first encoder feature of the current frame. The preset attention module is an existing channel-based attention module, and the attention mechanism processing process is a mature existing technology and will not be repeated here.

[0066] Step S105 : Using a preset skip connection module, the fusion feature maps of other scales are processed respectively to obtain multiple skip connection features of the current frame.

[0067] Specifically, the fused feature map of the smallest scale is input into the preset attention module, while the fused feature maps of other scales are input into the preset skip connection module to obtain the corresponding skip connection features respectively.

[0068] Step S106: Decode the encoded features and the multiple skip connection features to obtain an output feature map of the current frame.

[0069] Specifically, based on the linkage relationship between the key points and the skeleton of the bipedal humanoid robot, a decoder with an asymmetric convolutional structure is designed to reduce the number of parameters and alleviate overfitting, enhance the ability of the decoder module to process spatial features, and effectively improve the accuracy of the final posture detection.

[0070] In the posture detection of bipedal humanoid robots, an attention mechanism is added between the encoder module and the decoder module so that the importance of each channel of the features finally output by the encoder to the decoding work of the decoder varies according to different images, thereby improving the feature decoding capability of the decoder and helping to improve the accuracy of the final detection results.

[0071] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided in this embodiment increases the robustness of robot recognition at different positions and sizes through temporal multi-scale feature fusion, adds an attention mechanism between encoding and decoding, improves the feature decoding capability of the decoder, utilizes the decoder to enhance the ability to process spatial features, and improves the accuracy and recall rate of bipedal humanoid robot posture detection.

[0072] In some optional embodiments, the method further comprises:

[0073] Multiple key points and skeleton of the target bipedal humanoid robot are extracted based on the output feature map of the current frame.

[0074] Specifically, if the size of the output feature map of the current frame is Where H represents the height of the original image after preprocessing, W represents the length of the original image after preprocessing, p represents the number of key points of the bipedal humanoid robot, and s represents the number of skeletons of the bipedal humanoid robot.

[0075] Each channel of the output feature map corresponds to each key point or each skeleton of the robot. In each output feature map, the feature value at the image position corresponding to the key point or skeleton will be larger, and the response at other positions will be smaller. Therefore, the key points can be extracted by the maximum value suppression method, and the different key points of the bipedal humanoid robot can be combined by post-processing the skeleton feature map. Figure 2 Shown is a schematic diagram of the locations of the key points and skeleton of a bipedal humanoid robot, where a red point represents a key point and an orange line between two red points represents the skeleton.

[0076] The output features are visualized according to the key points and skeleton to obtain the heat map of the target bipedal humanoid robot.

[0077] Specifically, each channel of the output feature map is visualized separately and displayed in the form of a heat map, such as Figure 3 The following are the key point heat maps and skeleton heat maps of the bipedal humanoid robot. It can be seen that the responses of the feature maps of the key points and skeleton corresponding positions are relatively large, while the responses of the feature maps of the other positions are relatively small.

[0078] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided in this embodiment visualizes the output feature map, making it easier to intuitively observe the specific features captured by the network model at different levels, and better understand the process of the model extracting information from input data and making predictions, so as to better optimize the model and improve the accuracy of the model's posture detection of bipedal humanoid robots.

[0079] In this embodiment, a bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion is provided, which can be used in computer systems. Figure 4 FIG. 1 is a flow chart of a method for detecting a bipedal humanoid robot posture based on temporal multi-scale feature fusion according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0080] Step S201 : continuously acquire multiple frames of original images of the target bipedal humanoid robot, and pre-process each original image to extract multiple feature maps of different scales.

[0081] Specifically, the above step S201 includes:

[0082] Step S2011: normalize the original image to obtain a standard image suitable for deep network input.

[0083] Specifically, the original color image of the tth frame is I t , will I t After normalization, the size is adjusted to H×W, where H is the height and W is the length, and a standard image I suitable for deep network input is obtained. t ', the height and length of the standard image must be greater than 200 pixels. It can be an image containing one or more complete bipedal humanoid robots, a part of one or more bipedal humanoid robots, or an image that does not contain a bipedal humanoid robot.

[0084] Step S2012: extract features of different scales from the standard image to obtain multiple feature maps of different scales.

[0085] Specifically, any feature extraction module can be used to extract features at different scales from standard images. Smaller-scale feature maps represent deeper semantic features, while larger-scale feature maps represent simpler features (such as texture and shape). Because bipedal humanoid robots can appear at varying positions and distances in images, the primary task of feature extraction is to extract rotationally invariant, scale-invariant, stable, and versatile features for pose estimation.

[0086] like Figure 5 The figure shows a schematic diagram of the neural network structure based on which the execution process of this embodiment is based. The feature extraction module may include multiple feature extractors of different scales. In this embodiment, five feature extractors are used as an example for illustration only, but the invention is not limited thereto. The standard input image I is obtained after the original image is normalized / resized. t ′, change I t ’ is input into the first feature extractor to obtain the first feature map The scale (size) of the first feature map is Where C1 is the number of feature channels of the first feature map; the first feature map is input into the second feature extractor to obtain the second feature map The scale of the second feature map is Where C2 is the number of feature channels of the second feature map; the second feature map is input into the third feature extractor to obtain the third feature map The scale of the third feature map is Where C3 is the number of feature channels of the third feature map; the third feature map is input into the fourth feature extractor to obtain the fourth feature map The scale of the fourth feature map is Where C4 is the number of feature channels of the fourth feature map; the fourth feature map is input into the fifth feature extractor to obtain the fifth feature map The scale of the fifth feature map is Where C5 is the number of feature channels of the fifth feature map. In summary, the feature extraction module finally outputs the first feature map corresponding to the tth frame Second feature map The third characteristic map Fourth characteristic map Fifth characteristic map These five feature maps of different scales.

[0087] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided in this embodiment facilitates subsequent deep network processing through normalization processing. By extracting features of different scales, multiple features of the original image are understood from different levels, which is conducive to improving the accuracy of subsequent bipedal humanoid robot posture detection.

[0088] Step S202 , iteratively performing temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame, to obtain multiple fused feature maps of the current frame.

[0089] Specifically, the above step S202 includes:

[0090] Step S2021 , performing feature fusion on the minimum-scale feature map of the current frame and the minimum-scale feature map of the previous frame to obtain a fused feature map of the minimum scale of the current frame.

[0091] Specifically, the main component structure of the preset feature fusion module is FLNet, which is used to fuse the kth feature map of the tth frame, the k-1th feature map of the tth frame (if any), and the kth feature map of the t-1th frame, and fuse the same-scale features of adjacent time frames and the different-scale features of the same time frame to enhance the robustness of feature expression. For the feature map of the minimum scale of the current frame, there is no feature map of the previous scale of the current frame, so it is only necessary to fuse the feature map of the minimum scale of the current frame with the feature map of the minimum scale of the previous frame to obtain the fused feature map of the minimum scale of the current frame. The minimum scale of the current frame is equal to the minimum scale of the previous frame.

[0092] In step S2022, the fused feature map of the minimum scale of the current frame is used as the fused feature map of the previous scale of the current frame in the first temporal multi-scale feature fusion process, and the temporal multi-scale feature fusion is repeatedly performed based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of different scales in the current frame.

[0093] Specifically, taking five feature maps of different scales as an example, the fused feature map of the smallest scale of the tth frame is used as the fifth fused feature map The fourth feature map of the tth frame, the fifth fused feature map, and the fourth feature map of the t-1th frame are fused to obtain the fourth fused feature map The third feature map of the tth frame, the fourth fused feature map, and the third feature map of the t-1th frame are fused to obtain the third fused feature map The second feature map of the tth frame, the third fused feature map, and the second feature map of the t-1th frame are fused to obtain the second fused feature map The first feature map of the tth frame, the second fused feature map, and the first feature map of the t-1th frame are fused to obtain the first fused feature map Finally, five fused feature maps of different scales are obtained for the t-th frame. This is only an example, but not limited to this. It should be noted that the scale of the k-th feature map of the t-th frame is the same as the scale of the k-th feature map of the t-th frame.

[0094] In some optional implementations, step S2022 performs temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame, including:

[0095] In step a1, a first preset convolution process is performed on the feature map of the current frame at the current scale to obtain a first feature map, a preset upsampling process is performed on the fused feature map of the previous scale of the current frame to obtain a second feature map, and a second preset convolution process is performed on the feature map of the previous frame at the current scale to obtain a third feature map.

[0096] Specifically, the feature fusion module uses an FLNet architecture for feature fusion. FLNet has three inputs: Input 1 is the kth feature map of frame t, Input 2 is the k-1th fused feature map of frame t (if any), and Input 3 is the kth feature map of frame t-1. FLNet processes these three inputs by first performing a 1×1 convolution (Convolution, Conv) on Input 1 to obtain the first feature map, then performing a 2x bilinear interpolation upsampling on Input 2 to obtain the second feature map, and finally performing a 3x3 convolution on Input 3 to obtain the third feature map.

[0097] Step a2: Add the first feature map and the second feature map and then splice them with the third feature map to obtain a spliced ​​feature map.

[0098] Specifically, the first feature map and the second feature map are added and then spliced ​​with the third feature map in the channel dimension to obtain a spliced ​​feature map.

[0099] In step a3, a depth-wise separable convolution is performed on the spliced ​​feature map to obtain a fused feature map of the current frame and the current scale.

[0100] Specifically, the spliced ​​feature map is processed by two depth-wise separable convolutions to obtain the output feature map, where the depth-wise separable convolution can significantly reduce the model parameters and computational complexity after the features are spliced ​​in the channel dimension.

[0101] For feature maps with five different scales, the overall processing of the preset feature fusion module includes: the fifth feature map of the tth frame The fifth feature map of the t-1th frame After processing by the fifth FLNet, the fifth fusion feature of the t-th frame is obtained The fourth feature map of the tth frame The fourth feature map of the t-1th frame The fifth fusion feature of the t-th frame After the fourth FLNet processing, the fourth fusion feature of the t-th frame is obtained The third feature map of the tth frame The third feature map of the t-1th frame The fourth fusion feature of the t-th frame After the third FLNet processing, the third fusion feature of the tth frame is obtained The second feature map of frame t The second feature map of frame t-1 The third fusion feature of the t-th frame After the second FLNet processing, the second fusion feature of the tth frame is obtained The first feature map of frame t The first feature map of frame t-1 Second fusion feature After the first FLNet processing, the first fusion feature of the t-th frame is obtained

[0102] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided in this embodiment enhances the robustness of feature expression by fusing the same-scale features of adjacent temporal frames and the different-scale features of the same temporal frame. The deep separable convolution processing can significantly reduce the model parameters and calculation amount after the features are spliced ​​in the channel dimension, thereby improving the model processing speed.

[0103] Step S203: sort the multiple fused feature maps of the current frame according to their scales to obtain the fused feature map of the smallest scale and multiple fused feature maps of other scales. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0104] Step S204: Use the preset attention module to process the minimum-scale fusion feature map to obtain the encoding features of the current frame. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0105] Step S205 : Using a preset skip connection module, the fusion feature maps of other scales are processed respectively to obtain multiple skip connection features of the current frame.

[0106] Specifically, the above step S205 includes:

[0107] The fused feature maps of other scales are respectively input into the preset skip connection module, and the first preset convolution processing is performed to obtain the skip connection features of the scale corresponding to the current frame.

[0108] Specifically, the preset skip connection module includes multiple convolution layers with 1×1 convolution kernels, and the first fusion feature of the tth frame After the convolution with a convolution kernel of 1×1, the first skip connection feature of the t-th frame is obtained. The second fusion feature of the t-th frame After convolution processing with a convolution kernel of 1×1, the second skip connection feature of the t-th frame is obtained The third fusion feature of the t-th frame After convolution processing with a convolution kernel of 1×1, the third skip connection feature of the t-th frame is obtained The fourth fusion feature of the t-th frame After convolution processing with a convolution kernel of 1×1, the fourth jump connection feature of the t-th frame is obtained

[0109] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided in this embodiment directly transmits low-level features (such as edges, textures, etc.) in the network to higher-level feature maps by performing jump connections on fused features of other scales, which is beneficial to maintaining spatial information and improving the accuracy of target detection.

[0110] Step S206: Decode the encoded features and the multiple skip connection features to obtain an output feature map of the current frame.

[0111] Specifically, the above step S206 includes:

[0112] Step S2061 , sorting multiple skip connection features from small to large scale to obtain a skip connection feature sequence.

[0113] Specifically, if Figure 5 As shown, the scale of the fourth jump connection feature is 1 / 16, the scale of the third jump connection feature is 1 / 8, the scale of the second jump connection feature is 1 / 4, and the scale of the first jump connection feature is 1 / 2. Then the jump connection feature sequence is: fourth jump connection feature, third jump connection feature, second jump connection feature, first jump connection feature, which is only an example but not limited to this.

[0114] The decoding process is implemented through a preset decoding module, which includes multiple DCNet asymmetric convolution structures. The DCNet processing process includes:

[0115] (1) Receive the input feature map input and divide the input into the first branch and the second branch for separate processing.

[0116] (2) The first branch is processed with convolution kernels of 1×3 and 3×1 to obtain the first branch feature, and the second branch is processed with convolution kernels of 3×1 and 1×3 to obtain the second branch feature.

[0117] (3) After adding the first branch feature and the second branch feature, the intermediate feature is obtained.

[0118] (4) The intermediate features are processed by the third and fourth branches to obtain the third branch features and the fourth branch features: the third branch is composed of convolutions with convolution kernels of 1×3 and 3×1, and the fourth branch is processed by convolutions with convolution kernels of 3×1 and 1×3.

[0119] (5) After adding the third branch features and the fourth branch features, batch normalization and ReLu activation function processing are performed to finally obtain the output feature map output corresponding to the input feature map input.

[0120] Step S2062: Iteratively perform a preset asymmetric convolution process based on the coding features and the skip connection feature sequence to obtain the output coding features of the current frame.

[0121] Specifically, if Figure 5 As shown, the processing process of the preset decoding module includes: the first encoder feature of the tth frame After the first DCNet processing, the bilinear interpolation double upsampling is performed and the fourth jump connection feature is connected. The result of the addition is processed by the second DCNet, and then upsampled twice by bilinear interpolation and combined with the third jump connection feature. The result of the addition is processed by the third DCNet, and then upsampled twice by bilinear interpolation and combined with the second jump connection feature. The result of the addition is processed by the fourth DCNet, and then upsampled twice by bilinear interpolation and combined with the first jump connection feature. Add them together to get the output encoding features of the t-th frame.

[0122] Step S2063: Perform a second preset convolution process on the output coding features of the current frame to obtain an output feature map of the current frame.

[0123] Specifically, the output encoding feature of the tth frame is subjected to convolution processing with a convolution kernel of 3×3 to obtain the output feature map of the current frame.

[0124] The bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion provided in this embodiment uses jump connections and feature addition to achieve more powerful learning capabilities without significantly increasing the number of parameters. Since the amount of data required to be processed is reduced, the computing cost can be reduced and the reasoning speed can be accelerated.

[0125] In a specific embodiment, the camera collects a color image I t The size of the image is 1920×1080. The three channels of RGB are subtracted from the mean [123.68, 116.779, 103.939], and then divided by the variance [58.393, 57.12, 57.375] for normalization. After resizing, the standard input image I of the deep network is obtained.t '', its size is H = 384, W = 384. The feature extraction module is composed of ResNet34, and the sizes of the first feature map, second feature map, third feature map, fourth feature map, and fifth feature map output by its first feature extractor, second feature map, third feature map, fourth feature map, and fifth feature map are 192×192×64, 96×96×64, 48×48×128, 24×24×256, and 12×12×512 respectively.

[0126] In the preset feature fusion module, the parameters of the two sets of depthwise separable convolutions in FLNet are: the convolution kernel size of the depthwise convolution is 3×3, and the number of output channels of the pointwise convolution is 128. The process of temporal multi-scale feature fusion includes:

[0127] In the fifth FLNet, the size of Input1 is 12×12×512, Input2 does not exist, and the size of Input3 is 12×12×512; the feature map size obtained after Input1 is processed by 1×1 convolution is 12×12×512, and the feature map size obtained after Input3 is processed by 3×3 convolution is 12×12×512; the size of the output fifth fusion feature is 12×12×128.

[0128] In the fourth FLNet, the size of Input1 is 24×24×256, the size of Input2 is 12×12×512, and the size of Input3 is 24×24×256. The feature map size obtained after Input1 is processed by 1×1 convolution is 24×24×512, and the feature map size obtained after Input3 is processed by 3×3 convolution is 24×24×512. The size of the output fourth fusion feature is 24×24×128.

[0129] In the third FLNet, the size of Input1 is 48×48×128, the size of Input2 is 24×24×256, and the size of Input3 is 48×48×128; the feature map size obtained after Input1 is processed by 1×1 convolution is 48×48×256, and the feature map size obtained after Input3 is processed by 3×3 convolution is 48×48×256; the size of the output third fusion feature is 48×48×128.

[0130] In the second FLNet, the size of Input1 is 96×96×64, the size of Input2 is 48×48×128, and the size of Input3 is 96×96×64; the feature map size obtained after Input1 is processed by 1×1 convolution is 96×96×128, and the feature map size obtained after Input3 is processed by 3×3 convolution is 96×96×128; the size of the output second fused feature is 96×96×128.

[0131] In the first FLNet, the size of Input1 is 192×192×64, the size of Input2 is 96×96×64, and the size of Input3 is 192×192×64; the feature map size obtained after Input1 is processed by 1×1 convolution is 192×192×64, and the feature map size obtained after Input3 is processed by 3×3 convolution is 192×192×64; the size of the output first fused feature is 192×192×128.

[0132] The sizes of the first skip connection feature, the second skip connection feature, the third skip connection feature, and the fourth skip connection feature in the preset skip connection module are 192×192×64, 96×96×64, 48×48×64, 24×24×64, and 12×12×64, respectively.

[0133] In the preset decoding module, the sizes of the output encoding features of the first DCNet, the second DCNet, the third DCNet, and the fourth DCNet are 12×12×64, 24×24×64, 48×48×64, and 96×96×64, respectively.

[0134] Finally, the size of the output feature map is 192×192×(13+14).

[0135] In order to obtain the parameters of each module in the above-mentioned deep network, the mean square error loss function is constructed using the output of the network and the labeled true value, and training is performed through training samples to obtain the parameters of each module of the above-mentioned neural network.

[0136] In this embodiment, a bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation by hardware, or a combination of software and hardware, is also possible and conceivable.

[0137] This embodiment provides a bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion. Figure 6 Shown, including:

[0138] The multi-scale feature extraction module 601 is used to continuously acquire multiple frames of original images of the target bipedal humanoid robot, and extract multiple feature maps of different scales after pre-processing each original image.

[0139] The temporal multi-scale feature fusion module 602 is used to iteratively perform temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of the current frame.

[0140] The scale sorting module 603 is used to sort the multiple fused feature maps of the current frame according to the scale to obtain the fused feature map of the minimum scale and multiple fused feature maps of other scales.

[0141] The attention mechanism processing module 604 is used to use a preset attention module to process the minimum-scale fusion feature map to obtain the encoding features of the current frame.

[0142] The skip connection processing module 605 is used to use a preset skip connection module to process fusion feature maps of other scales respectively to obtain multiple skip connection features of the current frame.

[0143] The feature decoding module 606 is used to decode the encoded features and multiple skip connection features to obtain an output feature map of the current frame.

[0144] In some optional implementations, the multi-scale feature extraction module 601 includes:

[0145] The image normalization unit is used to normalize the original image to obtain a standard image suitable for deep network input.

[0146] The feature extraction unit is used to extract features of different scales from the standard image to obtain multiple feature maps of different scales.

[0147] In some optional implementations, the temporal multi-scale feature fusion module 602 includes:

[0148] The minimum fusion unit is used to fuse the feature map of the minimum scale of the current frame with the feature map of the minimum scale of the previous frame to obtain the fused feature map of the minimum scale of the current frame.

[0149] The iterative fusion unit is used to use the fused feature map of the minimum scale of the current frame as the fused feature map of the previous scale of the current frame in the first temporal multi-scale feature fusion process, and repeatedly perform temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of different scales in the current frame.

[0150] In some optional implementations, the iterative fusion unit includes:

[0151] The feature convolution processing subunit is used to perform a first preset convolution process on the feature map of the current frame and the current scale to obtain a first feature map, perform a preset upsampling process on the fused feature map of the previous scale of the current frame to obtain a second feature map, and perform a second preset convolution process on the feature map of the current scale of the previous frame to obtain a third feature map.

[0152] The feature splicing subunit is used to add the first feature map and the second feature map and then splice them with the third feature map to obtain a spliced ​​feature map.

[0153] The depth-wise separable convolution processing subunit is used to perform depth-wise separable convolution processing on the spliced ​​feature map to obtain a fused feature map of the current frame and current scale.

[0154] In some optional implementations, the feature decoding module 606 includes:

[0155] The skip connection feature sorting unit is used to sort multiple skip connection features according to scale from small to large to obtain a skip connection feature sequence.

[0156] The iterative decoding unit is used to iteratively perform a preset asymmetric convolution process according to the coding feature and the jump connection feature sequence to obtain the output coding feature of the current frame.

[0157] The output feature determination unit is used to perform a second preset convolution process on the output coding features of the current frame to obtain an output feature map of the current frame.

[0158] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0159] The bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0160] The embodiment of the present invention also provides a computer device having the above Figure 6 The bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion is shown.

[0161] See also Figure 7 , Figure 7 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0162] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0163] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0164] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0166] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0167] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0168] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A bipedal humanoid robot posture detection method based on temporal multi-scale feature fusion, characterized in that: The method comprises: Continuously acquire multiple frames of original images of the target bipedal humanoid robot, and preprocess each original image to extract multiple feature maps of different scales; Iteratively performing temporal multi-scale feature fusion according to the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of the current frame, including: performing feature fusion on the feature map of the minimum scale of the current frame and the feature map of the minimum scale of the previous frame to obtain the fused feature map of the minimum scale of the current frame; using the fused feature map of the minimum scale of the current frame as the fused feature map of the previous scale of the current frame in the first temporal multi-scale feature fusion process, repeatedly performing temporal multi-scale feature fusion according to the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of different scales of the current frame; The temporal multi-scale feature fusion is performed according to the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame, including: performing a first convolution process on the feature map of the current scale of the current frame to obtain a first feature map, performing an upsampling process on the fused feature map of the previous scale of the current frame to obtain a second feature map, and performing a second convolution process on the feature map of the current scale of the previous frame to obtain a third feature map; adding the first feature map and the second feature map and then splicing them with the third feature map to obtain a spliced ​​feature map; performing a depth-separable convolution process on the spliced ​​feature map to obtain a fused feature map of the current scale of the current frame; Sort the multiple fused feature maps of the current frame by scale to obtain the fused feature map of the smallest scale and multiple fused feature maps of other scales; Using the attention module to process the minimum-scale fusion feature map to obtain the encoding features of the current frame; Use the skip connection module to process the fusion feature maps of other scales separately to obtain multiple skip connection features of the current frame; Decoding is performed based on the encoding features and multiple skip connection features to obtain an output feature map of the current frame.

2. The method according to claim 1, characterized in that After preprocessing each original image, multiple feature maps of different scales are extracted, including: Normalize the original image to obtain a standard image suitable for deep network input; Feature extraction of different scales is performed on the standard image to obtain multiple feature maps of different scales.

3. The method according to claim 1, characterized in that The skip connection module is used to process the fusion feature maps of other scales separately to obtain multiple skip connection features of the current frame, including: The fused feature maps of other scales are input into the skip connection module respectively, and the first convolution process is performed to obtain the skip connection features of the scale corresponding to the current frame.

4. The method according to claim 1, wherein Decoding is performed based on the encoding features and multiple skip connection features to obtain an output feature map of the current frame, including: Sort multiple skip connection features from small to large scale to obtain a skip connection feature sequence; Iteratively performing asymmetric convolution processing according to the coding feature and the skip connection feature sequence to obtain an output coding feature of the current frame; A second convolution process is performed on the output coding features of the current frame to obtain an output feature map of the current frame.

5. The method according to claim 1 or 4, characterized in that The method further comprises: Extract multiple key points and skeleton of the target bipedal humanoid robot based on the output feature map of the current frame; The output features are visualized according to the key points and the skeleton to obtain a heat map of the target bipedal humanoid robot.

6. A bipedal humanoid robot posture detection device based on temporal multi-scale feature fusion, characterized in that: The device comprises: A multi-scale feature extraction module is used to continuously acquire multiple frames of original images of the target bipedal humanoid robot and extract multiple feature maps of different scales after preprocessing each original image; The temporal multi-scale feature fusion module is used to iteratively perform temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of the current frame, including: performing feature fusion on the feature map of the minimum scale of the current frame and the feature map of the minimum scale of the previous frame to obtain the fused feature map of the minimum scale of the current frame; using the fused feature map of the minimum scale of the current frame as the fused feature map of the previous scale of the current frame in the first temporal multi-scale feature fusion process, and repeatedly performing temporal multi-scale feature fusion based on the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame to obtain multiple fused feature maps of different scales of the current frame; The temporal multi-scale feature fusion is performed according to the feature map of the current scale of the current frame, the fused feature map of the previous scale of the current frame, and the feature map of the current scale of the previous frame, including: performing a first convolution process on the feature map of the current scale of the current frame to obtain a first feature map, performing an upsampling process on the fused feature map of the previous scale of the current frame to obtain a second feature map, and performing a second convolution process on the feature map of the current scale of the previous frame to obtain a third feature map; adding the first feature map and the second feature map and then splicing them with the third feature map to obtain a spliced ​​feature map; performing a depth-separable convolution process on the spliced ​​feature map to obtain a fused feature map of the current scale of the current frame; The scale sorting module is used to sort the multiple fused feature maps of the current frame according to the scale size to obtain the fused feature map of the smallest scale and multiple fused feature maps of other scales; An attention mechanism processing module, configured to process the minimum-scale fusion feature map using an attention module to obtain a coding feature of the current frame; A skip connection processing module is used to use the skip connection module to process the fusion feature maps of other scales respectively to obtain multiple skip connection features of the current frame; The feature decoding module is used to decode the encoding feature and multiple jump connection features to obtain an output feature map of the current frame.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images