Key point detection method and device, equipment, medium and product

Abstract

The invention discloses a key point detection method and device, equipment, a medium and a product, and the method comprises the steps: carrying out the feature extraction of a to-be-detected image after the to-be-detected image comprising one or more objects is obtained, and obtaining a feature extraction result, so as to enable the feature extraction result to represent the information carried by the to-be-detected image, the information comprises the position and state of each object in the to-be-detected image; according to the feature extraction result, at least one bounding box and key point information corresponding to each bounding box are predicted at the same time, so that different bounding boxes are used for representing the positions of different objects in the to-be-detected image, and for any bounding box, the key point information corresponding to each bounding box is obtained. The key point information corresponding to the bounding box is used for describing the state of the object framed by the bounding box in the to-be-detected image, so that multi-object key point detection can be quickly and accurately realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a key point detection method, apparatus, device, medium, or product. Background Technology

[0002] Multi-object keypoint detection is used to detect keypoints of multiple objects in an image, so that these keypoints can represent the state of each object in the image. For example, when there are multiple people in the image, multi-object keypoint detection can be used to detect the keypoints of each person in the image, such as skeletal keypoints, so that these keypoints can represent the state of each person in the image, such as body posture.

[0003] In addition, multi-object keypoint detection is an important technology in the field of computer vision; moreover, multi-object keypoint detection has a wide range of applications, such as virtual reality or sports motion analysis.

[0004] Furthermore, how to achieve multi-object key point detection is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a key point detection method, apparatus, equipment, medium, and product that can quickly and accurately detect key points of multiple objects.

[0006] To achieve the above objectives, the technical solution provided in this application is as follows:

[0007] This application provides a key point detection method, the method comprising:

[0008] Obtain an image to be detected, wherein at least one object exists in the image to be detected;

[0009] The image to be detected is subjected to feature extraction processing to obtain the feature extraction result;

[0010] Based on the feature extraction results, at least one bounding box and key point information corresponding to each bounding box are predicted simultaneously; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object enclosed by the bounding box in the image to be detected.

[0011] In one possible implementation, the step of simultaneously predicting at least one bounding box and key point information corresponding to each bounding box based on the feature extraction result includes:

[0012] The feature extraction results are subjected to at least one convolutional process to obtain the processing result;

[0013] The processing results are simultaneously subjected to bounding box prediction and keypoint prediction to obtain bounding box prediction results and keypoint prediction results.

[0014] Based on the bounding box prediction results and the key point prediction results, at least one bounding box and key point information corresponding to each bounding box are determined.

[0015] In one possible implementation, the feature extraction result includes a feature map; the bounding box prediction result includes bounding box prediction information corresponding to each pixel in the feature map; and the keypoint prediction result includes keypoint prediction information corresponding to each pixel in the feature map.

[0016] The method further includes:

[0017] For any pixel in the feature map, construct the correspondence between the bounding box prediction information corresponding to the pixel and the key point prediction information corresponding to the pixel;

[0018] The step of determining the at least one bounding box and the key point information corresponding to each bounding box based on the bounding box prediction result and the key point prediction result includes:

[0019] Based on the bounding box prediction results and the correspondence, the at least one bounding box and the key point information corresponding to each bounding box are determined.

[0020] In one possible implementation, the simultaneous prediction is achieved using a detection module; the detection module includes a convolution submodule, a bounding box prediction submodule, and a keypoint prediction submodule; the convolution submodule is used to implement the at least one convolution process; the bounding box prediction submodule is used to implement the bounding box prediction process; and the keypoint prediction submodule is used to implement the keypoint prediction process.

[0021] In one possible implementation, the feature extraction process on the image to be detected to obtain the feature extraction result includes:

[0022] The image to be detected is subjected to feature extraction processing at multiple scales to obtain the feature extraction results at the multiple scales.

[0023] In one possible implementation, the process of determining the at least one bounding box and the key point information corresponding to each bounding box includes:

[0024] For any of the scales, based on the feature extraction results of that scale, the bounding box prediction result and the key point prediction result of that scale are determined;

[0025] Based on the bounding box prediction results at multiple scales and the key point prediction results at multiple scales, at least one bounding box and the key point information corresponding to each bounding box are determined.

[0026] In one possible implementation, the process of determining the at least one bounding box includes:

[0027] For any of the scales, the bounding box prediction result for that scale is mapped to the image to be detected to obtain the bounding box mapping result corresponding to that scale;

[0028] Non-maximum suppression (NMS) is applied to the bounding box mapping results corresponding to the multiple scales to obtain at least one bounding box.

[0029] In one possible implementation, after obtaining the feature extraction results at the multiple scales, the method further includes:

[0030] For any of the scales, the feature extraction result of that scale is fused with the feature extraction results of one or more other scales besides that scale to obtain the feature fusion result of that scale.

[0031] The step of simultaneously predicting at least one bounding box and key point information corresponding to each bounding box based on the feature extraction results includes:

[0032] Based on the feature fusion results of the multiple scales, at least one bounding box and key point information corresponding to each bounding box are predicted simultaneously.

[0033] In one possible implementation, the feature extraction process is implemented using the feature extraction module in the multi-information detection model; the simultaneous prediction is implemented using the detection module in the multi-information detection model.

[0034] In one possible implementation, the method further includes:

[0035] The model loss of the multi-information detection model is determined based on the at least one bounding box, the key point information corresponding to the at least one bounding box, and the ground truth annotation information corresponding to the image to be detected; the ground truth annotation information is used to describe the actual position and state of each object in the image to be detected.

[0036] The multi-information detection model is updated based on the model loss.

[0037] In one possible implementation, the key point information includes prediction information of the target key points, and the truth value annotation information includes the truth value information of the target key points;

[0038] The process of determining the model loss includes:

[0039] Calculate the difference between the predicted information and the true information of the target key point;

[0040] If the absolute value of the gap is less than a preset threshold, the predicted loss of the target key point is determined based on the first loss function and the absolute value of the gap; in the first loss function, the dependent variable and the independent variable are negatively correlated, and in the derivative function of the first loss function, the dependent variable and the independent variable are negatively correlated.

[0041] If the absolute value of the gap is not less than a preset threshold, the predicted loss of the target key point is determined based on the second loss function and the absolute value of the gap; in the second loss function, the dependent variable and the independent variable are positively correlated, and the derivative of the second loss function is a constant;

[0042] Based on the prediction loss of the target key points, the model loss of the multi-information detection model is determined.

[0043] This application provides a key point detection device, including:

[0044] An acquisition unit is used to acquire an image to be detected, wherein at least one object exists in the image to be detected;

[0045] The extraction unit is used to perform feature extraction processing on the image to be detected to obtain feature extraction results;

[0046] The prediction unit is used to predict at least one bounding box and key point information corresponding to each bounding box based on the feature extraction results; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object framed by the bounding box in the image to be detected.

[0047] This application provides an electronic device, the device comprising: a processor and a memory;

[0048] The memory is used to store instructions or computer programs;

[0049] The processor is configured to execute the instructions or computer program in the memory, so that the electronic device performs the key point detection method provided in this application.

[0050] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the key point detection method provided in this application.

[0051] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the key point detection method provided in this application.

[0052] Compared with related technologies, this application has at least the following advantages:

[0053] In the technical solution provided in this application, after obtaining an image to be detected that includes one or more objects, feature extraction processing is first performed on the image to be detected to obtain feature extraction results, so that the feature extraction results can represent the information carried by the image to be detected, such as the position and state of each object in the image to be detected; then, based on the feature extraction results, at least one bounding box and key point information corresponding to each bounding box are predicted simultaneously, so that different bounding boxes are used to represent the position of different objects in the image to be detected, and for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected, so that these bounding boxes and the key point information corresponding to these bounding boxes can represent the position and state of each object in the image to be detected. In this way, object detection and key point detection can be performed simultaneously on an image, thereby effectively overcoming the defects of some related key point detection schemes, such as long time consumption or low accuracy, and thus achieving fast and accurate multi-object key point detection. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of a multi-object image provided in an embodiment of this application;

[0056] Figure 2 A schematic diagram illustrating the bounding box detection result of a multi-object image provided in an embodiment of this application;

[0057] Figure 3 A flowchart of a key point detection method provided in an embodiment of this application;

[0058] Figure 4 A schematic diagram of a multi-object key point detection process provided in an embodiment of this application;

[0059] Figure 5This is a schematic diagram of the structure of a detection head provided in an embodiment of this application;

[0060] Figure 6 A schematic diagram of bounding box prediction information corresponding to a pixel provided in an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of the structure of a key point detection device provided in an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] Research has found that in some scenarios, such as motion guidance applications that require real-time feedback, the speed requirements for multi-object keypoint detection are usually quite high, making it a challenging task to ensure a balance between speed and accuracy in multi-object keypoint detection.

[0064] The study also found that for keypoint detection schemes based on a top-down approach, this scheme includes: first, using a pre-trained model with bounding box detection capabilities to process the image, such as... Figure 1 Image 100 is shown, and bounding box detection processing is performed to obtain the bounding box of each object in the image, such as... Figure 2 The bounding boxes 201-203 are shown. Then, based on each bounding box, a region image of each object is extracted from the image. Next, keypoint detection processing is performed on each region image to obtain the keypoint detection result for each object. This method can obtain keypoint detection results with relatively high accuracy. However, because this scheme requires keypoint detection processing for each object's region image separately, the inference time increases linearly with the number of objects in the image. Therefore, this scheme is more suitable for real-time inference in single-object keypoint detection scenarios, but not for real-time inference in multi-object keypoint detection scenarios.

[0065] Further research revealed that for a bottom-up keypoint detection scheme, which directly predicts keypoints for all objects in an image, while the inference speed is relatively fast, its accuracy is lower than that of the top-down keypoint detection scheme mentioned above. The reasons for this lower accuracy are as follows: ① Because this scheme directly predicts keypoints for the entire image, and the image contains a large amount of information unrelated to the objects, such as background information, this information can significantly interfere with the keypoint prediction process, thus affecting accuracy. ② The prediction results obtained from this scheme can only describe the distribution of keypoints in the image, but cannot describe which keypoints belong to the same object or which belong to different objects. Therefore, subsequent methods, such as clustering, are needed to estimate which object each keypoint belongs to, further impacting accuracy.

[0066] Based on the above research, in order to achieve fast and accurate multi-object keypoint detection, this application provides a keypoint detection method. The method includes: after acquiring an image to be detected containing one or more objects, firstly performing feature extraction processing on the image to obtain feature extraction results, so that the feature extraction results can represent the information carried by the image to be detected, such as the position and state of each object in the image to be detected; then, based on the feature extraction results, simultaneously predicting at least one bounding box and the keypoint information corresponding to each bounding box, so that different bounding boxes are used to represent the positions of different objects in the image to be detected, and for any bounding box, the keypoint information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected. Thus, these bounding boxes and the keypoint information corresponding to these bounding boxes can represent the position and state of each object in the image to be detected. This enables simultaneous object detection and keypoint detection for a single image, effectively overcoming the shortcomings of some related keypoint detection schemes, such as long processing time or low accuracy, thereby achieving fast and accurate multi-object keypoint detection.

[0067] Furthermore, this application does not limit the executing entity of the key point detection method provided in the embodiments of this application. For example, the key point detection method provided in the embodiments of this application can be applied to a terminal device or a server. Alternatively, the key point detection method provided in the embodiments of this application can also be implemented through a data interaction process between a terminal device and a server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0068] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0069] To better understand the technical solution provided in this application, the key point detection method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 3 As shown, the key point detection method provided in this application includes S1-S3 as described below. Wherein, the... Figure 3 This is a flowchart of a key point detection method provided in an embodiment of this application.

[0070] S1: Obtain the image to be detected, which contains at least one object.

[0071] Here, the image to be detected refers to the image that needs to undergo key point detection processing, such as... Figure 1 The image shown is 100 or Figure 4 The input image is shown; and the image to be detected may contain at least one object, such as... Figure 1 The objects shown include black-lined figures, red-lined figures, and green-lined figures. It should be noted that this application does not limit the implementation method of these objects; for example, the objects can be implemented as humans, digital humans, or virtual avatars.

[0072] Furthermore, this application does not limit the implementation method of the image to be detected. For example, in some scenarios, such as keypoint detection using a model, in order to better meet the image size requirements of that scenario, the size of the image to be detected can be a preset image size, such as... Figure 4 The image size shown is 800×800. This preset image size is determined in advance based on the image size requirements of the current application scenario to ensure that the preset image size meets those requirements; moreover, this application does not limit the preset image size, for example, the preset image size can be 800×800.

[0073] Furthermore, this application does not limit the method of acquiring the image to be detected. For example, when the size of the image to be detected is a preset image size, and the long side of the preset image size is equal to the short side size, the acquisition process of the image to be detected may include steps 11-15 below.

[0074] Step 11: After obtaining the original image, compare the size of the original image with the preset image size to obtain a comparison result, so that the comparison result can be used to indicate whether the size of the original image is equal to the preset image size.

[0075] The original image refers to an image acquired in a certain way, such as an image acquired by a certain image acquisition device, an image uploaded by a user using a certain input device, or a sample image randomly extracted from a certain training image set in a model training scenario, etc. Moreover, this application does not limit the acquisition method of the original image.

[0076] Step 12: If the comparison result above indicates that the size of the original image is equal to the preset image size, then directly determine the original image as the image to be detected.

[0077] Step 13: If the comparison result above indicates that the long side of the original image is equal to the long side of the preset image size, but the short side of the original image is smaller than the short side of the preset image size, then add some pixels that cannot provide semantic information, such as black pixels, to the short side of the original image to obtain the image to be detected. This makes the image to be detected include all the pixels in the original image and these added pixels, and makes the size of the image to be detected equal to the preset image size. In this way, the size adjustment can be achieved while fully preserving the image information carried by the original image.

[0078] Step 14: If the comparison result above indicates that the long side of the original image is smaller than the long side of the preset image size, then add some pixels that cannot provide semantic information to both the long and short sides of the original image to obtain the image to be detected. This makes the image to be detected include all the pixels in the original image and these added pixels, and makes the size of the image to be detected equal to the preset image size. In this way, the size adjustment can be achieved while fully preserving the image information carried by the original image.

[0079] Step 15: If the comparison result above indicates that the long side of the original image is larger than the long side of the preset image size, then, while ensuring that the aspect ratio of the original image remains unchanged, the original image is first resized, such as through resizing or downsampling, to obtain a processed image. This processed image has a long side equal to the long side of the preset image size, and its aspect ratio is equal to that of the original image. Next, it is determined whether the short side of the processed image is equal to the short side of the preset image size. If it is, the processed image is directly identified as the image to be detected. If the short side of the processed image is smaller than the short side of the preset image size, some pixels that cannot provide semantic information are added to the short side of the processed image to obtain the image to be detected. This ensures that the image to be detected includes all pixels in the processed image and these added pixels, and that the size of the image to be detected is equal to the preset image size.

[0080] Based on the relevant content of steps 11 to 15 above, it can be seen that in some scenarios, after obtaining the original image, such as an image provided by the user, an image provided by other upstream tasks, or an image randomly extracted from the training image set, if the size of the original image is not equal to the preset image size for the scenario, the original image can be processed to obtain the image to be detected, so that the size of the image to be detected is equal to the preset image size, thereby enabling the image to be detected to meet the image size requirements for the scenario, so that subsequent key point detection processing for the scenario can be completed based on the image to be detected.

[0081] S2: Perform feature extraction processing on the image to be detected to obtain the feature extraction results.

[0082] The feature extraction results are used to represent the information carried by the image to be detected, such as the position and pose of each object in the image.

[0083] Furthermore, this application does not limit the implementation method of the above feature extraction results; for example, it can be implemented using feature maps.

[0084] Furthermore, this application does not limit the implementation of S2. For example, it can be implemented using any existing or future method capable of feature extraction for an image, such as a backbone network with feature extraction capabilities.

[0085] S3: Based on the feature extraction results, simultaneously predict at least one bounding box and the key point information corresponding to each bounding box; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected.

[0086] In this process, at least one bounding box is used to describe the location of an object in the image to be detected; and the at least one bounding box satisfies the following constraint: different bounding boxes are used to represent the locations of different objects in the image to be detected.

[0087] Furthermore, if at least one bounding box mentioned above includes I bounding boxes, then for the i-th bounding box, as follows: Figure 2 For the bounding box 202 shown, the key point information corresponding to the i-th bounding box is used to describe the state of the object bounded by the i-th bounding box in the image to be detected, such as its pose. The object bounded by the i-th bounding box refers to an object in the image to be detected that is located within the region described by the i-th bounding box, such as... Figure 2 The red lines represent people, etc. Here, i is a positive integer, i ≤ I, and I is a positive integer.

[0088] Furthermore, this application does not limit the implementation method of the key point information corresponding to the i-th bounding box mentioned above. For example, the key point information corresponding to the i-th bounding box may include at least the position coordinates of at least one key point corresponding to the i-th bounding box. Here, the at least one key point is used to describe the state of an object, such as its pose; and this application does not limit the implementation method of the at least one key point, for example, it may be implemented using 17 skeletal key points. The position coordinates of the j-th key point corresponding to the i-th bounding box are used to represent the position of the j-th key point of the object bounded by the i-th bounding box in the image to be detected, where j is a positive integer, j≤J, J is a positive integer, and J represents the number of key points among the at least one key point.

[0089] Furthermore, this application does not limit the implementation method of S3 above. For example, it can be implemented using any machine learning model that can simultaneously perform bounding box detection and key point detection processing on the same feature extraction result.

[0090] In fact, in order to better improve the detection accuracy, this application also provides a possible implementation of S3 above, in which S3 may include steps 21-23 below.

[0091] Step 21: Perform at least one convolution process on the feature extraction results above to obtain the processing result, so that the processing result can better represent the relevant information of the object carried in the image to be detected, such as position or pose.

[0092] It should be noted that this application does not limit the implementation method of the above-described at least one convolutional processing. For example, the at least one convolutional processing can be determined according to the actual application scenario. For example, in some scenarios, the at least one convolutional processing may include 1×1 convolution processing and 3×3 convolution processing, so that step 21 above can specifically be: first, perform 1×1 convolution processing on the feature extraction result above to obtain an intermediate result; then, perform 3×3 convolution processing on the intermediate result to obtain a processing result, so that the processing result can better represent the relevant information of the object carried by the image to be detected.

[0093] It should also be noted that this application does not limit the implementation of step 21 above. For example, in some scenarios, to better improve the detection effect, step 21 can be implemented using a convolutional submodule in the detection module. This detection module is used to simultaneously perform bounding box prediction and keypoint prediction processing on the input data of the detection module, such as feature maps. Furthermore, this application does not limit the detection module; for example, the detection module can use... Figure 4 The detection head shown and Figure 5 The detection head shown is used for implementation. Furthermore, the convolutional submodule refers to a module within the detection module that performs at least one convolutional process; and the convolutional submodule can be used to perform at least one convolutional process on the input data of the detection module. Moreover, this application does not limit the implementation of the convolutional submodule. For example, in some scenarios, the convolutional submodule may include a 1×1 convolutional layer and a 3×3 convolutional layer arranged sequentially, and the input data of the 3×3 convolutional layer includes the output data of the 1×1 convolutional layer.

[0094] It should also be noted that this application does not limit the implementation method of the processing result in step 21 above. For example, when the feature extraction result above is implemented using an H×W feature map, the size of the processing result can be H×W×256. Here, H represents the height of the feature map; W represents the width of the feature map.

[0095] Step 22: Simultaneously perform bounding box prediction and keypoint prediction on the above processing results to obtain bounding box prediction results and keypoint prediction results.

[0096] The bounding box prediction result describes relevant information about the bounding boxes predicted for the image to be detected, such as location, confidence level, and classification result. Furthermore, this application does not limit the implementation method of the bounding box prediction result. For example, when the size of the above processing result is H×W×256, the size of the bounding box prediction result can be H×W×6, so that the bounding box prediction result includes the above feature extraction result, such as... Figure 6The feature map shown contains bounding box prediction information for each pixel. Specifically, the bounding box prediction information for the k-th pixel in the feature extraction result represents the bounding box information predicted for that k-th pixel; however, this application does not limit the implementation method of the bounding box prediction information for the k-th pixel. Here, k is a positive integer, k≤K, and K is a positive integer representing the total number of pixels in the feature extraction result.

[0097] Additionally, in some scenarios, to better improve detection performance, for the k-th pixel in the feature extraction results above, such as... Figure 6 For the red pixel shown, the bounding box prediction information corresponding to the k-th pixel can be implemented using a 6-dimensional vector: [x1, y1, x2, y2, y2, prediction confidence, and classification result of the bounding box]. Here, the bounding box refers to the predicted bounding box for the k-th pixel. Figure 6 The purple bounding box is shown; and the k-th pixel satisfies the following positional relationship between the bounding box and the k-th pixel: the k-th pixel is located inside the bounding box. The coordinates (x1, y1) represent the position of the upper left corner of the bounding box in the feature extraction result. The coordinates (x2, y2) represent the position of the lower right corner of the bounding box in the feature extraction result. The bounding box prediction confidence is used to indicate the probability that the bounding box belongs to the bounding box, and the higher the bounding box prediction confidence, the greater the probability that the bounding box belongs to the bounding box. The bounding box classification result is used to indicate whether the target bounded by the bounding box belongs to the object of interest in the current application scenario, such as a person. This application does not limit the implementation of the bounding box classification result. For example, the bounding box classification result may include the bounding box object confidence. The bounding box object confidence is used to indicate the probability that the target bounded by the bounding box belongs to the object of interest in the current application scenario, so that the bounding box object confidence can, to a certain extent, indicate the probability that the bounding box belongs to the bounding box to be detected in the current application scenario, such as a human body bounding box.

[0098] The keypoint prediction result is used to describe relevant information such as location and confidence level of some keypoints predicted for the image to be detected. Furthermore, this application does not limit the implementation method of the keypoint prediction result. For example, when the size of the above processing result is H×W×256, the size of the keypoint prediction result can be H×W×51, so that the keypoint prediction result includes the keypoint prediction information corresponding to each pixel in the above feature extraction result. The keypoint prediction information corresponding to the k-th pixel in the feature extraction result is used to describe the distribution of keypoints predicted within the bounding box for the k-th pixel. Furthermore, this application does not limit the implementation method of the keypoint prediction information corresponding to the k-th pixel. Here, k is a positive integer, k≤K, and K is a positive integer.

[0099] Additionally, in some scenarios, to improve detection performance, for the k-th pixel in the feature extraction results above, when this application uses 17 skeletal keypoints {keypoint 1, keypoint 2, ..., keypoint 17} to describe the pose of an object, the keypoint prediction information corresponding to the k-th pixel can be [x 1 y 1 Score 1 x 2 y 2 Score 2 x 3 y 3 Score 3 ... (and so on), x 17 y 17 Score 17 This 51-dimensional vector is used for implementation. Where (x j y j This coordinate represents the predicted position of keypoint j within the bounding box predicted for the k-th pixel; Score j It refers to (x) j y j The confidence level of this coordinate is used to determine the score. j It can represent the result of (x) j y j The probability that the pixel described by this coordinate belongs to the keypoint j. Where j is a positive integer, j≤17.

[0100] Furthermore, this application does not limit the implementation of step 22 above. For example, in some scenarios, to improve the effect, step 22 can be implemented using a bounding box prediction submodule and a key point prediction submodule in the detection module. The relevant content of the detection module is described above. The bounding box prediction submodule is used to perform bounding box prediction processing on the output results of the convolution submodule in the detection module; and this application does not limit the implementation of the bounding box prediction submodule. The key point prediction submodule is used to perform key point prediction processing on the output results of the convolution submodule in the detection module; and this application does not limit the implementation of the key point prediction submodule.

[0101] Based on the above content and Figure 5 As shown in the detection head, in some scenarios, the detection module can include a 1×1 convolutional layer, a 3×3 convolutional layer, a bounding box prediction submodule, and a keypoint prediction submodule. The working principle of this detection module is as follows: After receiving a feature map, such as an H×W feature map, the 1×1 convolutional layer first performs a 1×1 convolution on the feature map to obtain an intermediate result, such as an H×W×256 intermediate result. Then, the 3×3 convolutional layer performs a 3×3 convolution on the intermediate result to obtain a processed result, such as an H×W×256 processed result. Next, the bounding box prediction submodule performs bounding box prediction on the processed result to obtain a bounding box prediction result, such as an H×W×6 bounding box prediction result. Simultaneously, the keypoint prediction submodule performs keypoint prediction on the processed result to obtain a keypoint prediction result, such as an H×W×51 keypoint prediction result. This allows for the subsequent determination of at least one bounding box and the corresponding keypoint information based on the bounding box prediction result and the keypoint prediction result.

[0102] Step 23: Based on the bounding box prediction results and key point prediction results above, determine at least one bounding box and the key point information corresponding to each bounding box.

[0103] It should be noted that this application does not limit the implementation of step 23 above. For example, in some scenarios, step 23 may specifically include steps 231-232 below.

[0104] Step 231: Based on the bounding box prediction results above, perform bounding box deduplication to obtain at least one bounding box and the corresponding pixel points of each bounding box.

[0105] The deduplication of bounding boxes is used to remove some bounding boxes from multiple bounding boxes that define the same object, keeping only one bounding box, so that the retained bounding box renders the object better than the other bounding boxes.

[0106] Furthermore, this application does not limit the implementation method of the bounding box deduplication process described above. For example, the bounding box deduplication process can be implemented using non-maximum suppression (NMS).

[0107] As can be seen, in one possible implementation, the bounding box deduplication process described above can be as follows: For any two bounding boxes described by the bounding box prediction results above, if the regions defined by these two bounding boxes have an intersection, the intersection-union ratio (IUR) between these two bounding boxes can be calculated first, for example, the ratio between the intersection of the regions defined by these two bounding boxes and the union of the regions defined by these two bounding boxes; then it is determined whether the IUR exceeds a threshold. If it does, it can be determined that the objects defined by these two bounding boxes are the same. Therefore, the bounding boxes with higher bounding box prediction confidence can be retained, and the bounding boxes with lower bounding box prediction confidence can be deleted.

[0108] Furthermore, for any bounding box in at least one of the bounding boxes mentioned above, the pixel corresponding to that bounding box refers to the pixel referenced when predicting that bounding box. For example, when the bounding box is... Figure 6 When the purple bounding box is shown, the pixel corresponding to the bounding box can refer to... Figure 6 The red pixels are shown.

[0109] Step 232: For any bounding box in at least one bounding box above, query the key point prediction information corresponding to the pixel point of the bounding box from the key point prediction results above, and obtain the key point information corresponding to the bounding box, so that the key point information corresponding to the bounding box includes part or all of the key point prediction information.

[0110] Based on the relevant content of steps 231 to 232 above, it can be seen that in some scenarios, after obtaining the bounding box prediction results and the key point prediction results above, the bounding box deduplication process can be performed first based on the bounding box prediction results to obtain at least one bounding box and the corresponding pixels of each bounding box; then, based on the pixels corresponding to these bounding boxes, the key point information corresponding to each bounding box can be queried from the key point prediction results, so that the key point information can accurately describe the state of the object bounded by the corresponding bounding box, thereby making the matching degree between the finally obtained bounding box and its corresponding key point information very high, thus effectively overcoming the defects of the key point detection scheme based on the first stage above, such as low accuracy, which is conducive to improving the detection effect.

[0111] Based on the relevant content of steps 21 to 23 above, in some scenarios, after obtaining the feature extraction results of the image to be detected, the feature extraction results are first subjected to multiple convolution processes to obtain processing results, so that the processing results can better represent the relevant information of each object in the image to be detected, such as position and pose information; then, bounding box prediction processing and key point prediction processing are performed on the processing results to obtain bounding box prediction results and key point prediction results, so that these two results can represent the predicted bounding boxes and their key points based on pixel level prediction information; then, based on these prediction information, at least one bounding box and the key point information corresponding to each bounding box are determined, so that these bounding boxes and their corresponding key point information can more accurately represent the position and pose of each object in the image to be detected.

[0112] In addition, in some scenarios, in order to improve efficiency, this application also provides a possible implementation of S3 above. In this way, when the feature extraction result above includes a feature map, S3 may include steps 31-34 below.

[0113] Step 31: Perform at least one convolution process on the feature extraction results above to obtain the processing result, so that the processing result can better represent the relevant information of the object carried in the image to be detected, such as position or pose.

[0114] It should be noted that the relevant content of step 31 can be found in step 21 above.

[0115] Step 32: Simultaneously perform bounding box prediction and keypoint prediction processing on the above processing results to obtain bounding box prediction results and keypoint prediction results, so that the bounding box prediction results include the bounding box prediction information corresponding to each pixel in the above feature extraction results, and the keypoint prediction results include the keypoint prediction information corresponding to each pixel in the feature extraction results.

[0116] It should be noted that the relevant content of step 32 can be found in step 22 above.

[0117] Step 33: For any pixel in the feature extraction results above, construct the correspondence between the bounding box prediction information corresponding to the pixel and the key point prediction information corresponding to the pixel, so as to bind multiple prediction information of the same pixel, so that the other prediction information of the pixel can be quickly found based on the prediction information of the pixel, which is beneficial to improving efficiency.

[0118] Step 34: Based on the bounding box prediction results and the corresponding relationships mentioned above, determine at least one bounding box and the key point information corresponding to each bounding box.

[0119] It should be noted that this application does not limit the implementation of step 34 above. For example, it can specifically be: after determining at least one bounding box based on the bounding box prediction results above, for any bounding box, first query the key point prediction information that corresponds to the bounding box; then, based on the key point prediction information, determine the key point information corresponding to the bounding box, for example, determine some or all of the data in the key point prediction information as the key point information corresponding to the bounding box.

[0120] Based on the relevant content of steps 31 to 34 above, in some scenarios, after obtaining the feature extraction result of the image to be detected, the feature extraction result is first subjected to multiple convolution processes to obtain the processing result; then, bounding box prediction processing and keypoint prediction processing are performed simultaneously based on the processing result to obtain bounding box prediction results and keypoint prediction results, so that these two results can represent the multiple prediction information corresponding to each pixel in the feature extraction result; then, a mapping relationship is constructed using the bounding box prediction result and the keypoint prediction result, so that the mapping relationship is used to record the correspondence between different prediction information of the same pixel; finally, based on the bounding box prediction result and the mapping relationship, at least one bounding box and the keypoint information corresponding to each bounding box are determined, which helps to improve efficiency.

[0121] Based on the relevant content of S1 to S3 above, the key point detection method provided in this application, after obtaining an image to be detected including one or more objects, firstly performs feature extraction processing on the image to be detected to obtain feature extraction results, so that the feature extraction results can represent the information carried by the image to be detected, such as the position and state of each object in the image to be detected; then, based on the feature extraction results, at least one bounding box and the key point information corresponding to each bounding box are predicted simultaneously, so that different bounding boxes are used to represent the position of different objects in the image to be detected, and for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected, so that these bounding boxes and the key point information corresponding to these bounding boxes can represent the position and state of each object in the image to be detected. In this way, object detection and key point detection can be performed simultaneously on an image, thereby effectively overcoming the defects of some related key point detection schemes, such as long time consumption or low accuracy, and thus achieving fast and accurate multi-object key point detection.

[0122] Research has found that for images containing multiple objects, such as... Figure 1In the image shown, different objects may be of different sizes. Therefore, to improve the keypoint detection effect, keypoint detection processing for different objects can be performed based on image information at different scales. Based on this, this application also provides a possible implementation of the keypoint detection method, in which the keypoint detection method may include steps 41-43 below.

[0123] Step 41: Obtain the image to be detected, which contains at least one object.

[0124] It should be noted that for details regarding step 41, please refer to the relevant content of S1 above.

[0125] Step 42: Perform feature extraction processing on the image to be detected at multiple scales to obtain feature extraction results at multiple scales.

[0126] The multiple scales refer to the scales required for feature extraction. This application does not limit the implementation of the multiple scales. For example, in some scenarios, when the size of the image to be detected is 800×800, the multiple scales may include three scales: 100×100, 50×50, and 25×25.

[0127] Furthermore, this application does not limit the acquisition method of the multiple scales mentioned above. For example, in some scenarios, such as those with high real-time inference requirements, the multiple scales may refer to those pre-set according to the actual application scenario. As another example, in some scenarios, such as those with high inference accuracy requirements, the multiple scales may be determined based on the size of each object in the image to be detected, so that the multiple scales can include the size of each object.

[0128] Furthermore, when the above multiple scales include Q scales, for the q-th scale, the feature extraction result at the q-th scale is obtained by performing feature extraction processing on the image to be detected at the q-th scale. This ensures that the size of the feature extraction result at the q-th scale is equal to the q-th scale, and that the feature extraction result at the q-th scale can better represent the image information presented by the image to be detected at the q-th scale. Here, q is a positive integer, q≤Q, and Q is a positive integer.

[0129] Furthermore, this application does not limit the implementation of the feature extraction process at the q-th scale described above. For example, it can be implemented using a model that can extract feature maps with the q-th scale from an image, such as a backbone network.

[0130] Therefore, in one possible implementation, step 42 above can specifically be: using the backbone network at the q-th scale to perform feature extraction processing on the image to be detected, obtaining the feature extraction result at the q-th scale, where q is a positive integer, q≤Q, and Q is a positive integer. Here, the backbone network at the q-th scale is used to perform feature extraction on the image to be detected by a certain factor, such as 8=2. 3 times, 16=2 4 2 times or 32 = 2 5 The feature extraction result at the q-th scale is obtained by performing a downsampling process of 10 times.

[0131] In practice, to improve efficiency in some scenarios, this application also provides a possible implementation of step 42 above. In this implementation, when the size of the image to be detected is a preset image size, step 42 may specifically include: using the first backbone network to perform feature extraction processing on the image to be detected, obtaining a feature extraction result at the first scale; using the second backbone network to perform feature extraction processing on the feature extraction result at the first scale, obtaining a feature extraction result at the second scale; using the third backbone network to perform feature extraction processing on the feature extraction result at the second scale, obtaining a feature extraction result at the third scale; ... (and so on); using the Qth backbone network to perform feature extraction processing on the feature extraction result at the Q-1th scale, obtaining a feature extraction result at the Qth scale. Wherein, the downsampling network in the first backbone network is used to downsample data with the preset image size to data with the first scale; the downsampling network in the qth backbone network is used to downsample data with the (q-1th)th scale to data with the qth scale, where q is a positive integer, 2≤q≤Q.

[0132] Based on the above, in some scenarios, such as those with high timeliness requirements for inference, after acquiring the image to be detected, multiple backbone networks can be used sequentially to perform feature extraction processing at multiple scales on the image to be detected, thereby obtaining feature extraction results at multiple scales. It is evident that for non-first backbone networks in this sequential sequence, feature extraction is based on the output data of the previous backbone network, rather than on the image to be detected. This is because the amount of data in the output data of the previous backbone network is less than the amount of data in the image to be detected, thus reducing the amount of data that the current backbone network needs to process, thereby improving efficiency. To better understand this, the following will use... Figure 4 The feature extraction process shown is illustrated using an example.

[0133] As an example, when the size of the image to be detected is 800×800, step 42 above can specifically be as follows: The backbone network B3 is used to perform feature extraction processing on the image to be detected, resulting in a 100×100 feature extraction result; the backbone network B4 is used to perform feature extraction processing on the 100×100 feature extraction result, resulting in a 50×50 feature extraction result; and the backbone network B5 is used to perform feature extraction processing on the 50×50 feature extraction result, resulting in a 25×25 feature extraction result. Here, the downsampling network in the backbone network B3 is used to achieve 8 = 2 3 The backbone network B3 performs a 2x downsampling, enabling it to downsample 800×800 data to 100×100 data. The backbone network B4 performs a 2x downsampling, enabling it to downsample 100×100 data to 50×50 data. The backbone network B5 performs a 2x downsampling, enabling it to downsample 50×50 data to 25×25 data.

[0134] Based on the relevant content of step 42 above, after obtaining the image to be detected, feature extraction processing at multiple scales can be performed on the image to be detected to obtain feature extraction results at multiple scales, so that the feature extraction results at different scales can be used to describe the information presented by the image to be detected at different scales, thereby enabling these feature extraction results to better represent the information carried by the image to be detected.

[0135] Step 43: Based on the feature extraction results at multiple scales above, simultaneously predict at least one bounding box and the key point information corresponding to each bounding box; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected.

[0136] It should be noted that this application does not limit the implementation of step 43 above. For example, it can be implemented using any machine learning model that can simultaneously perform bounding box detection and key point detection processing on feature extraction results at multiple scales.

[0137] In fact, in order to better improve the detection effect, this application also provides a possible implementation of step 43 above, in which step 43 may specifically include steps 431-432 below.

[0138] Step 431: For any of the multiple scales mentioned above, based on the feature extraction results of that scale, determine the bounding box prediction result and the key point prediction result of that scale.

[0139] In this application, after obtaining the feature extraction result at the q-th scale, the bounding box prediction result and the keypoint prediction result at the q-th scale can be determined based on the feature extraction result at the q-th scale. The bounding box prediction result at the q-th scale describes relevant information about the bounding boxes predicted based on the feature extraction result at the q-th scale, such as location, confidence level, and classification result. The implementation method of the bounding box prediction result at the q-th scale is similar to that of the bounding box prediction result in step 22 above, and will not be repeated here for the sake of brevity. The keypoint prediction result at the q-th scale describes relevant information about the keypoints predicted based on the feature extraction result at the q-th scale, such as location and confidence level. The implementation method of the keypoint prediction result at the q-th scale is similar to that of the keypoint prediction result in step 22 above, and will not be repeated here for the sake of brevity. Here, q is a positive integer, and q ≤ Q.

[0140] It should be noted that this application does not limit the determination process of "the bounding box prediction result at the q-th scale and the key point prediction result at the q-th scale" in the above paragraph. For example, it can be implemented using the determination process shown in steps 21 to 22 above.

[0141] As can be seen, in one possible implementation, when the multiple scales mentioned above include Q scales, step 431 above may specifically include: first, performing at least one convolution processing on the feature extraction result of the q-th scale to obtain a processing result, so that the processing result can better represent the information carried by the feature extraction result of the q-th scale, such as object position or object pose; then, simultaneously performing bounding box prediction processing and keypoint prediction processing on the processing result to obtain the bounding box prediction result and the keypoint prediction result of the q-th scale. Where q is a positive integer, q≤Q.

[0142] Step 432: Based on the bounding box prediction results at multiple scales and the key point prediction results at multiple scales, determine at least one bounding box and the key point information corresponding to each bounding box.

[0143] It should be noted that this application does not limit the implementation of step 432 above. For example, the implementation of step 432 is similar to the implementation of step 23 above.

[0144] Research has found that the size of the feature extraction results at different scales is different, which causes size differences between the prediction information determined based on the feature extraction results at different scales. Therefore, in order to improve the detection effect, this application also provides a method for determining at least one bounding box as described above. In this method, the process of determining at least one bounding box may specifically include steps 4321-4322 below.

[0145] Step 4321: For any of the multiple scales mentioned above, map the bounding box prediction result of that scale to the image to be detected to obtain the bounding box mapping result corresponding to that scale.

[0146] The bounding box mapping result corresponding to the q-th scale is obtained by mapping the bounding box prediction result at the q-th scale to the image to be detected. This mapping result represents the state of the bounding box described by the bounding box prediction result at the q-th scale in the image to be detected, such as its position and size. This allows the bounding box predicted by the feature extraction result at the q-th scale to be mapped to the image to be detected, thus obtaining the bounding box predicted for the image at the q-th scale. Here, q is a positive integer, q ≤ Q.

[0147] Furthermore, this application does not limit the implementation of step 4321 above. For example, it can adopt any existing or future method that can map data from a small feature map to a large image, such as by using a pre-built machine learning model with such mapping function.

[0148] Step 4322: Perform bounding box deduplication processing on the bounding box mapping results corresponding to multiple scales, such as NMS, to obtain at least one bounding box.

[0149] It should be noted that the implementation method of step 4322 above is similar to the implementation method of step 231 above, and will not be repeated here for the sake of brevity.

[0150] Based on the relevant content of steps 4321 to 4322 above, for some scenarios, after obtaining the bounding box prediction results at multiple scales, the bounding box prediction results at each scale are first mapped to the image to be detected to obtain the bounding box mapping results corresponding to each scale, so that these bounding box mapping results can represent the bounding boxes predicted for the image to be detected at multiple scales, thereby ensuring that there is no size difference between the bounding box mapping results corresponding to these scales; then, the bounding box deduplication processing is performed on the bounding box mapping results corresponding to these scales to obtain at least one bounding box, so that the at least one bounding box can represent the position of objects of different sizes in the image to be detected as accurately as possible.

[0151] Based on the above-mentioned content regarding at least one bounding box, it can be understood that, in one possible implementation, in order to better improve the detection effect, step 432 above may specifically include steps 4323-4326 below.

[0152] Step 4323: For any of the multiple scales mentioned above, map the bounding box prediction result of that scale to the image to be detected to obtain the bounding box mapping result corresponding to that scale.

[0153] It should be noted that for the relevant content of step 4323 above, please refer to the relevant content of step 4321 above.

[0154] Step 4324: Perform bounding box deduplication on the bounding box mapping results corresponding to multiple scales to obtain at least one bounding box and the pixel description information corresponding to each bounding box.

[0155] Specifically, for any bounding box in at least one of the bounding boxes mentioned above, the pixel description information corresponding to that bounding box describes which pixel in the feature extraction results at which scale the bounding box was obtained before mapping. Furthermore, this application does not limit the implementation of the pixel description information; for example, the pixel description information may include the scale corresponding to the bounding box and the position coordinates of the pixel corresponding to the bounding box. The "scale corresponding to the bounding box" indicates which scale of feature extraction results was used to process the bounding box before mapping. The "position coordinates of the pixel corresponding to the bounding box" indicates which pixel in the feature extraction results at that scale was used to process the bounding box before mapping.

[0156] Step 4325: For any bounding box in the above at least one bounding box, based on the pixel description information corresponding to the bounding box, query at least one key point corresponding to the bounding box from the key point prediction results of multiple scales above, so that the at least one key point and the bounding box satisfy the following constraint: the state before mapping of the at least one key point and the bounding box are obtained by predicting the same pixel in the feature extraction results of the same scale.

[0157] Step 4326: For any bounding box in the above at least one bounding box, according to the scale corresponding to the bounding box, at least one key point corresponding to the bounding box is mapped to the image to be detected to obtain the key point information corresponding to the bounding box, so that the key point information can represent the state of the at least one key point in the image to be detected, such as its position.

[0158] Based on the relevant content of steps 4323 to 4326 above, it can be seen that in some scenarios, the prediction data at multiple scales can be mapped to the image to be detected to determine at least one bounding box of the image to be detected and the key point information corresponding to each bounding box. This can effectively avoid the defects caused by the size difference between the prediction data at multiple scales, thereby improving the detection accuracy.

[0159] Based on the relevant content of steps 431 to 432 above, it can be seen that for some scenarios, after obtaining the feature extraction results of the image to be detected at multiple scales, the prediction data at each scale, such as bounding boxes and key points, can be determined first based on the feature extraction results at each scale; then the prediction data at all scales can be integrated to obtain at least one bounding box of the image to be detected and the key point information corresponding to each bounding box. This can effectively overcome the defects caused by the different sizes of different objects in the image to be detected, such as the poor detection accuracy of certain sizes, such as small objects, thereby improving the detection effect.

[0160] Based on the relevant content of steps 41 to 43 above, it can be seen that for some scenarios, after obtaining the image to be detected, the feature extraction results of the image to be detected at multiple scales are first obtained; then, based on these feature extraction results, at least one bounding box and the key point information corresponding to each bounding box are predicted simultaneously. This can effectively overcome the defects caused by the presence of objects of different sizes in the image to be detected, such as low accuracy, thereby improving the detection accuracy.

[0161] In fact, in order to improve the detection accuracy, this application also provides a possible implementation of the key point detection method, in which the key point detection method may include steps 51-54 below.

[0162] Step 51: Obtain the image to be detected, which contains at least one object.

[0163] It should be noted that for details regarding step 51, please refer to the relevant content of S1 above.

[0164] Step 52: Perform feature extraction processing on the image to be detected at multiple scales to obtain feature extraction results at multiple scales.

[0165] It should be noted that for details regarding step 52, please refer to the details regarding step 42 above.

[0166] Step 53: For any scale among the multiple scales mentioned above, fuse the feature extraction results of that scale with the feature extraction results of one or more other scales besides that scale to obtain the feature fusion result of that scale.

[0167] The feature fusion result at the q-th scale is obtained by fusing the feature extraction result at the q-th scale with the feature extraction results at other scales, so that the feature fusion result at the q-th scale can better represent the information carried by the image to be detected. Here, q is a positive integer, q≤Q.

[0168] Furthermore, this application does not limit the process of determining the feature fusion result at the q-th scale mentioned above. For example, it can specifically be as follows: first, according to the q-th scale, the feature extraction results of each scale other than the q-th scale in the above multiple scales are scale-adjusted to obtain the adjusted features corresponding to each scale, so that the size of the adjusted features is equal to the q-th scale; then, the adjusted features corresponding to each scale are fused with the feature extraction results at the q-th scale to obtain the feature fusion result at the q-th scale, so that the size of the feature fusion result is equal to the q-th scale.

[0169] Furthermore, this application does not limit the implementation of step 53 above. For example, step 53 can be implemented using Feature Pyramid Networks (FPN), such as... Figure 4 The three networks P3, P4, and P5 shown are implemented. Network P3 fuses the 100×100, 50×50, and 25×25 feature maps to obtain and output a 100×100 feature map. Network P4 fuses the 100×100, 50×50, and 25×25 feature maps to obtain and output a 50×50 feature map. Network P5 fuses the 100×100, 50×50, and 25×25 feature maps to obtain and output a 25×25 feature map.

[0170] Furthermore, to further improve prediction accuracy, this application also provides a possible implementation of step 53 above. In this implementation, step 53 can specifically be as follows: after obtaining feature extraction results at multiple scales, perform 1×1 convolution processing on the feature extraction results at each scale to obtain the processing results corresponding to each scale, so that the number of channels in the processing results corresponding to each scale is kept consistent, thereby unifying the number of channels of features at all scales; then, for any scale, fuse the processing results corresponding to that scale with the processing results corresponding to one or more other scales besides that scale to obtain the feature fusion result of that scale. This can effectively overcome the increased fusion difficulty caused by the inconsistent number of channels of features at different scales, thereby improving detection efficiency.

[0171] Step 54: Based on the feature fusion results of multiple scales above, simultaneously predict at least one bounding box and the key point information corresponding to each bounding box; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected.

[0172] It should be noted that the implementation of step 54 is similar to the implementation of step 43 above, except that "feature extraction result" in the implementation of step 43 is replaced with "feature fusion result".

[0173] Based on the relevant content of steps 51 to 54 above, for some scenarios, after obtaining the image to be detected, the multi-scale feature extraction results of the image to be detected are first obtained; then, for any scale, the feature extraction results of other scales are used to supplement the feature extraction results of that scale to obtain the feature fusion result of that scale; then, based on the feature fusion results of these scales, at least one bounding box and the key point information corresponding to each bounding box are predicted simultaneously, which helps to improve the detection accuracy.

[0174] In fact, to better improve the detection effect, the key point detection method provided in this application can utilize a multi-information detection model, such as... Figure 4 The multi-information detection model shown is implemented as follows. This multi-information detection model is used to detect input data such as... Figure 4 The input image shown is subjected to multi-information detection processing to obtain and output at least one bounding box of the input data and the key point information corresponding to each bounding box.

[0175] Furthermore, this application does not limit the implementation of the aforementioned multi-information detection model. For example, the aforementioned multi-information detection model may include a feature extraction module and a detection module. The feature extraction module can be used to perform feature extraction processing on the image to be detected, obtaining feature extraction results; the detection module can be used to predict at least one bounding box and the key point information corresponding to each bounding box based on the feature extraction results. It should be noted that the working principle of this multi-information detection model is explained in the processing flow shown in any embodiment of the key point detection method described above.

[0176] Furthermore, this application does not limit the implementation method of the feature extraction module described above. For example, in some scenarios, such as single-scale feature extraction, the feature extraction module can be implemented using a backbone network. Similarly, in some scenarios, such as multi-scale feature extraction, the feature extraction module may include multiple backbone networks, such as... Figure 4 The B3, B4, and B5 shown are examples of different backbone networks used to output feature extraction results at different scales.

[0177] Furthermore, this application does not limit the implementation method of the detection module described above. For example, in some scenarios, such as single-scale feature extraction scenarios, the detection module may include at least one detection head, such as... Figure 5 The detection head shown is an example. In some scenarios, such as multi-scale feature extraction, this detection module includes at least multiple detection heads. Different detection heads are used to perform multi-information detection processing on feature maps of different scales, such as bounding box detection and keypoint detection.

[0178] In practice, in some scenarios, such as multi-scale feature extraction, to improve detection performance, the multi-information detection model described above includes not only Q backbone networks and Q detection heads, but also Q fusion networks. The q-th fusion network is used to fuse the output data of the q-th backbone network with the output data of some or all of the other backbone networks in the Q backbone networks, excluding the q-th backbone network itself. The q-th detection head is used to perform multi-information detection processing on the output data of the q-th fusion network. Here, q is a positive integer, q ≤ Q, and Q is a positive integer.

[0179] Furthermore, in some scenarios, such as multi-scale feature extraction, to better improve detection performance, the multi-information detection model described above includes not only Q backbone networks, Q fusion networks, and Q detection heads, but also Q 1×1 convolutional networks. The q-th 1×1 convolutional network is used to perform convolution processing on the output data of the q-th backbone network. The q-th fusion network is specifically used to fuse the output data of the q-th 1×1 convolutional network with the output data of some or all of the other 1×1 convolutional networks in the Q network except for the q-th 1×1 convolutional network. Here, q is a positive integer, q ≤ Q, and Q is a positive integer.

[0180] In addition, this application also provides an update process for the above-mentioned multi-information detection model, which may specifically include steps 61-63 below.

[0181] Step 61: Obtain the image to be detected and the corresponding ground truth annotation information. There is at least one object in the image to be detected. The ground truth annotation information is used to describe the actual position and state of each object in the image to be detected.

[0182] For details regarding the image to be detected, please refer to the relevant content in S1 above. It can be seen that, in one possible implementation, the process of acquiring the image to be detected in step 61 above can specifically be as follows: First, randomly select an image from a pre-constructed training image set as a sample image, ensuring that at least one object exists in the sample image; then determine whether the size of the sample image is equal to a preset image size. If it is equal, then the sample image is determined as the image to be detected; if it is not equal, then perform size adjustment processing on the sample image, as shown in steps 12-15 above, to obtain the image to be detected, so that the size of the image to be detected is equal to the preset image size.

[0183] The ground truth annotation information corresponding to the image to be detected is pre-annotated for the image to be detected, used to indicate the actual position and actual state of each object in the image to be detected, such as pose; and this application does not limit the implementation method of the ground truth annotation information. For example, the ground truth annotation information may include at least one ground truth box and key point information of each ground truth box. Among them, different ground truth boxes are used to describe the actual position of different objects in the image to be detected; for any ground truth box, the key point information of the ground truth box is used to describe the actual position of each key point in the object framed by the ground truth box in the image to be detected.

[0184] In practice, in some scenarios, such as multi-scale feature extraction, to better improve model training performance, this application also provides a possible implementation of the ground truth annotation information corresponding to the image to be detected mentioned above. In this method, the ground truth annotation information may include bounding box annotation information at multiple scales and key point annotation information at multiple scales. Specifically, the bounding box annotation information at the q-th scale is used to represent the actual bounding box related information presented by the image to be detected at the q-th scale, such as the actual location, actual confidence level, and actual category. The implementation method of the bounding box annotation information at the q-th scale is similar to the implementation method of the bounding box prediction result at the q-th scale mentioned above. Moreover, this application does not limit the method of obtaining the bounding box annotation information at the q-th scale; for example, it can be implemented by manual annotation or by other methods. The key point annotation information at the q-th scale is used to represent the actual key point related information presented by the image to be detected at the q-th scale, such as the actual location and the actual confidence level. The implementation method of the key point annotation information at the q-th scale is similar to the implementation method of the key point prediction result at the q-th scale above. Moreover, this application does not limit the acquisition method of the key point annotation information at the q-th scale. For example, it can be implemented by manual annotation or by other methods.

[0185] Step 62: Input the image to be detected into the multi-information detection model to obtain at least one bounding box output from the image to be detected and the key point information corresponding to each bounding box.

[0186] In this application, for the current round of training, after obtaining the image to be detected, the image can be input into the multi-information detection model, such as... Figure 4 The model shown enables the multi-information detection model to simultaneously perform bounding box detection and key point detection on the image to be detected, and to obtain and output at least one bounding box and key point information corresponding to each bounding box, so that the output data can be used to represent the predicted position and predicted state of each object in the image to be detected, such as the predicted pose.

[0187] Step 63: Based on at least one bounding box, the key point information corresponding to the at least one bounding box, and the ground truth annotation information corresponding to the image to be detected, update the multi-information detection model, and return to execute step 61 above and subsequent steps until the preset stopping condition is reached.

[0188] It should be noted that this application does not limit the implementation of the update process in step 63 above. For example, it may include steps 631-632 below.

[0189] Step 631: Determine the model loss of the multi-information detection model based on at least one bounding box, the key point information corresponding to the at least one bounding box, and the ground truth annotation information corresponding to the image to be detected.

[0190] The model loss of the multi-information detection model is used to characterize the performance of the multi-information detection model; and this application does not limit the calculation method of the model loss of the multi-information detection model.

[0191] In addition, in order to better improve the model training effect, this application also provides a possible implementation of step 631 above, in which step 631 may include steps a-b below.

[0192] Step a: Based on the key point information corresponding to at least one bounding box mentioned above and the ground truth annotation information corresponding to the image to be detected, determine the key point prediction loss of the multi-information detection model.

[0193] The key point prediction loss of the multi-information detection model is used to characterize the performance of the multi-information detection model in key point prediction; and this application does not limit the calculation method of the key point loss.

[0194] In addition, in order to better improve the model training effect, this application also provides a possible implementation of step a above. In this way, when the key point information corresponding to at least one bounding box above includes the prediction information of the target key point, and the ground truth annotation information corresponding to the image to be detected above includes the ground truth information of the target key point, step a may specifically include steps a1-a4 below.

[0195] Step a1: Calculate the difference between the predicted information of the target key point and the true information of the target key point, so that the difference can represent the difference between the predicted information of the target key point and the true information of the target key point.

[0196] Here, the target key point refers to any key point in the key point information corresponding to at least one bounding box in the above text, or any key point in the ground truth annotation information corresponding to the image to be detected in the above text.

[0197] The prediction information of a target key point refers to the predicted content for that target key point, such as predicted location and prediction confidence. Furthermore, this application does not limit the implementation method of the prediction information for the target key point. For example, the prediction information of the target key point may include the predicted location coordinates of the target key point, as mentioned above (x...). j y j This coordinate. For example, the prediction information for the target key point can include the predicted location coordinates of the target key point and the confidence level of those predicted location coordinates, such as the Score mentioned above. j This level of confidence.

[0198] The truth information of the target key point is used to describe the actual state of the target key point, such as the actual location and the actual confidence level; and the implementation method of the truth information of the target key point is similar to the implementation method of the prediction information of the target key point mentioned above.

[0199] Furthermore, this application does not limit the implementation of step a1 above. For example, it can be implemented using the formula (1) below.

[0200]

[0201] In the formula, D score This represents the difference between the predicted information and the true value information of a target key point; Represents the truth value information of key target points; The prediction information represents the target key point; diff(·) refers to the formula for calculating the difference between two data points, and this application does not limit the implementation of diff(·). For example, it can be implemented using any difference calculation formula, such as Euclidean distance.

[0202] Step a2: If the absolute value of the above gap is less than the preset threshold, then the predicted loss of the target key point is determined based on the first loss function and the absolute value of the gap; in the first loss function, the dependent variable and the independent variable are negatively correlated, and in the derivative function of the first loss function, the dependent variable and the independent variable are negatively correlated.

[0203] The absolute value of the above difference is used to represent the magnitude of the difference between the predicted information of the target key point and the true information of the target key point.

[0204] The preset threshold refers to a pre-determined difference threshold; moreover, this application does not limit the method of obtaining the preset threshold. For example, the preset threshold can be pre-specified by relevant personnel. Alternatively, the preset threshold can be used as a hyperparameter so that it can be determined through the training process of hyperparameters.

[0205] The first loss function is a pre-defined loss function used to guide the model to pay more attention to small changes in loss, so that the first loss function is applicable to training periods when the predicted information is closer to the true information, such as the later stages of training.

[0206] Furthermore, the first loss function described above satisfies the following constraints: the dependent variable and the independent variable are negatively correlated in the first loss function, and the derivative of the first loss function is also negatively correlated with the independent variable, so that the smaller the independent variable is, the larger the gradient and the greater the loss. The derivative of the first loss function represents the derivative of the first loss function under different independent variables.

[0207] Furthermore, this application does not limit the implementation of the first loss function described above. For example, the first loss function can be implemented using the loss function shown in formula (2) below, so that the derivative of the first loss function is shown in formula (3) below.

[0208]

[0209] In the formula, |D score | represents the absolute value of the difference mentioned above; w represents the preset threshold mentioned above; ln(·) represents the natural logarithm function; c represents a predetermined parameter, which can be specified in advance by relevant personnel or used as a hyperparameter so that c can be determined through the hyperparameter training process; L(|D score |) represents the predicted loss for the target key point.

[0210] The prediction loss of the target key point is used to characterize the prediction performance of the multi-information detection model for the target key point.

[0211] Based on the relevant content in section 63112 above, for the current training round, if the absolute value of the difference between the predicted information of the target key point and the ground truth information of the target key point is less than a preset threshold, it can be determined that the prediction error for the target key point is relatively small. Therefore, the first loss function can be used to calculate the prediction loss of the target key point, so that the prediction loss is negatively correlated with the absolute value of the difference, and the corresponding gradient is negatively correlated with the absolute value of the difference. Thus, as the absolute value of the difference becomes smaller, the prediction loss becomes larger, and the corresponding gradient becomes larger, thereby enabling the model to pay more attention to small changes in loss, which is beneficial to improving the prediction accuracy of the model for the target key point.

[0212] Step a3: If the absolute value of the above gap is not less than the preset threshold, then the predicted loss of the target key point is determined based on the second loss function and the absolute value of the gap; the dependent variable and the independent variable in the second loss function are positively correlated, and the derivative of the second loss function is a constant.

[0213] The second loss function refers to a pre-defined loss function used to guide the model to more stably pass through the early stages of training, so that the second loss function is applicable to training periods where the difference between predicted information and true information is relatively large, such as the early stage of training.

[0214] In addition, the second loss function mentioned above can satisfy the following constraints: the dependent variable and the independent variable are positively correlated in the second loss function, and the derivative of the second loss function is a constant, so as to ensure that the gradient is constant and the loss is not too large, thereby effectively avoiding the occurrence of training instability due to too large loss in the early stage of training.

[0215] Furthermore, this application does not limit the implementation of the second loss function described above. For example, the second loss function can be implemented using the loss function shown in formulas (4)-(5) below, so that the derivative of the first loss function is shown in formula (6) below.

[0216] L(|D score |)=|D score |-d ,|D score |≥w (4)

[0217]

[0218] L′(|D score |)=1 (6)

[0219] Based on the relevant content in section 63113 above, for the current training round, if the absolute value of the difference between the predicted information of the target key point and the true information of the target key point is greater than or equal to a preset threshold, it can be determined that the prediction error for the target key point is relatively large. Therefore, the second loss function can be used to calculate the prediction loss for the target key point so that the prediction loss is not too large and the corresponding gradient remains constant. This can effectively avoid the instability of learning caused by the large loss in the early stage of learning the prediction ability for the target key point, thereby enabling the model to learn the prediction ability for the target key point more stably, which is conducive to improving the model performance.

[0220] Step a4: Determine the key point prediction loss of the multi-information detection model based on the prediction loss of the target key points.

[0221] It should be noted that this application does not limit the implementation of step a4 above. For example, it can be implemented using formula (7) below.

[0222]

[0223] In the formula, L kpts This represents the keypoint prediction loss of the multi-information detection model, so that the L kpts This demonstrates the performance of the multi-information detection model in keypoint detection; D n,m This represents the difference between the predicted information of the nth keypoint corresponding to the mth bounding box and the ground truth information of the nth keypoint, and this D n,m The calculation method is similar to that of D above. score The calculation method; L(|D n,m |) represents the prediction loss for the nth keypoint, and the L(|D) n,m The calculation method for |) is similar to that of L(|D) mentioned above. score The calculation method of |); N represents the number of key points corresponding to the m-th bounding box, such as 17; M represents the number of at least one bounding box mentioned above; n is a positive integer, n≤N, N is a positive integer; m is a positive integer, m≤M, M is a positive integer.

[0224] Based on steps a1 to a4 above, for the current training round, after obtaining keypoint information corresponding to at least one bounding box and ground truth annotation information corresponding to the image to be detected, we can first determine the prediction information and ground truth information of some keypoints from this information; then, based on the prediction information and ground truth information of each keypoint, we calculate the prediction loss of each keypoint; then, we calculate the sum of the prediction losses of all keypoints as the keypoint prediction loss of the multi-information detection model, so that the keypoint prediction loss can represent the performance of the multi-information detection model in keypoint detection. Specifically, for any keypoint, the loss function corresponding to that keypoint is selected based on the prediction error of that keypoint. This ensures that the loss and its corresponding gradient are constant in the early stages of training based on that keypoint, and that the loss and its corresponding gradient increase as the prediction error decreases in the later stages of training based on that keypoint. This allows the model to be stably trained in the early stages of training based on that keypoint and to pay more attention to small changes in loss in the later stages of training based on that keypoint, thus enabling the finally trained model to have better prediction performance for that keypoint.

[0225] Based on the relevant content of step a above, for the current round of training, after obtaining the key point information corresponding to at least one bounding box and the ground truth label information corresponding to the image to be detected, the prediction information and ground truth information of the key points recorded in these two pieces of information can be used to calculate the key point prediction loss of the multi-information detection model, so that the key point prediction loss can represent the performance of the multi-information detection model in key point detection in the current round.

[0226] Step b: Determine the model loss of the multi-information detection model based on the key point prediction loss of the multi-information detection model, at least one bounding box, and the ground truth annotation information corresponding to the image to be detected.

[0227] It should be noted that this application does not limit the implementation of step b above. For example, when the ground truth annotation information corresponding to the image to be detected includes at least one ground truth box, step b can specifically be: first, determine the ground truth boxes corresponding to each bounding box from the at least one ground truth box, so that the object bounded by the bounding box and the object bounded by the ground truth box corresponding to the bounding box are the same object; then, calculate the bounding box loss of the multi-information detection model based on the at least one bounding box and the ground truth boxes corresponding to each bounding box, so that the bounding box loss can represent the performance of the multi-information detection model in bounding box detection in the current round; then, determine the model loss of the multi-information detection model based on the bounding box loss and the key point prediction loss of the multi-information detection model. It should be noted that this application does not limit the calculation method of the bounding box loss. For example, it can be implemented using any existing or future loss function that can calculate the difference between the predicted value of the bounding box and the ground truth value of the bounding box, such as L1 loss.

[0228] In addition, in order to better improve the model training effect, when the ground truth annotation information corresponding to the image to be detected above includes the category ground truth of at least one ground truth box, step b above can specifically include at least steps b1-b2 below.

[0229] Step b1: Based on the classification results of at least one bounding box and the class ground truth of the corresponding ground truth boxes, calculate the classification loss of the multi-information detection model so that the classification loss can represent the performance of the multi-information detection model in classification in the current round. It should be noted that this application does not limit the calculation method of the classification loss; for example, it can be implemented using any existing or future loss function capable of calculating the difference between the predicted value and the ground truth of a class.

[0230] Step b2: Determine the model loss of the multi-information detection model based on the classification loss, the bounding box loss, and the key point prediction loss of the multi-information detection model.

[0231] It should be noted that this application does not limit the implementation of step b2 above. For example, step b2 can be implemented using formula (8) below.

[0232] L all =L cls +L bbox +L kpts (8)

[0233] In the formula, L all L represents the model loss of the multi-information detection model. cls L represents the classification (cls) loss of this multi-information detection model;bbox This represents the bounding box loss of the multi-information detection model.

[0234] Based on the relevant content of steps b1 to b2 above, for the current round of training, after obtaining the data output by the multi-information detection model for the image to be detected, such as at least one bounding box and the key point information corresponding to the at least one bounding box, the model loss of the multi-information detection model is determined according to the difference between the output data and the ground truth annotation information corresponding to the image to be detected, so that the model loss can represent the performance of the multi-information detection model in the current round.

[0235] Step 632: Update the multi-information detection model based on the model loss described above.

[0236] It should be noted that this application does not limit the implementation of step 632 above. For example, step 632 can specifically be: updating all networks in the multi-information detection model based on the model loss described above. Furthermore, in some scenarios, such as when some networks in the multi-information detection model have high data processing performance, to improve efficiency, step 632 can specifically be: updating the remaining networks in the multi-information detection model based on the model loss. It should be noted that this application does not limit the method of this update. For example, it can be implemented by updating some or all network parameters in the multi-information detection model. Alternatively, it can be implemented by adding new parameters to some or all networks in the multi-information detection model, such as through LoRa fine-tuning.

[0237] Based on the relevant content of steps 631 to 632 above, for the current round of training, after obtaining the data output by the multi-information detection model for the image to be detected, the multi-information detection model can be updated according to the difference between the output data and the ground truth annotation information corresponding to the image to be detected, so that the updated multi-information detection model has better performance, so that the next round of training process can be started based on the updated multi-information detection model to continue executing step 61 above and subsequent steps, and thus start the next round of training process for the multi-information detection model. This iterative loop continues until the preset stopping condition is met.

[0238] A preset stopping condition refers to a pre-set condition that must be met when the model training process ends; however, this application does not limit the preset stopping condition. For example, the preset stopping condition may include: the model loss of the multi-information detection model is lower than a preset loss threshold. Another example is: the rate of change of the model loss of the multi-information detection model is lower than a preset rate of change threshold. Yet another example is: the number of updates to the multi-information detection model reaches a preset number threshold.

[0239] Based on the relevant content in steps 61 to 63 above, it is known that in some scenarios, the keypoint detection method provided in this application can be implemented using a multi-information detection model. Specifically, because the multi-information detection model can use a second loss function to calculate the relevant loss of keypoints in the early stages of training, it avoids training instability due to excessive losses in the early stages. This allows the multi-information detection model to stably optimize keypoint prediction performance in the early stages of training, resulting in a better keypoint prediction performance in the final trained multi-information detection model. Furthermore, because the multi-information detection model can use a first loss function to calculate the relevant loss of keypoints in the later stages of training, it allows the model to pay more attention to subtle changes in loss, resulting in a better keypoint prediction performance in the final trained model.

[0240] Based on the key point detection method provided in the embodiments of this application, the embodiments of this application also provide a key point detection device, which is described below in conjunction with... Figure 7 Explanation and clarification will be provided. Among them, Figure 7 This is a schematic diagram of a key point detection device provided in an embodiment of this application. It should be noted that for technical details of the key point detection device provided in this embodiment, please refer to the relevant content of the key point detection method above.

[0241] like Figure 7 As shown, the key point detection device 700 provided in this application embodiment includes:

[0242] Acquisition unit 701 is used to acquire an image to be detected, wherein at least one object exists in the image to be detected;

[0243] Extraction unit 702 is used to perform feature extraction processing on the image to be detected to obtain feature extraction results;

[0244] The prediction unit 703 is used to predict at least one bounding box and key point information corresponding to each bounding box based on the feature extraction result; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object enclosed by the bounding box in the image to be detected.

[0245] In one possible implementation, the prediction unit 703 is specifically configured to: perform at least one convolutional process on the feature extraction result to obtain a processing result; simultaneously perform bounding box prediction processing and keypoint prediction processing on the processing result to obtain bounding box prediction results and keypoint prediction results; and determine the at least one bounding box and the keypoint information corresponding to each bounding box based on the bounding box prediction results and the keypoint prediction results.

[0246] In one possible implementation, the feature extraction result includes a feature map; the bounding box prediction result includes bounding box prediction information corresponding to each pixel in the feature map; and the keypoint prediction result includes keypoint prediction information corresponding to each pixel in the feature map.

[0247] The prediction unit 703 is specifically used to: construct a correspondence between the bounding box prediction information corresponding to any pixel in the feature map and the key point prediction information corresponding to the pixel; and determine the at least one bounding box and the key point information corresponding to each bounding box based on the bounding box prediction result and the correspondence.

[0248] In one possible implementation, the simultaneous prediction is achieved using a detection module; the detection module includes a convolution submodule, a bounding box prediction submodule, and a keypoint prediction submodule; the convolution submodule is used to implement the at least one convolution process; the bounding box prediction submodule is used to implement the bounding box prediction process; and the keypoint prediction submodule is used to implement the keypoint prediction process.

[0249] In one possible implementation, the extraction unit 702 is specifically used to: perform feature extraction processing on the image to be detected at multiple scales to obtain feature extraction results at the multiple scales.

[0250] In one possible implementation, the prediction unit 703 is specifically configured to: for any given scale, determine the bounding box prediction result and the key point prediction result for that scale based on the feature extraction result of that scale; and determine the at least one bounding box and the key point information corresponding to each bounding box based on the bounding box prediction results and the key point prediction results of the multiple scales.

[0251] In one possible implementation, the prediction unit 703 is specifically configured to: for any of the scales, map the bounding box prediction result of that scale to the image to be detected to obtain the bounding box mapping result corresponding to that scale; and perform non-maximum suppression (NMS) on the bounding box mapping results corresponding to the multiple scales to obtain the at least one bounding box.

[0252] In one possible implementation, the key point detection device 700 further includes:

[0253] The fusion unit is used to fuse the feature extraction results of any of the scales with the feature extraction results of one or more other scales besides the scale itself to obtain the feature fusion result of the scale.

[0254] The prediction unit 703 is specifically used to: predict at least one bounding box and key point information corresponding to each bounding box based on the feature fusion results of the multiple scales.

[0255] In one possible implementation, the feature extraction process is implemented using the feature extraction module in the multi-information detection model; the simultaneous prediction is implemented using the detection module in the multi-information detection model.

[0256] In one possible implementation, the key point detection device 700 further includes:

[0257] The determining unit is used to determine the model loss of the multi-information detection model based on the at least one bounding box, the key point information corresponding to the at least one bounding box, and the ground truth annotation information corresponding to the image to be detected; the ground truth annotation information is used to describe the actual position and state of each object in the image to be detected.

[0258] An update unit is used to update the multi-information detection model based on the model loss.

[0259] In one possible implementation, the key point information includes prediction information of the target key points, and the truth value annotation information includes the truth value information of the target key points;

[0260] The determining unit is specifically configured to: calculate the gap between the predicted information of the target key point and the true information of the target key point; if the absolute value of the gap is less than a preset threshold, determine the prediction loss of the target key point based on a first loss function and the absolute value of the gap; in the first loss function, the dependent variable and the independent variable are negatively correlated, and in the derivative function of the first loss function, the dependent variable and the independent variable are negatively correlated; if the absolute value of the gap is not less than a preset threshold, determine the prediction loss of the target key point based on a second loss function and the absolute value of the gap; in the second loss function, the dependent variable and the independent variable are positively correlated, and the derivative function of the second loss function is a constant; and determine the model loss of the multi-information detection model based on the prediction loss of the target key point.

[0261] Based on the aforementioned keypoint detection device 700, the working principle of the keypoint detection device 700 provided in this application includes: after acquiring an image to be detected that includes one or more objects, firstly, feature extraction processing is performed on the image to be detected to obtain feature extraction results, so that the feature extraction results can represent the information carried by the image to be detected, such as the position and state of each object in the image to be detected; then, based on the feature extraction results, at least one bounding box and keypoint information corresponding to each bounding box are predicted simultaneously, so that different bounding boxes are used to represent the position of different objects in the image to be detected, and for any bounding box, the keypoint information corresponding to the bounding box is used to describe the state of the object bounded by the bounding box in the image to be detected, so that these bounding boxes and the keypoint information corresponding to these bounding boxes can represent the position and state of each object in the image to be detected. In this way, object detection and keypoint detection can be performed simultaneously on an image, thereby effectively overcoming the defects of some related keypoint detection schemes, such as long time consumption or low accuracy, and thus achieving fast and accurate multi-object keypoint detection.

[0262] In addition, this application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device performs any implementation of the key point detection method provided in this application.

[0263] See Figure 8 This illustration shows a structural schematic of an electronic device 800 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0264] like Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0265] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0266] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0267] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0268] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the key point detection method provided in this application.

[0269] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0270] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0271] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0272] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0273] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0274] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0275] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0276] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0277] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0278] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0279] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0280] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0281] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0282] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A key point detection method, characterized in that, The method includes: Obtain an image to be detected, wherein at least one object exists in the image to be detected; The image to be detected is subjected to feature extraction processing to obtain the feature extraction result; Based on the feature extraction results, at least one bounding box and key point information corresponding to each bounding box are predicted simultaneously; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object enclosed by the bounding box in the image to be detected.

2. The method according to claim 1, characterized in that, The step of simultaneously predicting at least one bounding box and key point information corresponding to each bounding box based on the feature extraction results includes: The feature extraction results are subjected to at least one convolutional process to obtain the processing result; The processing results are simultaneously subjected to bounding box prediction and keypoint prediction to obtain bounding box prediction results and keypoint prediction results. Based on the bounding box prediction results and the key point prediction results, at least one bounding box and key point information corresponding to each bounding box are determined.

3. The method according to claim 2, characterized in that, The feature extraction results include feature maps; The bounding box prediction result includes the bounding box prediction information corresponding to each pixel in the feature map; The key point prediction result includes key point prediction information corresponding to each pixel in the feature map. The method further includes: For any pixel in the feature map, construct the correspondence between the bounding box prediction information corresponding to the pixel and the key point prediction information corresponding to the pixel; The step of determining the at least one bounding box and the key point information corresponding to each bounding box based on the bounding box prediction result and the key point prediction result includes: Based on the bounding box prediction results and the correspondence, the at least one bounding box and the key point information corresponding to each bounding box are determined.

4. The method according to claim 2, characterized in that, The simultaneous prediction is achieved using a detection module; The detection module includes a convolution submodule, a bounding box prediction submodule, and a key point prediction submodule; The convolution submodule is used to implement the at least one convolution process; The bounding box prediction submodule is used to implement the bounding box prediction processing; The key point prediction submodule is used to implement the key point prediction process.

5. The method according to claim 1, characterized in that, The feature extraction process on the image to be detected, to obtain the feature extraction result, includes: The image to be detected is subjected to feature extraction processing at multiple scales to obtain the feature extraction results at the multiple scales.

6. The method according to claim 5, characterized in that, The process of determining the at least one bounding box and the key point information corresponding to each bounding box includes: For any of the scales, based on the feature extraction results of that scale, the bounding box prediction result and the key point prediction result of that scale are determined; Based on the bounding box prediction results at multiple scales and the key point prediction results at multiple scales, at least one bounding box and the key point information corresponding to each bounding box are determined.

7. The method according to claim 6, characterized in that, The process of determining the at least one bounding box includes: For any of the scales, the bounding box prediction result for that scale is mapped to the image to be detected to obtain the bounding box mapping result corresponding to that scale; Non-maximum suppression (NMS) is applied to the bounding box mapping results corresponding to the multiple scales to obtain at least one bounding box.

8. The method according to claim 5, characterized in that, After obtaining the feature extraction results at the multiple scales, the method further includes: For any of the scales, the feature extraction result of that scale is fused with the feature extraction results of one or more other scales besides that scale to obtain the feature fusion result of that scale. The step of simultaneously predicting at least one bounding box and key point information corresponding to each bounding box based on the feature extraction results includes: Based on the feature fusion results of the multiple scales, at least one bounding box and key point information corresponding to each bounding box are predicted simultaneously.

9. The method according to any one of claims 1-8, characterized in that, The feature extraction process is implemented using the feature extraction module in the multi-information detection model; The simultaneous prediction is achieved using the detection module in the multi-information detection model.

10. The method according to claim 9, characterized in that, The method further includes: The model loss of the multi-information detection model is determined based on the at least one bounding box, the key point information corresponding to the at least one bounding box, and the ground truth annotation information corresponding to the image to be detected; the ground truth annotation information is used to describe the actual position and state of each object in the image to be detected. The multi-information detection model is updated based on the model loss.

11. The method according to claim 10, characterized in that, The key point information includes the prediction information of the target key points, and the truth value annotation information includes the truth value information of the target key points. The process of determining the model loss includes: Calculate the difference between the predicted information and the true information of the target key point; If the absolute value of the gap is less than a preset threshold, the predicted loss of the target key point is determined based on the first loss function and the absolute value of the gap; in the first loss function, the dependent variable and the independent variable are negatively correlated, and in the derivative function of the first loss function, the dependent variable and the independent variable are negatively correlated. If the absolute value of the gap is not less than a preset threshold, the predicted loss of the target key point is determined based on the second loss function and the absolute value of the gap; in the second loss function, the dependent variable and the independent variable are positively correlated, and the derivative of the second loss function is a constant; Based on the prediction loss of the target key points, the model loss of the multi-information detection model is determined.

12. A key point detection device, characterized in that, include: An acquisition unit is used to acquire an image to be detected, wherein at least one object exists in the image to be detected; The extraction unit is used to perform feature extraction processing on the image to be detected to obtain feature extraction results; The prediction unit is used to predict at least one bounding box and key point information corresponding to each bounding box based on the feature extraction results; different bounding boxes are used to represent the positions of different objects in the image to be detected; for any bounding box, the key point information corresponding to the bounding box is used to describe the state of the object framed by the bounding box in the image to be detected.

13. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-11.

14. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-11.

15. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method according to any one of claims 1-11.

Images