Key point detection method, computer device and storage medium

Abstract

This application discloses a keypoint detection method, computer device, and storage medium. The method includes: acquiring at least one target detection feature map corresponding to an image to be detected; performing comprehensive processing on the target detection feature map to obtain a comprehensive detection feature map, wherein the comprehensive detection feature map includes target features set according to channel dimensions and perceptron parameters; determining candidate targets based on the target features; setting a perceptron using the perceptron parameters corresponding to the candidate targets, and using the set perceptron to detect the target keypoints of the candidate targets. This approach can improve the accuracy of keypoint detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a key point detection method, a computer device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of computer technology, key point detection is an important branch of computer vision and has received much attention in the field.

[0003] Keypoint detection can be used to detect key points of target objects in images and is widely used in scenarios such as pose estimation, target tracking, autonomous driving, film and animation, human-computer interaction, video surveillance, and motion analysis.

[0004] In existing technologies, target key point detection is usually performed using a top-down or bottom-up approach. However, due to the influence of factors such as the background of the image being detected, the shooting environment, and the quality of the image to be detected, the accuracy of key point detection is often low. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide a key point detection method, computer equipment, and storage medium that can improve the accuracy of key point detection.

[0006] To address the aforementioned issues, the first aspect of this application provides a keypoint detection method, comprising: acquiring at least one target detection feature map corresponding to an image to be detected; performing comprehensive processing on the target detection feature map to obtain a comprehensive detection feature map, wherein the comprehensive detection feature map includes target features set according to channel dimensions and perceptron parameters; determining candidate targets based on the target features; setting a perceptron using the perceptron parameters corresponding to the candidate targets, and detecting target keypoints of the candidate targets using the set perceptron.

[0007] To address the aforementioned problems, a second aspect of this application provides a computer device comprising a memory and a processor coupled to each other, wherein the memory stores program data and the processor executes the program data to implement any step of the aforementioned key point detection method.

[0008] To address the aforementioned problems, a third aspect of this application provides a computer-readable storage medium storing program data executable by a processor, the program data being used to implement any step of the aforementioned key point detection method.

[0009] The above scheme obtains at least one target detection feature map corresponding to the image to be detected, performs comprehensive processing on the target detection feature map to obtain a comprehensive detection feature map, determines candidate targets based on target features, sets the perceptron parameters corresponding to the candidate targets, and uses the set perceptron to detect the target key points of the candidate targets. This integrates the target detection task and the key point detection task into one model, allowing the two tasks to share the feature extraction network, avoiding repeated feature extraction, and performing key point detection more simply and quickly. Since the comprehensive detection feature map obtained during the target detection process includes target features set according to the channel dimension and perceptron parameters, key point detection is performed through the perceptron set by the perceptron parameters, allowing the model to complete the target detection and key point detection tasks simultaneously. Furthermore, since the perceptron parameters encode the unique feature information of the target object, key point detection can be performed independently for each target object, improving the accuracy of key point detection. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0011] Figure 1 This is a flowchart illustrating the first embodiment of the training method for the image classification model of this application;

[0012] Figure 2 This is a schematic diagram of an embodiment of the image to be detected in this application;

[0013] Figure 3 This is a schematic diagram of the structure of an embodiment of the key point detection model of this application.

[0014] Figure 4 This is a schematic diagram of another embodiment of the key point detection model of this application;

[0015] Figure 5 This application Figure 1 A flowchart illustrating an embodiment of step S11;

[0016] Figure 6 This application Figure 1 A flowchart illustrating an embodiment of step S12;

[0017] Figure 7 This is a comprehensive detection feature of this application. Figure 1 Schematic diagram of the embodiment;

[0018] Figure 8 This application Figure 1 A flowchart illustrating an embodiment of step S14;

[0019] Figure 9 This is the second perceptual feature of this application. Figure 1 Schematic diagram of the embodiment;

[0020] Figure 10 This is a schematic diagram of an embodiment of the key objective of this application;

[0021] Figure 11 This is a schematic diagram of the structure of an embodiment of the key point detection model of this application;

[0022] Figure 12 This is a schematic diagram of an embodiment of key point related annotation in a sample image of this application;

[0023] Figure 13 This is a schematic diagram of the structure of an embodiment of the computer device of this application;

[0024] Figure 14 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0027] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] This application provides the following embodiments, and each embodiment is described in detail below.

[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the key point detection method of this application. The method may include the following steps:

[0031] S11: Obtain at least one target detection feature map corresponding to the image to be detected.

[0032] The image to be detected contains at least one target object that needs to be detected by key points. The target object can be a person, vehicle, animal, or object. The target object can be a target of interest in the application scenario. In actual application, the specific physical object referred to by the target object can be determined according to actual needs. This application does not impose any restrictions on this.

[0033] Please see Figure 2 This application takes a human as the target object as an example, but the image to be detected may contain multiple target objects.

[0034] Optionally, the image to be detected can be an image frame of a target video, wherein the target video can be a video stream obtained by capturing the target area using a camera device. The specific content of the aforementioned target area is not limited; those skilled in the art can set it according to actual needs. For example, the target area can be, but is not limited to, any one or more of the following: shopping mall, store, entrance / exit of a certain area, street venue, hospital, station, waiting room, airport, etc.; the aforementioned target video can contain one or more video image frames.

[0035] Feature extraction can be performed on the image to be detected to obtain at least one target detection feature map corresponding to the image to be detected. This application does not limit the method of feature extraction.

[0036] Optionally, multiple target detection feature maps corresponding to the image to be detected can be obtained. These multiple target detection feature maps may have different scales, or different downsampling factors relative to the image to be detected.

[0037] Optionally, please refer to Figures 3 to 4 A keypoint detection model can be used to detect keypoints in the image to be detected. The keypoint detection model includes a feature extraction module, an object detection module, and a keypoint detection module. The specific details of each module can be found in the following embodiments.

[0038] Optionally, the key point detection model includes a feature extraction module, which can be used to obtain at least one target detection feature map corresponding to the image to be detected.

[0039] In some embodiments, please refer to Figure 5 This embodiment can further extend step S11 of the above embodiment. Obtaining at least one target detection feature map corresponding to the image to be detected may include the following steps:

[0040] S111: Use the backbone network to extract downsampled features from the image to be detected, and obtain multiple sampled feature maps.

[0041] The feature extraction module can include a backbone network and a feature pyramid network. The backbone network can be a lightweight backbone network like MobileNetV2. By using the backbone network to perform downsampling feature extraction on the image to be detected, multiple sampled feature maps can be obtained.

[0042] Optionally, the multiple sampled feature maps may have different scales or different downsampling factors. The downsampling factors of the multiple sampled feature maps can be set based on the image to be detected, the target object to be detected, etc., and this application does not impose any restrictions on this.

[0043] For example, a backbone network can be used to extract downsampled features from the image to be detected, resulting in three sampled feature maps, C3, C4, and C5, with downsampling factors of 8x, 16x, and 32x respectively. Here, the downsampling factor is relative to the image to be detected.

[0044] S112: Use a feature pyramid network to perform pyramid feature fusion on multiple sampled feature maps to obtain at least one target detection feature map.

[0045] Since multiple sampled feature maps contain different semantic features of the image to be detected, the shallow sampled feature map (such as C3) contains richer feature details and can be fused with the deep sampled feature maps (such as C4 and C5) to obtain a feature map that fuses multi-scale semantic features.

[0046] Alternatively, multiple sampled feature maps can be input into a Feature Pyramid Network (FPN), and the feature pyramid network can be used to perform pyramid feature fusion on the multiple sampled feature maps to obtain at least one target detection feature map.

[0047] Optionally, feature fusion can be achieved through addition or superposition. For example, sampling feature maps C3 and C4 can be fused by upsampling C4 to make it the same scale as C3, and then adding or superimposing them to obtain the target detection feature map P3 for pyramid feature fusion. Similarly, multiple target detection feature maps P3, P4, and P5 can be extracted. The downsampling factors for the three target detection feature maps P3, P4, and P5 are 8x, 16x, and 32x, respectively.

[0048] In this embodiment, feature extraction is performed in the manner described above. By using a backbone network to obtain multiple sampled feature maps of the image to be detected, the feature extraction model can be made smaller and the feature extraction accuracy can be improved. In addition, a feature pyramid network is used to upsample the features at the bottom layer of the multiple sampled feature maps and fuse them with the bottom layer features to obtain high-resolution, semantically strong features, which can enhance feature extraction.

[0049] S12: Perform comprehensive processing on the target detection feature map to obtain a comprehensive detection feature map, which includes target features set according to the channel dimension and perceptron parameters.

[0050] At least one target detection feature map can be comprehensively processed, such as performing target detection on the target detection feature map to obtain a comprehensive detection feature map.

[0051] Optionally, the comprehensive detection feature map may include target features and perceptron parameters set according to the channel dimension. The perceptron parameters may represent feature information of multiple channel dimensions of the comprehensive detection feature map, and the target features may represent the detection result of the target object contained in the image to be detected or the target detection feature map.

[0052] Optionally, a keypoint detection model can be used to detect keypoints in the image to be detected. The keypoint detection model includes an object detection module, which can be used to comprehensively process the object detection feature map to obtain a comprehensive detection feature map.

[0053] In some embodiments, please refer to Figure 6 This embodiment can further extend step S12 of the above embodiment. To obtain a comprehensive detection feature map by comprehensively processing the target detection feature map, this embodiment may include the following steps:

[0054] S121: Perform convolution operations on the target detection feature map using depthwise separable convolutional layers.

[0055] The object detection module includes at least one depth-separable convolutional layer and a comprehensive processing convolutional layer.

[0056] In this process, at least one depthwise separable convolutional layer (e.g., a kernel size of 5×5 and a stride of 1) can be used to perform convolution operations on the target detection feature map to obtain a depthwise separable feature map.

[0057] Optionally, multiple object detection feature maps can be convolved sequentially using at least one depthwise separable convolutional layer to obtain multiple depthwise separable feature maps.

[0058] Optionally, the object detection module includes two cascaded depthwise separable convolutional layers, which perform convolution operations on multiple object detection feature maps to obtain multiple depthwise separable feature maps. For example, two consecutive depthwise separable convolutional layers (e.g., kernel size 5×5, stride 1) are used to further extract features from the object detection feature maps (P3, P4, and P5) to obtain depthwise separable feature maps F1, F2, and F3.

[0059] Optionally, each depthwise separable convolutional layer is followed by a batch normalization layer (BN layer) and an activation function (such as LeakyReLU) to ensure that the number of output channels of each depthwise separable convolution is consistent with the number of channels of the input features (object detection feature map).

[0060] S122: The target detection feature map is convolved using a comprehensive processing convolutional layer. The comprehensive processing convolutional layer performs convolution on the target detection feature map after it has been processed by the depth-separable convolutional layer.

[0061] A comprehensive detection feature map is obtained by convolving the object detection feature map (i.e., the depthwise separable feature map) with a comprehensive processing convolutional layer (e.g., a kernel size of 1×1 and a stride of 1). For example, convolving the depthwise separable feature maps F1, F2, and F3 with a comprehensive processing convolutional layer (e.g., a kernel size of 1×1 and a stride of 1) respectively yields three comprehensive detection feature maps at different scales. The comprehensive detection feature map represents the prediction result of the object detection module on the object detection feature map, that is, the object detection result containing the target object in the object detection feature map.

[0062] Please see Figure 7 The comprehensive detection feature map contains four target features, which can represent a detection box, and this detection box can be used to represent the target object. Target features can determine the location of the target object in the image to be detected. When there are multiple comprehensive detection feature maps, multiple comprehensive detection features of different scales can be combined to obtain the target detection result of the target object in the image to be detected. In the image to be detected, the target features indicate the region where the target object is located in the form of a detection box.

[0063] Optionally, the number of output channels of the convolutional layer is determined by at least the number of target features and the number of perceptron parameters.

[0064] Optionally, the number of output channels of the comprehensive processing convolutional layer is determined by at least the number of target features, the number of perceptron parameters, and the number of target categories. The number C of output channels of the comprehensive processing convolutional layer can be expressed as:

[0065] C = num convparams +num classes +a (1)

[0066] In the above formula (1), C represents the number of output channels of the comprehensive processing convolutional layer, and num convparams num represents the number of perceptron parameters, 'a' represents the number of target features, and num represents the number of parameters. classes This indicates the number of target categories.

[0067] The perceptron parameters can be the output features of each layer of the convolutional layer, i.e., features of each dimension. The output features of one or more convolutional layers can be used as the perceptron parameters. The number of target features can be up to four, representing the position of the target object in the image to be detected, such as the four coordinate points of the detection box: top left, bottom left, top right, and bottom right. The target category can be "person," and the number of target categories can be one.

[0068] In this embodiment, there can be one or more target detection feature maps. Since there may be target objects of different sizes in the image to be detected, multiple target detection features of different scales can be used to perform target detection separately to accommodate target objects of different sizes. In the process of using the comprehensive processing convolutional layer to perform target detection on the target detection features to obtain the comprehensive detection feature map, not only are the target features of the target object predicted, but also a set of perceptron parameters are predicted. The perceptron parameters can be used to subsequently set the perceptron for key point detection.

[0069] S13: Determine candidate targets based on target features.

[0070] Target objects can be determined based on target features. Target objects corresponding to the target features can be selected as candidate targets or extracted as candidate targets in turn, so as to perform key point detection on each detected target object.

[0071] Optionally, candidate targets can be determined based on target features, and a predetermined position (such as the center position) of the candidate target can be selected to represent the candidate target.

[0072] S14: Set the perceptron using the perceptron parameters corresponding to the candidate target, and use the set perceptron to detect the target key points of the candidate target.

[0073] By setting the perceptron parameters corresponding to the candidate target, that is, setting the perceptron parameters to the parameters contained in the perceptron, the perceptron can be used to detect the target key points of the candidate target.

[0074] In this embodiment, by acquiring at least one target detection feature map corresponding to the image to be detected, and performing comprehensive processing on the target detection feature map to obtain a comprehensive detection feature map, candidate targets are determined based on the target features. The perceptron parameters corresponding to the candidate targets are used to set the perceptron, and the target key points of the candidate targets are detected using the set perceptron. This allows the target detection task and the key point detection task to be integrated into one model, enabling the two tasks to share the feature extraction network, avoiding repeated feature extraction, and making key point detection simpler and faster. Since the comprehensive detection feature map obtained during the target detection process includes target features set according to the channel dimension and perceptron parameters, key point detection is performed using the perceptron set by the perceptron parameters, allowing the model to complete the target detection and key point detection tasks simultaneously. Furthermore, since the perceptron parameters encode the unique feature information of the target object, key point detection can be performed independently for each target object, improving the accuracy of key point detection.

[0075] Optionally, a keypoint detection model can be used to detect keypoints in the image to be detected. The keypoint detection model includes a keypoint detection module, which can be used to set the perceptron parameters corresponding to the candidate target, and then use the set perceptron to detect the target keypoints of the candidate target.

[0076] In some embodiments, please refer to Figure 8 This embodiment can be further extended to step S14 of the above embodiment. By setting the sensor using the sensor parameters corresponding to the candidate target, and using the set sensor to detect the target key points of the candidate target, this embodiment may include the following steps:

[0077] S141: Extract features from the comprehensive detection feature map along the channel dimension of the comprehensive detection feature map at the predetermined position of the candidate target.

[0078] Optionally, the perceptron may include at least one perceptual convolutional layer, which consists of perceptual weights and perceptual biases. The perceptual convolutional layer can be represented as: wx+b, where w represents the perceptual weights, b represents the perceptual biases, and x represents the features to be perceptually convolved.

[0079] Optionally, the perceptron includes a first perceptron convolutional layer and a second perceptron convolutional layer. The first perceptron convolutional layer includes a first perceptron weight and a first perceptron bias, and the second perceptron convolutional layer includes a second perceptron weight and a second perceptron bias. The number of perceptron parameters is not less than the sum of the number of the first perceptron weight, the first perceptron bias, the second perceptron weight, and the second perceptron bias.

[0080] Optionally, the first perceptual convolutional layer is a convolution with a kernel size of 1×1, a stride of 1, 8 input channels, and 8 output channels, and the second perceptual convolutional layer is a convolution with a kernel size of 1×1, a stride of 1, 8 input channels, and 1 output channel.

[0081] Optionally, the sum of the number of the first perception weight, the first perception bias, the second perception weight, and the second perception bias of the sensor, num. convparams It can be represented as:

[0082] num convparams =num conv1weight +num conv1bias +num conv2weight +num conv1bias (2)

[0083] In formula (2) above, num convlweight num represents the number of first-perception weights. conv1bias num represents the number of first-perception biases. conv2weight num represents the number of second-perception weights. conv1bias This represents the number of second perceptual biases. The first perceptual weight, first perceptual bias, second perceptual weight, and the number of second perceptual biases can be set based on specific application scenarios; for example, they can each be set to the value: num. conv1weight =64, num conv1bias =8, num conv2weight =8, num conv2bias =1.

[0084] When setting the sensor using the sensor parameters corresponding to the candidate target, at least some of the sensor parameters can be selected and set as the first perception weight, the first perception bias, the second perception weight, and the second perception bias, respectively.

[0085] First, the predetermined position of the candidate target is determined. The predetermined position can be the center position, at least one coordinate position in the target features, etc. This application takes the center position as the predetermined position for illustration, but this application does not limit it.

[0086] Then, a comprehensive detection feature map Ft is determined for the predetermined position of the candidate target. This selected comprehensive detection feature map Ft contains features with the number of output channels C. Features within a predetermined dimension range can be extracted from the comprehensive detection feature map Ft along the channel dimension at the predetermined position of the candidate target. The predetermined dimension range can be set according to the perceptron parameters or the perceptron itself; this application does not impose any restrictions on this.

[0087] Optionally, the predetermined dimensional range may include multiple consecutive dimensions distributed according to channel dimensions. For example, the first 83 dimensions of features can be extracted from the comprehensive detection feature map.

[0088] S142: Group the extracted features to serve as the first perception weight, first perception bias, second perception weight, and second perception bias of the perceptron corresponding to the candidate target.

[0089] The extracted features are grouped, with each group containing at least one dimension of features or multiple consecutive dimensions of features. For example, the first 83 dimensions of features can be grouped into: the first 64 dimensions, dimensions 65 to 73, dimensions 74 to 82, and dimension 83. Thus, the features contained in each group can be used as the first perceptual weight, the first perceptual bias, the second perceptual weight, and the second perceptual bias of the perceptron corresponding to the candidate target, respectively.

[0090] Optionally, the number of features in each group after grouping corresponds to the first perceptual weight, the first perceptual bias, the second perceptual weight, and the second perceptual bias of the corresponding perceptron.

[0091] Optionally, the grouped features can be reconstructed to match the perceptron's perceptron parameters and biases. For example, the reshape function can be used to reconstruct the array dimensions: B = Reshape(A, size). This returns an array B (the reconstructed features) with the same dimensions as the elements of A (perceptron parameters or biases), and the vector size determines the size of the reconstructed array dimensions.

[0092] Since the first perceptual convolutional layer is a 1×1 kernel with a stride of 1, 8 input channels, and 8 output channels, and the second perceptual convolutional layer is a 1×1 kernel with a stride of 1, 8 input channels, and 1 output channel, the first 64 dimensions, dimensions 65 to 73, dimensions 74 to 82, and dimension 83 can be transformed into matrices of shapes 8×8×1×1, 8×1, 1×8×1×1, and 1×1 respectively using the Reshape function. This yields the features after reconstructing the array dimensions for each set of features. Then, the 8×8×1×1, 8×1, 1×8×1×1, and 1×1 matrices are assigned the first perceptual weight, first perceptual bias, second perceptual weight, and second perceptual bias of the perceptron corresponding to the candidate target, respectively, thus completing the perceptron setup.

[0093] In some embodiments, the configured sensor can be used to detect target key points of candidate targets.

[0094] S143: Obtain the key point detection feature map corresponding to the candidate target.

[0095] The target detection feature map corresponding to the candidate target can be obtained. The target detection feature map used for key point detection can be any one of the above target detection feature maps.

[0096] Optionally, since the accuracy of key point detection is related to the size of the feature map, target detection feature map P3 is selected from the target detection feature maps (P3, P4 and P5) obtained above.

[0097] Optionally, the target detection feature map P3 corresponding to the candidate target can be obtained directly, or the candidate target can be feature extracted according to the above step S11 to obtain a target detection feature map of the same scale as the target detection feature map P3.

[0098] Next, the target detection feature map is upsampled by a preset factor to obtain the upsampled feature map K1. The preset upsampling factor can be based on the target detection feature map or the image to be detected, etc., with application settings. For example, the preset upsampling factor can be 2 times, which can yield a feature map K1 that is downsampled by 4 times relative to the image to be detected.

[0099] The upsampled feature map K1 is reduced in dimensionality along the channel dimension by using a dimensionality-reducing convolutional layer (e.g., kernel size 1×1, stride 1, output channels 8) to obtain the dimensionality-reduced feature map K2.

[0100] Then, normalization layers (such as BN layers) and activation functions (such as LeakyReLU) are used to process the dimensionality-reduced feature map in sequence to obtain the key point detection feature map, which can be used for subsequent key point detection.

[0101] S144: Perform first-perceptual convolution using the first-perceptual convolutional layer to obtain the first-perceptual feature map.

[0102] The first perceptual convolutional layer of the perceptron (e.g., kernel size 1×1, stride 1, input channels 8, output channels 8) can be used to perform a first perceptual convolution on the dimensionality-reduced feature map K2 or the key point detection feature map to obtain the first perceptual feature map.

[0103] S145: Use the second perceptual convolutional layer to perform a second perceptual convolution on the first perceptual feature map to obtain the second perceptual feature map.

[0104] The first perceptual feature map is obtained by performing a second perceptual convolution on the second perceptual convolutional layer of the perceptron (e.g., with a kernel size of 1×1, a stride of 1, 8 input channels, and 1 output channel). This second perceptual feature map can be used as a keypoint heatmap (HM) for candidate targets. Since the kernel size of the second perceptual convolutional layer of the perceptron is 1×1 and the output channel is 1, the resulting second perceptual feature map (keypoint heatmap HM) has only 1 channel.

[0105] Please see Figure 9 The second perceptual feature map can include multiple key points of the candidate target, and a large number of key points are detected in the second perceptual feature map.

[0106] S146: Select target key points from the second perceptual feature map.

[0107] Since there are many key points in the second perceptual feature map, the target key points can be selected from the second perceptual feature map (key point heatmap HM).

[0108] The second perceptual feature map can be normalized to obtain a normalized feature map. For example, the second perceptual feature map can be activated by the Sigmoid activation function and then normalized to a value between 0 and 1.

[0109] A sliding window is applied to the normalized feature map using a pooling layer (such as a max pooling layer with a kernel size of 3×3 and a stride of 1) to obtain local maxima within the sliding window. The local maxima are used as confidence scores, and the locations where the confidence scores are greater than a preset confidence threshold are selected as candidate keypoints. In other words, the location of the local maximum can be retained as a candidate keypoint for the target.

[0110] Please see Figure 10 Based on the confidence scores of candidate keypoints, a predetermined number of candidate keypoints with the highest confidence scores are selected as target keypoints. For example, the top 17 candidate keypoints with the highest confidence scores can be selected as the keypoint prediction results (i.e., target keypoints) for that candidate target.

[0111] Optionally, the feature map of the target keypoints can be upsampled to restore the same scale as the image to be detected. For example, since the prediction is performed on a feature map downsampled by 4 times relative to the image to be detected, the resulting keypoint prediction is multiplied by 4 or upsampled by 4 times to restore the scale.

[0112] In some embodiments, the above key point detection method can be applied to smart home scenarios, motion analysis scenarios, interaction scenarios, etc., and this application does not limit the application scenarios.

[0113] The above scheme, for a given image to be detected, taking a human as an example, only needs to extract the target detection features once to simultaneously perform human target detection and key point detection, thus completing the key point estimation of all human targets in the image to be detected in one step. This avoids the problem of cropping the detected human targets from the original input image and then re-inputting them into the feature extraction model for repeated feature extraction in the top-down method of human key point detection. It also avoids the problem that the time consumed by the top-down method of human key point detection increases with the number of human targets in the image, thus achieving a simpler and faster human key point detection.

[0114] Furthermore, by predicting an independent set of perceptron parameters for each candidate human target in the object detection model, and setting the perceptron using these parameters for keypoint detection, both human target detection and keypoint detection tasks can be completed simultaneously. Since a set of perceptron parameters is independently learned for each human target, encoding information unique to that target, independent keypoint detection can be performed on each human target in the image. This avoids the problem of clustering operations required to merge keypoints for each human target in a bottom-up approach, achieving fast and accurate human keypoint detection.

[0115] In some embodiments, the key point detection model can be used to detect key points in the image to be detected in order to achieve the key point detection method described above.

[0116] Please see Figure 11 , Figure 11 This is a schematic diagram of an embodiment of the keypoint detection model of this application. The keypoint detection model 20 can be used to detect keypoints in an image to be detected. The keypoint detection model 20 may include a feature extraction module 21, an object detection module 22, and a keypoint detection module 23.

[0117] The feature extraction module 21 is used to obtain at least one target detection feature map corresponding to the image to be detected.

[0118] The target detection module 22 is used to perform comprehensive processing on the target detection feature map to obtain a comprehensive detection feature map, wherein the comprehensive detection feature map includes target features set according to the channel dimension and perceptron parameters.

[0119] The key point detection module 23 is used to determine candidate targets based on target features; to set the perceptron using the perceptron parameters corresponding to the candidate targets; and to detect the target key points of the candidate targets using the set perceptron.

[0120] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0121] In addition to the aforementioned keypoint detection model, this application also provides a method for training the keypoint detection model. This method may include the following steps:

[0122] (1) Obtain a sample image set, wherein the sample image set includes several sample images containing the target object.

[0123] Among them, for the experimental area or experimental scene, the camera equipment installed at multiple angles, taking the target object as a human, takes several sample images containing human targets by using the camera equipment installed at multiple angles.

[0124] Optionally, the sample image set can be preprocessed by dividing it into a training set, a validation set, and a test set according to a preset ratio. For example, the preset ratio is 8:1:1, but this application does not limit this.

[0125] (2) Perform key point related annotation on the sample image set.

[0126] All or at least some of the target objects in the sample image (such as sample images of the validation set and / or test set) can be marked with a rectangle.

[0127] Please see Figure 12 It can also mark a set number of keypoint coordinates on the target object, such as 17 keypoint coordinates, each coordinate is (x, y, v). Where (x, y) represents the position, v represents the visibility of the keypoint, v=2 means that the keypoint has been marked and is visible, v=1 means that the keypoint has been marked but is not visible, v=0 means that the keypoint cannot be marked because it is not in the image or other reasons.

[0128] (3) Perform data augmentation processing on the training set of the labeled sample image set.

[0129] Keypoint detection methods rely on abundant training data to achieve excellent training results. Therefore, when the number of sample images containing human targets is limited, data augmentation processing can be performed on the sample images.

[0130] Optionally, data augmentation may include: performing affine transformations on sample images in the training set, such as translation, scaling, and rotation, and then stitching them together to form a single training sample image. The stitching process includes using Mosaic techniques with probability p to stitch multiple images into a single image, which can enrich the diversity of the training set sample images and reduce the learning difficulty for the keypoint detection model in dealing with diversity.

[0131] (4) Use the key point detection model to process the training sample images to obtain the training key point results, and train the key point detection model based on the training key point results.

[0132] The feature extraction module and target detection module of the key point detection model can be used to process the training sample images to obtain a training comprehensive detection feature map. The training comprehensive detection feature map includes training target features set according to the channel dimension and training perceptron parameters.

[0133] By utilizing the training comprehensive detection feature map to determine training candidate targets, C-dimensional features of preset anchor point positions (such as positions marked as positive samples) can be extracted from the training comprehensive detection feature map (i.e., prediction results). The keypoint detection module then performs subsequent keypoint extraction. During testing, C-dimensional features of the center positions of the training candidate targets can be extracted from the prediction results of the target detection module, and the keypoint detection module performs subsequent keypoint extraction.

[0134] After obtaining the key training results, the loss value of the key point detection model is obtained based on the loss function, and the key point detection model is trained based on the loss value to obtain the trained key point detection model.

[0135] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0136] Regarding the above embodiments, this application provides a computer device; please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device according to an embodiment of the present application. The computer device 30 includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled to each other. The memory 31 stores program data, and the processor 32 is used to execute the program data to implement the steps of any of the above embodiments of the key point detection method and the key point detection model training method.

[0137] In this embodiment, processor 32 can also be referred to as CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0138] The methods described in the above embodiments can be implemented as computer programs; therefore, this application proposes a computer-readable storage medium. Please refer to [link to relevant documentation]. Figure 14 , Figure 14 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 41 that can be executed by a processor. The program data 41 can be executed by the processor to implement the steps of any of the above embodiments of the key point detection method and the key point detection model training method.

[0139] In this embodiment, the computer-readable storage medium 40 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store program data 41. Alternatively, it can be a server that stores the program data 41. The server can send the stored program data 41 to other devices for execution, or it can run the stored program data 41 itself.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0144] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, and thus stored in a computer-readable storage medium for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this application is not limited to any particular hardware and software combination.

[0145] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A key point detection method, characterized in that, The method includes: Obtain at least one target detection feature map corresponding to the image to be detected; The target detection feature map is comprehensively processed to obtain a comprehensive detection feature map, wherein the comprehensive detection feature map includes target features set according to the channel dimension and perceptron parameters, including: performing a convolution operation on the target detection feature map using a depthwise separable convolutional layer, and performing a convolution operation on the target detection feature map processed by the depthwise separable convolutional layer using a comprehensive processing convolutional layer, wherein the number of output channels of the comprehensive processing convolutional layer is determined by at least the number of target features and the number of perceptron parameters; Candidate targets are determined based on the aforementioned target features; Setting the perceptron using the perceptron parameters corresponding to the candidate target, and detecting the target key points of the candidate target using the set perceptron, includes: extracting features of a predetermined dimension range from the comprehensive detection feature map along the channel dimension of the comprehensive detection feature map at a predetermined position of the candidate target; grouping the extracted features to serve as the first perception weight, the first perception bias, the second perception weight, and the second perception bias of the perceptron corresponding to the candidate target.

2. The method according to claim 1, characterized in that, The perceptron includes a first perceptron convolutional layer and a second perceptron convolutional layer. The first perceptron convolutional layer includes a first perceptron weight and a first perceptron bias. The second perceptron convolutional layer includes a second perceptron weight and a second perceptron bias. The number of perceptron parameters is not less than the sum of the number of the first perceptron weight, the first perceptron bias, the second perceptron weight, and the second perceptron bias.

3. The method according to claim 2, characterized in that, The step of setting the sensor using the sensor parameters corresponding to the candidate target, and using the set sensor to detect the target key points of the candidate target, further includes: Obtain the key point detection feature map corresponding to the candidate target; The first perceptual convolution is used to perform a first perceptual convolution to obtain a first perceptual feature map; The first perceptual feature map is subjected to a second perceptual convolution using the second perceptual convolution layer to obtain a second perceptual feature map. The target key points are selected from the second perceptual feature map.

4. The method according to claim 3, characterized in that, The step of obtaining the key point detection feature map corresponding to the candidate target includes: Obtain the target detection feature map corresponding to the candidate target; The target detection feature map is upsampled by a preset factor to obtain an upsampled feature map; The upsampled feature map is reduced in dimensionality along the channel dimension using a dimensionality-reducing convolutional layer to obtain a dimensionality-reduced feature map; The reduced-dimensional feature map is processed sequentially using a normalization layer and an activation function to obtain the key point detection feature map.

5. The method according to claim 3, characterized in that, The step of filtering the target key points from the second perceptual feature map includes: The second perceptual feature map is normalized to obtain a normalized feature map; The normalized feature map is subjected to a sliding window using a pooling layer to obtain local maxima within the sliding window; The local maxima are used as the confidence level, and the locations where the confidence level is greater than a preset confidence threshold are used as candidate key points. Based on the confidence ranking of the candidate key points, a predetermined number of candidate key points with the highest confidence are selected as the target key points.

6. The method according to claim 1, characterized in that, The step of obtaining at least one target detection feature map corresponding to the image to be detected includes: The image to be detected is downsampled and feature-extracted using a backbone network to obtain multiple sampled feature maps; The multiple sampled feature maps are fused using a feature pyramid network to obtain at least one target detection feature map.

7. A computer device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program data and the processor executing the program data to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The system stores program data that can be executed by a processor, the program data being used to implement the steps of the method according to any one of claims 1 to 6.

Images