Key point detection method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a key point detection method and apparatus, a device, and a medium. The method comprises: carrying out multi-scale feature extraction on a target image to obtain features of a plurality of scales; on the basis of a feature of the minimum scale among the features of the plurality of scales, obtaining a first detection position of a key point in the target image; and on the basis of a target-scale feature among the features of the plurality of scales, correcting the first detection position of the key point in the target image to obtain a second detection position of the key point in the target image, wherein the target-scale feature is a feature of some or all of the scales among the features of the plurality of scales.

Description

Key point detection method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese Patent Application No. 202311110469.6, filed on August 30, 2023. The entire teachings of the above application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a key point detection method, device, equipment, and medium. Background Art

[0004] In some image processing scenarios, keypoint detection is required to clearly identify the outlines of specific objects, such as people, or to perform image processing operations such as deformation based on keypoints. However, existing keypoint detection methods have low accuracy.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a key point detection method, device, equipment and medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a key point detection method, comprising: performing multi-scale feature extraction on a target image to obtain features at multiple scales; obtaining a first detection position of a key point in the target image based on a minimum-scale feature among the features at the multiple scales; and correcting the first detection position of the key point in the target image based on a target-scale feature among the features at the multiple scales to obtain a second detection position of the key point in the target image; wherein the target-scale feature is a feature of some or all of the scales among the features at the multiple scales.

[0008] In a second aspect, an embodiment of the present disclosure further provides a key point detection device, comprising: a multi-feature extraction module, configured to perform multi-scale feature extraction on a target image to obtain features at multiple scales; a first position acquisition module, configured to obtain a first detection position of a key point in the target image based on a feature at the minimum scale among the features at the multiple scales; and a first position correction module, configured to correct the first detection position of the key point in the target image based on a target scale feature among the features at the multiple scales to obtain a second detection position of the key point in the target image; wherein the target scale feature is a feature of some or all scales among the features at the multiple scales.

[0009] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement a key point detection method as provided in an embodiment of the present disclosure.

[0010] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the key point detection method provided by the embodiment of the present disclosure.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0013] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] FIG1 is a schematic diagram of a flow chart of a key point detection method provided by an embodiment of the present disclosure;

[0015] FIG2 is a schematic diagram of associated pixels provided by an embodiment of the present disclosure;

[0016] FIG3 is a schematic diagram of the structure of a key point detection model provided by an embodiment of the present disclosure;

[0017] FIG4 is a schematic diagram of the structure of a key point detection model provided by an embodiment of the present disclosure;

[0018] FIG5 is a schematic diagram of the structure of a key point detection model provided by an embodiment of the present disclosure;

[0019] FIG6 is a schematic structural diagram of a key point detection device provided by an embodiment of the present disclosure;

[0020] FIG7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0023] FIG1 is a flow chart of a key point detection method provided by an embodiment of the present disclosure. The method can be performed by a key point detection device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in FIG1 , the method mainly includes the following steps S102 to S106:

[0024] In step S102, multi-scale feature extraction is performed on the target image to obtain features at multiple scales, wherein multiple scales should be understood as two or more scales, which can be flexibly set according to needs and are not limited here.

[0025] The image to be processed for keypoint detection is the target image. The disclosed embodiments do not restrict the objects contained in the target image. Therefore, the target image can be any image requiring keypoint detection for a preset type of object, such as a human body image or a vehicle image. The disclosed embodiments do not restrict the keypoints; for example, keypoints can be the object's contour points or keypoints used to characterize key parts of the object. The keypoints to be detected can be flexibly set based on specific needs.

[0026] In practical applications, a feature extraction algorithm can be used to extract multi-scale features from a target image. For example, features can be extracted layer by layer through downsampling, thereby obtaining multiple features with scales ranging from large to small.

[0027] Step S104 : obtaining a first detection position of a key point in the target image based on a feature of the minimum scale among features of multiple scales.

[0028] The minimum-scale feature, such as a 1*1 feature, facilitates regression prediction of the first detected position of a keypoint in the target image based on the minimum-scale feature. The keypoints can be all keypoints in the target image, or they can be specified keypoints in the target image, such as keypoints of a target object of a specified type in the target image, without limitation.

[0029] Step S106 : Based on the target scale feature among the features of the multiple scales, correct the first detected position of the key point in the target image to obtain a second detected position of the key point in the target image.

[0030] The target scale feature is a feature of some or all scales among features at multiple scales. In other words, the target scale feature can be a feature of at least one scale among features at multiple scales, and the specific setting can be flexible. In practical applications, when there are multiple target scale features, each target scale feature can be processed separately, such as by establishing an association between each target scale feature and the relevant features of the key point based on a cross-attention mechanism. The processing results of the multiple target scale features are then fused to correct the first detected position of the key point based on the fusion result. Alternatively, the multiple target scale features can be fused first, and then the first detected position of the key point can be corrected directly based on the fused target scale feature. For example, the association between the fused target scale feature and the relevant features of the key point is established based on a cross-attention mechanism to correct the first detected position of the key point. In some specific implementation examples, features at multiple scales can be used as target scale features. Features at different scales have different corresponding receptive fields. In this way, the first detected position can be corrected based on multiple receptive fields. In this way, the accuracy of the first detected position regressed based on the minimum scale feature may not be high, mainly because the spatial structure of the minimum scale feature may be destroyed. To this end, the embodiment of the present disclosure also uses features of some or all scales among the features of multiple scales to correct the first detection position. That is, by combining local detail information of multiple scales, the accuracy of the first detection position can be further improved.

[0031] In summary, in the above method, the first detection position is not directly used as the final detection result. Instead, the first detection position of the key point can be further corrected by using the features of part or all scales of the target image, which can effectively improve the accuracy of the key point detection result.

[0032] In some implementation examples, the first detected position of the key point in the target image can be corrected based on the attention mechanism and the target scale features among the features of multiple scales. It is understandable that the attention mechanism can establish associations between different features, such as establishing associations between different key points based on the self-attention mechanism, and establishing associations between key point features and image scale features based on the cross-attention mechanism, thereby comprehensively improving the accuracy of key point position detection. For example, the following steps A and B can be referred to:

[0033] In step A, based on the self-attention mechanism and the first detected position of the key point in the target image, the associated feature vector corresponding to the key point is obtained; wherein the associated feature vector corresponding to each key point carries the first detected position information of the key point and the association information between the key point and other key points in the target image. The self-attention mechanism can reduce the dependence on external information and is better at capturing the internal correlation of data or features, thereby relatively reliably obtaining the association relationship between key points. In some specific implementation examples, step A can be performed with reference to the following steps A1 and A2:

[0034] Step A1: Encode the first detected position of the key point in the target image to obtain a position vector corresponding to the key point. The disclosed embodiment does not limit the encoding method of the first detected position. In actual applications, if there are N key points in total, each key point corresponds to a position vector, then N position vectors can be obtained.

[0035] Step A2, based on the self-attention mechanism, the position vector of the key point is processed to generate the associated feature vector corresponding to the key point. In practical applications, by performing self-attention processing on the position vectors of N key points based on the self-attention mechanism, the correlation between different key points can be extracted to obtain the associated feature vectors corresponding to each of the N key points. For a certain key point, the associated feature vector of the key point not only carries the position information of the key point itself, but also carries the association relationship between the key point and other key points. By introducing the self-attention mechanism to analyze the correlation between different key points, it is easier to correct the inaccurate position of the key point, and it also helps to further improve the position detection accuracy of the key point.

[0036] Step B: Correcting the first detected position of the key point in the target image based on the cross-attention mechanism, the target scale features among the features at multiple scales, and the associated feature vectors corresponding to the key points. In practical applications, processing the target scale features and the associated feature vectors corresponding to the key points based on the cross-attention mechanism can capture local information by applying attention to a local area in the image rather than the entire image. Correcting the first detected position is based on the global semantic information corresponding to the associated feature vector corresponding to the key point combined with the local detail information corresponding to the target scale features. Furthermore, the cross-attention mechanism can select representative local features and the associated features corresponding to the key points for attention processing. This improves the accuracy of key point position detection while requiring relatively low computational effort and achieving higher processing efficiency.

[0037] In some specific implementation examples, step B may be performed with reference to the following steps B1 and B3:

[0038] Step B1, determine the associated pixels corresponding to the key point in the target image. For example, the associated pixels can be the top N pixels with the highest correlation with the key point among the surrounding pixels of the key point. In practical applications, the relative position relationship between the key point and the associated pixels can be a pre-set fixed relationship. For example, taking a 3*3 pixel block as an example, the key point is taken as the center of the 3*3 pixel block, and the 8 pixels around the key point (that is, the remaining 8 pixels in the 3*3 pixel block except the center pixel) are all used as the associated pixels of the key point. The relative position relationship between the key point and the associated pixels can be a relationship learned in advance, such as, the associated pixels may not be the 8 pixels around the key point (that is, not the remaining 8 pixels in the above-mentioned 3*3 pixel block). Exemplarily, the associated pixels can be determined with reference to the following steps B1.1 and B1.2:

[0039] Step B1.1, obtain the offset parameter matrix corresponding to the key points in the target image.

[0040] The offset parameter matrix corresponding to each keypoint indicates the positional offset between a preset number of associated pixels with the highest correlation with the keypoint in the target image and the keypoint. The offset parameter matrices corresponding to different keypoints may be the same or different. For example, the offset parameter matrix may be a 3x3 parameter matrix, where each parameter in the offset parameter matrix represents the positional offset between an associated pixel and the keypoint.

[0041] For ease of understanding, refer to the associated pixel diagram shown in Figure 2. Assuming that the center point P is a key point, the eight gray dots surrounding P can be used as fixed associated pixels, or the eight black dots surrounding P can be determined as the most relevant associated pixels using an offset parameter matrix. In other words, if the offset parameter matrix is ​​not introduced, the associated pixels of each key point can be fixed in advance. However, by introducing the offset parameter matrix (which can be obtained through learning), the multiple pixels with the highest correlation degree around each key point can be more accurately selected as associated pixels. Taking the key points to be detected as human key points as an example, assuming that there are 63 key points in total, the corresponding offset parameter matrix for each key point can be learned during model training. This offset parameter matrix can be used to indicate the positional offset information between the multiple most relevant associated pixels of the key point and the key point. The offset parameter matrices corresponding to different key points can be the same or different, and the specific details can be determined through learning.

[0042] Step B1.2: Determine the associated pixels corresponding to the key points in the target image based on the offset parameter matrix. As mentioned above, based on the known offset parameter matrix and key points, the associated pixels corresponding to the key points can be directly determined.

[0043] Step B2: Determine the pixel features of the associated pixel based on the target scale feature among the features at multiple scales. The target scale feature may be some or all of the scale features among the features at multiple scales. If the target scale feature has multiple scales, the associated pixel also has multiple pixel features. In other words, the associated pixel has corresponding pixel features at each target scale.

[0044] Step B3, based on the cross-attention mechanism, the associated feature vectors corresponding to the key points and the pixel features of the associated pixels corresponding to the key points, corrects the first detection position of the key points in the target image. Based on the cross-attention mechanism, we can focus on the relationship between the associated pixels that are more closely associated with the key points and the key points, which requires lower computing resources and more efficient processing. Based on the pixel features of the associated pixels and the associated feature vectors of the key points (which can present the position information of the key points themselves and the association relationship between different key points), it can better assist in correcting the first detection position of the key points. Exemplarily, the first detection position can be corrected with reference to the following steps B3.1 and B3.2:

[0045] In step B3.1, if there are multiple associated pixels, the pixel features of the multiple associated pixels corresponding to the key point are fused to obtain a pixel fusion feature. Since each associated pixel has corresponding pixel features at different target scales, in some implementation examples, the pixel features of the multiple associated pixels at each target scale can be fused to obtain a pixel fusion feature corresponding to each target scale.

[0046] Step B3.2: Based on the cross-attention mechanism, the associated feature vector corresponding to the key point and the pixel fusion feature corresponding to the key point are processed to correct the first detection position of the key point in the target image.

[0047] If there are multiple target scales, in some implementation examples, based on the cross-attention mechanism, cross-attention processing can be performed on the associated feature vectors corresponding to the key points and the pixel fusion features corresponding to the key points corresponding to each target scale to obtain the cross-attention vector corresponding to each target scale. The cross-attention vectors corresponding to each target scale are then spliced ​​and fused to obtain the final vector. The final vector can indicate the corrected position (also called the second detection position) corresponding to the first detection position of the key point.

[0048] In practical applications, the key point detection method can be executed by a preset key point detection model. For example, referring to the structural diagram of a key point detection model shown in FIG3 , the key point detection model includes a first neural network and a second neural network; the target image is input to the first neural network, and the first neural network outputs the first detection position of the key point in the target image. The first detection position and the target scale feature of the first neural network are input to the second neural network, and the second neural network outputs the second detection position of the key point in the target image. Specifically: the first neural network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales; and based on the minimum scale feature among the features of multiple scales, the first detection position of the key point in the target image is obtained; the second neural network is used to correct the first detection position of the key point in the target image based on the target scale feature among the features of multiple scales to obtain the second detection position of the key point in the target image. The specific implementation method can be referred to the aforementioned related content and will not be repeated here.

[0049] The disclosed embodiment further provides a structural schematic diagram of a key point detection model as shown in FIG4 . Based on FIG3 , it illustrates that the first neural network includes a backbone network and a first prediction network, and the second neural network includes a position encoding network and a second prediction network; wherein the second prediction network includes a self-attention sub-network and a cross-attention sub-network. Specifically, the backbone network is used to extract multi-scale features of the target image to obtain features of multiple scales; the first prediction network is used to output the first detection position of the key point in the target image based on the minimum-scale features among the features of multiple scales; the position encoding network is used to encode the first detection position of the key point in the target image to obtain a position vector corresponding to the key point; the self-attention sub-network in the second prediction network is used to process the position vector of the key point based on the self-attention mechanism to generate an associated feature vector corresponding to the key point; the cross-attention sub-network in the second prediction network is used to correct the first detection position of the key point in the target image based on the cross-attention mechanism, the target scale features among the features of multiple scales, and the associated feature vector corresponding to the key point.

[0050] On the basis of the foregoing, the embodiment of the present disclosure further provides a structural diagram of a key point detection model as shown in Figure 5. Figure 5 simply illustrates that the backbone network has four stages, and each stage corresponds to a scale of features. Therefore, Figure 5 simply illustrates that the backbone network outputs features of four scales in total, and the features of the four scales are all input into the cross-attention sub-network as target scale features. Figure 5 further illustrates that the first prediction network includes a GAP sub-network and an FC sub-network, and based on Figure 4, Figure 5 not only illustrates that the second prediction network includes a self-attention sub-network and a cross-attention sub-network, but also illustrates the FFN sub-network and the MLP sub-network after the cross-attention sub-network. The final output result of the MLP sub-network is the second detection position. It should be noted that Figure 5 is only an implementation example. In actual applications, fewer or more sub-networks can be included in the key point detection model, and this is not limited here.

[0051] Furthermore, the embodiment of the present disclosure provides a method for constructing a key point detection model. That is, the key point detection model can be constructed through the following steps a to d:

[0052] Step a: obtaining an image sample; wherein the image sample is annotated with the real position information of the key points in the image sample.

[0053] Step b: Performing a one-stage training on the first initial network based on the image samples until a preset first training end condition is reached, thereby obtaining a first initial network after the one-stage training. The first initial network is an untrained first neural network.

[0054] Exemplarily, when the first initial network is trained in one stage based on image samples, the image samples can be input into the first initial network to obtain the first detection positions of the key points of the image samples output by the first initial network, and then based on the difference between the first detection positions of the key points of the image samples output by the first initial network and the actual position information of the key points annotated with the image samples, a preset first loss function is used to calculate the first loss value, and the parameters of the first initial network are adjusted according to the first loss value. Exemplarily, the first training end condition can be that the first loss value converges to a first preset threshold. The embodiment of the present disclosure does not limit the first loss function. Exemplarily, the first loss function can be an L2 loss function.

[0055] Step c: Performing a second-stage training on the second initial network and the first initial network after the first-stage training based on the image samples until a preset second training end condition is reached, thereby obtaining the first initial network after the second-stage training and the second initial network after the second-stage training. The second initial network is the untrained second neural network.

[0056] In which, the first initial network after one-stage training is connected in sequence with the second initial network. For example, when the second initial network and the first initial network after one-stage training are subjected to two-stage training based on the image sample, the image sample can be first input into the first initial network, and the first initial network is made to perform multi-scale feature extraction on the image sample to obtain features of multiple scales, and based on the minimum scale feature among the features of the multiple scales, the first detection position of the key point in the image sample is obtained; then, the first detection position output by the first initial network and the target scale feature among the multiple scale features extracted by the first initial network for the image sample are input into the second initial network, so that the second initial network outputs the second detection position of the key point of the image sample according to the first detection position of the key point in the image sample and the target scale feature corresponding to the image sample. Specifically, a first loss value can be calculated based on the difference between the first detection position of the key point of the image sample output by the first initial network and the actual position information of the key point annotated by the image sample, and a second loss value can be calculated based on the difference between the second detection position of the key point of the image sample and the actual position information of the key point annotated by the image sample, and then a total loss value can be calculated based on the first loss value and the second loss value. For example, the first loss value and the second loss value can be weighted to obtain the total loss value. The embodiment of the present disclosure does not restrict the weights corresponding to the first loss value and the second loss value, such as the weights of the two can be the same or different. Exemplarily, the second training end condition can be that the total loss value converges to a second preset threshold. The embodiment of the present disclosure does not restrict the first loss function and the second loss function. Exemplarily, the first loss function and the second loss function can both be L2 loss functions.

[0057] In step d, a first neural network is obtained based on the first initial network after the two-stage training, and a second neural network is obtained based on the second initial network after the two-stage training, and a key point detection model is constructed based on the first neural network and the second neural network. Specifically, the first initial network after the two-stage training can be directly used as the first neural network, and the second initial network after the two-stage training can be used as the second neural network.

[0058] In the embodiment of the present disclosure, the first initial network can be pre-trained (i.e., one-stage training) so that the first initial network can output a first detection position with a smaller error, thereby ensuring the accuracy of the first detection position and the multi-scale features extracted by the first initial network to a certain extent. On this basis, the first initial network after the one-stage training and the untrained second initial network are subjected to a second-stage training, which can effectively improve the training efficiency.

[0059] In summary, the key point detection method provided by the embodiments of the present disclosure can effectively improve the accuracy of the key point detection results by correcting the first detection position of the key point by utilizing the features of part or all scales of the target image.

[0060] Corresponding to the aforementioned key point detection method, the present disclosure further provides a key point detection device. FIG6 is a schematic structural diagram of a key point detection device provided by the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG6 , the key point detection device includes:

[0061] A multi-feature extraction module 602 is used to perform multi-scale feature extraction on the target image to obtain features at multiple scales;

[0062] A first position acquisition module 604 is configured to acquire a first detection position of a key point in a target image based on a minimum-scale feature among features of multiple scales;

[0063] A first position correction module 606 is configured to correct the first detected position of the key point in the target image based on a target scale feature among the features at multiple scales to obtain a second detected position of the key point in the target image. The target scale feature may be a feature of some or all of the scales among the features at multiple scales.

[0064] In the above device, the first detection position is not directly used as the final detection result. Instead, the first detection position of the key point can be corrected by further utilizing the features of part or all scales of the target image, which can effectively improve the accuracy of the key point detection result.

[0065] In some embodiments, the first position correction module 606 is specifically configured to correct the first detected position of the key point in the target image based on an attention mechanism and target scale features among the features of the multiple scales.

[0066] In some embodiments, the first position correction module 606 is specifically used to: obtain the associated feature vector corresponding to the key point based on the self-attention mechanism and the first detected position of the key point in the target image; wherein the associated feature vector corresponding to each key point carries the first detected position information of the key point and the association information between the key point and other key points in the target image; based on the cross-attention mechanism, the target scale feature among the features of the multiple scales and the associated feature vector corresponding to the key point, correct the first detected position of the key point in the target image.

[0067] In some embodiments, the first position correction module 606 is specifically used to: encode the first detection position of the key point in the target image to obtain the position vector corresponding to the key point; process the position vector of the key point based on the self-attention mechanism to generate an associated feature vector corresponding to the key point.

[0068] In some embodiments, the first position correction module 606 is specifically used to: determine the associated pixels corresponding to the key points in the target image; determine the pixel features of the associated pixels based on the target scale features among the multiple scale features; and correct the first detected position of the key points in the target image based on the cross-attention mechanism, the associated feature vectors corresponding to the key points, and the pixel features of the associated pixels corresponding to the key points.

[0069] In some embodiments, the first position correction module 606 is specifically used to: obtain an offset parameter matrix corresponding to the key point in the target image; wherein the offset parameter matrix corresponding to each key point is used to indicate the position offset information between a preset number of associated pixels with the highest correlation with the key point in the target image and the key point; the offset parameter matrices corresponding to different key points are the same or different; based on the offset parameter matrix, determine the associated pixels corresponding to the key point in the target image.

[0070] In some embodiments, the first position correction module 606 is specifically used to: when the number of associated pixels is multiple, fuse the pixel features of multiple associated pixels corresponding to the key point to obtain pixel fusion features; based on the cross-attention mechanism, process the associated feature vector corresponding to the key point and the pixel fusion features corresponding to the key point to correct the first detection position of the key point in the target image.

[0071] In some embodiments, the key point detection device is applied to a preset key point detection model, and the key point detection model includes a first neural network and a second neural network;

[0072] The first neural network is used to perform multi-scale feature extraction on the target image to obtain features at multiple scales; and, based on a minimum-scale feature among the features at the multiple scales, obtain a first detection position of a key point in the target image;

[0073] The second neural network is used to correct the first detected position of the key point in the target image based on the target scale feature among the multiple scale features to obtain the second detected position of the key point in the target image.

[0074] In some embodiments, the key point detection model is constructed by the following steps: obtaining an image sample; wherein the image sample is annotated with the real position information of the key points in the image sample; performing a first-stage training on a first initial network based on the image sample until a preset first training end condition is reached, thereby obtaining a first initial network after the first-stage training; performing a second-stage training on a second initial network and the first initial network after the first-stage training based on the image sample until a preset second training end condition is reached, thereby obtaining a first initial network after the second-stage training and a second initial network after the second-stage training;

[0075] A first neural network is obtained based on the first initial network after the two-stage training, and a second neural network is obtained based on the second initial network after the two-stage training, and a key point detection model is constructed based on the first neural network and the second neural network.

[0076] The key point detection device provided in the embodiments of the present disclosure can execute the key point detection method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0078] An embodiment of the present disclosure provides an electronic device, which includes: a storage device storing a computer program; and a processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0079] Reference is now made to FIG7 , which illustrates a schematic diagram of the structure of an electronic device 700 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG7 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0080] As shown in Figure 7, electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of electronic device 700 are also stored in RAM 703. Processing device 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0081] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 7 illustrates the electronic device 700 with various devices, it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0082] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0083] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0084] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is enabled to execute the key point detection method provided by the embodiment of the present disclosure.

[0085] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0086] The embodiments of the present disclosure further provide a computer program product, including a computer program / instruction, which implements the key point detection method in the embodiments of the present disclosure when executed by a processor.

[0087] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0088] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0089] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0090] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0091] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0092] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A key point detection method, comprising: Perform multi-scale feature extraction on the target image to obtain features at multiple scales; Acquire a first detection position of a key point in the target image based on a feature of a minimum scale among the features of the multiple scales; Based on the target scale feature among the features of the multiple scales, the first detected position of the key point in the target image is corrected to obtain the second detected position of the key point in the target image; wherein the target scale feature is a feature of some or all scales among the features of the multiple scales.

2. The method according to claim 1, wherein the correcting the first detected position of the key point in the target image based on the target scale feature among the features of the multiple scales comprises: Based on the attention mechanism and the target scale feature among the features of the multiple scales, a first detected position of the key point in the target image is corrected.

3. The method according to claim 2, wherein the first detected position of the key point in the target image is corrected based on the attention mechanism and the target scale feature among the features of the multiple scales, comprising: Based on the self-attention mechanism and the first detected position of the key point in the target image, obtaining the associated feature vector corresponding to the key point; wherein the associated feature vector corresponding to each key point carries the first detected position information of the key point and the association information between the key point and other key points in the target image; Based on a cross-attention mechanism, a target scale feature among the features of the multiple scales, and an associated feature vector corresponding to the key point, a first detected position of the key point in the target image is corrected.

4. The method according to claim 3, wherein the step of obtaining the associated feature vector corresponding to the key point based on the self-attention mechanism and the first detected position of the key point in the target image comprises: Encoding a first detected position of a key point in the target image to obtain a position vector corresponding to the key point; The position vector of the key point is processed based on the self-attention mechanism to generate an associated feature vector corresponding to the key point.

5. The method according to claim 3, wherein the first detected position of the key point in the target image is corrected based on the cross attention mechanism, the target scale feature among the features of the multiple scales, and the associated feature vector corresponding to the key point, comprising: Determine the associated pixels corresponding to the key points in the target image; Determining a pixel feature of the associated pixel based on a target scale feature among the features of the multiple scales; Based on the cross-attention mechanism, the associated feature vector corresponding to the key point and the pixel features of the associated pixels corresponding to the key point, the first detection position of the key point in the target image is corrected.

6. The method according to claim 5, wherein the determining of associated pixels corresponding to key points in the target image comprises: Obtaining an offset parameter matrix corresponding to a key point in the target image; wherein the offset parameter matrix corresponding to each key point is used to indicate position offset information between a preset number of associated pixels having the highest correlation with the key point in the target image and the key point; and the offset parameter matrices corresponding to different key points may be the same or different; Based on the offset parameter matrix, associated pixels corresponding to key points in the target image are determined.

7. The method according to claim 5, wherein the first detected position of the key point in the target image is corrected based on the cross attention mechanism, the associated feature vector corresponding to the key point, and the pixel features of the associated pixels corresponding to the key point, comprising: In the case where there are multiple associated pixels, the pixel features of the multiple associated pixels corresponding to the key point are fused to obtain a pixel fusion feature; Based on the cross-attention mechanism, the associated feature vector corresponding to the key point and the pixel fusion feature corresponding to the key point are processed to correct the first detection position of the key point in the target image.

8. The method according to claim 1, wherein the key point detection method is performed by a preset key point detection model, and the key point detection model includes a first neural network and a second neural network; in, The first neural network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales; and, based on the feature of the smallest scale among the features of the multiple scales, obtain a first detection position of a key point in the target image; The second neural network is used to correct the first detection position of the key point in the target image based on the target scale feature among the multiple scale features to obtain the second detection position of the key point in the target image.

9. The method according to claim 8, wherein the key point detection model is constructed by the following steps: Get an image sample; where, The image samples are annotated with real position information of key points in the image samples; Performing one-stage training on the first initial network based on the image samples until a preset first training end condition is reached, thereby obtaining a first initial network after the one-stage training; Based on the image samples, the second initial network and the first initial network after the first stage training are trained in two stages until a preset second training end condition is reached, thereby obtaining the first initial network after the two stage training and the second initial network after the two stage training; A first neural network is obtained based on the first initial network after the two-stage training, and a second neural network is obtained based on the second initial network after the two-stage training, and a key point detection model is constructed based on the first neural network and the second neural network.

10. A key point detection device, comprising: A multi-feature extraction module is used to extract multi-scale features from the target image to obtain features at multiple scales; A first position acquisition module, configured to acquire a first detection position of a key point in the target image based on a feature of a minimum scale among the features of the multiple scales; a first position correction module, configured to correct a first detected position of a key point in the target image based on a target scale feature among the features of the multiple scales, so as to obtain a second detected position of the key point in the target image; wherein the target scale feature is a feature of some or all scales among the features of the multiple scales.

11. An electronic device, comprising: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the key point detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute the key point detection method described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the key point detection method according to any one of claims 1 to 9 is implemented.

Images