Key point detection method and device, equipment and medium

Abstract

The embodiment of the invention relates to a key point detection method and device, equipment and a medium, and the method comprises the steps: carrying out the multi-scale feature extraction of a target image, and obtaining the features of a plurality of scales; obtaining a first detection position of a key point in the target image based on the feature of the minimum scale in the features of the plurality of scales; correcting a first detection position of a key point in the target image based on a target scale feature in the multiple scale features to obtain a second detection position of the key point in the target image; wherein the target scale features are features of partial scales or all scales in the features of the multiple scales. According to the embodiment of the invention, the accuracy of the key point detection result can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a key point detection method, device, equipment and medium. Background Art

[0002] In some image processing scenarios, it is necessary to detect key points in an image in order to clearly identify the outline of a specific type of target object such as a person, or to perform processing operations such as deformation on the image based on the key points. However, the accuracy of existing key point detection methods is low. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a key point detection method, device, equipment and medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a key point detection method, the method comprising: performing multi-scale feature extraction on a target image to obtain features of multiple scales; acquiring a first detection position of a key point in the target image based on a feature of a minimum scale among the features of the multiple scales; and correcting the first detection position of the key point in the target image based on a target scale feature among the features of the multiple scales to obtain a second detection position of the key point in the target image; wherein the target scale feature is a feature of some or all of the scales among the features of the multiple scales.

[0005] In a second aspect, an embodiment of the present disclosure further provides a key point detection device, comprising: a multi-feature extraction module, used to perform multi-scale feature extraction on a target image to obtain features of multiple scales; a first position acquisition module, used to obtain a first detection position of a key point in the target image based on a feature of the smallest scale among the features of the multiple scales; a first position correction module, used to correct the first detection position of the key point in the target image based on a target scale feature among the features of the multiple scales to obtain a second detection position of the key point in the target image; wherein the target scale feature is a feature of some or all scales among the features of the multiple scales.

[0006] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a key point detection method as provided in an embodiment of the present disclosure.

[0007] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the key point detection method provided by the embodiment of the present disclosure.

[0008] The above technical solution provided by the embodiment of the present disclosure can perform multi-scale feature extraction on the target image, obtain features of multiple scales, and obtain the first detection position of the key point in the target image based on the feature of the minimum scale among the features of multiple scales; on this basis, the first detection position of the key point in the target image can be further corrected based on the features of some scales or all scales among the features of multiple scales, thereby obtaining the second detection position of the key point in the target image. In the above method, the first detection position is not directly used as the final detection result, but the first detection position of the key point can be further corrected by using the features of some scales or all scales of the target image, which can effectively improve the accuracy of the key point detection result.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] Figure 1 A flowchart of a key point detection method provided by an embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram of associated pixels provided in an embodiment of the present disclosure;

[0014] Figure 3 A schematic diagram of the structure of a key point detection model provided in an embodiment of the present disclosure;

[0015] Figure 4 A structural diagram of a key point detection model provided in an embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram of the structure of a key point detection model provided in an embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram of the structure of a key point detection device provided in an embodiment of the present disclosure;

[0018] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0021] Figure 1 The present invention provides a flowchart of a key point detection method provided by an embodiment of the present invention. The method can be performed by a key point detection device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0022] Step S102, multi-scale feature extraction is performed on the target image to obtain features of multiple scales, wherein multiple scales should be understood as two or more scales, which can be flexibly set according to needs and are not limited here.

[0023] The image to be processed for key point detection is the target image. The disclosed embodiment does not limit the objects contained in the target image. Therefore, the target image can be any image that needs to perform key point detection on a preset type of object, such as a human body image, a vehicle image, etc. The disclosed embodiment does not limit the key points. For example, the key points can be the contour points of the object, or the key points used to characterize the key parts of the object. The key points to be detected can be flexibly set according to the needs.

[0024] In practical applications, a feature extraction algorithm may be used to perform multi-scale feature extraction on a target image. For example, features may be extracted layer by layer by downsampling to obtain multiple features with scales ranging from large to small.

[0025] Step S104: acquiring a first detection position of a key point in the target image based on a feature of a minimum scale among features of multiple scales.

[0026] The minimum scale feature may be, for example, a 1*1 feature. Based on the minimum scale feature, it is easier to regress and predict the first detection position of the key point in the target image. The key points may be all the key points in the target image, or may be specified key points in the target image, such as key points of a target object of a specified type in the target image, which is not limited here.

[0027] Step S106: Based on the target scale feature among the features of the multiple scales, the first detected position of the key point in the target image is corrected to obtain a second detected position of the key point in the target image.

[0028] Among them, the target scale feature is a feature of some or all scales among the features of multiple scales. In other words, the target scale feature can be a feature of at least one scale among the features of multiple scales, and can be set flexibly. In practical applications, when the number of target scale features is multiple, each target scale feature can be processed separately, such as establishing an association between each target scale feature and the relevant features of the key point based on a cross-attention mechanism, and then fusing the corresponding processing results of the multiple target scale features, so as to correct the first detection position of the key point based on the fusion result; in addition, multiple target scale features can be fused first, and then the first detection position of the key point can be corrected directly based on the fused target scale feature, such as establishing an association between the fused target scale feature and the relevant features of the key point based on a cross-attention mechanism, and then correcting the first detection position of the key point. In some specific implementation examples, features of multiple scales can be used as target scale features, and the receptive fields corresponding to features of different scales are different. Through the above method, the first detection position can be corrected based on multiple receptive fields. In the above method, it is fully considered that the accuracy of the first detection position obtained by regression based on the minimum scale feature may not be high, mainly because the spatial structure of the minimum scale feature may be destroyed. To this end, the embodiment of the present disclosure also uses features of some or all scales of features at multiple scales to correct the first detection position, that is, the accuracy of the first detection position can be further improved by combining local detail information at multiple scales.

[0029] In summary, in the above method, the first detection position is not directly used as the final detection result, but the first detection position of the key point can be further corrected by using the characteristics of part or all scales of the target image, which can effectively improve the accuracy of the key point detection result.

[0030] In some implementation examples, the first detected position of the key point in the target image can be corrected based on the attention mechanism and the target scale features in the features of multiple scales. It can be understood that the attention mechanism can establish associations between different features, such as establishing associations between different key points based on the self-attention mechanism, and establishing associations between key point features and image scale features based on the cross-attention mechanism, thereby comprehensively improving the position detection accuracy of the key points. Exemplarily, the following steps A and B can be referred to for execution:

[0031] Step A, based on the self-attention mechanism and the first detected position of the key point in the target image, obtain the associated feature vector corresponding to the key point; wherein the associated feature vector corresponding to each key point carries the first detected position information of the key point and the association information between the key point and other key points in the target image. The self-attention mechanism can reduce the dependence on external information and is better at capturing the internal correlation of data or features, so as to relatively reliably obtain the association relationship between key points. In some specific implementation examples, step A can be performed with reference to the following steps A1 and A2:

[0032] Step A1, encode the first detection position of the key point in the target image to obtain the position vector corresponding to the key point. The embodiment of the present disclosure does not limit the encoding method of the first detection position. In practical applications, if there are N key points in total, each key point corresponds to a position vector, then N position vectors can be obtained.

[0033] Step A2, based on the self-attention mechanism, the position vector of the key point is processed to generate the associated feature vector corresponding to the key point. In practical applications, the position vectors of N key points are processed based on the self-attention mechanism to extract the correlation between different key points, and the associated feature vectors corresponding to each of the N key points are obtained, wherein, for a certain key point, the associated feature vector of the key point carries not only the position information of the key point itself, but also the association relationship between the key point and other key points. By introducing the self-attention mechanism to analyze the correlation between different key points, it is easier to correct the inaccurate position of the key point, and it also helps to further improve the position detection accuracy of the key point.

[0034] Step B, based on the cross-attention mechanism, the target scale features in the features of multiple scales and the associated feature vectors corresponding to the key points, the first detection position of the key points in the target image is corrected. In practical applications, the target scale features and the associated feature vectors corresponding to the key points are processed based on the cross-attention mechanism, and attention can be applied to capture local information in the local area of ​​the image rather than the entire image, and the first detection position is corrected based on the global semantic information corresponding to the associated feature vector corresponding to the key point combined with the local detail information corresponding to the target scale feature. Moreover, the cross-attention mechanism can select representative local features and associated features corresponding to the key points for attention processing. On the basis of improving the position detection accuracy of the key points, the required amount of calculation is relatively low and the processing efficiency is higher.

[0035] In some specific implementation examples, step B may be performed with reference to the following steps B1 and B3:

[0036] Step B1, determine the associated pixels corresponding to the key point in the target image. For example, the associated pixels may be the first N pixels with the highest correlation with the key point among the surrounding pixels of the key point. In practical applications, the relative position relationship between the key point and the associated pixels may be a pre-set fixed relationship, such as, taking a 3*3 pixel block as an example, taking the key point as the center of the 3*3 pixel block, and taking the 8 pixels around the key point (that is, the remaining 8 pixels in the 3*3 pixel block except the center pixel) as the associated pixels of the key point. The relative position relationship between the key point and the associated pixels may be a relationship learned in advance, such as, the associated pixels may not be the 8 pixels around the key point (that is, not the remaining 8 pixels in the above-mentioned 3*3 pixel block). Exemplarily, the associated pixels may be determined with reference to the following steps B1.1 and B1.2:

[0037] Step B1.1, obtain the offset parameter matrix corresponding to the key points in the target image.

[0038] The offset parameter matrix corresponding to each key point is used to indicate the position offset information between a preset number of associated pixels with the highest correlation with the key point in the target image and the key point; the offset parameter matrices corresponding to different key points are the same or different. Exemplarily, the offset parameter matrix can be a 3*3 parameter matrix, and each parameter in the offset parameter matrix is ​​used to characterize the position offset information between an associated pixel and the key point.

[0039] For easier understanding, please refer to Figure 2As shown in the figure, assuming that the center point P is a key point, the 8 gray points around P can be used as fixed associated pixels, or the 8 black points around P can be determined as the most relevant associated pixels with P through the offset parameter matrix. In other words, if the offset parameter matrix is ​​not introduced, the associated pixels of each key point can be fixed in advance, and by introducing the offset parameter matrix (which can be obtained through learning), multiple pixels with the highest degree of association around each key point can be more accurately selected as associated pixels. Taking the key point to be detected as a human key point as an example, assuming that there are 63 human key points in total, the offset parameter matrix corresponding to each key point can be learned during the model training period. The offset parameter matrix can be used to indicate the position offset information between the multiple most relevant associated pixels of the key point and the key point. The offset parameter matrices corresponding to different key points can be the same or different, which can be determined specifically through learning.

[0040] Step B1.2, based on the offset parameter matrix, determine the associated pixels corresponding to the key points in the target image. As mentioned above, based on the known offset parameter matrix and the key points, the associated pixels corresponding to the key points can be directly determined.

[0041] Step B2: Determine the pixel features of the associated pixels based on the target scale features in the features of the multiple scales. The target scale features may be partial scale features or all scale features in the features of the multiple scales. When the target scale features have multiple scales, the pixel features of the associated pixels also have multiple scales. In other words, the associated pixels have corresponding pixel features at each target scale.

[0042] Step B3, based on the cross-attention mechanism, the associated feature vectors corresponding to the key points, and the pixel features of the associated pixels corresponding to the key points, the first detection position of the key points in the target image is corrected. Based on the cross-attention mechanism, the relationship between the associated pixels and the key points that are more strongly associated with the key points can be focused on, the required computing resources are lower, and the processing is more efficient. Based on the pixel features of the associated pixels and the associated feature vectors of the key points (which can present the position information of the key points themselves and the association relationship between different key points), the first detection position of the key points can be better assisted in correcting. Exemplarily, the first detection position can be corrected with reference to the following steps B3.1 and B3.2:

[0043] Step B3.1, when there are multiple associated pixels, the pixel features of the multiple associated pixels corresponding to the key point are fused to obtain a pixel fusion feature. Since each associated pixel also has corresponding pixel features at different target scales, in some implementation examples, the pixel features of the multiple associated pixels at each target scale can be fused to obtain a pixel fusion feature corresponding to each target scale.

[0044] Step B3.2, based on the cross attention mechanism, the associated feature vector corresponding to the key point and the pixel fusion feature corresponding to the key point are processed to correct the first detection position of the key point in the target image.

[0045] If there are multiple target scales, in some implementation examples, based on the cross-attention mechanism, cross-attention processing can be performed on the associated feature vectors corresponding to the key points and the pixel fusion features corresponding to the key points corresponding to each target scale to obtain the cross-attention vector corresponding to each target scale. The cross-attention vectors corresponding to each target scale are then spliced ​​and fused to obtain a final vector. The final vector can indicate the corrected position (also called the second detection position) corresponding to the first detection position of the key point.

[0046] In practical applications, the key point detection method can be performed by a preset key point detection model, for example, see Figure 3 A structural schematic diagram of a key point detection model is shown, the key point detection model includes a first neural network and a second neural network; a target image is input to the first neural network, the first neural network outputs a first detection position of a key point in the target image, the first detection position and the target scale feature of the first neural network are input to the second neural network, and the second neural network outputs a second detection position of the key point in the target image. Specifically: the first neural network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales; and based on the minimum scale feature among the features of multiple scales, the first detection position of the key point in the target image is obtained; the second neural network is used to correct the first detection position of the key point in the target image based on the target scale feature among the features of multiple scales to obtain the second detection position of the key point in the target image. The specific implementation method can refer to the aforementioned related content, which will not be repeated here.

[0047] The present disclosure further provides a Figure 4 The structural diagram of the key point detection model shown in Figure 3On the basis of, it is illustrated that the first neural network includes a backbone network and a first prediction network, and the second neural network includes a position encoding network and a second prediction network; wherein the second prediction network includes a self-attention sub-network and a cross-attention sub-network. Specifically, the backbone network is used to extract multi-scale features of the target image to obtain features of multiple scales; the first prediction network is used to output the first detection position of the key point in the target image based on the minimum scale feature among the features of multiple scales; the position encoding network is used to encode the first detection position of the key point in the target image to obtain the position vector corresponding to the key point; the self-attention sub-network in the second prediction network is used to process the position vector of the key point based on the self-attention mechanism to generate the associated feature vector corresponding to the key point, and the cross-attention sub-network in the second prediction network is used to correct the first detection position of the key point in the target image based on the cross-attention mechanism, the target scale feature among the features of multiple scales, and the associated feature vector corresponding to the key point.

[0048] Based on the foregoing, the present disclosure further provides a Figure 5 The structural diagram of the key point detection model shown in Figure 5 The backbone network has four stages, each corresponding to a scale feature. Figure 5 The figure simply shows that the backbone network outputs features of four scales, and the features of the four scales are input into the cross-attention sub-network as target scale features. Figure 5 It is further illustrated in FIG. 1 that the first prediction network includes a GAP subnetwork and a FC subnetwork, and in Figure 4 On the basis of Figure 5 In addition to illustrating that the second prediction network includes a self-attention sub-network and a cross-attention sub-network, the FFN sub-network and the MLP sub-network after the cross-attention sub-network are also illustrated, and the output result of the final MLP sub-network is the second detection position. It should be noted that Figure 5 This is only an implementation example. In actual applications, the key point detection model may include fewer or more sub-networks, which is not limited here.

[0049] Furthermore, the embodiment of the present disclosure provides a method for constructing a key point detection model, that is, the key point detection model can be constructed through the following steps a to d:

[0050] Step a, obtaining an image sample; wherein the image sample is annotated with the real position information of the key points in the image sample.

[0051] Step b: Performing a one-stage training on the first initial network based on the image samples until a preset first training end condition is reached, thereby obtaining a first initial network after the one-stage training. The first initial network is an untrained first neural network.

[0052] Exemplarily, when the first initial network is trained in one stage based on image samples, the image samples can be input into the first initial network to obtain the first detection positions of the key points of the image samples output by the first initial network, and then based on the difference between the first detection positions of the key points of the image samples output by the first initial network and the actual position information of the key points annotated by the image samples, a preset first loss function is used to calculate the first loss value, and the parameters of the first initial network are adjusted according to the first loss value. Exemplarily, the first training end condition may be that the first loss value converges to a first preset threshold. The disclosed embodiment does not limit the first loss function. Exemplarily, the first loss function may be an L2 loss function.

[0053] Step c, based on the image samples, the second initial network and the first initial network after the first stage training are trained in a second stage until the preset second training end condition is reached, thereby obtaining the first initial network after the second stage training and the second initial network after the second stage training. The second initial network is the untrained second neural network.

[0054] Among them, the first initial network after one-stage training is connected in sequence with the second initial network. Exemplarily, when the second initial network and the first initial network after one-stage training are subjected to two-stage training based on image samples, the image samples can be first input into the first initial network, and the first initial network is made to perform multi-scale feature extraction on the image samples to obtain features of multiple scales, and based on the minimum-scale features among the features of multiple scales, the first detection position of the key point in the image sample is obtained; then, the first detection position output by the first initial network and the target scale feature among the features of multiple scales extracted by the first initial network for the image sample are input into the second initial network, so that the second initial network outputs the second detection position of the key point in the image sample according to the first detection position of the key point in the image sample and the target scale feature corresponding to the image sample. Specifically, a first loss value can be calculated using a preset first loss function based on the difference between the first detection position of the key point of the image sample output by the first initial network and the actual position information of the key point annotated by the image sample, and a second loss value can be calculated using a preset second loss function based on the difference between the second detection position of the key point of the image sample and the actual position information of the key point annotated by the image sample, and then a total loss value can be calculated based on the first loss value and the second loss value, such as the first loss value and the second loss value can be weighted to obtain the total loss value. The embodiment of the present disclosure does not restrict the weights corresponding to the first loss value and the second loss value, such as the weights of the two can be the same or different. Exemplarily, the second training end condition can be that the total loss value converges to a second preset threshold. The embodiment of the present disclosure does not restrict the first loss function and the second loss function. Exemplarily, the first loss function and the second loss function can both be L2 loss functions.

[0055] Step d, obtaining a first neural network based on the first initial network after two-stage training, and obtaining a second neural network based on the second initial network after two-stage training, and constructing a key point detection model based on the first neural network and the second neural network. Specifically, the first initial network after two-stage training can be directly used as the first neural network, and the second initial network after two-stage training can be used as the second neural network.

[0056] In the disclosed embodiment, the first initial network can be pre-trained (i.e., one-stage training) so that the first initial network can output a first detection position with a smaller error, thereby ensuring the accuracy of the first detection position and the multi-scale features extracted by the first initial network to a certain extent. On this basis, the first initial network after the one-stage training and the untrained second initial network are subjected to a second-stage training, which can effectively improve the training efficiency.

[0057] In summary, the key point detection method provided in the embodiments of the present disclosure can effectively improve the accuracy of the key point detection result by correcting the first detection position of the key point by utilizing the features of a partial scale or all scales of the target image.

[0058] Corresponding to the aforementioned key point detection method, the embodiment of the present disclosure further provides a key point detection device, Figure 6 Schematic diagram of a key point detection device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 6 As shown, the key point detection device includes:

[0059] A multi-feature extraction module 602 is used to extract multi-scale features from the target image to obtain features at multiple scales;

[0060] A first position acquisition module 604, configured to acquire a first detection position of a key point in a target image based on a feature of a minimum scale among features of multiple scales;

[0061] The first position correction module 606 is used to correct the first detection position of the key point in the target image based on the target scale feature among the features of multiple scales to obtain the second detection position of the key point in the target image; wherein the target scale feature is a feature of some scales or all scales among the features of multiple scales.

[0062] In the above device, the first detection position is not directly used as the final detection result, but the first detection position of the key point can be further corrected by using the characteristics of part or all scales of the target image, which can effectively improve the accuracy of the key point detection result.

[0063] In some implementations, the first position correction module 606 is specifically configured to correct the first detected position of the key point in the target image based on an attention mechanism and target scale features among the features of the multiple scales.

[0064] In some embodiments, the first position correction module 606 is specifically used to: obtain the associated feature vector corresponding to the key point based on the self-attention mechanism and the first detection position of the key point in the target image; wherein the associated feature vector corresponding to each key point carries the first detection position information of the key point and the association information between the key point and other key points in the target image; and correct the first detection position of the key point in the target image based on the cross-attention mechanism, the target scale feature among the features of the multiple scales, and the associated feature vector corresponding to the key point.

[0065] In some embodiments, the first position correction module 606 is specifically used to: encode the first detection position of the key point in the target image to obtain the position vector corresponding to the key point; process the position vector of the key point based on the self-attention mechanism to generate an associated feature vector corresponding to the key point.

[0066] In some embodiments, the first position correction module 606 is specifically used to: determine the associated pixels corresponding to the key points in the target image; determine the pixel features of the associated pixels based on the target scale features among the multiple scale features; and correct the first detection position of the key points in the target image based on the cross-attention mechanism, the associated feature vectors corresponding to the key points, and the pixel features of the associated pixels corresponding to the key points.

[0067] In some embodiments, the first position correction module 606 is specifically used to: obtain an offset parameter matrix corresponding to a key point in the target image; wherein the offset parameter matrix corresponding to each key point is used to indicate the position offset information between a preset number of associated pixels with the highest correlation with the key point in the target image and the key point; the offset parameter matrices corresponding to different key points are the same or different; and based on the offset parameter matrix, determine the associated pixels corresponding to the key point in the target image.

[0068] In some embodiments, the first position correction module 606 is specifically used to: when the number of associated pixels is multiple, fuse the pixel features of multiple associated pixels corresponding to the key point to obtain a pixel fusion feature; based on the cross-attention mechanism, process the associated feature vector corresponding to the key point and the pixel fusion feature corresponding to the key point to correct the first detection position of the key point in the target image.

[0069] In some embodiments, the key point detection device is applied to a preset key point detection model, and the key point detection model includes a first neural network and a second neural network;

[0070] The first neural network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales; and based on the minimum-scale feature among the features of the multiple scales, obtain a first detection position of a key point in the target image;

[0071] The second neural network is used to correct the first detection position of the key point in the target image based on the target scale feature among the multiple scale features to obtain the second detection position of the key point in the target image.

[0072] In some embodiments, the key point detection model is constructed by the following steps: obtaining an image sample; wherein the image sample is annotated with the real position information of the key point in the image sample; performing a first-stage training on the first initial network based on the image sample until a preset first training end condition is reached, thereby obtaining a first initial network after the first-stage training; performing a second-stage training on the second initial network and the first initial network after the first-stage training based on the image sample until a preset second training end condition is reached, thereby obtaining a first initial network after the second-stage training and a second initial network after the second-stage training;

[0073] A first neural network is obtained based on the first initial network after the two-stage training, and a second neural network is obtained based on the second initial network after the two-stage training, and a key point detection model is constructed based on the first neural network and the second neural network.

[0074] The key point detection device provided in the embodiments of the present disclosure can execute the key point detection method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0076] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0077] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0078] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0079] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0080] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0081] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0082] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the key point detection method provided by the embodiment of the present disclosure.

[0083] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0084] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the key point detection method in the embodiment of the present disclosure when executed by a processor.

[0085] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0086] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0087] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0088] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0089] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0090] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A key point detection method, characterized in that: include: Perform multi-scale feature extraction on the target image to obtain features at multiple scales; Acquire a first detection position of a key point in the target image based on a feature of a minimum scale among the features of the multiple scales; Based on the target scale feature among the features of the multiple scales, the first detected position of the key point in the target image is corrected to obtain the second detected position of the key point in the target image; wherein the target scale feature is a feature of some or all scales among the features of the multiple scales.

2. The method according to claim 1, characterized in that The correcting the first detected position of the key point in the target image based on the target scale feature among the features of the multiple scales includes: Based on the attention mechanism and the target scale feature among the features of the multiple scales, a first detected position of the key point in the target image is corrected.

3. The method according to claim 2, characterized in that The method of correcting the first detected position of the key point in the target image based on the attention mechanism and the target scale feature among the features of the multiple scales includes: Based on the self-attention mechanism and the first detected position of the key point in the target image, obtaining the associated feature vector corresponding to the key point; wherein the associated feature vector corresponding to each key point carries the first detected position information of the key point and the association information between the key point and other key points in the target image; Based on a cross-attention mechanism, a target scale feature among the features of the multiple scales, and an associated feature vector corresponding to the key point, a first detected position of the key point in the target image is corrected.

4. The method according to claim 3, characterized in that The step of obtaining the associated feature vector corresponding to the key point based on the self-attention mechanism and the first detected position of the key point in the target image includes: Encoding a first detected position of a key point in the target image to obtain a position vector corresponding to the key point; The position vector of the key point is processed based on the self-attention mechanism to generate an associated feature vector corresponding to the key point.

5. The method according to claim 3, characterized in that: The method of correcting the first detected position of the key point in the target image based on the cross attention mechanism, the target scale feature among the features of the multiple scales, and the associated feature vector corresponding to the key point includes: Determine the associated pixels corresponding to the key points in the target image; Determining a pixel feature of the associated pixel based on a target scale feature among the features of the multiple scales; Based on the cross-attention mechanism, the associated feature vector corresponding to the key point and the pixel features of the associated pixels corresponding to the key point, the first detection position of the key point in the target image is corrected.

6. The method according to claim 5, characterized in that The determining of associated pixels corresponding to the key points in the target image comprises: Obtaining an offset parameter matrix corresponding to a key point in the target image; wherein the offset parameter matrix corresponding to each key point is used to indicate position offset information between a preset number of associated pixels having the highest correlation with the key point in the target image and the key point; and the offset parameter matrices corresponding to different key points may be the same or different; Based on the offset parameter matrix, associated pixels corresponding to key points in the target image are determined.

7. The method according to claim 5, characterized in that The method of correcting the first detected position of the key point in the target image based on the cross attention mechanism, the associated feature vector corresponding to the key point, and the pixel features of the associated pixels corresponding to the key point includes: In the case where there are multiple associated pixels, the pixel features of the multiple associated pixels corresponding to the key point are fused to obtain a pixel fusion feature; Based on the cross-attention mechanism, the associated feature vector corresponding to the key point and the pixel fusion feature corresponding to the key point are processed to correct the first detection position of the key point in the target image.

8. The method according to claim 1, characterized in that The key point detection method is performed by a preset key point detection model, and the key point detection model includes a first neural network and a second neural network; The first neural network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales; and based on the minimum-scale feature among the features of the multiple scales, obtain a first detection position of a key point in the target image; The second neural network is used to correct the first detection position of the key point in the target image based on the target scale feature among the multiple scale features to obtain the second detection position of the key point in the target image.

9. The method according to claim 8, characterized in that The key point detection model is constructed through the following steps: Acquire an image sample; wherein the image sample is annotated with real position information of key points in the image sample; Performing one-stage training on the first initial network based on the image samples until a preset first training end condition is reached, thereby obtaining a first initial network after the one-stage training; Based on the image samples, the second initial network and the first initial network after the first stage training are trained in two stages until a preset second training end condition is reached, thereby obtaining the first initial network after the two stage training and the second initial network after the two stage training; A first neural network is obtained based on the first initial network after the two-stage training, and a second neural network is obtained based on the second initial network after the two-stage training, and a key point detection model is constructed based on the first neural network and the second neural network.

10. A key point detection device, characterized in that: include: A multi-feature extraction module is used to extract multi-scale features from the target image to obtain features at multiple scales; A first position acquisition module, configured to acquire a first detection position of a key point in the target image based on a feature of a minimum scale among the features of the multiple scales; a first position correction module, configured to correct a first detected position of a key point in the target image based on a target scale feature among the features of the multiple scales, so as to obtain a second detected position of the key point in the target image; wherein the target scale feature is a feature of some or all scales among the features of the multiple scales.

11. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the key point detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the key point detection method described in any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the key point detection method according to any one of claims 1 to 9 when being executed by a processor.

Images