Key point detection method and key point detection device

Abstract

The invention discloses a key point detection method and a key point detection device.The key point detection method comprises the steps that a preliminary detection result and first variance information of a key point of a target object in a to-be-detected image are acquired, and the preliminary detection result comprises position information of the key point; performing key point verification on the preliminary detection result based on the first variance information; and determining a detection result of the target object based on a result of the key point verification.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and more specifically, to a key point detection method and a key point detection device for detecting key points of a target object. Background Art

[0002] Keypoint detection (such as human pose estimation) is a hot and important research topic in the field of computer vision, which aims to detect all object instances in an image and identify the key points of each object (such as the joints of the human body). Keypoint detection has a wide range of applications, such as action recognition, human-computer interaction, etc.

[0003] At present, although a variety of methods for human posture estimation have been proposed, these methods have the problems of low estimation accuracy, slow detection speed and inability to shield detection interference in complex scenes (such as crowded scenes with multiple people). Summary of the invention

[0004] The present disclosure provides a key point detection method and a corresponding key point detection device to at least solve the above-mentioned problems in the related art, and may not solve any of the above-mentioned problems.

[0005] According to one aspect of an exemplary embodiment of the present disclosure, a key point detection method is provided, wherein the key point detection method comprises: obtaining preliminary detection results and first variance information of key points of a target object in an image to be detected, wherein the preliminary detection results include position information of the key points; performing key point verification on the preliminary detection results based on the first variance information; and determining the detection result of the target object based on the result of the key point verification.

[0006] As described above, by performing key point verification on the preliminary detection results based on the first variance information, the detection results of the target object are determined based on the results of the key point verification, and the influence of invisible key points on visible key points during structural prediction is eliminated, thereby improving the accuracy of human posture estimation.

[0007] Optionally, performing key point verification on the preliminary detection result based on the first variance information includes: generating a mask matrix for determining key points based on the first variance information; and determining a first key point of the target object based on the mask matrix and the preliminary detection result.

[0008] Optionally, obtaining the preliminary detection results and first variance information of the key points of the target object includes: obtaining a target image block in the image to be detected, the target image block at least including the target object; performing feature extraction on the target image block to obtain feature information of the target image block; and obtaining the preliminary detection results and the first variance information of the key points of the target object based on the feature information.

[0009] Optionally, obtaining the preliminary detection results and the first variance information of the key points of the target object based on the feature information includes: using a first neural network to predict first position-related information of the key points of the target object based on at least part of the features in the feature information; and using a second neural network to predict the preliminary detection results and the first variance information of the key points of the target object based on the first position-related information and the at least part of the features.

[0010] Optionally, the second neural network includes at least a first self-attention network, a cross-attention network, a first position prediction network and a first variance prediction network, wherein, based on the first position-related information and the at least partial features, using the second neural network to predict the preliminary detection result and the first variance information of the key points of the target object includes: generating a first query vector based on the at least partial features; generating a first feature based on the first query vector and the first position-related information using the first self-attention network; generating a second feature based on the first feature and the at least partial features using the cross-attention network; and predicting the preliminary detection result and the first variance information based on the second feature using the first position prediction network and the first variance prediction network, respectively.

[0011] Optionally, determining the detection result of the target object based on the result of the key point verification includes: obtaining a target image block in the image to be detected, the target image block at least including the target object; performing feature extraction on the target image block to obtain feature information of the target image block; and predicting the position information of the final key point of the target object using a third neural network based on the feature information, the preliminary detection result and the first key point.

[0012] Optionally, the third neural network includes at least a second self-attention network, a deformable attention network, a second position prediction network and a second variance prediction network, wherein the method of using the third neural network to predict the position information of the final key point of the target object based on the feature information, the preliminary detection result and the first key point includes: using the second self-attention network to generate a third feature based on the first key point and the second position-related information and the second query vector output by the second neural network; using the deformable attention network to generate a fourth feature based on the third feature, the feature information and the preliminary detection result; and using the second position prediction network and the second variance prediction network respectively to predict the position information of the final key point of the target object based on the fourth feature.

[0013] Optionally, based on the fourth feature, using the second position prediction network and the second variance prediction network respectively to predict the position information of the final key point of the target object includes:

[0014] Based on the fourth feature, the second position prediction network is used to obtain the position information of the key points of the target object; based on the fourth feature, the second variance prediction network is used to obtain the second variance information of the key points of the target object; based on the comparison between the second variance information and a threshold, the final key points of the target object are determined, thereby obtaining the position information of the final key points of the target object.

[0015] As described above, the second variance information is used as a basis for determining whether a key point exists, and the key point recognition result of the target object is determined more accurately by comparing the second variance information with a threshold.

[0016] Optionally, the second neural network includes multiple neural network units, each of the multiple neural network units includes at least the first self-attention network, the cross-attention network, the first position prediction network and the first variance prediction network, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first position-related information, the at least part of the features, and the first query vector, and the output of the first neural network unit is the position-related information, the query vector, the position information and variance information of the key points as intermediate quantities, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the preliminary detection result and the first variance information.

[0017] Optionally, the third neural network includes multiple neural network units, each of the multiple neural network units includes at least the second self-attention network, the deformable attention network, the second position prediction network and the second variance prediction network, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first key point, the second position-related information and the second query vector, the feature information and the preliminary detection result, and the output of the first neural network unit is the position-related information, query vector, position information of the key point and variance information as intermediate quantities, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the final detection result.

[0018] According to another aspect of the exemplary embodiment of the present disclosure, a key point detection device is provided, which may include: a data acquisition module, configured to acquire preliminary detection results and first variance information of key points of a target object in an image to be detected, wherein the detection results include position information of the key points; a key point verification module, configured to perform key point verification on the preliminary detection results based on the first variance information; and a key point determination module, configured to determine the detection result of the target object based on the result of the key point verification.

[0019] Optionally, the key point verification module may be configured to: generate a mask matrix for determining key points based on the first variance information; and determine a first key point of the target object based on the mask matrix and the preliminary detection result.

[0020] Optionally, the data acquisition module can be configured to: obtain a target image block in the image to be detected, wherein the target image block at least includes the target object; perform feature extraction on the target image block to obtain feature information of the target image block; and obtain the preliminary detection result and the first variance information of the key points of the target object based on the feature information.

[0021] Optionally, the data acquisition module can be configured to: use a first neural network to predict first position-related information of the key points of the target object based on at least part of the features in the feature information; and use a second neural network to predict the preliminary detection results and the first variance information of the key points of the target object based on the first position-related information and the at least part of the features.

[0022] Optionally, the second neural network includes at least a first self-attention network, a cross-attention network, a first position prediction network and a first variance prediction network, wherein the data acquisition module can be configured to: generate a first query vector based on the at least part of the features; generate a first feature using the first self-attention network based on the first query vector and the first position-related information; generate a second feature using the cross-attention network based on the first feature and the at least part of the features; and predict the preliminary detection result and the first variance information based on the second feature using the first position prediction network and the first variance prediction network, respectively.

[0023] Optionally, the key point determination module can be configured to: obtain a target image block in the image to be detected, wherein the target image block includes at least the target object; perform feature extraction on the target image block to obtain feature information of the target image block; and use a third neural network to predict the position information of the final key point of the target object based on the feature information, the preliminary detection result and the first key point.

[0024] Optionally, the third neural network includes at least a second self-attention network, a deformable attention network, a second position prediction network and a second variance prediction network, wherein the key point determination module can be configured to: generate a third feature using the second self-attention network based on the first key point and the second position-related information and the second query vector output by the second neural network; generate a fourth feature using the deformable attention network based on the third feature, the feature information and the preliminary detection result; and predict the position information of the final key point of the target object based on the fourth feature using the second position prediction network and the second variance prediction network respectively.

[0025] Optionally, the key point determination module can be configured as: based on the fourth feature, using the second position prediction network, to obtain the position information of the key points of the target object; based on the fourth feature, using the second variance prediction network, to obtain the second variance information of the key points of the target object; based on the comparison of the second variance information with a threshold, determining the final key points of the target object, thereby obtaining the position information of the final key points of the target object.

[0026] Optionally, the second neural network includes multiple neural network units, each of the multiple neural network units includes at least the first self-attention network, the cross-attention network, the first position prediction network and the first variance prediction network, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first position-related information, the at least part of the features, and the first query vector, and the output of the first neural network unit is the position-related information, the query vector, the position information and variance information of the key points as intermediate quantities, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the preliminary detection result and the first variance information.

[0027] Optionally, the third neural network includes multiple neural network units, each of the multiple neural network units includes at least the second self-attention network, the deformable attention network, the second position prediction network and the second variance prediction network, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first key point, the second position-related information and the second query vector, the feature information and the preliminary detection result, and the output of the first neural network unit is the position-related information, query vector, position information of the key point and variance information as intermediate quantities, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the final detection result.

[0028] According to another aspect of an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by a processor, the key point detection method described in the present disclosure is implemented.

[0029] According to another aspect of the exemplary embodiment of the present disclosure, an electronic device is provided, which includes: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the key point detection method described in the present disclosure.

[0030] In summary, by using the key point detection method provided by the present disclosure, it is possible to effectively shield the interference of invisible points of objects in the image on the learning of structural information based on variance information, thereby improving recognition accuracy.

[0031] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other objects and features of exemplary embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate the embodiments, in which:

[0033] Figure 1 A flowchart showing a key point detection method according to an embodiment of the present disclosure is shown;

[0034] Figure 2 The overall architecture diagram of key point detection according to an embodiment of the present disclosure is shown;

[0035] Figure 3 A schematic diagram showing a second neural network according to an embodiment of the present disclosure;

[0036] Figure 4 is a schematic diagram of a third neural network according to an embodiment of the present disclosure;

[0037] Figure 5 A schematic diagram showing key point detection according to an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram showing a first neural network according to an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram showing a second neural network according to another embodiment of the present disclosure;

[0040] Figure 8 A schematic diagram showing a third neural network according to another embodiment of the present disclosure;

[0041] Fig. 9 A block diagram showing a key point detection device according to an embodiment of the present disclosure;

[0042] Fig.10 A structural schematic diagram of an electronic device applicable to an embodiment of the present disclosure is shown in FIG. DETAILED DESCRIPTION

[0043] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0044] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0045] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0046] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0047] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0048] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, embodiments may be implemented in various forms and are not limited to the examples described herein.

[0049] Figure 1 A flowchart of a key point detection method according to an embodiment of the present disclosure is shown. Figure 2 The overall architecture diagram of the key point detection according to the embodiment of the present disclosure is shown below. Figure 1 and Figure 2 To describe the key point detection method disclosed in the present invention.

[0050] In step 101, a preliminary detection result and first variance information of key points of a target object in an image to be detected are obtained, wherein the preliminary detection result includes position information of the key points. For example, the preliminary detection result may show which key points are detected and the position information of these key points. The position information may be coordinate information.

[0051] For example, the image to be detected can be a single picture or a frame in a video. The target object can be each human body or object in the image. By detecting the key points (or joint points) of the target object, the posture, shape, etc. of the target object can be identified for subsequent action recognition or human-computer interaction applications.

[0052] As an example, a target image block in the image to be detected can be obtained, the target image block at least includes the target object, and feature extraction is performed on the target image block to obtain feature information of the target image block, and a preliminary detection result and first variance information of the key points of the target object are obtained based on the feature information. For example, if the image to be detected includes a human body, all human body frames in the image can be first obtained through any human body detection network, and each human body frame is the target image block. For each target image block, the target image block is used as the input of the feature extraction network (for example, the target image block can be directly input or normalized to an image in img format of size W×H×3), so that the feature extraction network extracts feature information from the target image block. The extracted features can be multi-scale feature information (i.e., multi-scale feature representation), and the number of multi-scales can be adjusted according to actual needs and network structure. For example, feature representations of four scales can be extracted. The feature extraction network can be ResNet, HRNet, or HRFormer, etc.

[0053] After obtaining the feature information of the target image block, the first position-related information of the key point of the target object can be predicted using a first neural network based on at least part of the features in the feature information, and the preliminary detection result and first variance information of the key point of the target object can be predicted using a second neural network based on the first position-related information and the at least part of the features. For example, at least part of the features in the feature information may refer to the feature information of the minimum scale in the extracted multi-scale feature information. The first position-related information may be the coordinates of the key point of the target object or a position vector obtained by position encoding the coordinates, and the first variance information may include the variance of the key point of the target object in the horizontal and vertical coordinates.

[0054] Assuming that feature representations of four scales of the target image block are extracted, one of the feature representations of the four scales (such as the feature of the smallest size) can be used as the input of the first neural network to obtain the first position-related information of the target object (such as the coordinate information of the key points). For example, the first neural network may be composed of a global average pooling (GAP) layer and a fully connected (FC) layer. The above examples are only exemplary, and the first neural network of the present disclosure may be any neural network that can extract the first position-related information of the key points of the target object from the image. In addition, the present disclosure may also use two, more or all of the feature representations of multiple scales as the input of the first neural network.

[0055] In addition, the first neural network may be composed of two parts, the first part is used to predict the first position-related information (such as the coordinate information of the key points) of the target object, and the second part is used to predict the variance information of the key points of the target object. For example, each part may be composed of a GAP layer and an FC layer. In the training stage, the first position-related information and the variance information may be used to train the first neural network respectively; in the inference stage, only the first position-related information output by the first neural network may be used.

[0056] The second neural network can be used to correct key points with large errors in the preliminary positioning. The second neural network can include at least a first self-attention network module, a cross-attention network module, a first position prediction network module and a first variance prediction network module.

[0057] A first query vector can be generated based on at least part of the features in the feature information; a first feature can be generated using a first self-attention network module based on the first query vector and the first position-related information; a second feature can be generated using a cross-attention network module based on the first feature and the at least part of the features; a preliminary detection result can be predicted based on the second feature using a first position prediction network module, and a first variance prediction network module can be used to predict the first variance information.

[0058] For example, for the minimum size feature, the number of channels is first changed to 256 through a 1×1 convolution operation, and then it is flattened and reshaped into a feature with a dimension of [1, 64×48, 256]. The processed feature is used to generate a first query vector or the processed feature is used to obtain the first query vector through vector encoding. Based on the first query vector and the first position-related information, q (query), k (key), and v (value) as inputs of the first self-attention network module can be generated, and the first feature can be obtained through the first self-attention network module. Based on the first feature and the processed feature (i.e., the minimum scale feature after processing), the second feature is generated using the cross attention network module. Based on the second feature, the first position prediction network module is used to predict the preliminary detection results, such as the coordinate information of the key points of the object. Based on the second feature, the first variance prediction network module is used to predict the first variance information of the key points of the object.

[0059] In addition, the second neural network can also output a query vector and a position encoding vector for subsequent operations. For example, the query vector (such as the second query vector described below) and the position encoding vector (such as the second position related information described below) output by the second neural network can be used as inputs to the third neural network. For example, the second neural network can convolve, flatten, and re-deform the second feature output by the cross-attention network module to obtain a second query vector, and position encode the position information output by the first position prediction network module to obtain a position encoding vector. The second neural network includes each network that can use any form of neural network, and there is no limitation here.

[0060] According to another example of the present disclosure, the second neural network may include multiple neural network units, each of the multiple neural network units includes at least the first self-attention network module, the cross-attention network module, the first position prediction network module and the first variance prediction network module, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first position-related information, the at least part of the features, and the first query vector, and the output of the first neural network unit is the position-related information (position encoding vector) as an intermediate quantity, the query vector, the position information and variance information of the key point, and the latter neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the preliminary detection result and the first variance information. That is to say, the second neural network may be composed of a plurality of the above-mentioned neural network units stacked together, and key point correction is performed through a plurality of neural network units, and the position-related information output by the last neural network unit of the second neural network is used as the preliminary detection result.

[0061] Figure 3 A schematic diagram of a second neural network according to an embodiment of the present disclosure is shown. Figure 3 The second neural network includes multiple neural network units.

[0062] Reference Figure 3 , a first query vector may be generated based on at least part of the features in the feature information extracted from the target image block.

[0063] In the first neural network unit, a first feature is generated using a first self-attention network module based on a first query vector and first position-related information.

[0064] Based on the first feature and the at least part of the feature, a second feature is generated using a cross-attention network module.

[0065] Based on the second feature, the first position prediction network module is used to generate the position information of the key points of the target object, and based on the second feature, the first variance prediction network module is used to generate the variance information of the key points of the target object.

[0066] Next, based on the position information output by the first neural network unit, position-related information can be generated through position encoding, and based on the second feature, a query vector can be generated through vector encoding. The position-related information and the query vector are then input into the first self-attention network module of the second neural network unit. In addition, the second feature and position information can also be directly input into the first self-attention network module.

[0067] For the middle neural network unit, the feature information input to the cross-attention network module can all use the feature information extracted from the target image block, or the subsequent neural network unit can use the first feature or the second feature output by the previous neural network unit.

[0068] When the last neural network unit is executed, the preliminary detection results of the key points of the target object (such as the coordinates of the key points), the first variance information and the second query vector can be output. In addition, the second neural network can position encode the position information output by the last neural network unit to output the second position-related information. The output information can be used for the subsequent third neural network.

[0069] In the present disclosure, in the case where the input is location-related information, location information (such as coordinates) may also be used instead as a network input.

[0070] In step 102, the preliminary detection result is key-point verified based on the first variance information. For example, the key points in the preliminary detection result are verified to further determine whether to shield or supplement the key points of the target object, thereby ensuring the detection accuracy of the key points. In the present disclosure, the first variance information can be used to generate a mask to eliminate the influence of invisible key points on visible key points during structural prediction.

[0071] Specifically, based on the first variance information, a mask matrix for determining key points is generated, and based on the mask matrix and the preliminary detection result, a first key point of the target object is determined.

[0072] As an example, the module for generating a mask matrix may generate a mask matrix Mask based on the first variance information using the following equation (1):

[0073] Mask=binary((1- repeat(mean(V t ),N)) +eye,threshold) (1)

[0074] Among them, mean represents the variance V in the coordinate dimension t Calculate the mean; repeat means repeating the vector N times, where N is the number of key points; threshold is the threshold for binarization operation, binary can set the matrix to a [0, 1] matrix according to the threshold, and eye represents the unit matrix.

[0075] The module for generating the mask matrix may include the third neural network described below, or the module for generating the mask matrix may be separately provided.

[0076] In step 103, based on the result of the key point verification, the detection result of the target object is determined. For example, the final key points of the target object and their corresponding positions may be determined.

[0077] As an example, based on the feature information of the target image block, the preliminary detection result and the first key point, the third neural network is used to predict the position information of the final key point of the target object. This will be described in detail below.

[0078] For example, the multi-scale features extracted from the feature extraction network can be processed before being input into the third neural network. Taking a feature of one scale as an example, the number of channels is first changed to 256 through a 1×1 convolution operation, and then it is flattened and reshaped into a feature with a dimension of [1, 64×48, 256]. Then, the processed features of each scale are operated to obtain a memory vector containing multi-scale information (i.e., the feature information of the target image block) as the input of the third neural network.

[0079] Based on the memory vector, the preliminary detection result and the first key point, a third neural network can be used to predict the position information of the final key point of the target object.

[0080] For example, the third neural network may include at least a second self-attention network module, a deformable attention network module, a second position prediction network module, and a second variance prediction network module. Based on the first key point and the second position-related information output by the second neural network (such as information obtained by position encoding the key point coordinates predicted by the second neural network) and the second query vector, the second self-attention network module may be used to generate a third feature; based on the third feature, feature information (such as a memory vector) and the preliminary detection result, the deformable attention network module may be used to generate a fourth feature; based on the fourth feature, the second position prediction network module and the second variance prediction network module are used to predict the position information of the final key point of the target object. For example, based on the fourth feature, the second position prediction network is used to obtain the position information of the key point of the target object; based on the fourth feature, the second variance prediction network is used to obtain the second variance information of the key point of the target object; based on the comparison of the second variance information with the threshold, the final key point of the target object is determined, thereby obtaining the position information of the final key point of the target object.

[0081] As an example, q (query), k (key), and v (value) input to the second self-attention network module can be generated based on the second position-related information output by the second neural network and the second query vector, and the third feature can be generated using the second self-attention network module based on q, k, v and the first key point. Based on the third feature, feature information and preliminary detection results, a fourth feature is generated using a deformable attention network module. Based on the fourth feature, the second position prediction network module is used to obtain the position information of the key points of the target object; based on the fourth feature, the second variance prediction network module is used to obtain the second variance information of the key points of the target object; based on the comparison of the second variance information with the threshold, the final key point of the target object is determined, thereby obtaining the position information of the final key point of the target object. Key points with a variance less than the threshold can be screened out according to a given threshold, and for key points that are invisible or very difficult to identify, these key points can be effectively shielded. For example, the threshold can be set to 0.5. The threshold can be set according to actual needs.

[0082] According to another example of the present disclosure, a third neural network may include multiple neural network units, each of the multiple neural network units includes at least the second self-attention network module, the deformable attention network module, the second position prediction network module and the second variance prediction network module, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first key point, the second position-related information (position encoding vector) and the second query vector, the feature information and the preliminary detection result, and the output of the first neural network unit is the position-related information, query vector, position information of the key point and variance information as intermediate quantities, and the latter neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the final detection result.

[0083] Figure 4 is a schematic diagram of a third neural network according to an embodiment of the present disclosure. Figure 4 The third neural network shown includes multiple neural network units.

[0084] Reference Figure 4 In the first neural network unit, a second self-attention network module is used based on the first key point, the second position-related information (position encoding vector) and the second query vector to obtain a third feature. Based on the third feature, the feature information and the preliminary detection result, a deformable attention network module is used to generate a fourth feature. Based on the fourth feature, a second position prediction network module is used to generate position information, and a second variance prediction network module is used to generate variance information.

[0085] Based on the position information output by the first neural network unit, position-related information is generated through position encoding. Based on the variance information and position information output by the first neural network unit, the key points are determined using the above method of determining key points based on the mask matrix. Based on the fourth feature, a query vector is generated through vector encoding.

[0086] The query vector, key points, and position-related information output by the first neural network unit are input into the second self-attention network module of the second neural network unit.

[0087] For the middle neural network unit, the feature information input to the deformable attention network module can all use the feature information extracted from the target image block, or the subsequent neural network unit can use the third feature or the fourth feature output by the previous neural network unit. In addition, the position information input to the deformable attention network module can be the preliminary detection result or the subsequent neural network unit can use the position information output by the previous neural network unit.

[0088] When the last neural network unit is executed, the location information of the final key points of the target object can be output.

[0089] The above network structure is only exemplary, and the third neural network including each network may use any form of neural network, and there is no limitation here.

[0090] In the present disclosure, variance information can be used as a basis for determining whether a key point exists. For example, when the variance of a key point is greater than a threshold, it can be considered that the key point does not exist. When the variance of a key point is less than a threshold, it can be considered that the key point exists. In this way, the key points of the target object can be identified more accurately.

[0091] Next, according to the coordinates of the target image block in the image to be detected, the final key point coordinates of the target object are mapped to the image to be detected, so as to complete the key point detection of the image to be detected.

[0092] For example, in human body posture estimation, if a part of the image cannot be seen due to body occlusion, the above method can also accurately identify the key points of the occluded part. In addition, the method disclosed in the present invention can locate a certain part without unnecessary offset due to the offset of other parts, making the recognition result more accurate.

[0093] According to another embodiment of the present disclosure, the last neural network unit of the third neural network may not include the second variance prediction network, but directly outputs the position information of the key points of the target object as the final recognition result.

[0094] In the process of training the above neural network, residual log-likelihood estimation RLE loss can be used for supervision, where the key point coordinates R are fully supervised learning, the variance V is adaptive learning, and the key points that do not exist in the image due to occlusion and other reasons do not participate in the loss calculation. A loss function can be constructed based on the output of the first neural network, the output of the second neural network, and the output of the third neural network and the corresponding real data (such as the position of the real key point), and the parameters of the above neural network are trained by minimizing the constructed loss function. However, the above training method is only exemplary, and the present disclosure is not limited to this.

[0095] The key point detection (such as human pose estimation) method based on a variance-constrained deformable mask decoding network proposed in the present disclosure can eliminate the influence of invisible key points on visible key points during structural prediction, thereby improving the accuracy of human pose estimation.

[0096] According to the embodiments of the present disclosure, key point detection can be performed on any target object in an image or video to be detected. Taking the key point detection of a human body in an image or video as an example, all human frames in the image to be detected can be first obtained through an object detection network, and then for each human frame, the key point coordinates of each human body are predicted by the key point detection method according to the present disclosure, and finally, the key point coordinates of the corresponding human body are mapped to the image to be detected according to the coordinates of each human frame in the image to be detected.

[0097] In the present disclosure, any target object method / model capable of detecting a target object in an image or video can be used to detect / crop an object frame in an image or video. Any human body detection method can be used to detect / crop all human body frames in an image as target image blocks.

[0098] Assuming that the target image block is an image block including the target human body, taking the cropping of the target image block according to the human body frame as an example, for each detected human body frame (also referred to as a coordinate frame), considering that the output of the object detection model / object detection network may have errors, resulting in the human body frame not being able to completely surround the edge of the human body, the human body frame can be enlarged to be able to surround more human body areas. For example, the human body frame can be enlarged by 1.25 times. Then, all areas in the image except the enlarged human body frame can be replaced with 0 (i.e., irrelevant interference factors are removed) and the short side of the human body frame can be filled to meet the aspect ratio input by the subsequent key point detection network (such as a single-person key point detection network). Finally, the image to be detected is cropped according to the processed human body frame, and the cropped target image block is adjusted to a preset size, such as 256×192. Here, if the aspect ratio of the human body frame is consistent with the aspect ratio input by the single-person key point detection network or the size of the cropped image block meets the preset size, there is no need to fill the short side of the human body frame or resize the image block.

[0099] After obtaining the target image block, the key point detection method disclosed in the present invention may be used to detect the key points of the target object.

[0100] Figure 5 A schematic diagram of key point detection according to an embodiment of the present disclosure is shown. The following description is made by taking the estimation of key points of a human body in an image or video as an example.

[0101] As an example, Figure 5The method shown can be mainly performed by four parts, namely, a feature extraction module (backbone), a key point coordinates initialization module (key point coords init module), a decoding module (decoder module), and a deformable mask decoding module (mask deformabledecoder module). Here, the feature extraction module can correspond to the feature extraction network described above, the key point coordinates initialization module can correspond to the first neural network described above, the decoding module can correspond to the second neural network described above, and the deformable mask decoding module can correspond to the third neural network described above. The operations performed by each module are described in detail below.

[0102] The feature extraction module can perform feature extraction on the input image (such as the target image block) to provide a multi-scale feature representation of the input image. The number of multi-scales can be adjusted according to actual needs and the network structure. For example, based on the structure of the selected feature extraction module (such as ResNet, HRNet, HRFormer, etc.), feature representations of four scales can be extracted. The input of the feature extraction module is a normalized image of size W×H×3 (the processed target image block), and the output is a multi-scale feature vector S1-S4, where S1-S4 are features of different sizes.

[0103] For example, you can Figure 4 The scaled single-person image block shown is input into the feature extraction network, and the multi-scale features S1-S4 of the input image are extracted by the feature extraction network. Taking Resnet50 as an example, the size of the input image is 256×192, and the features of each scale output are: S1[1,256,64,48], S2[1,512,32,24], S3[1,1024,16,12], S4[1,2048,8,6]. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0104] The key point coordinate initialization module may be composed of a global average pooling (GAP) layer and a fully connected (FC) layer for preliminary pose estimation. For example, the key point coordinate initialization module may predict the position R of the key point of the target object based on one or more scale features in the multi-scale features. 1 and its variance V on the horizontal and vertical axes 1 .

[0105] Figure 6 A schematic diagram of a first neural network (ie, a key point coordinate initialization module) according to an embodiment of the present disclosure is shown. The key point coordinate initialization module can be used to predict variance information and coordinate information (ie, position information) of key points, respectively.

[0106] Reference Figure 6 After obtaining multi-scale features S1-S4 through the feature extraction network, the variance information and coordinate information of the key points of the object can be predicted using the key point coordinate initialization module based on the minimum scale feature S4. For example, S4 is input to the GAP layer and then to the FC layer to obtain the variance information V and coordinate information V of the key points. 1 and location information R 1 Alternatively, S4 is input into the GAP layer and then respectively input into different FC layers (i.e., the key point variance prediction network and the key point coordinate prediction network) to obtain the variance information and position information of the key points. The above examples are merely exemplary and the present disclosure is not limited thereto.

[0107] The decoding module can be composed of a plurality of decoders (such as Transformer) stacked together to correct the key points with large errors in the preliminary positioning. Here, each decoder can correspond to a neural network unit in the second neural network described above.

[0108] The deformable mask decoding module can be composed of multiple stacked neural network units. While learning structured features through the deformable mask decoding module, multi-scale information is introduced, and the negative impact of invisible points is reduced through mask generation constrained by key point variance, and the key point position is obtained.

[0109] Before inputting into the decoding module, the multi-scale features S1-S4 can be processed. Taking S1 as an example, the number of channels is first changed to 256 through a 1×1 convolution operation, and then a Flatten operation is performed on it and reshaped into a feature with a dimension of [1, 64×48, 256] to obtain the processed F1′. Using the same operation, F2′, F3′, and F4′ can be obtained.

[0110] Based on F4′ and R 1 Generate partial input of the decoding module. This partial input can be composed of the query vector Q and the position vector P, where Q can be generated by F4′ through 1×1 convolution and P can be generated by R 1 Finally, F4′, P and Q are input into the decoding module to learn the structural information of the key points of the human body and predict the 1 The key point offset ΔR and the key point variance V 2 , the corrected key point coordinates R 2 Calculate using the following equation (2):

[0111] R 2 = R 1 +ΔR (2)

[0112] Optionally, the decoding module may directly output the corrected keypoint coordinates and variances without the need for calculation based on equation (2) above.

[0113] The upper left corner of the above letter is only used to distinguish the information of the previous prediction from the current prediction. In the case of multiple neural network units, the output of the previous neural network can be used as the input of the next neural network.

[0114] Figure 7 FIG. 2 is a schematic diagram showing a second neural network (i.e., a decoding module) according to an embodiment of the present disclosure. Figure 7 Only one neural network unit is shown, but in the present disclosure, it can be implemented using one neural network unit or by stacking multiple neural network units, each of which may include a self-attention network, a cross-attention network, a position prediction network, and a variance prediction network.

[0115] Reference Figure 7 , we can firstly base on the query vector Q t and position vector P generate the input v, k, and q of the self-attention network. Here, in the case of multiple neural network units stacked, for the first neural network unit, Q t It can be the query vector generated by F4′ through 1×1 convolution, and P can be the query vector generated by R 1 The position vector generated by trigonometric functions; for the second neural network unit, Q t The query vector may be the output of the first neural network unit, P may be the position vector obtained by position encoding the position information output by the first neural network unit, and so on. Where t is related to the number of neural network units in the decoding module.

[0116] Q t Add k and q to P, and add Q t As v. Then the output of the self-attention network is combined with Q t Add and normalize, input the normalized result and F4′ into the cross attention network, add and normalize the output of the cross attention network and the previous normalized result, and obtain the query vector Q for subsequent operations through the feedforward neural network FNN t+1 , use the position prediction network to get the position information of the key points, and use the variance prediction network to get the variance information of the key points.

[0117] The decoding module may include multiple neural network units, each of which includes at least a self-attention network, a cross-attention network, a position prediction network, and a variance prediction network, and the multiple neural network units are connected in series. The input of the first neural network unit in the multiple neural network units may be a query vector generated by F4′ through 1×1 convolution, F4′, and R 1 The position vector is generated by trigonometric functions, and the output of the first neural network unit is position-related information (i.e., position vector / position encoding), query vector, position information and variance information of key points, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the position information and variance information of the key points.

[0118] Figure 7 The network structure shown is only exemplary, and the present disclosure is not limited thereto.

[0119] Figure 8 FIG. 2 is a schematic diagram showing a third neural network (i.e., a deformable mask decoding module) according to an embodiment of the present disclosure. Figure 8 Only one neural network unit is shown, but in the present disclosure, one neural network unit can be used for implementation, or multiple neural network units can be stacked for implementation. Each neural network unit may include a self-attention network, a deformable attention network, a position prediction network, and a variance prediction network. The final key point coordinates R are obtained through the deformable mask decoding module. t And the key point variance V t , where t is related to the number of neural network units in the deformable mask decoding module.

[0120] Return to reference Figure 5 , the F1′, F2′, F3′ and F4′ vectors are concatenated (Concat) to form a memory vector containing multi-scale information as the input M of the deformable mask decoding module.

[0121] Reference Figure 8 , the mask generation network can use the above equation (1) to calculate the mask matrix and mask the key points with large variance.

[0122] Q t Add k and q to P, and add Q t Here, for the first neural network unit in the deformable mask decoding module, the query vector and position vector output by the decoding module and the mask matrix output by the mask generation network can be used as the input of the self-attention network. For the second neural network unit, Q tThe query vector may be the output of the first neural network unit, P may be the position vector obtained by position encoding the position information output by the first neural network unit, and so on. Where t is related to the number of neural network units in the deformable mask decoding module.

[0123] Then, the output of the self-attention network is combined with Q t Add and normalize, input the normalized result, memory vector and the position information of the previous prediction into the deformable attention network, add and normalize the output of the deformable attention network and the normalized result again, pass through FNN, and output the query vector Q t+1 , use the position prediction network to get the position offset information of the key points, and use the variance prediction network to get the variance V of the key points t+1 , add the predicted offset information to the previously predicted position information to obtain the final position information R t+1 . Optionally, the position prediction network can be used to directly predict the final position information without adding the offset information to the previously predicted position information. Here, for the first neural network unit, the previously predicted position information can be the position information output by the decoding module, for the second neural network unit it can be the position information output by the first neural network unit, and so on. The memory vector input into the neural network unit each time can be the same vector.

[0124] After obtaining the position information and variance of the key points finally output by the deformable mask decoding module, the key points with variance less than the threshold can be screened out according to a given threshold, so that key points that are invisible or very difficult to identify can be effectively shielded. For example, the threshold can be set to 0.5. The threshold can be set according to actual needs.

[0125] In other words, the variance information can be used to generate a mask to eliminate the influence of invisible key points on visible key points during structural prediction; in addition, the variance of the final output of the network can be further used as evidence of the existence of key points to further determine more accurate key points.

[0126] In the training phase, the key point coordinates and variances output by the key point coords init module, the decoder module, and the mask deformable decoder module are all involved in the calculation of the RLE loss. In the inference phase, the output of the last module is used as the final inference result. In the inference phase, the variance information output by the key point coords init module and the variance information output by the mask deformable decoder module are not needed.

[0127] The key point recognition method proposed in the present disclosure can effectively shield the interference of invisible points on structural information learning based on variance, thereby improving recognition accuracy.

[0128] Fig. 9 A block diagram of a key point detection device according to an embodiment of the present disclosure is shown. Fig. 9 The modules shown are only exemplary, and the names and quantities of the modules may be changed according to actual conditions.

[0129] The key point detection device 900 may include: a data acquisition module 901, configured to obtain preliminary detection results and first variance information of key points of a target object in an image to be detected, wherein the detection results include position information of the key points; a key point verification module 902, configured to perform key point verification on the preliminary detection results based on the first variance information; and a key point determination module 903, configured to determine the detection result of the target object based on the result of the key point verification.

[0130] Optionally, the key point verification module 902 may be configured to: generate a mask matrix for determining key points based on the first variance information; and determine a first key point of the target object based on the mask matrix and the preliminary detection result.

[0131] Optionally, the data acquisition module 901 can be configured to: obtain a target image block in the image to be detected, wherein the target image block includes at least the target object; perform feature extraction on the target image block to obtain feature information of the target image block; and obtain the preliminary detection result and the first variance information of the key points of the target object based on the feature information.

[0132] Optionally, the data acquisition module 901 can be configured to: use a first neural network to predict first position-related information of the key points of the target object based on at least part of the features in the feature information; and use a second neural network to predict the preliminary detection results and the first variance information of the key points of the target object based on the first position-related information and the at least part of the features.

[0133] Optionally, the second neural network includes at least a first self-attention network, a cross-attention network, a first position prediction network and a first variance prediction network, wherein the data acquisition module 901 can be configured to: generate a first query vector based on the at least part of the features; generate a first feature using the first self-attention network based on the first query vector and the first position-related information; generate a second feature using the cross-attention network based on the first feature and the at least part of the features; and predict the preliminary detection result and the first variance information based on the second feature, using the first position prediction network and the first variance prediction network, respectively.

[0134] Optionally, the key point determination module 903 can be configured to: obtain a target image block in the image to be detected, wherein the target image block at least includes the target object; perform feature extraction on the target image block to obtain feature information of the target image block; and use a third neural network to predict the position information of the final key point of the target object based on the feature information, the preliminary detection result and the first key point.

[0135] Optionally, the third neural network includes at least a second self-attention network, a deformable attention network, a second position prediction network and a second variance prediction network, wherein the key point determination module 903 can be configured to: generate a third feature using the second self-attention network based on the first key point and the second position-related information and the second query vector output by the second neural network; generate a fourth feature using the deformable attention network based on the third feature, the feature information and the preliminary detection result; and predict the position information of the final key point of the target object using the second position prediction network and the second variance prediction network, respectively, based on the fourth feature.

[0136] Optionally, the key point determination module 903 can be configured as: based on the fourth feature, using the second position prediction network, to obtain the position information of the key points of the target object; based on the fourth feature, using the second variance prediction network, to obtain the second variance information of the key points of the target object; based on the comparison of the second variance information with a threshold, determining the final key points of the target object, thereby obtaining the position information of the final key points of the target object.

[0137] Optionally, the second neural network includes multiple neural network units, each of the multiple neural network units includes at least the first self-attention network, the cross-attention network, the first position prediction network and the first variance prediction network, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first position-related information, the at least part of the features, and the first query vector, and the output of the first neural network unit is the position-related information, the query vector, the position information and variance information of the key points as intermediate quantities, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the preliminary detection result and the first variance information.

[0138] Optionally, the third neural network includes multiple neural network units, each of the multiple neural network units includes at least the second self-attention network, the deformable attention network, the second position prediction network and the second variance prediction network, and the multiple neural network units are connected in series; wherein the input of the first neural network unit among the multiple neural network units is the first key point, the second position-related information and the second query vector, the feature information and the preliminary detection result, and the output of the first neural network unit is the position-related information, query vector, key point position information and variance information as intermediate quantities, and the subsequent neural network unit uses the output of the previous neural network unit as input to perform operations until the last neural network unit outputs the final detection result.

[0139] It should be understood that each unit / module in the key point detection device according to the exemplary embodiment of the present disclosure can be implemented as a hardware component and / or a software component. Those skilled in the art can implement each unit / module, for example, using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), according to the processing performed by each defined unit / module.

[0140] According to another aspect of the exemplary embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the key point detection method described in the present disclosure is implemented.

[0141] Specifically, the key point detection method for a host device according to an exemplary embodiment of the present disclosure may be written as a computer program, a code segment, an instruction, or any combination thereof, and recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. The computer-readable storage medium is any data storage device that can store data read out by a computer system. Examples of computer-readable storage media include: read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, optical data storage device, and carrier wave (such as data transmission over the Internet via a wired or wireless transmission path).

[0142] According to another aspect of the exemplary embodiments of the present disclosure, an electronic device is provided, wherein the electronic device includes: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the key point detection method described in the present disclosure.

[0143] Specifically, the electronic device can be broadly a tablet computer, a smart phone, a smart watch, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the electronic device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the electronic device can be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device can be used to connect and communicate with external devices via a network.

[0144] At least some functions of the device or electronic device provided in the embodiments of the present disclosure can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0145] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0146] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0147] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0148] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0149] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0150] According to each step of the present disclosure, an artificial intelligence model can be used to implement it. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an artificial intelligence model input. The artificial intelligence model can be obtained through training. Here, "obtained through training" means that a basic artificial intelligence model is trained with multiple training data through a training algorithm to obtain a predefined operation rule or artificial intelligence model configured to perform the desired feature (or purpose).

[0151] An electronic device is also provided in an embodiment of the present disclosure, which includes at least one processor and, optionally, may also include at least one transceiver and / or at least one memory coupled to the at least one processor, and the at least one processor is configured to execute the steps of the method provided in any optional embodiment of the present disclosure.

[0152] Fig.10 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present disclosure is shown in FIG. Fig.10 As shown, Fig.10The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, each of the processor 4001, the memory 4003 and the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure. Optionally, the electronic device may be a first network node, a second network node or a third network node.

[0153] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0154] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0155] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0156] The memory 4003 is used to store computer programs or executable instructions for executing the embodiments of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the above method embodiments.

[0157] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program or instructions stored thereon. When the computer program or instructions are executed by at least one processor, the steps and corresponding contents of the aforementioned method embodiment can be executed or implemented.

[0158] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0159] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that shown or described in the text.

[0160] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0161] The above text and drawings are provided only as examples to help readers understand the present disclosure. They are not intended and should not be interpreted as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present disclosure, and other similar implementation means based on the technical ideas of the present disclosure are adopted, which also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A key point detection method, characterized in that: The method comprises: Acquire a preliminary detection result and first variance information of a key point of a target object in an image to be detected, wherein the preliminary detection result includes position information of the key point; Performing key point verification on the preliminary detection result based on the first variance information; Based on the result of the key point verification, a detection result of the target object is determined.

2. The method according to claim 1, characterized in that The performing key point verification on the preliminary detection result based on the first variance information includes: Based on the first variance information, generating a mask matrix for determining key points; Based on the mask matrix and the preliminary detection result, a first key point of the target object is determined.

3. The method according to claim 1, characterized in that The obtaining of the preliminary detection result and the first variance information of the key points of the target object includes: Acquire a target image block in the image to be detected, wherein the target image block at least includes the target object; Performing feature extraction on the target image block to obtain feature information of the target image block; The preliminary detection result of the key point of the target object and the first variance information are obtained based on the feature information.

4. The method according to claim 3, characterized in that The obtaining the preliminary detection result and the first variance information of the key point of the target object based on the feature information includes: Based on at least some of the features in the feature information, using a first neural network to predict first position-related information of a key point of the target object; Based on the first position-related information and the at least part of the features, a second neural network is used to predict the preliminary detection result of the key point of the target object and the first variance information.

5. The method according to claim 4, characterized in that The second neural network at least includes a first self-attention network, a cross-attention network, a first position prediction network and a first variance prediction network, The step of predicting the preliminary detection result and the first variance information of the key points of the target object using a second neural network based on the first position-related information and the at least part of the features includes: Based on the at least part of the features, generating a first query vector; Based on the first query vector and the first position related information, using the first self-attention network to generate a first feature; Based on the first feature and the at least part of the feature, generating a second feature using the cross-attention network; Based on the second feature, the first position prediction network and the first variance prediction network are used respectively to predict the preliminary detection result and the first variance information.

6. The method according to claim 2, characterized in that The step of determining the detection result of the target object based on the result of the key point verification includes: Acquire a target image block in the image to be detected, wherein the target image block at least includes the target object; Performing feature extraction on the target image block to obtain feature information of the target image block; Based on the feature information, the preliminary detection result and the first key point, a third neural network is used to predict the position information of the final key point of the target object.

7. The method according to claim 6, characterized in that The third neural network includes at least a second self-attention network, a deformable attention network, a second position prediction network, and a second variance prediction network. The method of predicting the position information of the final key point of the target object using a third neural network based on the feature information, the preliminary detection result and the first key point includes: Generate a third feature using the second self-attention network based on the first key point and the second position-related information output by the second neural network and a second query vector; Based on the third feature, the feature information and the preliminary detection result, generating a fourth feature using the deformable attention network; Based on the fourth feature, the second position prediction network and the second variance prediction network are used respectively to predict the position information of the final key points of the target object.

8. A key point detection device, characterized in that: The device comprises: A data acquisition module is configured to acquire a preliminary detection result and first variance information of a key point of a target object in an image to be detected, wherein the detection result includes position information of the key point; A key point verification module, configured to perform key point verification on the preliminary detection result based on the first variance information; The key point determination module is configured to determine the detection result of the target object based on the result of the key point verification.

9. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor, the key point detection method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the key point detection method according to any one of claims 1 to 7.

Images