Key point detection method and device, and storage medium

Abstract

The application provides a key point detection method and device and a storage medium. The method comprises: acquiring a to-be-detected image; processing the to-be-detected image by using a preset convolutional network model to obtain key point position information in the to-be-detected image; wherein the preset convolutional network model is obtained by training based on key point heat map errors between a plurality of sample images and respectively corresponding real images and an elastic convolution module in each convolution stage. In the training process of the preset convolutional network model, the key point heat map errors between the plurality of sample images and the respectively corresponding real images are considered, and the sample images are trained by using the elastic convolution model, so that the parameter quantity of the network is reduced, and the accuracy and efficiency of key point detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a key point detection method, apparatus and storage medium. Background Technology

[0002] Faces play a crucial role in visual communication. In computer vision, facial landmark localization is essential for obtaining non-verbal facial information such as gender and emotion. High-resolution images often yield more accurate feature information; however, inconsistent image clarity across different facial landmark detection datasets can negatively impact detection accuracy. In related technologies, High-Resolution Representations for Labeling Pixels and Regions (HRNetV2) uses bilinear upsampling to amplify low-resolution data from different channels to the maximum possible resolution, then fuses these features to obtain the final high-resolution features for landmark detection. HRNetV2 employs numerous 3×3 convolutions, resulting in a large number of network parameters and demanding high-performance equipment, leading to low detection efficiency. Furthermore, HRNetV2 uses the common L2 loss function to optimize the network. While L2 loss is ideal for optimizing large errors in training samples, it is less effective for heatmap regression where the shape of landmark heatmaps varies. Using L2 to optimize errors can cause the model to misjudge some points, resulting in low detection accuracy. Summary of the Invention

[0003] The present invention provides a key point detection method, apparatus and storage medium that can improve the accuracy and efficiency of key point detection.

[0004] The technical solution of this invention is implemented as follows:

[0005] This invention provides a key point detection method, including:

[0006] Acquire the image to be tested;

[0007] The image to be tested is processed using a pre-defined convolutional network model to obtain the location information of multiple key points in the image; wherein...

[0008] The preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage.

[0009] In the above scheme, before processing the image to be tested using a preset convolutional network model to obtain the location information of multiple key points in the image to be tested, the method further includes:

[0010] Each acquired sample image and its corresponding real image are input into an initial convolutional network model. The elastic convolution module in the first convolutional stage processes each sample image to obtain a first feature matrix. The real image is an image in which multiple real key points are annotated for each sample image.

[0011] Based on the acquired user instructions, and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage, the first feature matrix is ​​processed to obtain the second feature matrix, and the Euler angle difference corresponding to each sample image is calculated based on the second feature matrix and the multiple sample images; N is an integer greater than or equal to 2;

[0012] The second feature matrix is ​​processed by combining the elastic convolution modules contained in each of the N+1 convolution branches in the third convolution stage to obtain the third feature matrix.

[0013] The fourth feature matrix is ​​obtained by processing the third feature matrix by combining the elastic convolution modules contained in each of the N+2 convolution branches in the fourth convolution stage.

[0014] The predicted keypoint heatmap is obtained by fusing the fourth feature matrix to determine the location information of multiple training keypoints, and the function loss is calculated based on the predicted keypoint heatmap, the real image and the Euler angle difference.

[0015] The initial convolutional network model is trained based on the function loss until a predetermined condition is met, at which point the preset convolutional network model is obtained.

[0016] In the above scheme, the step of processing each sample image to obtain the first feature matrix through the elastic convolution module in the first convolution stage includes:

[0017] Each sample image is converted into a first initial matrix;

[0018] The first initial matrix is ​​convolved using a first convolution kernel of a predetermined size, and then processed by the elastic convolution module to obtain the first feature matrix.

[0019] In the above scheme, the step of convolving the first initial matrix with a first convolution kernel of a predetermined size, and then processing it through the elastic convolution module to obtain the first feature matrix includes:

[0020] The first initial matrix is ​​convolved with M first convolution kernels to obtain the first intermediate feature matrix; M is an integer greater than or equal to 2.

[0021] The first intermediate feature matrix is ​​obtained by sequentially convolving it with a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolution kernels; K is the same as the number of input channels of each sample image.

[0022] In the above scheme, the step of processing the first feature matrix to obtain the second feature matrix based on the acquired user instructions and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage includes:

[0023] In response to the first user instruction, the first feature matrix is ​​processed by combining the first elastic convolution modules composed of K single-channel convolution kernels and multiple pointwise convolution kernels in each convolution branch to obtain the corresponding second feature matrix, and then N second feature matrices are obtained.

[0024] In the above scheme, the step of processing the first feature matrix to obtain the second feature matrix based on the acquired user instructions and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage includes:

[0025] In response to the acquired second user instruction, the first feature matrix is ​​processed by combining a pointwise convolution kernel, K single-channel convolution kernels and multiple pointwise convolutions in each convolution branch to obtain the corresponding second feature matrix, and then N second feature matrices are obtained.

[0026] In the above scheme, calculating the Euler angle difference corresponding to each sample image based on the second feature matrix and the plurality of sample images includes:

[0027] The second feature matrix is ​​processed by a pre-set auxiliary network model to obtain the predicted Euler angles;

[0028] An average sample image is calculated based on the multiple sample images, and multiple reference key point location information is extracted from the average sample image;

[0029] In each sample image, multiple key point location information is extracted, and a rotation matrix is ​​calculated using the multiple reference key point location information and the multiple key point location information.

[0030] The reference Euler angles are determined based on the rotation matrix;

[0031] The nutation angle difference, precession angle difference, and rotation angle difference are calculated using the predicted Euler angles and the reference Euler angles.

[0032] In the above scheme, the predicted key point heatmap includes: multiple predicted key points corresponding to the multiple real key points;

[0033] The loss function calculated based on the predicted keypoint heatmap, the real image, and the Euler angle difference includes:

[0034] The loss weight is calculated based on the nutation angle difference, the precession angle difference, and the rotation angle difference;

[0035] Based on the pixel values ​​of each real keypoint and the corresponding predicted keypoint, the intermediate function loss corresponding to each predicted keypoint is calculated by combining the preset value and the loss weight.

[0036] The average of the intermediate function losses of the multiple predicted key points is calculated to obtain the function loss.

[0037] In the above scheme, the loss weight is calculated based on the nutation angle difference, the precession angle difference, and the rotation angle difference, including:

[0038] Calculate the three cosine values ​​corresponding to the nutation angle difference, the precession angle difference, and the rotation angle difference for each predicted key point;

[0039] Calculate the three differences between the first preset value and the three cosine values ​​respectively, and add the three differences to obtain the sum of the differences for each predicted key point;

[0040] The sums of the multiple differences of the multiple predicted key points are added together to obtain the second difference sum for each sample image;

[0041] The sum of the multiple second differences of the multiple sample images is added together to obtain the third difference sum, and then the third difference sum is divided by the number of the multiple sample images to obtain the loss weight.

[0042] In the above scheme, the step of calculating the intermediate function loss corresponding to each predicted keypoint based on the pixel values ​​of each real keypoint and the corresponding predicted keypoint, combined with a preset value and the loss weight, includes:

[0043] If the absolute value of the difference between the pixel value of each real key point and the corresponding predicted key point is less than the second preset value, then the absolute value of the ratio of the difference to the third preset value is used to obtain the intermediate value.

[0044] Calculate the intermediate difference between the fourth preset value and the pixel value of each actual key point;

[0045] Calculate the intermediate difference power of the intermediate value, and then add it to the fifth preset value to obtain the second intermediate value;

[0046] Calculate the natural logarithm of the second intermediate value, multiply the natural logarithm by the loss weight and by the sixth preset value to obtain the intermediate function loss.

[0047] In the above scheme, the step of calculating the intermediate function loss corresponding to each predicted keypoint based on the pixel values ​​of each real keypoint and the corresponding predicted keypoint, combined with a preset value and the loss weight, includes:

[0048] If the absolute value of the difference between the pixel value of each real keypoint and the corresponding predicted keypoint is not less than the second preset value, then the fourth difference sum of the absolute value of the difference and the seventh preset value is calculated, and the eighth preset value is subtracted from the fourth difference sum to obtain the intermediate function loss; the seventh preset value and the eighth preset value are both calculated based on the pixel value of each real keypoint and in combination with the preset value.

[0049] This invention also provides a key point detection device, comprising:

[0050] An image acquisition unit is used to acquire the image to be tested.

[0051] The processing unit is used to process the image to be tested using a preset convolutional network model to obtain a key point heatmap; wherein, the preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage.

[0052] The processing unit is also used to obtain multiple key point location information based on the key point heatmap.

[0053] This invention also provides a key point detection device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above method.

[0054] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0055] In this embodiment of the invention, a test image is acquired; a preset convolutional network model is used to process the test image to obtain the location information of multiple key points in the test image; wherein, the preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage. Because this scheme considers the key point heatmap errors between multiple sample images and their corresponding real images during the training process of the preset convolutional network model, and uses an elastic convolution model to train the sample images, the number of network parameters is reduced, thereby improving the accuracy and efficiency of key point detection. Attached Figure Description

[0056] Figure 1An optional effect diagram of the related technology provided in the embodiments of the present invention;

[0057] Figure 2 An optional effect diagram of the related technology provided in the embodiments of the present invention;

[0058] Figure 3 An optional effect diagram of the related technology provided in the embodiments of the present invention;

[0059] Figure 4 A schematic diagram of an optional process for a key point detection method provided in an embodiment of the present invention;

[0060] Figure 5 This is an optional effect diagram of the key point detection method provided in the embodiment of the present invention;

[0061] Figure 6 This is an optional effect diagram of the key point detection method provided in the embodiment of the present invention;

[0062] Figure 7 This is an optional effect diagram of the key point detection method provided in the embodiment of the present invention;

[0063] Figure 8 A schematic diagram of an optional process for a key point detection method provided in an embodiment of the present invention;

[0064] Figure 9 This is an optional effect diagram of the key point detection method provided in the embodiment of the present invention;

[0065] Figure 10 This is an optional effect diagram of the key point detection method provided in the embodiment of the present invention;

[0066] Figure 11 This is a schematic diagram of the key point detection device provided in an embodiment of the present invention;

[0067] Figure 12 This is a schematic diagram of a hardware entity of the key point detection device provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0070] If similar descriptions such as "first / second" appear in the invention document, the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the invention described herein can be implemented in an order other than that illustrated or described herein.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0072] Facial landmark detection, also known as face alignment, aims to identify the locations of facial landmarks in an image or video. These locations are typically features with specific semantic information, such as the eyes, nose, mouth, cheeks, and eyebrows. Given a 2D face image, the face alignment algorithm predicts the locations of the landmarks L = {x1, y1, x2, y2, ... x}. D y D} Here, x and y represent the two-dimensional coordinates of facial key points, and the value of D varies in different datasets, with common values ​​being 68 and 98.

[0073] Currently, the main approach to keypoint detection in low-resolution face images is based on high-resolution representation learning strategies, primarily using HRNetV2. The HRNetV2 network consists of four main stages, such as... Figure 1 As shown in the diagram, the convolutional stages are divided into four phases: Phase 1, Phase 2, Phase 3, and Phase 4. Phase 1 includes one convolutional branch, Phase 2 includes two convolutional branches, Phase 3 includes three convolutional branches, and Phase 4 includes four convolutional branches. Each phase consists of a multi-resolution module, which has two convolutions: multi-resolution group convolution and multi-resolution convolution. The multi-resolution group convolution extends the group convolution by dividing the input channels into subsets of the original channels, and then performing regular convolution operations on each subset with different spatial resolutions. Features are extracted at each phase using encapsulated basic blocks and bottlenecks.

[0074] For facial landmark detection, HRNetV2 uses bilinear upsampling to amplify the low resolution of different channels to the maximum possible resolution, and then fuses these features to obtain the final high-resolution features for landmark detection. During HRNetV2 network training, the L2 loss function is used for optimization; L2 loss is a commonly used loss function in face alignment tasks.

[0075] The core components of HRNetV2 for feature extraction are the basic block and the bottleneck. The basic block structure is as follows: Figure 2 As shown, the basic block mainly consists of two 3×3 convolutions. The bottleneck structure is as follows: Figure 3 As shown, the bottleneck mainly consists of one 1×1 convolution, one 3×3 convolution, and one 1×1 convolution. HRNetV2 uses the bottleneck to extract features in the first stage, and uses four basic blocks to extract features in the following three stages.

[0076] The large number of 3×3 convolutions results in a large number of network parameters, placing high demands on memory and graphics processing units (GPUs) during training. Meanwhile, HRNetV2 uses the standard L2 loss function to optimize the network, which is ideal for optimizing large errors in training samples. Given an image, the ground truth keypoints can be represented as: S = {(x1, y1), ..., (x...} n y n )} belongs to R n The predicted point is: S*={(x1) * y1 * ), …(x n * y n * )} belongs to R n The distance between these two points can be expressed as: d = Si - Si*. The mathematical expression for L2 is formula (1):

[0077] L2 = 1 / 2d 2 (1)

[0078] However, for heatmap regression, the display shape of keypoint heatmaps varies, and directly using L2 error optimization can lead to misjudgments of some points. For devices with limited computing resources, models based on high-resolution representation learning strategies cannot be well deployed and applied. Therefore, the problems with keypoint detection in related technologies are low accuracy and efficiency.

[0079] This invention provides a key point detection method; please refer to [link / reference]. Figure 4This is an optional flowchart illustrating the key point detection method provided in an embodiment of the present invention, which will be combined with... Figure 4 The steps shown are explained.

[0080] S101. Obtain the image to be tested.

[0081] In this embodiment of the invention, the key point detection device acquires the image to be tested.

[0082] In this embodiment of the invention, the key point detection device can acquire the image to be tested through an image acquisition component. The key point detection device can also acquire images to be tested sent by other devices.

[0083] The key point detection device can be a terminal, server, or wearable terminal device.

[0084] In this embodiment of the invention, the image to be tested can be a picture or video frame including a predetermined target object. The target object can include predetermined objects such as cars, faces, animals, computers, and mobile phones.

[0085] S102. The image to be tested is processed using a pre-defined convolutional network model to obtain the location information of multiple key points in the image to be tested. The pre-defined convolutional network model is trained based on the key point heatmap error between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage.

[0086] In this embodiment of the invention, the key point detection device processes the image under test using a preset convolutional network model to obtain the location information of multiple key points in the image under test. The preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with elastic convolutional modules in each convolutional stage.

[0087] In this embodiment of the invention, the keypoint detection device processes the image under test using a preset convolutional network model to obtain a keypoint heatmap. Based on the keypoint heatmap, the preset convolutional network model of the keypoint detection device determines the location information of multiple keypoints.

[0088] In this embodiment of the invention, the keypoint detection device estimates the initial coordinates of multiple keypoints in the keypoint heatmap based on maximum likelihood estimation, and obtains the coordinates of adjacent points corresponding to each keypoint. The keypoint detection device uses an interpolation method to determine the position coordinate information of multiple keypoints based on the initial coordinates of the multiple keypoints and the coordinates of adjacent points.

[0089] In this embodiment of the invention, multiple key points may include: parts of a person in a face image that have specific semantic information, such as the eyes, nose, mouth, cheeks, and eyebrows. In other embodiments, key points may also be key points in other corresponding objects.

[0090] In this embodiment of the invention, the keypoint detection device uses an elastic convolution module to improve upon the 3×3 ordinary convolution in the basic block and bottleneck of HRNetV2. The elastic convolution module consists of depthwise separable convolution and inverse residual convolution, resulting in two new networks named HR-brief1 and HR-brief2. These two networks can be deployed on different hardware environments. For devices with lower GPU performance, the HR-brief1 model can be deployed. For devices with higher GPU performance and where higher detection accuracy is required, HR-brief2 can be deployed.

[0091] Depthwise separable convolution is a combination of two types of convolution: depthwise convolution and pointwise convolution. A regular convolution kernel has the same number of channels as the input feature map, typically 3×3×3. However, depthwise convolution splits the kernel into single-channel forms, resulting in a 3×3×1 kernel. Essentially, it performs convolution on a two-dimensional plane, with the number of kernels matching the number of channels in the input feature map. This convolution of each channel of the feature map results in an output feature map with the same number of channels. However, the resulting feature map has too few channels to fully capture the inter-channel relationships. Pointwise convolution, i.e., multiple 1×1 convolutions, is then used to change the dimensionality of the feature map, thus obtaining a multi-dimensional feature map that greatly preserves the effective spatial information between features. Inverse residual convolution adds a pointwise convolution before depthwise separable convolution to expand the channels, and the final output feature uses a Linear activation function instead of ReLU6. Depthwise separable convolution is suitable for ordinary devices; inverse residual convolution requires expanding channels at the beginning, has relatively high memory requirements, and produces a smaller overall model, making it suitable for devices with high computing performance. These two convolution methods are highly related and can make full use of computing resources, accelerating network training while ensuring detection accuracy.

[0092] During deployment, the keypoint detection device can select a specific convolution method based on the training script to obtain the corresponding model. The structure of the elastic convolution module is as follows: Figure 5 As shown. The keypoint detection device inputs the image to be tested and processes it through the corresponding elastic convolution module of GPU-low or GPU-high according to preset parameters. The elastic convolution module for GPU-low includes: Dwise 3x3 and Conv 1x1, stride=s convolutions; the elastic convolution module for GPU-high includes: Conv 1x1, stride=s, Dwise 3x3 and Conv 1x1, stride=s convolutions. The activation function of the elastic convolution module for GPU-high is Linear. The improved basicblock is as follows. Figure 6 As shown, the improved basicblock includes two elastic convolutional modules. The improved bottleneck is as follows: Figure 7 As shown, the improved bottleneck includes a 1x1.64 convolution, a resilient convolution module, and a 1x1.256 convolution. The activation function for the bottleneck is Linear.

[0093] In this embodiment of the invention, a test image is acquired; a preset convolutional network model is used to process the test image to obtain the location information of multiple key points in the key point heatmap; wherein, the preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage. Because this scheme considers the key point heatmap errors between multiple sample images and their corresponding real images during the training process of the preset convolutional network model, and uses an elastic convolution model to train the sample images, the number of network parameters is reduced, thereby improving the accuracy and efficiency of key point detection.

[0094] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of an optional flowchart of the key point detection method provided in an embodiment of the present invention. Figure 4 The steps shown in S102 may also include S103 to S108, which will be explained in conjunction with each step.

[0095] S103. Input each acquired sample image and its corresponding real image into the initial convolutional network model, and process each sample image through the elastic convolution module in the first convolution stage to obtain the first feature matrix.

[0096] In this embodiment of the invention, the key point detection device inputs each acquired sample image and the corresponding real image into the initial convolutional network model, and processes each sample image through the elastic convolution module in the first convolution stage to obtain the first feature matrix.

[0097] The real image is an image in which multiple real key points are annotated for each sample image.

[0098] In this embodiment of the invention, the key point detection device inputs each acquired sample image and the corresponding real image into the initial convolutional network model, and processes each sample image through the elastic convolution module in the first convolution stage to obtain the first feature matrix.

[0099] In this embodiment of the invention, a sample image can be viewed as a three-dimensional matrix, where each number represents the brightness of different colors at different locations in the image. However, the image is not stored directly as these three-dimensional matrices; instead, they are compressed and encoded before storage. Therefore, the initial process of the convolutional network model reading a sample image is actually a process of first reading its compressed and encoded result, and then decoding it.

[0100] S104. Based on the acquired user instructions, and combining the elastic convolution modules contained in each of the N convolution branches in the second convolution stage, process the first feature matrix to obtain the second feature matrix, and calculate the Euler angle difference corresponding to each sample image based on the second feature matrix and multiple sample images.

[0101] In this embodiment of the invention, the key point detection device, based on the acquired user instructions and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage, processes the first feature matrix to obtain a second feature matrix, and calculates the Euler angle difference corresponding to each sample image based on the second feature matrix and multiple sample images. N is an integer greater than or equal to 2.

[0102] In this embodiment of the invention, the user sends user instructions (which can be predetermined codes or predetermined characters) based on the device performance of the keypoint detection device. When the keypoint detection device is a GPU-low device, the user can send a first user instruction; when the keypoint detection device is a GPU-high device, the user can send a second user instruction. The first user instruction is used to selectively train the elastic convolution modules corresponding to the GPU-low stage from the second to the fourth convolution stages. The second user instruction is used to selectively train the elastic convolution modules corresponding to the GPU-high stage from the second to the fourth convolution stages.

[0103] In this embodiment of the invention, the second convolution stage may include two convolution branches, each convolution branch may include four elastic convolution modules corresponding to GPU-low and four elastic convolution modules corresponding to GPU-high.

[0104] S105. Combining the elastic convolution modules contained in each of the N+1 convolution branches in the third convolution stage, the second feature matrix is ​​processed to obtain the third feature matrix.

[0105] In this embodiment of the invention, the key point detection device combines the elastic convolution modules contained in each of the N+1 convolution branches in the third convolution stage to process the second feature matrix to obtain the third feature matrix.

[0106] In this embodiment of the invention, the key point detection device can fuse the obtained second feature matrix to obtain a feature representation map, which is then input into the third convolution stage for convolution processing to obtain the third feature matrix.

[0107] In this embodiment of the invention, the third convolution stage may include three convolution branches, each of which may include four elastic convolution modules corresponding to GPU-low and four elastic convolution modules corresponding to GPU-high. The keypoint detection device may selectively train either the elastic convolution modules corresponding to GPU-low or GPU-high based on the acquired user instructions.

[0108] S106. Combining the elastic convolution modules contained in each of the N+2 convolution branches in the fourth convolution stage, the third feature matrix is ​​processed to obtain the fourth feature matrix.

[0109] In this embodiment of the invention, the key point detection device combines the elastic convolution modules contained in each of the N+2 convolution branches in the fourth convolution stage to process the third feature matrix to obtain the fourth feature matrix.

[0110] In this embodiment of the invention, the key point detection device can fuse the obtained third feature matrix to obtain a feature representation map, which is then input into the fourth convolution stage for convolution processing to obtain the fourth feature matrix.

[0111] In this embodiment of the invention, the fourth convolutional stage may include four convolutional branches, each of which may include four elastic convolutional modules corresponding to GPU-low and four elastic convolutional modules corresponding to GPU-high. The keypoint detection device can selectively train the elastic convolutional modules corresponding to GPU-low or GPU-high according to the acquired user instructions.

[0112] S107. The predicted key point heatmap is obtained by fusing the fourth feature matrix to determine the location information of multiple training key points, and the function loss is calculated based on the predicted key point heatmap, the real image and the Euler angle difference.

[0113] In this embodiment of the invention, the key point detection device obtains a predicted key point heatmap by fusing the fourth feature matrix to determine the location information of multiple training key points, and calculates the function loss based on the predicted key point heatmap, the real image and the Euler angle difference.

[0114] In this embodiment of the invention, after the keypoint detection device obtains a predicted keypoint heatmap through fusion of the fourth feature matrix, it estimates the initial coordinates of multiple training keypoints in the predicted keypoint heatmap using maximum likelihood estimation and obtains the coordinates of adjacent points corresponding to each training keypoint. The keypoint detection device then uses interpolation to determine the position coordinates of the multiple training keypoints based on their initial coordinates and the coordinates of their adjacent points.

[0115] S108. Train the initial convolutional network model based on the function loss until the predetermined conditions are met, and then stop to obtain the preset convolutional network model.

[0116] In this embodiment of the invention, the key point detection device trains the initial convolutional network model based on function loss until a predetermined condition is met, at which point the preset convolutional network model is obtained.

[0117] In this embodiment of the invention, if the function loss is greater than the loss threshold, the key point detection device continues to train the initial convolutional network model until a certain number of training iterations are reached or the function converges, at which point the preset convolutional network model is obtained.

[0118] In this embodiment of the invention, the training of the initial convolutional network model can be represented by the following stages: forward propagation, backward propagation, and weight update. The forward propagation stage involves transmitting the sample image from the input layer to the output layer. The backward propagation stage involves transmitting the image from the output layer to the input layer. The initial convolutional network model calculates the function loss of the corresponding sample image based on the loss function. If this loss is greater than a loss threshold, the network structure of the initial convolutional network model will backpropagate from the output layer to the intermediate layers and then to the input layer, adjusting the weights of each layer using gradient descent. After the weights of each layer of the initial convolutional network model are adjusted, the network structure will continue to train on newly acquired sample images. The process of training the initial convolutional network model to obtain the preset convolutional network model continues until the current loss calculated by the initial convolutional network model is no greater than the loss threshold, or until the number of training iterations of the initial convolutional network model reaches a preset number of training iterations, thus obtaining the preset convolutional network model.

[0119] In this embodiment of the invention, each sample image and the corresponding real image are input into an initial convolutional network model for training. During the training process, the corresponding elastic convolutional module can be trained according to the user's instructions to adapt to devices with different performance. Moreover, Euler angle difference is introduced into the calculation function loss during the training process, which makes the key point error between each sample image and the corresponding real image calculated by the initial convolutional network model more accurate. As a result, a preset convolutional network model with high detection accuracy and efficiency can be obtained.

[0120] Figure 8 The shown S103 can also be implemented by S109 to S110, which will be explained in conjunction with each step.

[0121] S109. Convert each sample image into a first initial matrix.

[0122] In this embodiment of the invention, the key point detection device converts each sample image into a first initial matrix.

[0123] S110. After convolving the first initial matrix with a first convolution kernel of a predetermined size, the first feature matrix is ​​obtained by further processing it through the elastic convolution module.

[0124] In this embodiment of the invention, the key point detection device uses a first convolution kernel of a predetermined size to convolve the first initial matrix, and then processes it through the elastic convolution module to obtain the first feature matrix.

[0125] In this embodiment of the invention, the key point detection device can use a 3×3 first convolution kernel to convolve the first initial matrix, and then process it through an elastic convolution module selected according to user instructions to obtain the first feature matrix.

[0126] The shown S110 can also be implemented by S111 to S112, which will be explained in conjunction with each step.

[0127] S111. The first initial matrix is ​​convolved using M first convolution kernels to obtain the first intermediate feature matrix.

[0128] In this embodiment of the invention, the key point detection device uses M first convolution kernels to convolve the first initial matrix to obtain a first intermediate feature matrix. M is an integer greater than or equal to 2.

[0129] The first convolution kernel can be a 3×3 convolution kernel.

[0130] S112. The first intermediate feature matrix is ​​sequentially convolved using a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolution kernels to obtain the first feature matrix.

[0131] In this embodiment of the invention, the keypoint detection device uses a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolution kernels to sequentially convolve the first intermediate feature matrix to obtain the first feature matrix. K is the same as the number of input channels for each sample image, and K is a positive integer.

[0132] The pointwise convolution kernel can be a 1x1.64 kernel. The single-channel convolution kernel can be a 3x3x1 kernel. Multiple pointwise convolution kernels can be 1x1.256 kernels respectively.

[0133] In this embodiment of the invention, a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolution kernels constitute the elastic convolution module of the first convolution stage, which is smaller than conventional convolution kernels and improves detection and processing efficiency.

[0134] Figure 8 The shown S104 can also be implemented through S113, which will be explained in conjunction with each step.

[0135] S113. In response to the acquired first user instruction, the first elastic convolution module, composed of K single-channel convolution kernels and multiple pointwise convolution kernels in each convolution branch, is processed to obtain the corresponding second feature matrix, thereby obtaining N second feature matrices.

[0136] In this embodiment of the invention, the key point detection device responds to the acquired first user instruction and combines each first elastic convolution module, which consists of K single-channel convolution kernels and multiple point-by-point convolution kernels in each convolution branch in sequence, to process the first feature matrix to obtain the corresponding second feature matrix, and then obtains N second feature matrices.

[0137] The first user instruction can be the user instruction for the corresponding GPU-low device. The keypoint detection device combines K single-channel convolutional kernels and multiple point-by-point convolutional kernels in N convolutional branches to form each first elastic convolutional module, processes the first feature matrix to obtain the corresponding second feature matrix, and then obtains N second feature matrices corresponding to the N convolutional branches.

[0138] The number of each first elastic convolution module can be four or five, and there is no limitation in this embodiment of the invention.

[0139] Figure 8 The shown S104 can also be implemented via S114, which will be explained in conjunction with each step.

[0140] S114. In response to the acquired second user instruction, combine each second elastic convolution module consisting of a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolutions in each convolution branch in sequence to process the first feature matrix to obtain the corresponding second feature matrix, and then obtain N second feature matrices.

[0141] In this embodiment of the invention, the key point detection device responds to the acquired second user instruction and combines each second elastic convolution module, which consists of a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolutions arranged in sequence in each convolution branch, to process the first feature matrix to obtain the corresponding second feature matrix, and then obtains N second feature matrices.

[0142] The second user instruction can be the user instruction corresponding to the GPU-high device. The key point detection device combines a pointwise convolutional kernel from N convolutional branches, K single-channel convolutional kernels, and multiple pointwise convolutional kernels arranged in sequence to form various second elastic convolutional modules. It processes the first feature matrix to obtain the corresponding second feature matrix, and then obtains N second feature matrices corresponding to the N convolutional branches.

[0143] The number of the second elastic convolutional modules is the same as the number of the first elastic convolutional modules.

[0144] In this embodiment of the invention, elastic convolutional modules corresponding to different GPU performance devices can be trained according to a first user instruction or a second user instruction, which can adapt to different types of devices and improve detection efficiency.

[0145] Figure 8 The shown S105 can also be implemented via S115, which will be explained in conjunction with each step.

[0146] S115. The key point detection device combines the first elastic convolution modules composed of K single-channel convolution kernels and multiple point-by-point convolution kernels in N+1 convolution branches in sequence to process the second feature matrix to obtain the corresponding third feature matrix, and then obtains the N+1 third feature matrices corresponding to the N+1 convolution branches.

[0147] The key point detection device combines a pointwise convolution kernel from N+1 convolution branches, K single-channel convolution kernels, and multiple pointwise convolution kernels arranged in sequence to form various second elastic convolution modules. It processes the second feature matrix to obtain the corresponding third feature matrix, and then obtains the N+1 third feature matrices corresponding to the N+1 convolution branches.

[0148] Figure 8 The shown S106 can also be implemented via S116, which will be explained in conjunction with each step.

[0149] S116. The key point detection device combines the first elastic convolution modules composed of K single-channel convolution kernels and multiple point-by-point convolution kernels in N+2 convolution branches in sequence to process the third feature matrix to obtain the corresponding fourth feature matrix, and then obtains N+2 fourth feature matrices corresponding to the N+2 convolution branches.

[0150] The key point detection device combines a pointwise convolutional kernel from N+2 convolutional branches, K single-channel convolutional kernels, and multiple pointwise convolutional kernels arranged in sequence to form various second elastic convolutional modules. It processes the third feature matrix to obtain the corresponding fourth feature matrix, and then obtains the N+2 fourth feature matrices corresponding to the N+2 convolutional branches.

[0151] Figure 8 The calculation of the Euler angle difference corresponding to each sample image based on the second feature matrix and the plurality of sample images can also be achieved through S117 to S121, which will be explained in conjunction with each step.

[0152] S117. The second feature matrix is ​​processed by a preset auxiliary network model to obtain the predicted Euler angles.

[0153] In this embodiment of the invention, the key point detection device processes the second feature matrix through a preset auxiliary network model to obtain predicted Euler angles.

[0154] In this embodiment of the invention, the key point detection device fuses the second feature matrix to obtain a corresponding feature map, and inputs the feature map into an auxiliary network model to predict the Euler angles. For example, the auxiliary network model can be determined in conjunction with Table 1.

[0155] input operate out S 28x28x40 conv,3x3 128 2 14x14x128 conv,3x3 128 1 14x14x128 conv,3x3 32 2 7x7x32 conv,7x7 128 1 1x1x128 FC1 32 1 1x1x32 FC2 3 -

[0156] Table 1 shows the auxiliary network model structure.

[0157] S118. Calculate an average sample image based on the multiple sample images, and extract multiple reference key point location information from the average sample image.

[0158] In this embodiment of the invention, the key point detection device calculates an average sample image based on the multiple sample images, and extracts multiple reference key point location information from the average sample image.

[0159] In this embodiment of the invention, the keypoint detection device normalizes the sample image through coordinate transformation. The keypoint detection device can warp the region of the sample image to a size of 600*600. The keypoint detection device averages the grayscale values ​​of corresponding pixels in all sample images to obtain the average sample image.

[0160] S119. Extract the corresponding key point position information from each sample image, and calculate the rotation matrix using the multiple reference key point position information and the multiple key point position information.

[0161] In this embodiment of the invention, the key point detection device extracts the corresponding key point position information of each sample image, and calculates the rotation matrix using the multiple reference key point position information and the multiple key point position information.

[0162] In this embodiment of the invention, the keypoint detection device acquires multiple reference keypoint position information and multiple keypoint position information. The keypoint detection device can use relevant algorithms to calculate the corresponding rotation matrix.

[0163] S120. Determine the reference Euler angles based on the rotation matrix.

[0164] In this embodiment of the invention, the key point detection device determines the reference Euler angles based on the rotation matrix.

[0165] Given a rotation matrix R, the keypoint detection device can calculate the Euler angles ψ, θ, φ by equating each element in R with the corresponding element in matrix multiplications Rz(φ), Ryθ, Rx(ψ). Here, Rz(φ) is defined as the rotation ψ in radians about the x-axis, Ryθ as the rotation in radians about the y-axis, and Rx(ψ) as the rotation in radians along the z-axis.

[0166] S121. Calculate the nutation angle difference, precession angle difference, and rotation angle difference using the predicted Euler angles and the reference Euler angles.

[0167] In this embodiment of the invention, the key point detection device calculates the nutation angle difference, precession angle difference, and rotation angle difference using the predicted Euler angles and the reference Euler angles.

[0168] Figure 8 The loss calculated based on the predicted key point heatmap, the real image, and the Euler angle difference can also be implemented through steps S122 to S124, which will be explained in conjunction with each step.

[0169] S122. Based on the nutation angle difference, the precession angle difference, and the rotation angle difference, the loss weight is calculated.

[0170] In this embodiment of the invention, the key point detection device calculates the loss weight based on the nutation angle difference, the precession angle difference, and the rotation angle difference.

[0171] In this embodiment of the invention, the key point detection device calculates three cosine values ​​corresponding to the nutation angle difference, the precession angle difference, and the rotation angle difference for each predicted key point. The device calculates the three differences between a first preset value and the three cosine values, and adds these three differences to obtain the sum of differences for each predicted key point. The sum of multiple sums of differences for multiple predicted key points is then added to obtain a second sum of differences for each sample image. The sum of multiple second sums of differences for multiple sample images is then added to obtain a third sum of differences. Finally, the third sum of differences is divided by the number of sample images to obtain the loss weight.

[0172] The first preset value can be 1.

[0173] In real-world scenarios, faces often exhibit significant changes in head pose, corresponding to the problem of in-plane and out-of-plane face rotation. Previous methods typically rotate the face uniformly to a frontal pose before allowing the network to learn and detect it. This method is effective for in-plane face rotation, but its effectiveness is minimal for out-of-plane rotation, such as pitch and 360° rotation. We borrow from 3D face landmark detection methods and introduce Euler angles to address both types of face rotation. Here, an auxiliary network is used to predict three Euler angles for face rotation: Roll, Pitch, and Yaw.

[0174] The method for calculating the true Euler angles for each face is as follows: First, the average face is calculated using all frontal faces in the dataset, and 11 points are fixed at the main positions as references for all training faces; second, the rotation matrix is ​​calculated using the 11 corresponding markers and reference markers for each face; finally, the Euler angles are calculated from the rotation matrix. Then, the difference between the predicted and true Euler angles is calculated and used as the weight of the nonlinear part of the awing loss function to form a new loss function. For images with large changes in face pose, we increase the penalty. The angle weights can be expressed as formula (2):

[0175]

[0176] Where M represents the number of training sample images in each round, and N represents the number of face annotation points. Here, X = 3, which means there are 3 Euler angles, and θ1, θ2, and θ3 correspond to the nutation angle difference, the precession angle difference, and the rotation angle difference, respectively.

[0177] S123. Based on the pixel values ​​of each real key point and the corresponding predicted key point, the intermediate function loss corresponding to each predicted key point is calculated by combining the preset value and the loss weight.

[0178] In this embodiment of the invention, the key point detection device calculates the intermediate function loss corresponding to each predicted key point based on the pixel values ​​of each real key point and the corresponding predicted key point, combined with a preset value and the loss weight.

[0179] All preset values ​​are positive.

[0180] In this embodiment of the invention, if the absolute value of the difference between the pixel value of each real keypoint and the corresponding predicted keypoint is less than a second preset value, the keypoint detection device calculates the absolute value of the ratio of the difference to a third preset value to obtain an intermediate value; the keypoint detection device calculates the intermediate difference between a fourth preset value and the pixel value of each real keypoint; the keypoint detection device calculates the power of the intermediate difference of the intermediate value and adds it to a fifth preset value to obtain a second intermediate value; the keypoint detection device calculates the natural logarithm of the second intermediate value, multiplies the natural logarithm by the loss weight and a sixth preset value to obtain the intermediate function loss.

[0181] In this embodiment of the invention, if the absolute value of the difference between the pixel value of each real key point and the corresponding predicted key point is not less than the second preset value, the key point detection device calculates the fourth difference sum of the absolute value of the difference and the seventh preset value, and subtracts the eighth preset value from the fourth difference sum to obtain the intermediate function loss.

[0182] Both the seventh preset value and the eighth preset value are calculated based on the pixel value of each real key point, combined with the preset value.

[0183] In this embodiment of the invention, the loss function is called geometric-awing (gawing), and its expression is formula (3):

[0184]

[0185] Where θ is the second preset value, ∈ is the third preset value, α is the fourth preset value, and w is the fifth preset value. ω, θ, ∈, and α are positive numbers. The values ​​of the parameters α, w, ∈, and θ can be 2.1, 14, 1, and 0.5, respectively. y represents the pixel grayscale value of the actual keypoint. This represents the pixel grayscale value of the predicted keypoint corresponding to the real keypoint.

[0186] A can be calculated using formula (4):

[0187] A=w(1 / (1+(θ / ∈) (α-y) ))(α-y)((θ / ∈) (α-y-1) (1 / ∈) (4)

[0188] C can be calculated using formula (5):

[0189] C=(θA-wln(1+(θ / ∈) α-y (5)

[0190] The parameters A and C ensure that the loss function is within the range specified by the parameter A. The function is continuous and smooth at zero. The variable θ serves as a threshold to switch between the linear and nonlinear parts of the function. Because heatmap regression regresses to a value between 0 and 1, the threshold is chosen within this range. The exponential term α-y is used to fit the shape of the loss function, making it smooth at zero. The nonlinear part penalizes smaller errors more severely. As the error approaches zero, the effect of the nonlinear part rapidly decreases to zero.

[0191] S124. The average value of the intermediate function losses of the multiple prediction key points is obtained by summing the sums.

[0192] In this embodiment of the invention, the key point detection device adds up the average of the intermediate function losses of the multiple predicted key points to obtain the function loss.

[0193] In this embodiment of the invention, the key point detection process is as follows: Figure 10 As shown, inputting a face image, the network predicts and aligns the resulting face. See also... Figure 9 We visualized some test images on the 300W public test set. Our invented network can still perform well in detecting various challenging images, such as those with large pose changes, glasses occlusion, complex facial expressions, low light, and blurred images.

[0194] For example, in combination Figure 10 Keypoint detection training process:

[0195] Step 1: The key point detection device can complete the initialization by cropping the face image using the bounding box.

[0196] Step 2: The key point detection device inputs the cropped face image (sample image) and the real key point annotations into the initial convolutional network model for training.

[0197] Step 3: The intermediate feature extraction process is the same as HRNetV2. At the first convolutional node, the feature is first reduced by two 3×3 convolutions, followed by an improved bottleneck to extract features. This is then fed into the second convolutional stage. The intermediate features output from the second convolutional stage are input into the auxiliary network model to predict the three Euler angles Roll, Pitch, and Yaw, and the angle differences are calculated as weights for the loss function.

[0198] Step 4: Both the third and fourth convolutional stages use four improved basic blocks to extract features. Low-resolution features are upsampled using bilinear interpolation to recover high-resolution features.

[0199] Step 5: In the fourth convolutional stage, features are extracted simultaneously from the four convolutional branches. Finally, the low-resolution features are restored to the highest resolution for prediction to obtain a heatmap of predicted key points. The coordinate position information of the key points is calculated based on the heatmap. The network is optimized using the Gawing loss function.

[0200] In this embodiment of the invention, the Euler angle difference between the key points in each sample image and the corresponding key points in the real image is taken into account when calculating the function loss. This makes the key point error between each sample image and the corresponding real image calculated by the initial convolutional network model more accurate, thereby obtaining a preset convolutional network model with high detection accuracy and efficiency.

[0201] Please see, Figure 11 A schematic diagram of the key point detection device provided in an embodiment of the present invention.

[0202] This invention also provides a key point detection device 800, including: an image acquisition unit 803 and a processing unit 804.

[0203] Image acquisition unit 803 is used to acquire the image to be tested;

[0204] Processing unit 804 is used to process the image to be tested using a preset convolutional network model to obtain the location information of multiple key points in the image to be tested; wherein,

[0205] The preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage.

[0206] In this embodiment of the invention, the keypoint detection device 800 is used to input each acquired sample image and the corresponding real image into an initial convolutional network model, and to process each sample image through the elastic convolution module in the first convolution stage to obtain a first feature matrix; the real image is an image in which multiple real keypoints are annotated for each sample image; according to the acquired user instructions, and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage, the first feature matrix is ​​processed to obtain a second feature matrix, and the Euler angle difference corresponding to each sample image is calculated based on the second feature matrix and the multiple sample images; N is greater than or equal to The integer 2; combining the elastic convolution modules contained in each of the N+1 convolution branches in the third convolution stage, the second feature matrix is ​​processed to obtain the third feature matrix; combining the elastic convolution modules contained in each of the N+2 convolution branches in the fourth convolution stage, the third feature matrix is ​​processed to obtain the fourth feature matrix; the predicted keypoint heatmap is obtained by fusing the fourth feature matrix to determine the location information of multiple training keypoints, and the function loss is calculated based on the predicted keypoint heatmap, the real image and the Euler angle difference; the initial convolutional network model is trained based on the function loss until a predetermined condition is met to obtain the preset convolutional network model.

[0207] In this embodiment of the invention, the key point detection device 800 is used to convert each sample image into a first initial matrix; after convolving the first initial matrix with a first convolution kernel of a predetermined size, it is then processed by the elastic convolution module to obtain the first feature matrix.

[0208] In this embodiment of the invention, the key point detection device 800 is used to convolve the first initial matrix with M first convolution kernels to obtain a first intermediate feature matrix; M is an integer greater than or equal to 2; the first intermediate feature matrix is ​​sequentially convolved with one pointwise convolution kernel, K single-channel convolution kernels and multiple pointwise convolution kernels to obtain the first feature matrix; K is consistent with the number of input channels of each sample image.

[0209] In this embodiment of the invention, the key point detection device 800 is used to respond to the acquired first user instruction, and combine each first elastic convolution module composed of K single-channel convolution kernels and multiple point-by-point convolution kernels in each convolution branch in sequence to process the first feature matrix to obtain the corresponding second feature matrix, and then obtain N second feature matrices.

[0210] In this embodiment of the invention, the key point detection device 800 is used to respond to the acquired second user instruction, and combine each second elastic convolution module composed of a pointwise convolution kernel, K single-channel convolution kernels and multiple pointwise convolutions in each convolution branch to process the first feature matrix to obtain the corresponding second feature matrix, and then obtain N second feature matrices.

[0211] In this embodiment of the invention, the key point detection device 800 is used to process the second feature matrix through a preset auxiliary network model to obtain predicted Euler angles; calculate an average sample image based on the multiple sample images, and extract multiple reference key point position information from the average sample image; extract multiple corresponding key point position information from each sample image, and calculate a rotation matrix using the multiple reference key point position information and the multiple key point position information; determine reference Euler angles based on the rotation matrix; and calculate the nutation angle difference, precession angle difference, and rotation angle difference using the predicted Euler angles and the reference Euler angles.

[0212] In this embodiment of the invention, the key point detection device 800 is used to calculate the loss weight based on the nutation angle difference, the precession angle difference, and the rotation angle difference; calculate the intermediate function loss corresponding to each predicted key point based on the pixel value of each real key point and the corresponding predicted key point, combined with the preset value and the loss weight; and add the intermediate function losses of the multiple predicted key points together and calculate the average value to obtain the function loss.

[0213] In this embodiment of the invention, the key point detection device 800 is used to calculate three cosine values ​​corresponding to the nutation angle difference, the precession angle difference, and the rotation angle difference of each predicted key point; calculate the three differences between a first preset value and the three cosine values ​​respectively; add the three differences to obtain the sum of differences for each predicted key point; add the sums of multiple differences for multiple predicted key points to obtain a second sum of differences for each sample image; add the sums of multiple second sums of differences for multiple sample images to obtain a third sum of differences; and then divide the third sum of differences by the number of multiple sample images to obtain the loss weight.

[0214] In this embodiment of the invention, the key point detection device 800 is used to: if the absolute value of the difference between the pixel value of each real key point and the corresponding predicted key point is less than a second preset value, then calculate the absolute value of the ratio of the difference to a third preset value to obtain an intermediate value; calculate the intermediate difference between a fourth preset value and the pixel value of each real key point; calculate the intermediate difference raised to the power of the intermediate value, and then add it to a fifth preset value to obtain a second intermediate value; calculate the natural logarithm of the second intermediate value, and multiply the natural logarithm by the loss weight and a sixth preset value to obtain the intermediate function loss.

[0215] In this embodiment of the invention, the key point detection device 800 is used to calculate the fourth difference sum of the absolute value of the difference between the pixel value of each real key point and the corresponding predicted key point, if the absolute value of the difference is not less than a second preset value, and subtract an eighth preset value from the fourth difference sum to obtain the intermediate function loss; the seventh preset value and the eighth preset value are both calculated based on the pixel value of each real key point and in combination with the preset value.

[0216] In this embodiment of the invention, an image acquisition unit 803 acquires a test image; a processing unit 804 processes the test image using a preset convolutional network model to obtain the location information of multiple key points in the test image; wherein, the preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage. Because this scheme considers the key point heatmap errors between multiple sample images and their corresponding real images during the training process of the preset convolutional network model, and uses an elastic convolution model to train the sample images, the number of network parameters is reduced, thereby improving the accuracy and efficiency of key point detection.

[0217] It should be noted that, in the embodiments of the present invention, if the above-described key point detection method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a key point detection device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

[0218] Correspondingly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0219] Correspondingly, this embodiment of the invention provides a key point detection device, including a memory 802 and a processor 801. The memory 802 stores a computer program that can run on the processor 801. When the processor 801 executes the program, it implements the steps in the above method.

[0220] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of the present invention, please refer to the descriptions of the method embodiments of the present invention for understanding.

[0221] It should be noted that, Figure 12 This is a schematic diagram of a hardware entity of the key point detection device provided in an embodiment of the present invention, such as... Figure 12 As shown, the hardware entity of the key point detection device 800 includes: a processor 801 and a memory 802, wherein;

[0222] The processor 801 typically controls the overall operation of the key point detection device 800.

[0223] The memory 802 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 801 and the key point detection device 800. It can be implemented by flash memory or random access memory (RAM).

[0224] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of the invention, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the invention. The sequence numbers of the above-described embodiments of the invention are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0225] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0226] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0227] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0228] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0229] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0230] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0231] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A key point detection method, characterized in that, include: Each acquired sample image and its corresponding real image are input into the initial convolutional network model. The first feature matrix is ​​obtained by processing each sample image through the elastic convolution module in the first convolution stage. The real image is an image in which multiple real key points are annotated for each sample image. Based on the acquired user instructions, and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage, the first feature matrix is ​​processed to obtain the second feature matrix, and the Euler angle difference corresponding to each sample image is calculated based on the second feature matrix and multiple sample images; N is an integer greater than or equal to 2; By combining the elastic convolution modules contained in each of the N+1 convolution branches in the third convolution stage, the second feature matrix is ​​processed to obtain the third feature matrix. By combining the elastic convolution modules contained in each of the N+2 convolution branches in the fourth convolution stage, the third feature matrix is ​​processed to obtain the fourth feature matrix. The predicted keypoint heatmap is obtained by fusing the fourth feature matrix to determine the location information of multiple training keypoints. The function loss is calculated based on the predicted keypoint heatmap, the real image and the Euler angle difference. The initial convolutional network model is trained based on the function loss until a predetermined condition is met to obtain the preset convolutional network model. Acquire the image to be tested; The image to be tested is processed using a pre-defined convolutional network model to obtain the location information of multiple key points in the image; wherein... The preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage.

2. The key point detection method according to claim 1, characterized in that, The process of obtaining the first feature matrix by processing each sample image through the elastic convolution module in the first convolution stage includes: Each sample image is converted into a first initial matrix; The first initial matrix is ​​convolved using a first convolution kernel of a predetermined size, and then processed by the elastic convolution module to obtain the first feature matrix.

3. The key point detection method according to claim 2, characterized in that, The process of convolving the first initial matrix with a first convolution kernel of a predetermined size, and then processing it through the elastic convolution module to obtain the first feature matrix includes: The first initial matrix is ​​convolved with M first convolution kernels to obtain the first intermediate feature matrix; M is an integer greater than or equal to 2. The first intermediate feature matrix is ​​obtained by sequentially convolving it with a pointwise convolution kernel, K single-channel convolution kernels, and multiple pointwise convolution kernels; K is the same as the number of input channels of each sample image.

4. The key point detection method according to claim 1, characterized in that, The step of processing the first feature matrix to obtain the second feature matrix based on the acquired user instructions and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage includes: In response to the first user instruction, the first feature matrix is ​​processed by combining the first elastic convolution modules composed of K single-channel convolution kernels and multiple pointwise convolution kernels in each convolution branch to obtain the corresponding second feature matrix, and then N second feature matrices are obtained.

5. The key point detection method according to claim 1, characterized in that, The step of processing the first feature matrix to obtain the second feature matrix based on the acquired user instructions and in conjunction with the elastic convolution modules contained in each of the N convolution branches in the second convolution stage includes: In response to the acquired second user instruction, the first feature matrix is ​​processed by combining a pointwise convolution kernel, K single-channel convolution kernels and multiple pointwise convolutions in each convolution branch to obtain the corresponding second feature matrix, and then N second feature matrices are obtained.

6. The key point detection method according to claim 1, characterized in that, The step of calculating the Euler angle difference corresponding to each sample image based on the second feature matrix and multiple sample images includes: The second feature matrix is ​​processed by a pre-set auxiliary network model to obtain the predicted Euler angles; An average sample image is calculated based on the multiple sample images, and multiple reference key point location information is extracted from the average sample image; In each sample image, multiple key point location information is extracted, and a rotation matrix is ​​calculated using the multiple reference key point location information and the multiple key point location information. The reference Euler angles are determined based on the rotation matrix; The nutation angle difference, precession angle difference, and rotation angle difference are calculated using the predicted Euler angles and the reference Euler angles.

7. The key point detection method according to claim 6, characterized in that, The predicted key point heatmap includes: multiple predicted key points corresponding to the multiple real key points; The loss function calculated based on the predicted keypoint heatmap, the real image, and the Euler angle difference includes: The loss weight is calculated based on the nutation angle difference, the precession angle difference, and the rotation angle difference; Based on the pixel values ​​of each real keypoint and the corresponding predicted keypoint, the intermediate function loss corresponding to each predicted keypoint is calculated by combining the preset value and the loss weight. The average of the intermediate function losses of the multiple predicted key points is calculated to obtain the function loss.

8. A key point detection device, characterized in that, include: The processing unit is used to input each acquired sample image and its corresponding real image into the initial convolutional network model, and process each sample image through the elastic convolution module in the first convolution stage to obtain a first feature matrix; the real image is an image in which multiple real key points are annotated for each sample image; The processing unit is configured to process the first feature matrix to obtain a second feature matrix based on the acquired user instructions and the elastic convolution modules contained in each of the N convolution branches in the second convolution stage, and to calculate the Euler angle difference corresponding to each sample image based on the second feature matrix and multiple sample images; N is an integer greater than or equal to 2. The processing unit is used to combine the elastic convolution modules contained in each of the N+1 convolution branches in the third convolution stage to process the second feature matrix to obtain a third feature matrix, and to combine the elastic convolution modules contained in each of the N+2 convolution branches in the fourth convolution stage to process the third feature matrix to obtain a fourth feature matrix. The processing unit is configured to obtain a predicted keypoint heatmap by fusing the fourth feature matrix to determine the location information of multiple training keypoints, and calculate the function loss based on the predicted keypoint heatmap, the real image and the Euler angle difference; and train the initial convolutional network model based on the function loss until a predetermined condition is met to obtain a preset convolutional network model. An image acquisition unit is used to acquire the image to be tested. The processing unit is used to process the image to be tested using a preset convolutional network model to obtain the location information of multiple key points in the image to be tested; wherein, The preset convolutional network model is trained based on the key point heatmap errors between multiple sample images and their corresponding real images, combined with the elastic convolution module in each convolution stage.

9. A key point detection device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

Images