Key point detection method, device, electronic device and storage medium for target image

Abstract

The present invention discloses a method, device, electronic device and storage medium for detecting key points of a target image. During the distillation learning process of the present invention, the correct prediction distribution corresponding to the correct samples in the first network is determined according to the flag function, and the second network is subjected to distillation learning according to the correct prediction distribution, and the second network is pruned to obtain a second network model with fewer model parameters as the key point detection model, which improves the calculation speed and detection efficiency of the key point detection model and reduces the requirement for the computing power of the terminal; in addition, through the flag function, the second network can perform knowledge distillation on the prediction results corresponding to the correct samples in the first network, without spending other computing loads to perform knowledge distillation on the prediction process of the wrong samples, further improving the calculation speed of the key point detection model and reducing the requirement for the computing power of the terminal, and can be widely applied to the field of computer vision technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to a method, apparatus, electronic device, and storage medium for detecting key points of a target image. Background Art

[0002] Detecting key points of a target image means detecting a plurality of key points from a given image for further analysis and processing of the target image. Taking a face image as an example, key points such as the center of the eyeball, the corners of the eyes, the tip of the nose, the corners of the mouth, and the face contour are detected in the face image. The accurate detection of face key points has a decisive impact on the application of face recognition. With the continuous development of artificial intelligence technology, the current key point detection technology based on deep learning has become increasingly mature. Currently, all key point detection technologies based on deep learning need to train a key point detection model before detection, and then perform key point detection through this model.

[0003] With the continuous enrichment of key point detection scenarios, more and more key point detection processes need to be implemented on different terminals. For example, face key point detection is implemented on terminals such as computer devices, smart home devices, and intelligent monitoring devices, and then face recognition, etc. is realized. However, since key point detection involves high-computation processes such as model training, in order to pursue detection accuracy, the related technology increases the deep learning training volume. The excessive model training computation volume will affect the device performance of the terminal, resulting in an overloaded terminal. Therefore, how to improve the key point detection efficiency of the terminal, reduce the load and deployment cost of the terminal has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for detecting key points of a target image to improve the key point detection efficiency of the target image in the terminal and reduce the load pressure of the terminal deploying the model.

[0005] One aspect of the present invention provides a method for detecting key points of a target image, including:

[0006] Obtaining training data;

[0007] Inputting the training data into a first network to train a first network model;

[0008] Inputting the training data into a second network, performing distillation learning on the second network according to the first network model, and performing pruning processing on the second network while performing the distillation learning to obtain a key point detection model, where the knowledge distillation loss function in the distillation learning includes a flag function, and the flag function is used to flag correct samples and incorrect samples in the training data;

[0009] Input the target image to be detected into the key point detection model to obtain the key points in the target image.

[0010] On the other hand, the present invention provides a key point detection device for a target image, including:

[0011] An acquisition module for acquiring training data;

[0012] A first training module for inputting the training data into a first network to train and obtain a first network model;

[0013] A second training module for inputting the training data into a second network, performing distillation learning on the second network according to the first network model, and pruning the second network during the distillation learning to obtain a key point detection model, where the knowledge distillation loss function in the distillation learning includes a flag function for flagging correct samples and incorrect samples in the training data;

[0014] A detection module for inputting the target image to be detected into the key point detection model to obtain the key points in the target image.

[0015] On the other hand, the present invention also provides an electronic device, including a processor and a memory;

[0016] The memory is used for storing programs;

[0017] The processor executes the program to implement the method as described above.

[0018] On the other hand, the present invention also provides a computer-readable storage medium, where the storage medium stores a program, and the program is executed by a processor to implement the method as described above.

[0019] On the other hand, the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method as described above.

[0020] Embodiments of the present invention obtain training data; input the training data into a first network to train a first network model; input the training data into a second network and perform distillation learning according to the first network model to obtain a key point detection model, and input a target image to be detected into the key point detection model to obtain key points in the target image; wherein, the distillation learning includes: determining a correct prediction distribution corresponding to correct samples in the first network according to the flag function, and performing distillation learning on the second network according to the correct prediction distribution, and pruning the second network to obtain a key point detection model; Embodiments of the present invention perform distillation learning on a second network based on a trained first network model to obtain a second network model with fewer model parameters as the key point detection model, improving the calculation speed and detection efficiency of the key point detection model, reducing the requirement for the computing power of the terminal, and enabling real-time operation on a terminal with low computing power; In addition, the knowledge distillation loss function in the distillation learning includes a flag function, and the flag function is used to flag correct samples and incorrect samples in the training data. Embodiments of the present invention can enable the second network to perform knowledge distillation on the prediction results corresponding to correct samples in the first network through the flag function, without spending other computing loads to perform knowledge distillation on the prediction process of incorrect samples, further improving the calculation speed of the key point detection model and reducing the requirement for the computing power of the terminal; Furthermore, embodiments of the present invention also perform pruning processing on the second network, which helps to reduce the scale of the key point detection model, reducing the requirement for the computing power of the terminal and the load pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 Schematic diagram of the implementation environment for the key point detection method of the target image provided by the embodiments of the present invention;

[0023] Figure 2 Flowchart of the key point detection method of the target image provided by the embodiments of the present invention;

[0024] Figure 3 Flowchart of the steps for generating the first network model provided by the embodiments of the present invention;

[0025] Figure 4 Schematic diagram of the structure of the first network provided by the embodiments of the present invention;

[0026] Figure 5Schematic diagram of the distillation learning process provided by an embodiment of the present invention;

[0027] Figure 6 Flowchart of steps for generating a key point detection model provided by an embodiment of the present invention;

[0028] Figure 7 Flowchart of steps for obtaining key points in a target image provided by an embodiment of the present invention;

[0029] Figure 8 Schematic diagram of the process for determining a visualization image based on a target image to be detected;

[0030] Figure 9 Flowchart of steps for a key point detection method for a target image in a specific embodiment;

[0031] Figure 10 Schematic diagram of the structure of a detection device provided by an embodiment of the present invention. Detailed implementation manners

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] First, relevant noun terms involved in the embodiments of the present invention are introduced and explained:

[0034] Artificial Intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, a theory, method, technology, and application system that can perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence is also to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning, and decision-making.

[0035] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] Computer Vision Technology (CV): Computer vision is a science that studies how to enable machines to "see". More specifically, it refers to machine vision that uses cameras and computers to replace human eyes for tasks such as object recognition, tracking, and measurement, and further performs image processing to make the computer-processed images more suitable for human eyes to observe or be transmitted to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to build artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc., and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0037] Machine Learning (ML): An interdisciplinary field that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0038] Knowledge distillation: Also known as dark knowledge extraction, it refers to the process of guiding the training of a relatively simple and computationally less-intensive student neural network by a teacher neural network with a complex structure and large computational load but excellent performance, in order to improve the performance of the student neural network and achieve knowledge transfer. Knowledge distillation can make the model lightweight (convenient for deployment) while minimizing performance loss.

[0039] Kullback-Leibler divergence: Also known as relative entropy, it is a method to describe the difference between two probability distributions. In the field of machine learning, KL divergence is used to measure the similarity or closeness of two functions.

[0040] Model pruning is to generate a model with a smaller scale, lower memory utilization, faster inference speed, and less loss of inference accuracy by pruning the model structure and model parameters.

[0041] With the continuous enrichment of key point detection scenarios, more and more key point detection processes need to be implemented on different terminals. And the key point detection technology based on deep learning needs to train a key point detection model before detection and then use this model for key point detection. First, since key point detection involves high-computation processes such as model training, in related technologies, in order to pursue detection accuracy and increase the amount of deep learning training, the excessive model training computation will, on the one hand, affect the device performance of the terminal, resulting in an overloaded terminal and reducing the calculation speed and detection efficiency of the key point detection model. Moreover, in the process of knowledge distillation in related technologies, the total samples including correct samples and wrong samples are usually used for knowledge distillation, resulting in a part of the computational load being consumed for the knowledge distillation of wrong samples, which further reduces the calculation speed of the key point detection model. Finally, related technologies do not involve the relevant processing of the scale of the key point detection model, resulting in a large scale of the key point detection model, high memory utilization rate, slow inference speed, and much loss of inference accuracy.

[0042] In the embodiments of the present invention, first, based on the trained first network model, distillation learning is performed on the second network to obtain a second network model with fewer model parameters as the key point detection model, which improves the calculation speed and detection efficiency of the key point detection model. Applying the key point detection model to the terminal can reduce the requirement for the computing power of the terminal, enabling key point detection to run in real time on terminals with low computing power. Moreover, in the knowledge distillation loss function in distillation learning, a flag function is introduced to flag the correct samples and wrong samples in the training data. The flag function enables the second network to perform knowledge distillation on the prediction results corresponding to the correct samples in the first network, without spending a part of the computational load on the knowledge distillation of the prediction process of wrong samples, further improving the calculation speed of the key point detection model and reducing the requirement for the computing power of the terminal. Finally, the embodiments of the present invention also perform pruning processing on the second network, which can reduce the scale of the key point detection model. When the key point detection model is applied to the terminal, it can reduce the requirement for the computing power of the terminal and the load pressure on the terminal. Optionally, the first network can be a teacher network, and the second network can be a student network. The structure of the first network is complex and the computation amount is large; the structure and computational complexity of the second network are less than those of the first network. The first network can be used to guide the training of the second network to improve the performance of the second network and achieve knowledge transfer. In the present invention, the teacher network refers to a pre-trained neural network to be compressed; and the student network is a neural network with the same model structure as the teacher network and a smaller model parameter scale than that of the teacher network.

[0043] Reference Figure 1, which shows a schematic diagram of the implementation environment of a key point detection method for a target image provided by an embodiment of the present invention. In this implementation environment, it includes at least one server 101 and at least one terminal device 102. Among them, the terminal device 102 can be any electronic product that can perform human-computer interaction through one or more ways such as a keyboard, a touchpad, a touch screen, a remote control, voice interaction, or a handwriting device. This electronic product can obtain the target image to be detected and use the key point detection model of the embodiment of the present invention to detect the key points in the target image. Optionally, the key point detection model can be pre-stored in the terminal device 102 or sent to the terminal device 102 by the server 101. This electronic product can be, for example, a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, etc. The server 101 can be an independent server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It should be noted that the server 101 can store the first network model and the second network, and has the functions of obtaining training data, inputting the training data into the first network to train the first network model, inputting the training data into the second network, performing distillation learning on the second network according to the first network model, and performing pruning processing on the second network while performing distillation learning to obtain the key point detection model, and transmitting the key point detection model to the terminal device 102 through a network connection. It should be noted that the server 101 can also store the first network model, the terminal device 102 can store the second network, and the server 101 can have the functions of obtaining training data, inputting the training data into the first network to train the first network model, and transmitting the training data to the second network, performing distillation learning on the second network according to the first network model, and performing pruning processing on the second network while performing distillation learning. Optionally, the pruning processing of the second network can be performed by the server 101 or by the terminal device 102. Among them, the server 101 can have the function of directly obtaining training data or the function of receiving training data obtained by other devices.

[0044] Based on Figure 1 the implementation environment, as Figure 2 shown, an embodiment of the present invention provides a key point detection method for a target image. The detection method specifically includes but is not limited to steps S100 - S400:

[0045] S100. Obtain training data.

[0046] Specifically, the training data is image data with known key-point detection results. The training data has labels, and the labels are the true coordinate values of the key points of the training data, that is, they represent the known key-point detection results. The key points include but are not limited to the key points of a human face. The image data includes but is not limited to the data of a human face image. Optionally, the training data can be the face region image extracted after obtaining the target image through an image acquisition device or a computer device, or the human face image from the Internet or a third-party application (such as an image processing software), or the human face image pre-stored in a database. This application does not make any limitations in this regard.

[0047] S200. Input the training data into the first network and train to obtain the first network model.

[0048] Specifically, both the network width and depth of the first network are larger than those of the second network. In the embodiments of the present invention, the structure of the first network can be designed according to actual needs. For example, the resnet50 network can be used. During the training process of the first network, the training data is input into the first network, and the network parameters of the first network are optimized through the training data, so as to obtain the first network model.

[0049] S300. Input the training data into the second network, perform distillation learning on the second network according to the first network model, and perform pruning processing on the second network while performing distillation learning to obtain the key-point detection model.

[0050] Specifically, the second network refers to a network model with a simpler structure and a smaller amount of calculation than the first network. The second network is trained by inputting training data into the second network, and in the process of training the second network, the trained first network model is used to guide the training process of the second network, thereby improving the performance of the second network, realizing knowledge transfer, so that the second network with a simple structure after training can have the performance of the first network model to a certain extent. In addition, it should be noted that the "pruning the second network while distilling and learning" described in the embodiment of the present invention means that the scheme of the present invention combines the distillation learning technology and the pruning technology, and the pruning process can be performed before the second network is trained, or can be performed while the second network is trained, or can also be performed after the second network is trained, and the timing of the pruning process is not specifically limited here. It should be noted that the pruning process can cut the model structure and model parameters of the second network to generate a model with a smaller scale, lower memory utilization, faster inference speed, and less loss of inference accuracy. Among them, the key point detection model refers to the second network model obtained by distilling and learning the second network through the first network model and pruning the second network, which can be used for key point detection of the target image to be detected.

[0051] S400: Input the target image to be detected into a key point detection model to obtain key points in the target image.

[0052] Specifically, the target image to be detected is input into the key point detection model, and the key point detection model outputs the detection result of the key points, that is, the key points in the target image can be obtained through the key point detection model.

[0053] like Figure 3 As shown, optionally, in the above step S200, inputting the training data into the first network to train and obtain the first network model includes the following steps S210-S240:

[0054] S210, inputting the training data into the first network for separable convolution processing to obtain a first feature map set of different scales;

[0055] In an embodiment of the present invention, the first network includes a feature extraction network and at least one fully connected layer. Specifically, the feature extraction network includes a plurality of separable convolution blocks. The feature extraction network can perform separable convolution processing on training data, efficiently encode different scales of the images of the training data to obtain a set of first feature maps of different scales. Each scale of the set of first feature maps can have at least one first feature map, and the separable convolution processing can enable each first feature map of different scales in the set of first feature maps to contain rich information of different scales. Among them, the separable convolution processing can include at least one separable convolution sub-processing and downsampling processing. Each separable convolution block can perform separable convolution sub-processing on the input training data, or on the training data after being processed by the previous separable convolution block, or on the training data after being downsampled by a certain multiple (for example, including but not limited to 2-fold downsampling). Optionally, the separable convolution sub-processing includes but is not limited to performing grouped convolution operation, convolution operation, and at least one batch normalization / ReLu (Rectified Linear Unit) processing on the training data. Optionally, the sizes of the convolution kernels used in the grouped convolution operation and the convolution operation can be set according to actual needs without specific limitation. For example, the size of the convolution kernel used in the convolution operation is 1×1, and the size of the convolution kernel used in the grouped convolution operation is 3×3, reducing the resolution of the processed feature map by the corresponding multiple to obtain feature maps of different scales.

[0056] As Figure 4 shown, for the convenience of description, a schematic structural diagram of the first network 301 in an embodiment of the present invention is exemplarily given. It can be understood that this does not limit the specific structure of the first network 301, and the actual structure of the first network 301 can be more complex and refined, that is Figure 4The structure of the first network 301 shown in [description] is a basic framework for explaining the principle of the first network 301. The first network 301 has four separable convolution blocks and a fully connected layer 302. The four separable convolution blocks are the first separable convolution block 303, the second separable convolution block 304, the third separable convolution block 305, and the fourth separable convolution block 306 respectively. Among them, the composition of each separable convolution block is the same, and it is used to perform grouped convolution operations, convolution operations, and at least one batch normalization / ReLu (Rectified Linear Unit) processing on the images in the training data. Among them, the convolution kernel size used in the grouped convolution operation of the embodiments of the present application is 3×3, that is, the grouped convolution operation refers to the grouped convolution operation with a convolution kernel size of 3×3. In the embodiments of the present invention, the training data is input into the first network 301. The first network first performs downsampling processing on the images in the training data. For example, the image is downsampled through a 3×3 convolution kernel, and then the downsampled image is input into the first separable convolution module 303, as Figure 5 shown. Since the first separable convolution module 303 includes a downsampling operation, a plurality of first feature maps 3031 at the first scale can be obtained. Similarly, the first feature maps at the first scale are input into the second separable convolution block 304 for processing to obtain a plurality of first feature maps 3041 at the second scale. Then, the first feature maps at the second scale are input into the third separable convolution block 305 for processing to obtain a plurality of first feature maps 3051 at the third scale. Then, the first feature maps at the third scale are input into the fourth separable convolution block 306 for processing to obtain a plurality of first feature maps 3061 at the fourth scale. Among them, the first feature maps at the first scale, the first feature maps at the second scale, the first feature maps at the third scale, and the first feature maps at the fourth scale constitute a set of first feature maps at different scales.

[0057] S220. Predict the first key point coordinate values for the set of first feature maps through the fully connected layer of the first network to obtain a first prediction result;

[0058] Specifically, the fully connected layer can perform regression of the key point coordinate values on the set of first feature maps with high-dimensional features after separable convolution processing, predict the first key point coordinate values, and obtain the predicted coordinate values of the first key point as the first prediction result. For example, if it is necessary to obtain the coordinates of 94 first key points of a human face in a human face image, and each first key point coordinate includes two values of x and y, then the number of neurons in the last fully connected layer can be designed to be 94×2 = 188. Through network learning, the predicted coordinate values of these 188 first key points can be directly regressed and predicted.

[0059] As Figure 4 and Figure 5As shown, the embodiment of the present invention has a fully connected layer. The fully connected layer includes a first sub-fully connected layer for processing the first feature map at the fourth scale obtained by the fourth separable convolution module, and a second sub-fully connected layer for further processing the output of the first sub-fully connected layer to obtain a first prediction result. It can be understood that the first prediction result includes the predicted coordinate values of the first key points.

[0060] S230. Determine a first loss value between the first prediction result and the label by using a first loss function.

[0061] Specifically, the first loss function is:

[0062]

[0063] where y i,j is the true coordinate value of the j-th first key point of the i-th sample, and y' i,j is the predicted coordinate value of the j-th first key point of the i-th sample. It should be noted that both y i,j and y' i,j include the above two values of x and y. n is the number of samples in the training data, m is the number of first key points, MSE(y, y') represents the first loss value, that is, the loss value. The mean square error between the two is directly used to judge the difference between the true coordinate value and the predicted coordinate value. The smaller the distance, that is, the smaller the loss value, the closer the predicted coordinate value is to the true coordinate value. The larger the distance, that is, the larger the loss value, the farther the predicted coordinate value is from the true coordinate value.

[0064] S240. Modify the first network model parameters according to the first loss value, and generate a first network model according to the modified first network model parameters.

[0065] Specifically, the first network model parameters are modified according to the above first loss value. The first network model parameters include, but are not limited to, data processing (or preprocessing) related parameters, training process and training related parameters, or network related parameters. For example, data processing (or preprocessing) related parameters include, but are not limited to, parameters for enriching the database (enrich data), parameters for data generalization processing (feature normalization and scaling), and parameters for batch normalization (batchnormalization); training process and training related parameters include, but are not limited to, training momentum, learning rate, decay function, weight initialization, and regularization related methods; network related parameters include, but are not limited to, classifier selection parameters, number of neurons, number of filters, and number of network layers. By modifying the first network model parameters, the first network is optimized. For example, when the first loss value between the first prediction result of the first network after modifying the first network model parameters and the true coordinate value is less than the first threshold, the currently modified first network model parameters are used as the first network model parameters of the first network, thereby obtaining the first network model.

[0066] Optionally, before the above step S220, there is also step S211, specifically:

[0067] S211. Perform global average pooling processing on each first feature map in the first feature map set to obtain the first channel weight of each first feature map.

[0068] It should be noted that since the separable convolution block included in the first network contains a grouped convolution operation, that is, the image data input to the separable convolution block is grouped, and then each group performs a convolution operation respectively. For example, each group uses a convolution kernel of size 3×3 for convolution operation, then the number of convolution kernels is the number of convolution channels. Correspondingly, each convolution kernel performs convolution calculation on the grouped image data, that is, the feature maps of the convolution channels corresponding to each convolution kernel can be obtained. For example, as Figure 5 shown, the first feature map 3031 at the first scale has three channels, and the number of channels is 3. In the embodiment of the present invention, global average pooling (GAP) processing is performed on each first feature map at different scales in the first feature map set. For each GAP processing, assuming the dimension of the first feature map is C×H×W (C is the number of channels, H is the height of the feature map, and W is the width of the feature map), after GAP processing, the dimension becomes C, which represents the weight size of each channel. Therefore, by performing GAP processing on each first feature map respectively, the first channel weight of each first feature map can be obtained.

[0069] As Figure 6As shown, optionally, the above step S300 may include the following steps S310 - S350:

[0070] S310. Input the training data into the second network for separable convolution processing to obtain a set of second feature maps of different scales;

[0071] Specifically, the structure of the second network is designed similar to the basic framework of the first network shown in Figure 4 . Therefore, similarly, the second network also has a feature extraction network and a fully connected layer. The feature extraction network can also perform separable convolution processing on the training data, efficiently encode different scales of the images of the training data to obtain a set of second feature maps of different scales. The set of second feature maps of different scales can have at least one second feature map at each scale, and the separable convolution processing can make each second feature map of different scales in the set of second feature maps contain rich information of different scales. Similarly, as shown in Figure 5 , through separable convolution processing, the second network 401 can obtain a plurality of second feature maps 402 at the first scale. Similarly, input the second feature maps at the first scale into the separable convolution block for processing to obtain a plurality of second feature maps 403 at the second scale, then input the second feature maps at the second scale into the separable convolution block for processing to obtain a plurality of second feature maps 404 at the third scale, and then input the second feature maps at the third scale into the separable convolution block for processing to obtain a plurality of second feature maps 405 at the fourth scale. Among them, the second feature maps at the first scale, the second feature maps at the second scale, the second feature maps at the third scale, and the second feature maps at the fourth scale constitute a set of second feature maps of different scales.

[0072] S320. Predict the second key point coordinate values for the set of second feature maps through the fully connected layer of the second network to obtain a second prediction result;

[0073] As shown in Figure 4 , similar to step S220, through the fully connected layer 406 of the second network, perform regression on the key point coordinate values for the set of second feature maps with high - dimensional features, predict the second key point coordinate values, and obtain the predicted coordinate values of the second key points as the second prediction result. It can be understood that the design of the fully connected layer 406 of the second network can be the same as that of the fully connected layer 302 of the first network.

[0074] S330. Use the second loss function to determine the second loss value between the second prediction result and the label;

[0075] Similarly, similar to the principle of step S230, the mean square error between the true coordinate value and the predicted coordinate value is calculated using the second loss function to determine the difference between the true coordinate value and the predicted coordinate value, thereby determining the second loss value between the second prediction result and the label.

[0076] S340. Modify the second network model parameters according to the second loss value and the knowledge distillation loss function, and generate a second network model according to the modified second network model parameters;

[0077] Similarly, the second network model parameters include, but are not limited to, parameters for enriching the database (enrich data), parameters for data generalization processing (feature normalization and scaling), and parameters for BN processing (batch normalization), training momentum, learning rate, decay function, weight initialization and regularization related methods, selection of classifiers, number of neurons, number of filters, and number of network layers. Specifically, the knowledge distillation loss function includes a flag function, which is used to flag correct samples and incorrect samples in the training data. In the embodiment of the present invention, the second network model parameters are adjusted according to the second loss value and the knowledge distillation loss function including the flag function to optimize the second network. When the second network model parameters of the second network are optimized to the actual required accuracy, the current modified second network model parameters are used as the second network model parameters of the second network, thereby obtaining a second network model.

[0078] S350. Prune the second network model to obtain a key point detection model.

[0079] Specifically, by pruning the second network model, the model structure and model parameters of the second network model can be trimmed, thereby generating a model with a smaller scale, lower memory utilization, faster inference speed, and smaller loss of inference accuracy. When the second network model is applied to a terminal, it can reduce the computing power requirements of the terminal and the load pressure on the terminal.

[0080] Optionally, step S340 includes steps S341 - S343:

[0081] S341. Determine a first prediction distribution according to the first prediction result, and determine a second prediction distribution according to the second prediction result; specifically, the formula for the prediction distribution is:

[0082]

[0083] Among them, p represents the predicted distribution, a is the output result of the network, and T is the temperature parameter. When T is larger, the output result of the network is smoother, which can play a role in retaining similar information. If T is equal to infinity, it is equivalent to a uniform distribution. While performing distillation learning, the second network fits the soft-target of the first network model (that is, minimizes the output distributions of the first network model and the second network). It can be understood that when the output result of the network is the first prediction result, the first predicted distribution can be determined according to the above formula, and when the output result of the network is the second prediction result, the second predicted distribution can be determined according to the above formula.

[0084] S342. Determine the third loss value between the first predicted distribution and the second predicted distribution by using the knowledge distillation loss function;

[0085] In the embodiment of the present invention, the formula of the knowledge distillation loss function is:

[0086]

[0087] Among them, KD optimize(s,t) is the third loss value, is the flag function, which is used to flag the correct samples and incorrect samples in the training data, represents the first predicted distribution of the first network model, i represents the predicted coordinate value of the i-th sample, represents measuring the difference in the output probability distributions between the first predicted distribution and the second predicted distribution of the first network and the second network by using the KL divergence, represents the first predicted distribution of the i-th sample, represents the second predicted distribution of the i-th sample, and n is the number of samples. In the embodiment of the present invention, the third loss value between the first predicted distribution and the second predicted distribution is determined according to the above formula by using the knowledge distillation loss function.

[0088] S343. Modify the parameters of the second network model according to the second loss value and the third loss value, and generate the second network model according to the modified parameters of the second network model.

[0089] Specifically, by modifying the parameters of the second network model, the second network is optimized. For example, the parameters of the second network model can be modified until the second loss value is less than the second threshold and the third loss value is less than the third threshold, and the current parameters of the second network model are used as the parameters of the second network model of the second network, so as to obtain the second network model.

[0090] Optionally, step S340 further includes step S344, specifically:

[0091] S344. Convolve the first feature map according to the number of channels of the first feature map and the number of channels of the second feature map.

[0092] It should be noted that since the number of channels of the first feature map and the number of channels of the second feature map of the first network model may be different, and when the number of channels of the first feature map and the number of channels of the second feature map are different, in order to improve the effect of the first network model in supervising the learning of the second network, convolution processing can be performed, for example, convolution processing can be performed using a convolution kernel of size 1×1, so that the number of channels of the first feature map is the same as the number of channels of the second feature map.

[0093] Specifically, step S342 may include steps S361 - S363:

[0094] S361. Determine the correct samples in the training data;

[0095] Specifically, taking the image data of the face class and the image data of the non - face class included in the training data as an example, the correct samples refer to the positive samples, that is, the image data including the face class; correspondingly, the incorrect samples refer to the negative samples, that is, the image data of the non - face class (such as animal head image data, human hand image data, etc.).

[0096] S362. Set the flag function corresponding to the correct samples in the first prediction distribution to 1, and set the flag function corresponding to the incorrect samples in the first prediction distribution to 0, to obtain the assigned flag function.

[0097] In the embodiments of the present invention, after determining the correct samples of the training data, the correct samples of the training data can be compared with the prediction results in the first prediction distribution, so as to determine the samples with correct predictions in the first prediction distribution, that is, the correct samples in the first prediction distribution; and determine the samples with incorrect predictions in the first prediction distribution, that is, the incorrect samples in the first prediction distribution. It can be understood that the correct samples in the first prediction distribution include the samples that predict that the image data contains a face and the prediction results are correct, and the incorrect samples in the first prediction distribution include the samples that predict that the image data contains a face but the prediction results are incorrect. In the embodiments of the present invention, during the distillation learning process, the flag function corresponding to the correct samples in the first prediction distribution is set to 1, and the flag function corresponding to the incorrect samples in the first prediction distribution is set to 0, to obtain the assigned flag function. It can be understood that the model parameters obtained by training with correct samples have a positive effect on the accuracy of model prediction, while the model parameters obtained by training with incorrect samples have a negative effect on the accuracy of model prediction. Therefore, the embodiments of the present invention obtain correct samples through the above - mentioned method and perform distillation learning on the second network according to the model parameters corresponding to the correct samples, which can reduce the training amount of the second network and improve the training efficiency.

[0098] S363. Determine the knowledge distillation loss function according to the labeled flag function after assignment, and use the knowledge distillation loss function to calculate the difference value between the first prediction distribution and the second prediction distribution, so as to obtain the third loss value.

[0099] Specifically, after determining the labeled flag function for the correct samples and incorrect samples in the first prediction distribution, the labeled flag function can be substituted into the formula of the knowledge distillation loss function in step S342 to calculate the difference value between the first prediction distribution and the second prediction distribution, so as to determine the third loss value KD. optimize(s,t) . It should be noted that although theoretically the first network model will output more accurate prediction results than the second network, there will also be incorrect samples with incorrect predictions in the first network model. Therefore, if the existing KD loss value calculation method is used, the second network will learn the ability of the first network model to predict incorrect samples, thereby reducing the prediction accuracy of the second network. If such a second network is applied to the detection of facial key points, the prediction accuracy of facial key points will be reduced. Therefore, in the embodiments of the present invention, by setting the flag function corresponding to the correct samples in the first prediction distribution to 1 and setting the flag function corresponding to the incorrect samples in the first prediction distribution to 0, the prediction distribution corresponding to the correct samples in the first network (model) is determined according to the flag function, and the second network is distilled and learned using the correct prediction distribution. Thus, during the process of distilling and learning the first network model to the second network, only the correct prediction distribution of the first network model is transferred to the second network, while the incorrect prediction distribution is ignored, thereby reducing or even avoiding the second network from learning the ability of the first network model to predict incorrect samples, thereby improving the prediction accuracy of the second network and the prediction accuracy of facial key points.

[0100] Optionally, before step S350, there is also step S370, specifically:

[0101] S370. Perform global average pooling on each second feature map in the second feature map set to obtain the second channel weights of each second feature map.

[0102] Similarly, as described in step S211, since the second network also includes grouped convolution operations, the image data input to the separable convolution block is grouped, and then convolution operations are performed on each group respectively. For example, each group uses a convolution kernel of size 3×3 for convolution operation. Then the number of convolution kernels is the number of convolution channels. Correspondingly, each convolution kernel performs convolution calculation on the grouped image data, and the feature maps of the convolution channels corresponding to each convolution kernel can be obtained. For example, Figure 5As shown, the number of the second feature maps 402 at the first scale is three, that is, there are three channels, and the number of channels is 3. In the embodiments of the present invention, Global Average Pooling (GAP) is performed on each of the second feature maps with different scales in the set of second feature maps, and thus the second-channel weights of each of the second feature maps can be obtained.

[0103] Optionally, step S350 includes the following steps S351-S355:

[0104] S351. Obtain a preset pruning ratio;

[0105] Specifically, the pruning ratio can be set according to actual needs without specific limitations. The preset pruning ratio can refer to the ratio of pruning the second network or the ratio of retention after pruning. In the embodiments of the present invention, taking the preset pruning ratio as the ratio of pruning the second network as an example for illustration, for example, if the preset pruning ratio is 0.5, the number of channels of the pruning layer corresponding to the second network that needs to be pruned is halved.

[0106] S352. Sort the absolute values of the second-channel weights;

[0107] In the embodiments of the present invention, sorting can be performed in descending order of the absolute values of the second-channel weights. It can be understood that sorting can also be performed in ascending order. The second-channel weights can be, for example, the weights of each convolution kernel used for grouped convolution operations. The absolute values of the second-channel weights are the absolute values of the second-channel weights obtained after the second network is trained. When the absolute value of the second-channel weight is lower, for example, closer to 0, it means that the effect on the second network is lower and the importance is lower, and it is considered that pruning can be performed. It should be noted that the model pruning in the embodiments of the present invention adopts an iterative training method. In each round of training, it is necessary to sort the absolute values of the second-channel weights and determine the preset pruning ratio according to step S351.

[0108] S353. Determine the target second-channel weights according to the pruning ratio and the sorting result;

[0109] Specifically, according to the pruning ratio and the result of sorting the absolute values of the second-channel weights in descending order, the target second-channel weights are determined, where the target second-channel weights can refer to the second-channel weights corresponding to the channels to be retained or the second-channel weights corresponding to the channels to be pruned.

[0110] For example, when the pruning ratio is 0.5 and the result of sorting the absolute values of the second-channel weights in descending order is 0.7, 0.6, 0.5, 0.3, 0.2, 0.1, the second-channel weights corresponding to the channels to be retained are 0.7, 0.6, 0.5.

[0111] S354. Determine the target second feature map according to the target second channel weight;

[0112] Specifically, as in the example in step S353 above, the second channels corresponding to 0.3, 0.2, and 0.1, that is, the second feature maps corresponding to 0.3, 0.2, and 0.1, are used as the target second feature map.

[0113] S355. Delete the target second feature map from the second network model to obtain a key point detection model.

[0114] Specifically, deleting the target second feature map from the second network model is equivalent to deleting the second feature maps that have little effect on the prediction results of the second network model during the training process, while retaining the second feature maps that have a greater impact on the prediction results of the second network model during the training process, and using this second network model as the key point detection model. It can be understood that this key point detection model can perform key point detection of the face, and the model structure and model parameters of this key point detection model are smaller, the memory utilization rate is lower, the inference speed is faster, and the loss of inference accuracy is smaller compared to the initial second network model.

[0115] Optionally, after step S370, the following steps S371 - S374 are further included:

[0116] S371. Obtain the number of training samples in the training data;

[0117] Specifically, in the embodiments of the present invention, the number of training samples refers to the total number of correct samples and wrong samples in the training data.

[0118] S372. Obtain the number of channels of the first feature map;

[0119] Specifically, in the embodiments of the present invention, the number of channels of the first feature map can be determined by the method in step S211, which will not be elaborated here.

[0120] S373. Calculate the channel attention loss value according to the number of training samples, the number of channels of the first feature map, the first channel weight of the first feature map, and the second channel weight of the second feature map;

[0121] Specifically, in the embodiments of the present invention, the following formula is used to calculate the channel attention loss value:

[0122]

[0123] Among them, \(CA(s,t)\) is the channel attention loss value between the first network model and the second network, \(s\) represents the second network, \(t\) represents the first network model, \(n\) is the number of training samples, \(c\) represents the number of channels of the first feature map, represents the weight size of the \(j\)-th channel of the \(i\)-th sample of the second network, represents the weight magnitude of the j-th channel of the i-th sample of the first network model. Therefore, by substituting the number of training samples, the number of channels of the first feature map, the first channel weight of the first feature map, and the second channel weight of the second feature map, the channel attention loss value can be calculated. It should be noted that, as Figure 5 shown, CA is Channel Attention, that is, channel attention includes but is not limited to having convolution operations, batch normalization / ReLu (Rectified Linear Unit), and global average pooling operations. The feature map calculated by the first network model through CA is the feature map of the first network model and the second network at the same resolution. For example, the first feature map at the first scale is calculated with the second feature map at the first scale through CA. During the distillation learning process, the first network model supervises the second network to learn the attention information of each of its channels, thereby improving the effect of key point detection of the second network. It should be noted that Figure 5 the same number of channels of the first feature map and the second feature map shown in

[0124] S374. Update the second channel weight according to the channel attention loss value.

[0125] Specifically, the second channel weight can be updated and adjusted according to the channel attention loss value. For example, a channel attention loss threshold can be set to update and adjust the second channel weight until the channel attention loss value is less than the channel attention loss threshold, that is, taking the currently updated and adjusted second channel weight as the second channel weight of the second network.

[0126] As Figure 7 shown, specifically, step S400 may include the following steps S410 - S430:

[0127] S410. Obtain the face box information of the target image to be detected;

[0128] Specifically, since the target image to be detected may not include a human face, or may include a human face and a background image, a pre-trained face detection network can be used to detect the human face in the target image. For example, when the target image is input into the face detection network, if there is a human face in the target image, the output of the face detection network at this time is the face bounding box information, and the face bounding box information is bbox = [x, y, w, h], where x is the coordinate value of the upper left corner of the face bounding box, y is the coordinate value of the lower right corner of the face bounding box, w is the width value of the face bounding box, and h is the height value of the face bounding box. It can be understood that when there are multiple human faces in the target image, there are multiple bboxes for the face bounding box information. Among them, the face detection network includes but is not limited to MTCNN (Multi-task convolutional neural network).

[0129] S420. Crop the target image according to the face bounding box information to determine the face image;

[0130] Specifically, the part of the human face can be determined according to the coordinate value of the upper left corner of the face bounding box and the coordinate value of the lower right corner of the face bounding box, or can be determined according to one of the width value of the face bounding box, the height value of the face bounding box, combined with the coordinate value of the upper left corner of the face bounding box and the coordinate value of the lower right corner of the face bounding box. For example, the corresponding pixel points in the target image are determined, and then the target image is cropped to crop out the face area and determine it as the face image.

[0131] S430. Input the face image into the key point detection model to obtain the key points in the face image.

[0132] Specifically, the face image obtained in the above steps is output to the key point detection model, so as to detect the key points in the face image. In the embodiment of the present invention, the face key point coordinate set J = [x 1 , y 1 , x 2 , y 2 , …, x n , y n is obtained, where n is the number of key points, and x n , y n represents the coordinates of the nth face key point. For example, if the key points of the human face obtained are 94 points, then n is 94, and the face key point coordinate set has the coordinates of the 94 key points of the human face.

[0133] Optionally, the key point detection method for the target image in the embodiment of the present invention may further include the following step S500 or step S600:

[0134] S500. Perform key point annotation on the target image according to the key points in the target image to obtain a first visualized image with key points annotated;

[0135] Specifically, after the key point detection model detects a face image to obtain the key points in the face image, the key points are labeled, and the labeled face image is used as the first visualization image. It should be noted that the labeling method is not specifically limited and can be performed in the form of a combination of one or more colors and one or more shapes; the first visualization image can be an image with key points labeled in the face image or an image with key points labeled in the target image, which is not specifically limited.

[0136] For example Figure 8 As shown in the figure, the target image 801 is input into the face detection network 802 to determine the face frame information and crop the target image to obtain the face image 803. The face image 803 is input into the key point detection model 804 to obtain the key points, and the key points are labeled to obtain the visualization image 805.

[0137] S600. Perform key point elimination on the target image according to the key points in the face image to obtain a second visualization image with the key points eliminated.

[0138] Specifically, after the key point detection model detects a face image to obtain the key points in the face image, the key points can be eliminated to obtain a second visualization image with the key points eliminated. It should be noted that the second visualization image can be an image with key points eliminated in the face image or an image with key points eliminated in the target image, which is not specifically limited.

[0139] For example Figure 9 As shown in the figure, taking the first network as the teacher network and the second network as the student network as an example, the key point detection method for the target image in the embodiment of the present invention is described in detail. The detection method specifically includes steps S701 - S715:

[0140] S701. Obtain training data.

[0141] S702. Input the training data into the teacher network and input the training data into the student network.

[0142] It should be noted that the following steps are described by taking the teacher network and the student network as an example of being trained in an online distillation manner through parallel training. Among them, after inputting the training data into the teacher network, the following steps S703 - S705 are included:

[0143] S703. The teacher network performs separable convolution processing on the training data to obtain a set of first feature maps of different scales.

[0144] Specifically, the implementation manner of step S703 can refer to step S210.

[0145] S704. Use the fully connected layer of the teacher network to predict the first key point coordinate values for the first set of feature maps, obtaining a first prediction result.

[0146] Specifically, the implementation of step S704 can refer to step S220.

[0147] S705. Use the first loss function to determine the first loss value between the first prediction result and the label, and modify the parameters of the teacher network model according to the first loss value, thereby generating a teacher network model.

[0148] Specifically, the determination of the first loss value can be determined by the formula in step S230, and the method of modifying the parameters of the teacher network model according to the first loss value to generate a teacher network model can refer to step S240.

[0149] Among them, after inputting the training data into the student network, the following steps S706 - S711 are included:

[0150] S706. Input the training data into the student network for separable convolution processing to obtain a second set of feature maps with different scales;

[0151] Specifically, the implementation of step S706 can refer to step S310.

[0152] S707. Use the fully connected layer of the student network to predict the second key point coordinate values for the second set of feature maps, obtaining a second prediction result;

[0153] Specifically, the implementation of step S707 can refer to step S320.

[0154] S708. Use the second loss function to determine the second loss value between the second prediction result and the label;

[0155] Specifically, the determination of the second loss value can be determined by a formula similar to that in step S230.

[0156] S709. Determine the first prediction distribution according to the first prediction result, and determine the second prediction distribution according to the second prediction result;

[0157] Specifically, the implementation of step S709 refers to the formula in step S341. Among them, step S709 can be executed after step S704 and step S708.

[0158] S710. Determine the correct samples in the training data, set the flag function corresponding to the correct samples in the first prediction distribution to 1, and set the flag function corresponding to the incorrect samples in the first prediction distribution to 0, obtaining an assigned flag function; determine the knowledge distillation loss function according to the assigned flag function, and calculate the difference value between the first prediction distribution and the second prediction distribution to obtain a third loss value;

[0159] Specifically, the implementation of step S710 refers to step S342.

[0160] S711. Modify the parameters of the student network model according to the second loss value and the third loss value to generate a student network model.

[0161] Specifically, the implementation of step S711 refers to step S343.

[0162] It should be noted that steps S706 - S711 and steps S703 - S705 in the embodiments of the present invention do not limit the order of execution.

[0163] S712. Perform global average pooling on each second feature map in the second feature map set to obtain the second channel weights of each second feature map.

[0164] Among them, step S712 can be executed after step S706.

[0165] S713. Obtain a preset pruning ratio, sort the absolute values of the second channel weights, determine the target second channel weights according to the pruning ratio and the sorting result, thereby determine the target second feature map, and delete the target second feature map from the student network model to obtain a key point detection model.

[0166] Specifically, the implementation of step S713 refers to step S350. Among them, step S713 can be executed after step S712 and step S711.

[0167] S714. Obtain the face box information of the target image to be detected, crop the target image according to the face box information to determine a face image, and input the face image into the key point detection model to obtain the key points in the face image;

[0168] Specifically, the implementation of step S713 refers to step S400.

[0169] S715. Perform key point annotation on the target image according to the key points in the target image to obtain a visualized image with key points annotated.

[0170] Specifically, the implementation of step S713 refers to step S500.

[0171] Among them, the key point detection model of the embodiments of the present invention can reduce the inference speed of the model while improving the face key point detection effect index. As shown in Table 1, compared with the existing key point detection models, the key point detection model of the embodiments of the present invention can reduce the number of parameters and the number of floating-point operations per second, and obtain better effect indexes on the common training data, such as the normalized mean error, which is used to measure the difference between the predicted key point coordinate values and the true key point coordinate values. The lower the value, the better the model effect, which fully shows that the solution of the present invention can more effectively transfer the knowledge of the first network to the second network for learning, and then more effectively detect the face key points, and improve the model inference speed without reducing the detection effect through model pruning, so as to more efficiently meet the needs of client applications.

[0172] Table 1

[0173]

[0174] Figure 10 is a schematic structural diagram of a key point detection device 1000 for a target image provided by an embodiment of the present invention, as Figure 10 shown, the detection device 1000 may include:

[0175] An acquisition module 1100, configured to acquire training data;

[0176] A first training module 1200, configured to input the training data into a first network to train a first network model;

[0177] A second training module 1300, configured to input the training data into a second network, perform distillation learning on the second network according to the first network model, and perform pruning processing on the second network while performing distillation learning to obtain a key point detection model, where the knowledge distillation loss function in the distillation learning includes a flag function, and the flag function is used to flag correct samples and incorrect samples in the training data; performing distillation learning on the second network according to the first network model includes: determining a correct prediction distribution corresponding to the correct samples in the first network according to the flag function, and performing distillation learning on the second network according to the correct prediction distribution;

[0178] A detection module 1400, configured to input a target image to be detected into the key point detection model to obtain key points in the target image.

[0179] In summary, the detection device provided by the embodiment of the present invention performs distillation learning on the second network based on the trained first network model, and obtains the second network model with fewer model parameters as the key point detection model, which improves the calculation speed and detection efficiency of the key point detection model; through the flag function, the second network can perform knowledge distillation on the prediction results corresponding to the correct samples in the first network, without spending other computational loads to perform knowledge distillation on the prediction process of the wrong samples, further improving the calculation speed of the key point detection model; furthermore, the embodiment of the present invention also performs pruning processing on the second network, which helps to reduce the scale of the key point detection model.

[0180] The embodiment of the present invention also provides an electronic device, which includes a processor and a memory;

[0181] The memory stores a program;

[0182] The processor executes the program to execute the foregoing key point detection method for the target image.

[0183] This electronic device has the function of carrying out the key point detection of the target image in the embodiment of the present invention, such as a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, etc.

[0184] The embodiment of the present invention also provides a computer-readable storage medium, which stores a program, and the program is executed by the processor to complete the foregoing key point detection method for the target image.

[0185] The embodiment of the present invention also provides a computer program product including instructions, which when running on a computer, causes the computer to execute the foregoing key point detection method for the target image.

[0186] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0187] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item) of the following" or its similar expressions refer to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0188] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0191] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0192] In the above, the above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of various embodiments of this application.

Claims

1. A method for detecting key points of a target image, characterized in that, it includes: Obtain training data; Input the training data into a first network to train a first network model; Input the training data into a second network for separable convolution processing to obtain a set of second feature maps of different scales; predict the second key point coordinate values through the fully connected layer of the second network to obtain a second prediction result; use a second loss function to determine the second loss value between the second prediction result and the label; modify the parameters of the second network model according to the second loss value and the knowledge distillation loss function, and generate a second network model according to the modified parameters of the second network model; perform pruning processing on the second network model to obtain a key point detection model, wherein the knowledge distillation loss function in distillation learning includes a flag function, and the flag function is used to flag correct samples and incorrect samples in the training data; Input the target image to be detected into the key point detection model to obtain the key points in the target image; Among them, the step of modifying the parameters of the second network model according to the second loss value and the knowledge distillation loss function, and generating a second network model according to the modified parameters of the second network model includes: Determine a first prediction distribution according to the first prediction result, and determine a second prediction distribution according to the second prediction result, where the first prediction result is obtained after the first network predicts the first key point coordinate values based on the training data; determine the correct samples in the training data; set the flag function corresponding to the correct samples in the first prediction distribution to 1, and set the flag function corresponding to the incorrect samples in the first prediction distribution to 0 to obtain an assigned flag function; determine the knowledge distillation loss function according to the assigned flag function, and use the knowledge distillation loss function to calculate the difference value between the first prediction distribution and the second prediction distribution to obtain a third loss value; modify the parameters of the second network model according to the second loss value and the third loss value, and generate a second network model according to the modified parameters of the second network model.

2. The method for detecting key points of a target image according to claim 1, characterized in that, the step of inputting the training data into a first network to train a first network model includes: Input the training data into the first network for separable convolution processing to obtain a set of first feature maps of different scales; the training data has a label, and the label is the true coordinate value of the key points of the training data; Predict the first key point coordinate values through the fully connected layer of the first network to obtain a first prediction result; Use a first loss function to determine the first loss value between the first prediction result and the label; Modify the parameters of the first network model according to the first loss value, and generate the first network model according to the modified parameters of the first network model.

3. The method for detecting key points of a target image according to claim 2, characterized in that, Before the step of predicting the first key point coordinate values for the first set of feature maps through the fully connected layer of the first network to obtain a first prediction result, the following steps are further included: Perform global average pooling on each first feature map in the first set of feature maps to obtain the first channel weights of each first feature map.

4. A method for detecting key points of a target image according to claim 1, wherein, Before the step of pruning the second network model to obtain a key point detection model, the following steps are further included: Perform global average pooling on each second feature map in the second set of feature maps to obtain the second channel weights of each second feature map.

5. A method for detecting key points of a target image according to claim 4, wherein, The step of pruning the second network model to obtain the key point detection model includes: Obtain a preset pruning ratio; Sort the absolute values of the second channel weights; Determine target second channel weights according to the pruning ratio and the sorting result; Determine target second feature maps according to the target second channel weights; Delete the target second feature maps from the second network model to obtain the key point detection model.

6. A method for detecting key points of a target image according to claim 4, wherein, After the step of performing global average pooling on each second feature map in the second set of feature maps to obtain the second channel weights of each second feature map, the following steps are further included: Obtain the number of training samples in the training data; Obtain the number of channels of the first feature map; Calculate a channel attention loss value according to the number of training samples, the number of channels of the first feature map, the first channel weights of the first feature map, and the second channel weights of the second feature map; Update the second channel weights according to the channel attention loss value.

7. A method for detecting key points of a target image according to claim 1, wherein, The step of modifying the parameters of the second network model according to the second loss value and the knowledge distillation loss function and generating a second network model according to the modified parameters of the second network model further includes: Perform convolution processing on the first feature map according to the number of channels of the first feature map and the number of channels of the second feature map, wherein the convolution processing is used to make the number of channels of the first feature map the same as the number of channels of the second feature map.

8. A method for detecting key points of a target image according to claim 1, wherein, The step of inputting the target image to be detected into the key point detection model to obtain the key points in the target image includes: Obtain the face box information of the target image to be detected, wherein the face box information includes the upper left coordinate value of the face box, the width value of the face box, and the height value of the face box; Crop the target image according to the face box information to determine a face image; Input the face image into the key point detection model to obtain the key points in the face image.

9. A method for detecting key points of a target image according to any one of claims 1-8, characterized in that, the method further includes: performing key point annotation on the target image according to the key points in the target image to obtain a first visualized image annotated with key points; or, performing key point removal on the target image according to the key points in the face image to obtain a second visualized image with key points removed.

10. A device for detecting key points of a target image, characterized in that, it includes: an acquisition module for acquiring training data; a first training module for inputting the training data into a first network to train and obtain a first network model; a second training module for inputting the training data into a second network for separable convolution processing to obtain a set of second feature maps of different scales; predicting second key point coordinate values for the set of second feature maps through a fully connected layer of the second network to obtain a second prediction result; determining a second loss value between the second prediction result and a label by using a second loss function; modifying the parameters of the second network model according to the second loss value and a knowledge distillation loss function, and generating a second network model according to the modified parameters of the second network model; pruning the second network model to obtain a key point detection model, wherein the knowledge distillation loss function in distillation learning includes a flag function for flagging correct samples and incorrect samples in the training data; a detection module for inputting the target image to be detected into the key point detection model to obtain the key points in the target image; wherein, the modifying the parameters of the second network model according to the second loss value and the knowledge distillation loss function, and generating a second network model according to the modified parameters of the second network model includes: determining a first prediction distribution according to a first prediction result, and determining a second prediction distribution according to the second prediction result, the first prediction result being obtained after the first network predicts first key point coordinate values based on the training data; determining correct samples in the training data; setting the flag function corresponding to the correct samples in the first prediction distribution to 1, and setting the flag function corresponding to the incorrect samples in the first prediction distribution to 0 to obtain an assigned flag function; determining a knowledge distillation loss function according to the assigned flag function, and calculating a difference value between the first prediction distribution and the second prediction distribution by using the knowledge distillation loss function to obtain a third loss value; modifying the parameters of the second network model according to the second loss value and the third loss value, and generating a second network model according to the modified parameters of the second network model.

11. An electronic device, characterized in that, it includes a processor and a memory; the memory is used for storing a program; the processor executes the program to implement the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, the storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1-9.

Images