Key Point Detection Method, Device, Electronic Device and Computer-Readable Storage Medium

Abstract

An embodiment of the present invention discloses a key point detection method, device, electronic device, and computer-readable storage medium. After obtaining an image sample set, the embodiment of the present invention trains a preset depth detection model using the image sample set, compresses the preset depth detection model multiple times, and trains a compressed transition detection model using the trained depth detection model. According to the trained transition detection model, a preset lightweight detection model is trained to obtain a candidate trained lightweight detection model. The candidate trained lightweight detection model is used to detect key points of the image sample. Based on the detection results, a target trained lightweight detection model is selected from the candidate trained lightweight detection models, and the target trained lightweight detection model is used to detect key points of the image to be detected. This solution can improve the detection efficiency of key point detection. The embodiment of the present invention can be applied to various scenarios such as cloud technology, artificial intelligence, and intelligent transportation.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular, to a key point detection method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of Internet technologies, the detection of images or videos has become more accurate. For example, key points in a video or an image can be detected. Taking a video / image of a human face as an example, the so-called key points can be eyes, nose, mouth, etc. in the human face contour. Detecting key points often requires a large model. To reduce the size of the detection model, existing key point detection methods often obtain a lightweight detection model through knowledge distillation and use the lightweight detection model to detect key points.

[0003] In the process of researching and practicing the prior art, the inventors of the present invention found that when the lightweight detection model is reduced to a certain extent, the feature gap is too large, so that knowledge distillation cannot be performed, which limits the size of the lightweight detection model. Therefore, the detection efficiency of key point detection is reduced. Summary of the Invention

[0004] Embodiments of the present invention provide a key point detection method, apparatus, electronic device, and computer-readable storage medium, which can improve the detection efficiency of key point detection.

[0005] A key point detection method includes:

[0006] Obtaining a set of image samples, and training a preset depth detection model with the set of image samples to obtain a trained depth detection model;

[0007] Compressing the preset depth detection model multiple times, and training each obtained transition detection model after compression with the set of image samples and the trained depth detection model to obtain multiple trained transition detection models;

[0008] Training a preset lightweight detection model with the set of image samples and the trained transition detection models to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model, where the preset lightweight detection model is a detection model obtained by lightweight processing of a preset depth detection model;

[0009] Detecting key points of the image samples in the set of image samples with the candidate trained lightweight detection models;

[0010] Based on the detection results, screening out a target trained lightweight detection model from the candidate trained lightweight detection models, and detecting key points of the image to be detected with the target trained lightweight detection model.

[0011] Correspondingly, an embodiment of the present invention provides a key point detection device, including:

[0012] An acquisition unit, configured to acquire an image sample set, and train a preset depth detection model using the image sample set to obtain a trained depth detection model;

[0013] A compression unit, configured to compress the preset depth detection model multiple times, and train the obtained transition detection model after each compression using the image sample set and the trained depth detection model to obtain multiple trained transition detection models;

[0014] A training unit, configured to train a preset lightweight detection model according to the image sample set and the trained transition detection model to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model, where the preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model;

[0015] A detection unit, configured to perform key point detection on the image samples in the image sample set using the candidate trained lightweight detection model;

[0016] A screening unit, configured to screen out a target trained lightweight detection model from the candidate trained lightweight detection models based on the detection results, and perform key point detection on the image to be detected using the target trained lightweight detection model.

[0017] Optionally, in some embodiments, the compression unit may specifically be configured to perform key point prediction on the image samples in the image sample set using the trained depth detection model to obtain a first key point prediction result; perform key point prediction on the image samples in the image sample set using the transition detection model to obtain a second key point prediction result; and converge the obtained transition detection model after each compression according to the first key point prediction result and the second key point prediction result to obtain multiple trained transition detection models.

[0018] Optionally, in some embodiments, the compression unit may specifically be configured to obtain a preset key point position interval of each image sample in the image sample set; determine target loss information of the image sample set according to the preset key point position interval, the first key point prediction result, and the second key point prediction result; and converge the obtained transition detection model after each compression based on the target loss information to obtain multiple trained transition detection models.

[0019] Optionally, in some embodiments, the compression unit may be specifically configured to determine the key point loss information of the image sample set according to the preset key point position interval and the second key point prediction result; determine the distillation loss information of the image sample set based on the first key point prediction result and the second key point prediction result; and fuse the key point loss information and the distillation loss information to obtain the target loss information of the image sample set.

[0020] Optionally, in some embodiments, the compression unit may be specifically configured to extract the position coordinates of each predicted key point from the second key prediction result; determine the position relationship between the predicted key point and the preset key point position interval according to the position coordinates; and calculate the initial key point loss information of the predicted key point based on the position relationship, and fuse the initial key point loss information to obtain the key point loss information of the image sample set.

[0021] Optionally, in some embodiments, the compression unit may be specifically configured to compare the position coordinates with the interval coordinates of the predicted key point position interval; when the position coordinates are within the interval coordinates, determine that the predicted key point is within the preset key point position interval; and when the position coordinates are outside the interval coordinates, determine that the predicted key point is outside the preset key point position interval.

[0022] Optionally, in some embodiments, the compression unit may be specifically configured to, when the predicted key point is within the preset key point position interval, adjust the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point; and when the predicted key point is outside the preset key point position region, calculate the position distance between the predicted key point and the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point.

[0023] Optionally, in some embodiments, the compression unit may be specifically configured to determine the adjustment parameter of the preset key point position interval according to the position coordinates of the predicted key point; and fuse the adjustment parameter with the preset key point position interval to obtain the initial key point loss information of the predicted key point.

[0024] Optionally, in some embodiments, the compression unit may specifically be configured to calculate a curvature ratio between the preset key point position interval and the preset interval curvature; screen out the target position coordinates of the target key points in the preset key point position interval according to the curvature ratio; and calculate a position distance between the predicted key points and the target key points based on the position coordinates and the target position coordinates, so as to obtain the initial key point loss information of the predicted key points.

[0025] Optionally, in some embodiments, the compression unit may specifically be configured to screen out the first predicted position of each predicted key point in the first key point prediction result, and screen out the second predicted position of the corresponding predicted key point in the second key point prediction result; calculate a position distance between the first predicted position and the second predicted position of the predicted key point to obtain the initial distillation loss information corresponding to the predicted key point; and fuse the initial distillation loss information of each predicted key point to obtain the distillation loss information of the image sample set.

[0026] Optionally, in some embodiments, the training unit may specifically be configured to perform key point prediction on the image samples in the image sample set by using the trained transition detection model to obtain a third key point prediction result; perform key point prediction on the image samples in the image sample set by using the preset lightweight detection model to obtain a fourth key point prediction result; and converge the preset lightweight detection model according to the third key point prediction result and the fourth key point prediction result to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model.

[0027] Optionally, in some embodiments, the screening unit may specifically be configured to obtain the key point annotation positions of each image sample in the image sample set; extract the key point prediction positions of each image sample from the detection results; and screen out a target trained lightweight detection model from the candidate trained lightweight detection models according to the key point annotation positions and the key point prediction positions.

[0028] Optionally, in some embodiments, the screening unit may specifically be configured to compare the key point prediction positions with the key point annotation positions; calculate the prediction accuracy of each candidate trained lightweight model for the image sample set based on the comparison result; and screen out a target trained lightweight detection model from the candidate trained lightweight detection models according to the prediction accuracy.

[0029] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, where the memory stores an application program, and the processor is configured to run the application program in the memory to implement the key point detection method provided by the embodiment of the present invention.

[0030] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing multiple instructions adapted to be loaded by a processor to execute the steps in any of the key point detection methods provided by the embodiments of the present invention.

[0031] After obtaining an image sample set in an embodiment of the present invention, training a preset depth detection model with the image sample set to obtain a trained depth detection model, compressing the preset depth detection model multiple times, and training a transition detection model obtained after each compression with the image sample set and the trained depth detection model to obtain multiple trained transition detection models. Then, training a preset lightweight detection model according to the image sample set and the trained transition detection models to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model. Then, using the candidate trained lightweight detection model to perform key point detection on the image samples in the image sample set, screening out a target trained lightweight detection model from the candidate trained lightweight detection models based on the detection results, and using the target trained lightweight detection model to perform key point detection on the image to be detected; since this solution does not directly distill a lightweight detection model from the trained depth detection model, but adds multiple transition detection models between the depth detection model and the lightweight detection model, and the transition models are obtained by compressing the depth detection model, so that the knowledge learned by the depth detection model can be gradually transferred to the lightweight detection model through the transition detection models, making the lightweight detection model lighter. Therefore, the detection efficiency of key point detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0033] Figure 1 It is a schematic diagram of the scenario of the key point detection method provided by the embodiment of the present invention;

[0034] Figure 2 It is a schematic flowchart of the key point detection method provided by the embodiment of the present invention;

[0035] Figure 3 It is a schematic structural diagram of knowledge distillation transfer of the transition detection model provided by the embodiment of the present invention;

[0036] Figure 4 It is a detection schematic diagram of performing key point detection using the candidate trained lightweight detection model provided by the embodiment of the present invention;

[0037] Figure 5 is another schematic flowchart of key point detection provided by an embodiment of the present invention;

[0038] Figure 6 is a schematic structural diagram of a key point detection device provided by an embodiment of the present invention;

[0039] Figure 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0041] An embodiment of the present invention provides a key point detection method, device, electronic device, and computer-readable storage medium. Among them, the key point detection device can be integrated in the electronic device, and the electronic device can be a server or a device such as a terminal.

[0042] Among them, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0043] For example, refer to Figure 1, taking the integration of the key point detection device in the electronic device as an example, after the electronic device obtains the image sample set and trains the preset depth detection model with the image sample set to obtain the trained depth detection model, the preset depth detection model is compressed multiple times, and the image sample set and the trained depth detection model are used to train the transition detection model obtained after each compression to obtain multiple trained transition detection models. Then, according to the image sample set and the trained transition detection models, the preset lightweight detection model is trained to obtain the candidate trained lightweight detection models corresponding to each trained transition detection model. The preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model. The candidate trained lightweight detection models are used to perform key point detection on the image samples in the image sample set. Based on the detection results, the target trained lightweight detection model is selected from the candidate trained lightweight detection models, and the target trained lightweight detection model is used to perform key point detection on the image to be detected, thereby improving the detection efficiency of key point detection.

[0044] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0045] This embodiment will be described from the perspective of the key point detection device. The key point detection device can be specifically integrated in an electronic device, which can be a server or a terminal device, etc. Among them, the terminal can include devices such as tablet computers, laptop computers, personal computers (PCs), wearable devices, virtual reality devices, or intelligent devices that can perform key point detection.

[0046] A key point detection method includes:

[0047] Obtain an image sample set, and use the image sample set to train a preset depth detection model to obtain a trained depth detection model. Compress the preset depth detection model multiple times, and use the image sample set and the trained depth detection model to train the transition detection model obtained after each compression to obtain multiple trained transition detection models. According to the image sample set and the trained transition detection models, train a preset lightweight detection model to obtain candidate trained lightweight detection models corresponding to each trained transition detection model. The preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model. Use the candidate trained lightweight detection models to perform key point detection on the image samples in the image sample set. Based on the detection results, select the target trained lightweight detection model from the candidate trained lightweight detection models, and use the target trained lightweight detection model to perform key point detection on the image to be detected.

[0048] As Figure 2 shown, the specific process of this key point detection method is as follows:

[0049] 101. Obtain an image sample set and use the image sample set to train a preset depth detection model to obtain a trained depth detection model.

[0050] Among them, the preset depth detection model can be a preset large model for key point detection.

[0051] Among them, there are various ways to obtain the image sample set. Specifically, it can be as follows:

[0052] For example, it is possible to directly receive the image sample set uploaded by the user through the terminal, or extract at least one video frame from the video, label the key points in the video frame to obtain the image sample set, or obtain the original image from the Internet or the image database, identify the target original image with key points in the original image sample, and label the key points existing in the target original image to obtain the image sample. By fusing the image samples, the image sample set can be obtained. Or receive the detection model training request sent by the terminal. The detection model training request carries the storage address of the image sample. According to the storage address, obtain the image sample in the memory, cache or third-party database of the terminal, so as to obtain the image sample set.

[0053] After obtaining the image sample set, the preset depth detection model can be trained using the image sample set. There are various training methods. For example, the preset depth detection model can be used to predict the key points of each image sample in the image sample set to obtain the target key point prediction result. Obtain the preset key point position interval of each image sample in the image sample set. According to the preset key point position interval and the target key point prediction result, determine the target key point loss information of the image sample set. Based on the target key point loss information, converge the preset depth detection model to obtain the trained depth detection model.

[0054] Among them, there are various ways to determine the target key point loss information of the image sample set according to the preset key point position interval and the target key point prediction result. For example, extract the basic position coordinates of each predicted key point in the target key point prediction result. Based on the basic position coordinates, determine the target position relationship between the predicted key point and the preset key point interval. Based on the target position relationship, calculate the loss information of the predicted key point to obtain the first key point loss information, and fuse the first key point loss information to obtain the target key point loss information.

[0055] Among them, the target position relationship can be understood as predicting whether the key point is within the preset key point position interval. Therefore, there can be multiple ways to determine the target position relationship. For example, the basic position coordinates are compared with the interval coordinates of the predicted key point position interval. When the basic position coordinates are within the interval coordinates, it is determined that the predicted key point is within the preset key point position interval. When the basic position coordinates are outside the interval coordinates, it is determined that the predicted key point is outside the preset key point position interval.

[0056] After the target position relationship, based on the target position relationship, the loss information of the predicted key point can be calculated to obtain the first key point loss information. There can be multiple calculation methods. For example, when the predicted key point is within the preset key point position interval, the preset key point position interval is adjusted according to the basic position coordinates of the predicted key point to obtain the first key point loss information of the predicted key point. When the predicted key point is outside the preset key point position interval, the position distance between the predicted key point and the preset key point position interval is calculated according to the basic position coordinates of the predicted key point to obtain the first key point loss information of the predicted key point. Specifically, it can be as shown in formula (1):

[0057]

[0058] Among them, wing(x) is the loss information of the first key point, x is the basic position coordinates of the predicted key point, w is the preset key point position interval, ∈ is the preset interval curvature, and C is a constant, C = w - wln(1 + w / ∈).

[0059] It can be found from formula (1) that when the predicted key point is within the preset key point position interval, there can be multiple ways to adjust the preset key point position interval. For example, the adjustment parameter of the preset key point position interval can be determined according to the position coordinates of the predicted key point, and the adjustment parameter is fused with the preset key point position interval to obtain the first key point loss information of the preset key point. When the predicted key point is outside the preset key point position interval, there can be multiple ways to calculate the position distance between the predicted key point and the preset key point position interval. For example, the curvature ratio of the preset key point position interval and the preset interval curvature can be calculated, and based on the curvature ratio, the target position coordinates of the target key point are selected from the preset key point position interval. Here, the target position coordinates can be understood as the constant C in formula (1). Based on the position coordinates and the target position coordinates, the position distance between the predicted key point and the target key point is calculated to obtain the first key point loss information of the predicted key point.

[0060] Among them, there are various ways to converge the preset depth detection model based on the target key point loss information. For example, the gradient descent algorithm or other convergence algorithms can be used to converge the preset depth detection model based on the target key point loss information to obtain the trained depth detection model. The trained depth detection model here can be understood as the teacher detection model for key point detection.

[0061] 102. Compress the preset depth detection model multiple times, and use the image sample set and the trained depth detection model to train the intermediate detection model obtained after each compression to obtain multiple trained intermediate detection models.

[0062] Among them, the intermediate detection model is used for intermediate detection in knowledge distillation. Since the lightweight detection model is small, if knowledge distillation is directly performed, the feature gap is too large, which will lead to the failure of distillation. The role of using the intermediate detection model is to build a knowledge distillation bridge between the depth detection model and the lightweight detection model for transmitting and transitioning the learned knowledge.

[0063] Among them, there are various ways to compress the preset depth detection model multiple times. Specifically, it can be as follows:

[0064] For example, the model structure of the preset depth detection model can be obtained. Based on this model structure, some repeated or adjustable network layers of the preset depth detection model are deleted. Each time, a different number of network layers are deleted, so that multiple depth detection models after deletion can be obtained. The depth detection model after deletion is used as the compressed intermediate detection model. Or, the model depth of the preset depth detection model can be directly reduced, so that preset depth detection models with different model depths can be obtained. The preset depth detection model with reduced depth is used as the compressed intermediate detection model. Or, the attribute information of the preset depth detection model can be obtained, and the size of the preset depth detection model is extracted from this attribute information. Then, according to the size of the preset depth detection model, the preset depth detection model is directly compressed using multiple compression ratios. For example, it can be compressed to 0.9, 0.8 of the original size or the model size corresponding to any compression ratio. The preset depth detection models corresponding to different model sizes are used as the compressed intermediate detection models.

[0065] After adjusting the preset depth detection model to obtain the transition detection model, the transition detection model obtained after each compression can be trained (knowledge distillation) using the image sample set and the trained depth detection model. There can be multiple training processes. For example, the trained depth detection model can be used to predict key points of the image samples in the image sample set to obtain the first key point prediction result, and the transition detection model can be used to predict key points of the image samples in the image sample set to obtain the second key point prediction result. Based on the first key point prediction result and the second key point prediction result, the transition detection model obtained after each compression is converged to obtain multiple trained transition detection models.

[0066] Among them, there can be multiple ways to converge the transition detection model obtained after each compression based on the first key point prediction result and the second key point prediction result. For example, the preset key point position interval of each image sample in the image sample set can be obtained, and based on the preset key point position interval, the first key point prediction result, and the second key point prediction result, the target loss information of the image sample set is determined. Based on the target loss information, the transition detection model obtained after each compression is converged to obtain multiple trained transition detection models.

[0067] Among them, there can be multiple ways to determine the target loss information of the image sample set based on the preset key point position interval, the first key point prediction result, and the second key point prediction result. For example, the key point loss information of the image sample can be determined based on the preset key point position interval and the second key point prediction result, the target loss information of the image sample set is determined based on the first key point prediction result and the second key point prediction result, and the key point loss information and the distillation loss information are fused to obtain the target loss information of the image sample set.

[0068] Among them, the key point loss information can be the loss information generated by the transition detection model during key point prediction. There can be multiple ways to determine the key point loss information of the image sample set based on the preset key point position interval and the second key point prediction result. For example, the position coordinates of each predicted key point can be extracted from the second key point prediction result, and based on the position coordinates, the position relationship between the predicted key point and the preset key point position interval is determined. Based on this position relationship, the initial key point loss information of the predicted key point is calculated, and the initial key point loss information is fused to obtain the key point loss information of the image sample set.

[0069] Among them, the positional relationship can be understood as whether the predicted key point is located within the preset key point position interval. Therefore, there can be multiple ways to determine the positional relationship. For example, compare the position coordinates with the interval coordinates of the preset key point position interval. When the position coordinates are within the interval coordinates, it is determined that the predicted key point is within the preset key point position interval. When the position coordinates are outside the interval coordinates, it is determined that the predicted key point is outside the preset key point position interval.

[0070] After determining the positional relationship, the initial key point loss information of the predicted key point can be calculated based on the positional relationship. There can be multiple calculation methods. For example, when the predicted key point is within the preset key point position interval, adjust the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point. When the predicted key point is outside the preset key point position interval, calculate the positional distance between the predicted key point and the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point.

[0071] Among them, calculating the initial key point loss information of the predicted key point can be as shown in formula (1). There can be multiple ways to adjust the preset key point position interval to obtain the initial key point loss information. For example, determine the adjustment parameter of the preset key point position interval according to the position coordinates of the predicted key point, and fuse the adjustment parameter with the preset key point position interval to obtain the initial key point loss information of the predicted key point. There can be multiple ways to calculate the positional distance between the predicted key point and the preset key point position interval to obtain the initial key point loss information. For example, the curvature ratio of the preset key point position interval and the preset interval curvature can be calculated. According to the curvature ratio, the target position coordinates of the target key point are screened out in the preset key point position interval. Based on the position coordinates and the target position coordinates, the positional distance between the predicted key point and the target key point is calculated to obtain the initial key point loss information of the predicted key point.

[0072] Among them, the distillation loss information can be understood as the loss information generated by supervising the transition detection model with the feature supervision signal of the trained depth detection model. The role of this distillation loss information is to constrain the output of the transition detection model to be as close as possible to the output of the trained depth detection model. There are various ways to determine the distillation loss information of the image sample set according to the first key point prediction result and the second key point prediction result. For example, the first predicted position of each predicted key point can be selected from the first key point prediction result, and the second predicted position of the corresponding predicted key point can be selected from the second key point prediction result. The position distance between the first predicted position and the second predicted position of the predicted key point is calculated to obtain the initial distillation loss information corresponding to the predicted key point. The initial distillation loss information of each predicted key point is fused to obtain the distillation loss information of the image sample set, which can be specifically shown in formula (2):

[0073] l dis =∑(x T -x s ) 2 (2)

[0074] Among them, l dis is the distillation loss information, x T is the first predicted position, and x S is the second predicted position.

[0075] After determining the key point loss information and the distillation loss information, the key point loss information and the distillation loss information can be fused. There are various ways of fusion. For example, the key point loss information and the distillation loss information can be directly added to obtain the target loss information of the image sample set, which can be specifically shown in formula (3):

[0076] loss=l dis (x)+wing(x) (3)

[0077] Among them, loss is the target loss information, l dis (x) is the distillation loss information, and wing(x) is the key point loss information.

[0078] Optionally, the fusion method can also include obtaining the weighting coefficients of the key point loss information and the distillation loss information, weighting the key point loss information and the distillation loss information respectively based on the weighting coefficients, and then fusing the weighted key point loss information and the distillation loss information to obtain the target loss information of the image sample set.

[0079] After calculating the target loss information of the image sample set, each transition detection model can be converged based on this target loss information. There are various ways to converge, for example, the gradient descent algorithm or other convergence algorithms can be used to update the network parameters of the transition detection model through the target loss information to converge the transition detection model, thereby obtaining multiple trained transition detection models.

[0080] 103. Train the preset lightweight detection model according to the image sample set and the trained transition detection model to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model.

[0081] Among them, the preset lightweight detection model can be a model with a small size for key point detection. The network structures of the preset lightweight detection model, the transition detection model, and the preset depth detection model can be the same or similar, or different. The preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model. Lightweight processing can be understood as reducing the depth or channels of the preset depth detection model so that the preset lightweight detection model is much smaller than the preset depth detection model.

[0082] Among them, there are various ways to train the preset lightweight detection model according to the image sample set and the trained transition detection model. Specifically, it can be as follows:

[0083] For example, the trained transition detection model can be used to predict key points of the image samples in the image sample set to obtain a third key point prediction result. The preset lightweight detection model can be used to predict key points of the image samples in the image sample set to obtain a fourth key point prediction result. According to the third key point prediction result and the fourth key point prediction result, the preset lightweight detection model is converged to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model.

[0084] Among them, the process of training the preset lightweight detection model using the trained transition detection model and the image sample set can be regarded as using the trained transition detection model to perform knowledge distillation on the preset lightweight detection model, so that the preset lightweight detection model can learn valuable features of the transition detection model. The process of this knowledge distillation is the same as the way the trained depth detection model performs knowledge distillation on the transition detection model. Refer to the above description and will not be elaborated here one by one.

[0085] Among them, there are two stages of knowledge distillation from the trained depth detection model to the trained lightweight detection model, and it is transmitted through multiple transition detection models, as Figure 3 shown, thereby overcoming the capacity limitation of the lightweight detection model before and being able to gradually transfer the knowledge learned by the depth detection model to the preset lightweight detection model well.

[0086] 104. Use the candidate-trained lightweight detection model to perform key point detection on the image samples in the image sample set.

[0087] For example, the candidate-trained lightweight detection model can be used to extract features of multiple dimensions from the image samples in the image sample set to obtain local image features of multiple dimensions. Then, the local image features are fused to obtain the global image feature of the image sample. Based on the global image feature, the key point coordinates of at least one key point in the image sample are determined, and the key point coordinates are used as the detection result. Specifically, it can be as Figure 4 shown.

[0088] 105. Based on the detection result, select the target-trained lightweight detection model from the candidate-trained lightweight detection models, and use the target-trained lightweight detection model to perform key point detection on the image to be detected.

[0089] Among them, the target-trained lightweight detection model can be understood as the trained lightweight detection model with the best key point detection effect among multiple candidate lightweight detection models. There are various ways to select the target-trained lightweight detection model. Specifically, it can be as follows:

[0090] For example, the key point annotation positions of each image sample in the image sample set can be obtained, the key point prediction positions of each image sample are extracted from the detection result, and the target-trained lightweight detection model is selected from the candidate-trained lightweight detection models according to the key point annotation positions and the key point prediction positions.

[0091] Among them, there are various ways to select the target-trained lightweight detection model from the candidate-trained lightweight detection models according to the key point annotation positions and the key point prediction positions. For example, the key point prediction positions can be compared with the key point annotation positions. Based on the comparison result, the prediction accuracy rate of each candidate-trained lightweight detection model for the image sample set is calculated, and the target-trained lightweight detection model is selected from the candidate-trained lightweight detection models according to the prediction accuracy rate.

[0092] Among them, the prediction accuracy can be understood as the accuracy of the candidate trained lightweight detection model in predicting the key points of each image sample in the image sample set. There are various ways to calculate the prediction accuracy. For example, when the position error and quantity error between the position of the key points marked in the image sample and the position of the predicted key points are within the preset error threshold, it can be determined that the image sample is predicted accurately. Then, count the number of image samples predicted accurately, and then calculate the ratio between the number of image samples predicted accurately and the total number of the image sample set, so as to obtain the prediction accuracy. Or, when the position error between the position of the key points marked in the image sample and the position of the predicted key points is within the preset error threshold, it can be determined that the key point is predicted accurately. Then, count the number of key points predicted accurately, calculate the ratio between the number of key points predicted accurately and the total number of the key points marked in the image sample set, so as to obtain the prediction accuracy of the image sample set.

[0093] After calculating the prediction accuracy, the target trained lightweight detection model can be selected from the candidate trained lightweight detection models. There are various selection methods. For example, the candidate trained lightweight detection model with the highest prediction accuracy can be directly selected as the target trained lightweight detection model. Or, the candidate trained lightweight detection models with the top K prediction accuracies can also be selected, and then a new image sample set is obtained. The top K candidate trained lightweight detection models are used to predict the key points of the image samples in the new image sample set. Then, calculate the prediction accuracy again, weight the prediction accuracies calculated twice, and sort the weighted prediction accuracies, so as to select the candidate trained lightweight detection model with the largest weighted prediction accuracy as the target trained lightweight detection model.

[0094] Optionally, the target trained lightweight detection model can also be selected from the candidate trained lightweight detection models by using the neural network architecture search (NAS) method, so as to select the target trained lightweight detection model.

[0095] Among them, after obtaining the trained deep detection model, the trained transition detection model, and the target trained lightweight detection model, the image samples can also be tested. The visualization results of the test can be shown in Table 1:

[0096] Table 1

[0097] Model Normalized Mean Error Optimization Ratio Trained Depth Detection Model 0.032317 - Trained Transition Model 0.032575 - Preset Lightweight Detection Model 0.03472 - Lightweight Detection Model after Basic Training 0.03486 -0.4％ Lightweight Detection Model after Target Training 0.033168 4.5％

[0098] Among them, the lightweight detection model after basic training is the lightweight detection model after training obtained by directly performing knowledge distillation on the preset lightweight detection model using the deep model after training. The target lightweight detection model after training is the lightweight detection model after training obtained by performing knowledge distillation twice using the intermediate detection model. As can be seen from Table 1, directly distilling the preset lightweight detection model using an extremely large deep detection model after training may still result in negative optimization. However, using the intermediate detection model to distill the preset lightweight detection model can obtain a 4.5% positive gain. Therefore, it shows that this solution can be well used in lightweight model distillation, mainly to solve the problem of insufficient capacity of the lightweight model. Using the intermediate network is equivalent to performing intermediate screening on the network, which can gradually perform transitions in the feature dimensions and can better learn the feature fitting ability of the deep detection model after training (teacher network).

[0099] After the target lightweight detection model after training is selected, the target lightweight detection model after training can be used to perform key point detection on the image to be detected. There can be various detection methods. For example, the image to be detected can be obtained, and the target lightweight detection model after training is used to perform multi-dimensional feature extraction on the image to be detected to obtain target image features in multiple dimensions. The target image features are fused to obtain the fused image features. Based on the fused image features, the key point positions of the key points in the image to be detected are determined. The key point positions can be one or more, thereby completing the key point detection of the image to be detected.

[0100] As can be seen from the above, in the embodiment of the present application, after obtaining the image sample set and training the preset deep detection model using the image sample set to obtain the deep detection model after training, the preset deep detection model is compressed multiple times, and the intermediate detection model obtained after each compression is trained using the image sample set and the deep detection model after training to obtain multiple intermediate detection models after training. Then, based on the image sample set and the intermediate detection model after training, the preset lightweight detection model is trained to obtain the candidate lightweight detection model after training corresponding to each intermediate detection model after training. Then, the candidate lightweight detection model after training is used to perform key point detection on the image samples in the image sample set. Based on the detection results, the target lightweight detection model after training is selected from the candidate lightweight detection models after training, and the target lightweight detection model after training is used to perform key point detection on the image to be detected; since this solution does not directly distill the lightweight detection model by using the deep detection model after training, but adds multiple intermediate detection models between the deep detection model and the lightweight detection model, and the intermediate model is obtained by compressing the deep detection model, the knowledge learned by the deep detection model can be gradually transferred to the lightweight detection model through the intermediate detection model, making the lightweight detection model lighter. Therefore, the detection efficiency of key point detection can be improved.

[0101] According to the method described in the above embodiments, further detailed examples will be given below.

[0102] In this embodiment, it will be described by taking the case where the key point detection device is specifically integrated in an electronic device, the electronic device is a server, the trained deep detection model is the teacher detection model, and the trained lightweight detection model is the student detection model.

[0103] As Figure 5 shown, a key point detection method has the following specific process:

[0104] 201. The server obtains an image sample set.

[0105] For example, the server can directly receive the image sample set uploaded by the user through the terminal, or extract at least one video frame from the video, annotate the key points in the video frame to obtain the image sample set, or obtain the original images from the Internet or the image database, identify the target original images with key points in the original image samples, and annotate the key points existing in the target original images to obtain the image samples. By fusing the image samples, the image sample set can be obtained. Or, receive the detection model training request sent by the terminal. The detection model training request carries the storage address of the image sample. According to the storage address, obtain the image sample in the memory, cache or third-party database of the terminal, so as to obtain the image sample set.

[0106] 202. The server uses the image sample set to train the preset deep detection model to obtain the teacher detection model.

[0107] For example, the server uses the preset deep detection model to predict the key points of each image sample in the image sample set to obtain the target key point prediction result, obtains the preset key point position interval of each image sample in the image sample set, and extracts the basic position coordinates of each predicted key point from the target key point prediction result. Compare the basic position coordinates with the interval coordinates of the predicted key point position interval. When the basic position coordinates are within the interval coordinates, it is determined that the predicted key point is within the preset key point position interval. When the basic position coordinates are outside the interval coordinates, it is determined that the predicted key point is outside the preset key point position interval.

[0108] When the predicted key point is within the preset key point position interval, the server adjusts the preset key point position interval according to the basic position coordinates of the predicted key point to obtain the first key point loss information of the predicted key point. When the predicted key point is outside the preset key point position interval, the server calculates the position distance between the predicted key point and the preset key point position interval according to the basic position coordinates of the predicted key point to obtain the first key point loss information of the predicted key point, which can be specifically shown in formula (1).

[0109] It can be found from formula (1) that when the predicted key point is within the preset key point position interval, the server can determine the adjustment parameter of the preset key point position interval according to the position coordinates of the predicted key point, and fuse the adjustment parameter with the preset key point position interval to obtain the first key point loss information of the preset key point. When the predicted key point is outside the preset key point position interval, the server can calculate the curvature ratio between the preset key point position interval and the preset interval curvature, and screen out the target position coordinates of the target key point in the preset key point position interval according to the curvature ratio. Here, the target position coordinates can be understood as the constant C in formula (1). Based on the position coordinates and the target position coordinates, the position distance between the predicted key point and the target key point is calculated to obtain the first key point loss information of the predicted key point.

[0110] The server uses the gradient descent algorithm or other convergence algorithms to converge the preset depth detection model based on the target key point loss information to obtain the teacher detection model.

[0111] 203. The server compresses the preset depth detection model multiple times to obtain multiple compressed intermediate detection models.

[0112] For example, the server can obtain the model structure of the preset depth detection model, and based on this model structure, delete some repeated or adjustable network layers of the preset depth detection model. Each time, different numbers of network layers are deleted, so that multiple depth detection models after deletion can be obtained. The depth detection models after deletion are used as the compressed intermediate detection models. Or, the model depth of the preset depth detection model can be directly reduced, so that preset depth detection models with different model depths can be obtained. The preset depth detection model with reduced depth is used as the compressed intermediate detection model. Or, the server can also obtain the attribute information of the preset depth detection model, extract the size of the preset depth detection model from this attribute information, and then directly compress the preset depth detection model with multiple compression ratios according to the size of the preset depth detection model to obtain multiple compressed intermediate detection models.

[0113] 204. The server uses the image sample set and the teacher detection model to train the intermediate detection model obtained after each compression to obtain multiple trained intermediate detection models.

[0114] For example, the server can use the teacher detection model to predict the key points of the image samples in the image sample set to obtain the first key point prediction result, and use the intermediate detection model to predict the key points of the image samples in the image sample set to obtain the second key point prediction result. The server obtains the preset key point position interval of each image sample in the image sample set.

[0115] The server can extract the position coordinates of each predicted key point from the second key point prediction result, compare the position coordinates with the interval coordinates of the preset key point position interval. When the position coordinates are within the interval coordinates, it is determined that the predicted key point is within the preset key point position interval; when the position coordinates are outside the interval coordinates, it is determined that the predicted key point is outside the preset key point position interval.

[0116] When the predicted key point is within the preset key point position interval, the server determines the adjustment parameter of the preset key point position interval according to the position coordinates of the predicted key point, and fuses the adjustment parameter with the preset key point position interval to obtain the initial key point loss information of the predicted key point. When the predicted key point is outside the preset key point position interval, the server calculates the curvature ratio of the preset key point position interval to the preset interval curvature, and based on the curvature ratio, filters out the target position coordinates of the target key point in the preset key point position interval. Based on the position coordinates and the target position coordinates, the position distance between the predicted key point and the target key point is calculated to obtain the initial key point loss information of the predicted key point. The initial key point loss information is fused to obtain the key point loss information of the image sample set.

[0117] The server can filter out the first predicted position of each predicted key point in the first key point prediction result, and filter out the second predicted position of the corresponding predicted key point in the second key point prediction result, calculate the position distance between the first predicted position and the second predicted position of the predicted key point to obtain the initial distillation loss information corresponding to the predicted key point, and fuse the initial distillation loss information of each predicted key point to obtain the distillation loss information of the image sample set, which can be specifically shown in formula (2).

[0118] The server directly adds the key point loss information and the distillation loss information to obtain the target loss information of the image sample set, which can be specifically shown in formula (3). Or, it can also obtain the weighting coefficients of the key point loss information and the distillation loss information, weight the key point loss information and the distillation loss information respectively based on the weighting coefficients, and then fuse the weighted key point loss information and the distillation loss information to obtain the target loss information of the image sample set.

[0119] The server uses the gradient descent algorithm or other convergence algorithms to update the network parameters of the transition detection model through the target loss information to converge the transition detection model, thereby obtaining multiple trained transition detection models.

[0120] 205. The server trains the preset lightweight detection model according to the image sample set and the trained transition detection model to obtain a candidate student detection model corresponding to each trained transition detection model.

[0121] For example, the server uses the trained transition detection model to predict key points of the image samples in the image sample set, obtaining the third key point prediction result, and uses the preset lightweight detection model to predict key points of the image samples in the image sample set, obtaining the fourth key point prediction result. Based on the third key point prediction result and the fourth key point prediction result, the preset lightweight detection model is converged to obtain a candidate student detection model corresponding to each trained transition detection model. The training process here is the same as the process of the teacher detection model training the transition detection model, so it will not be elaborated one by one here.

[0122] 206. The server uses the candidate student detection model to detect key points of the image samples in the image sample set.

[0123] For example, the server can use the candidate student detection model to extract multi-dimensional features of the image samples in the image sample set, obtaining local image features in multiple dimensions. Then, the local image features are fused to obtain the global image feature of the image sample. Based on the global image feature, the key point coordinates of at least one key point in the image sample are determined, and the key point coordinates are used as the detection result.

[0124] 207. The server filters out the target student detection model from the candidate student detection models based on the detection results.

[0125] For example, the server can obtain the key point annotation positions of each image sample in the image sample set, and extract the key point prediction positions of each image sample from the detection results. The key point prediction position is compared with the key point annotation position. When the position error and quantity error between the position of the annotated key point and the predicted key point in the image sample are within the preset error threshold, it can be determined that the prediction of this image sample is accurate. Then, the number of image samples with accurate predictions is counted, and the ratio between the number of image samples with accurate predictions and the total number of the image sample set is calculated to obtain the prediction accuracy rate. Or, when the position error between the position of the annotated key point and the predicted key point in the image sample is within the preset error threshold, it can be determined that the prediction of this key point is accurate. Then, the number of accurately predicted key points is counted, and the ratio between the number of accurately predicted key points and the total number of the annotated key points in the image sample set is calculated to obtain the prediction accuracy rate of the image sample set.

[0126] The server can directly select the candidate student detection model with the highest prediction accuracy as the target student detection model. Alternatively, it can also select the top K candidate student detection models with the highest prediction accuracy, then obtain a new set of image samples, use the top K candidate student detection models to perform key point prediction on the image samples in the new set of image samples, and then calculate the prediction accuracy again, weight the prediction accuracies calculated twice, and sort the weighted prediction accuracies, so as to select the candidate student detection model with the largest weighted prediction accuracy as the target trained lightweight detection model. Or, it can also be selected by means of neural architecture search (NAS), so as to select the target student detection model.

[0127] 208. The server uses the target student detection model to perform key point detection on the image to be detected.

[0128] For example, the server can obtain the image to be detected, use the target student detection model to perform multi-dimensional feature extraction on the image to be detected to obtain target image features in multiple dimensions, fuse the target image features to obtain the fused image features, and determine the key point positions of the key points in the image to be detected according to the fused image features. The key point positions can be one or more, so as to complete the key point detection of the image to be detected.

[0129] As can be seen from the above, in this embodiment, after the server obtains the set of image samples and uses the set of image samples to train the preset depth detection model to obtain the teacher detection model, it compresses the preset depth detection model multiple times, and uses the set of image samples and the trained depth detection model to train the transitional detection models obtained after each compression to obtain multiple trained transitional detection models. Then, according to the set of image samples and the trained transitional detection models, it trains the preset lightweight detection model to obtain candidate student detection models corresponding to each trained transitional detection model. Then, it uses the candidate student detection models to perform key point detection on the image samples in the set of image samples, and based on the detection results, selects the target student detection model from the candidate student detection models, and uses the target student detection model to perform key point detection on the image to be detected. Since this solution does not directly use the teacher detection model for distillation to obtain the student detection model, but adds multiple transitional detection models between the teacher detection model and the student detection model, and the transitional models are obtained by compressing the teacher detection model, the knowledge learned by the teacher detection model can be gradually transferred to the student detection model through the transitional detection models, making the student detection model more lightweight. Therefore, the detection efficiency of key point detection can be improved.

[0130] To better implement the above method, an embodiment of the present invention further provides a key point detection device, which can be integrated in an electronic device, such as a server or a terminal device, etc. The terminal may include a tablet computer, a notebook computer, and / or a personal computer, etc.

[0131] For example, as Figure 6 shown, the key point detection device may include an acquisition unit 301, a compression unit 302, a training unit 303, a detection unit 304, and a screening unit 305, as follows:

[0132] (1) Acquisition unit 301;

[0133] The acquisition unit 301 is used to acquire an image sample set and train a preset depth detection model with the image sample set to obtain a trained depth detection model.

[0134] For example, the acquisition unit 301 can specifically be used to acquire an image sample set, predict the key points of each image sample in the image sample set using the preset depth detection model to obtain a target key point prediction result, acquire the preset key point position interval of each image sample in the image sample set, determine the target key point loss information of the image sample set according to the preset key point position interval and the target key point prediction result, and converge the preset depth detection model based on the target key point loss information to obtain a trained depth detection model.

[0135] (2) Compression unit 302;

[0136] The compression unit 302 is used to compress the preset depth detection model multiple times and train the transition detection model obtained after each compression with the image sample set and the trained depth detection model to obtain multiple trained transition detection models.

[0137] For example, the compression unit 302 can specifically be used to compress the preset depth detection model multiple times, predict the key points of the image samples in the image sample set using the trained depth detection model to obtain a first key point prediction result, predict the key points of the image samples in the image sample set using the transition detection model to obtain a second key point prediction result, and converge the transition detection model obtained after each compression according to the first key point prediction result and the second key point prediction result to obtain multiple trained transition detection models.

[0138] (3) Training unit 303;

[0139] The training unit 303 is configured to train a preset lightweight detection model according to an image sample set and a trained transition detection model, so as to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model. The preset trained lightweight detection model is a detection model obtained by cleaning a preset deep detection model.

[0140] For example, the training unit 303 can be specifically configured to perform key point prediction on the image samples in the image sample set by using the trained transition detection model to obtain a third key point prediction result, perform key point prediction on the image samples in the image sample set by using the preset lightweight detection model to obtain a fourth key point prediction result, and converge the preset lightweight detection model according to the third key point prediction result and the fourth key point prediction result, so as to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model.

[0141] (4) The detection unit 304;

[0142] The detection unit 304 is configured to perform key point detection on the image samples in the image sample set by using the candidate trained lightweight detection model.

[0143] For example, the detection unit 304 can be specifically configured to perform feature extraction on the image samples in the image sample set in multiple dimensions by using the candidate trained lightweight detection model to obtain local image features in multiple dimensions, then fuse the local image features to obtain a global image feature of the image sample, determine the key point coordinates of at least one key point in the image sample according to the global image feature, and use the key point coordinates as the detection result.

[0144] (5) The screening unit 305;

[0145] The screening unit 305 is configured to screen out a target trained lightweight detection model from the candidate trained lightweight detection models based on the detection result, and perform key point detection on the image to be detected by using the target trained lightweight detection model.

[0146] For example, the screening unit 305 can be specifically configured to obtain the key point annotation positions of each image sample in the image sample set, extract the key point prediction positions of each image sample from the detection result, and screen out the target trained lightweight detection model from the candidate trained lightweight detection models according to the key point annotation positions and the key point prediction positions. Perform key point detection on the image to be detected by using the target trained lightweight detection model.

[0147] In specific implementation, each of the above units can be implemented as an independent entity, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of each of the above units, reference can be made to the foregoing method embodiments, which will not be elaborated herein.

[0148] As can be seen from the above, in the embodiment of the present application, after the acquisition unit 301 acquires the image sample set and trains the preset depth detection model with the image sample set to obtain the trained depth detection model, the compression unit 302 compresses the preset depth detection model multiple times, and trains the obtained transition detection model after each compression with the image sample set and the trained depth detection model to obtain multiple trained transition detection models. Then, the training unit 303 trains the preset lightweight detection model according to the image sample set and the trained transition detection models to obtain the candidate trained lightweight detection models corresponding to each trained transition detection model. Then, the detection unit 304 uses the candidate trained lightweight detection models to perform key point detection on the image samples in the image sample set, and the screening unit 305 screens out the target trained lightweight detection model from the candidate trained lightweight detection models based on the detection results, and uses the target trained lightweight detection model to perform key point detection on the image to be detected. Since this solution does not directly use the trained depth detection model to distill the lightweight detection model, but adds multiple transition detection models between the depth detection model and the lightweight detection model, and the transition models are obtained by compressing the depth detection model, the knowledge learned by the depth detection model can be gradually transferred to the lightweight detection model through the transition detection models, making the lightweight detection model lighter. Therefore, the detection efficiency of key point detection can be improved.

[0149] An embodiment of the present invention also provides an electronic device, such as Figure 7 shown, which shows a schematic structural diagram of the electronic device involved in the embodiment of the present invention. Specifically:

[0150] The electronic device may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input unit 404 and other components. Those skilled in the art can understand that Figure 7 the structure of the electronic device shown in

[0151] The processor 401 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 402, and by invoking the data stored in the memory 402, it executes various functions of the electronic device and processes data, thereby performing an overall detection of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 401 either.

[0152] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.); the data storage area can store data created according to the use of the electronic device. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. Correspondingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0153] The electronic device further includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 may also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0154] The electronic device may further include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0155] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402 to realize various functions as follows:

[0156] Obtain a set of image samples, and use the set of image samples to train a preset depth detection model to obtain a trained depth detection model. Compress the preset depth detection model multiple times, and use the set of image samples and the trained depth detection model to train the transitional detection models obtained after each compression to obtain multiple trained transitional detection models. According to the set of image samples and the trained transitional detection models, train a preset lightweight detection model to obtain a candidate trained lightweight detection model corresponding to each trained transitional detection model, where the preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model; use the candidate trained lightweight detection model to perform key point detection on the image samples in the set of image samples; based on the detection results, select a target trained lightweight detection model from the candidate trained lightweight detection models, and use the target trained lightweight detection model to perform key point detection on the image to be detected.

[0157] For example, an electronic device obtains a set of image samples, uses a preset depth detection model to predict key points of each image sample in the set of image samples to obtain a target key point prediction result, obtains a preset key point position interval of each image sample in the set of image samples, determines the target key point loss information of the set of image samples according to the preset key point position interval and the target key point prediction result, and converges the preset depth detection model based on the target key point loss information to obtain a trained depth detection model. The preset depth detection model is compressed multiple times. The trained depth detection model is used to predict key points of the image samples in the set of image samples to obtain a first key point prediction result. A transition detection model is used to predict key points of the image samples in the set of image samples to obtain a second key point prediction result. The transition detection model obtained after each compression is converged according to the first key point prediction result and the second key point prediction result to obtain multiple trained transition detection models. The trained transition detection models are used to predict key points of the image samples in the set of image samples to obtain a third key point prediction result. A preset lightweight detection model is used to predict key points of the image samples in the set of image samples to obtain a fourth key point prediction result. The preset lightweight detection model is converged according to the third key point prediction result and the fourth key point prediction result to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model. The candidate trained lightweight detection model is used to extract local image features in multiple dimensions from the image samples in the set of image samples, and then the local image features are fused to obtain global image features of the image samples. At least one key point coordinate in the image samples is determined according to the global image features, and the key point coordinate is used as a detection result. The key point annotation position of each image sample in the set of image samples is obtained, the key point prediction position of each image sample is extracted from the detection results, and a target trained lightweight detection model is selected from the candidate trained lightweight detection models according to the key point annotation position and the key point prediction position. The target trained lightweight detection model is used to detect key points of the image to be detected.

[0158] For the specific implementation of each of the above operations, reference may be made to the previous embodiments, which will not be elaborated here.

[0159] As can be seen from the above, in the embodiment of the present invention, after obtaining an image sample set and training a preset depth detection model with the image sample set to obtain a trained depth detection model, the preset depth detection model is compressed multiple times, and the image sample set and the trained depth detection model are used to train the transitional detection models obtained after each compression to obtain multiple trained transitional detection models. Then, according to the image sample set and the trained transitional detection models, the preset lightweight detection model is trained to obtain a candidate trained lightweight detection model corresponding to each trained transitional detection model. Then, the candidate trained lightweight detection model is used to perform key point detection on the image samples in the image sample set. Based on the detection results, a target trained lightweight detection model is selected from the candidate trained lightweight detection models, and the target trained lightweight detection model is used to perform key point detection on the image to be detected. Since this solution does not directly use the trained depth detection model to distill a lightweight detection model, but adds multiple transitional detection models between the depth detection model and the lightweight detection model, and the transitional models are obtained by compressing the depth detection model, the knowledge learned by the depth detection model can be gradually transferred to the lightweight detection model through the transitional detection models, making the lightweight detection model lighter. Therefore, the detection efficiency of key point detection can be improved.

[0160] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0161] Therefore, the embodiment of the present invention provides a computer-readable storage medium, which stores multiple instructions that can be loaded by a processor to execute the steps in any one of the key point detection methods provided by the embodiment of the present invention. For example, the instructions can execute the following steps:

[0162] Obtain a set of image samples, and use the set of image samples to train a preset depth detection model to obtain a trained depth detection model. Compress the preset depth detection model multiple times, and use the set of image samples and the trained depth detection model to train the transitional detection models obtained after each compression to obtain multiple trained transitional detection models. According to the set of image samples and the trained transitional detection models, train a preset lightweight detection model to obtain a candidate trained lightweight detection model corresponding to each trained transitional detection model, where the preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model; use the candidate trained lightweight detection models to perform key point detection on the image samples in the set of image samples; based on the detection results, select a target trained lightweight detection model from the candidate trained lightweight detection models, and use the target trained lightweight detection model to perform key point detection on the image to be detected.

[0163] For example, obtain a set of image samples, use a preset depth detection model to predict the key points of each image sample in the set of image samples to obtain a target key point prediction result, obtain the preset key point position interval of each image sample in the set of image samples, and determine the target key point loss information of the set of image samples based on the preset key point position interval and the target key point prediction result. Based on the target key point loss information, converge the preset depth detection model to obtain a trained depth detection model. Perform multiple compressions on the preset depth detection model, use the trained depth detection model to predict the key points of the image samples in the set of image samples to obtain a first key point prediction result, use a transition detection model to predict the key points of the image samples in the set of image samples to obtain a second key point prediction result, and converge the transition detection model obtained after each compression based on the first key point prediction result and the second key point prediction result to obtain multiple trained transition detection models. Use the trained transition detection model to predict the key points of the image samples in the set of image samples to obtain a third key point prediction result, use a preset lightweight detection model to predict the key points of the image samples in the set of image samples to obtain a fourth key point prediction result, and converge the preset lightweight detection model based on the third key point prediction result and the fourth key point prediction result to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model. Use the candidate trained lightweight detection model to extract local image features in multiple dimensions from the image samples in the set of image samples, then fuse the local image features to obtain the global image features of the image samples, determine the key point coordinates of at least one key point in the image samples based on the global image features, and use the key point coordinates as the detection result. Obtain the key point annotation positions of each image sample in the set of image samples, extract the key point prediction positions of each image sample from the detection results, and screen out the target trained lightweight detection model from the candidate trained lightweight detection models based on the key point annotation positions and the key point prediction positions. Use the target trained lightweight detection model to perform key point detection on the image to be detected.

[0164] For the specific implementation of each of the above operations, reference may be made to the previous embodiments and will not be elaborated here.

[0165] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0166] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the key point detection methods provided by the embodiments of the present invention, the beneficial effects achievable by any of the key point detection methods provided by the embodiments of the present invention can be realized. For details, see the previous embodiments and will not be repeated here.

[0167] Wherein, according to one aspect of the present application, there is provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in various alternative implementations of the above key point detection aspect or knowledge distillation aspect.

[0168] The above has introduced in detail a key point detection method, apparatus, electronic device, and computer-readable storage medium provided by the embodiments of the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A key point detection method, characterized in that Including: Obtain a set of image samples, and use the set of image samples to train a preset depth detection model to obtain a trained depth detection model; Compress the preset depth detection model multiple times, and use the set of image samples and the trained depth detection model to train the transitional detection models obtained after each compression to obtain multiple trained transitional detection models; According to the set of image samples and the trained transitional detection models, train a preset lightweight detection model to obtain a candidate trained lightweight detection model corresponding to each trained transitional detection model, where the preset lightweight detection model is a detection model obtained by lightweight processing of the preset depth detection model; Use the candidate trained lightweight detection model to perform key point detection on the image samples in the set of image samples; Based on the detection results, select a target trained lightweight detection model from the candidate trained lightweight detection models, and use the target trained lightweight detection model to perform key point detection on the image to be detected; Among them, the step of using the set of image samples and the trained depth detection model to train the transitional detection models obtained after each compression to obtain multiple trained transitional detection models includes: using the trained depth detection model to perform key point prediction on the image samples in the set of image samples to obtain a first key point prediction result; using the transitional detection model to perform key point prediction on the image samples in the set of image samples to obtain a second key point prediction result; according to the first key point prediction result and the second key point prediction result, converge the transitional detection models obtained after each compression to obtain multiple trained transitional detection models.

2. The key point detection method according to claim 1, wherein The step of according to the first key point prediction result and the second key point prediction result, converging the transitional detection models obtained after each compression to obtain multiple trained transitional detection models includes: Obtain the preset key point position interval of each image sample in the set of image samples; According to the preset key point position interval, the first key point prediction result and the second key point prediction result, determine the target loss information of the set of image samples; Based on the target loss information, converge the transitional detection models obtained after each compression to obtain multiple trained transitional detection models.

3. The key point detection method according to claim 2, wherein The step of according to the preset key point position interval, the first key point prediction result and the second key point prediction result, determining the target loss information of the set of image samples includes: According to the preset key point position interval and the second key point prediction result, determine the key point loss information of the set of image samples; Based on the first key point prediction result and the second key point prediction result, determine the distillation loss information of the set of image samples; Fuse the key point loss information and the distillation loss information to obtain the target loss information of the set of image samples.

4. The key point detection method according to claim 3, wherein The step of according to the preset key point position interval and the second key point prediction result, determining the key point loss information of the set of image samples includes: Extract the position coordinates of each predicted key point from the second key prediction result; Determine the positional relationship between the predicted key point and the preset key point position interval according to the position coordinates; Based on the positional relationship, calculate the initial key point loss information of the predicted key point, and fuse the initial key point loss information to obtain the key point loss information of the image sample set.

5. The key point detection method according to claim 4, wherein The determining the positional relationship between the predicted key point and the preset key point position interval according to the position coordinates includes: Compare the position coordinates with the interval coordinates of the predicted key point position interval; When the position coordinates are within the interval coordinates, determine that the predicted key point is within the preset key point position interval; When the position coordinates are outside the interval coordinates, determine that the predicted key point is outside the preset key point position interval.

6. The key point detection method according to claim 5, wherein The calculating the initial key point loss information of the predicted key point based on the positional relationship includes: When the predicted key point is within the preset key point position interval, adjust the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point; When the predicted key point is outside the preset key point position area, calculate the positional distance between the predicted key point and the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point.

7. The key point detection method according to claim 6, wherein The adjusting the preset key point position interval according to the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point includes: Determine the adjustment parameter of the preset key point position interval according to the position coordinates of the predicted key point; Fuse the adjustment parameter with the preset key point position interval to obtain the initial key point loss information of the predicted key point.

8. The key point detection method according to claim 6, wherein The calculating the positional distance between the predicted key point and the preset key point position interval according to the preset key point position area and the position coordinates of the predicted key point to obtain the initial key point loss information of the predicted key point includes: Calculate the curvature ratio of the preset key point position interval to the preset interval curvature; According to the curvature ratio, screen out the target position coordinates of the target key point in the preset key point position interval; Based on the position coordinates and the target position coordinates, calculate the positional distance between the predicted key point and the target key point to obtain the initial key point loss information of the predicted key point.

9. The key point detection method according to claim 3, wherein The determining the distillation loss information of the image sample set based on the first key point prediction result and the second key point prediction result includes: Screen out the first prediction position of each predicted key point in the first key point prediction result, and screen out the second prediction position of the corresponding predicted key point in the second key point prediction result; Calculate the positional distance between the first prediction position and the second prediction position of the predicted key point to obtain the initial distillation loss information corresponding to the predicted key point; Fuse the initial distillation loss information of each predicted key point to obtain the distillation loss information of the image sample set.

10. The key point detection method according to any one of claims 1 to 9, characterized in that, Training the preset lightweight detection model according to the image sample set and the trained transition detection model to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model, including: Using the trained transition detection model to perform key point prediction on the image samples in the image sample set to obtain a third key point prediction result; Using the preset lightweight detection model to perform key point prediction on the image samples in the image sample set to obtain a fourth key point prediction result; Converging the preset lightweight detection model according to the third key point prediction result and the fourth key point prediction result to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model.

11. The key point detection method according to any one of claims 1 to 9, characterized in that, Based on the detection results, screening out the target trained lightweight detection model from the candidate trained lightweight detection models, including: Obtaining the key point annotation positions of each image sample in the image sample set; Extracting the key point prediction positions of each image sample from the detection results; Screening out the target trained lightweight detection model from the candidate trained lightweight detection models according to the key point annotation positions and the key point prediction positions.

12. The key point detection method according to claim 11, wherein The screening out the target trained lightweight detection model from the candidate trained lightweight detection models according to the key point annotation positions and the key point prediction positions includes: Comparing the key point prediction positions with the key point annotation positions; Based on the comparison results, calculating the prediction accuracy rate of each candidate trained lightweight model for the image sample set; Screening out the target trained lightweight detection model from the candidate trained lightweight detection models according to the prediction accuracy rate.

13. A key point detection device, characterized in that, Including: An acquisition unit, configured to acquire an image sample set and use the image sample set to train a preset depth detection model to obtain a trained depth detection model; A compression unit, configured to perform multiple compressions on the preset depth detection model and use the image sample set and the trained depth detection model to train the transition detection models obtained after each compression to obtain multiple trained transition detection models, where the using the image sample set and the trained depth detection model to train the transition detection models obtained after each compression to obtain multiple trained transition detection models includes: using the trained depth detection model to perform key point prediction on the image samples in the image sample set to obtain a first key point prediction result; using the transition detection model to perform key point prediction on the image samples in the image sample set to obtain a second key point prediction result; converging the transition detection models obtained after each compression according to the first key point prediction result and the second key point prediction result to obtain multiple trained transition detection models; A training unit, configured to train a preset lightweight detection model according to the image sample set and the trained transition detection model to obtain a candidate trained lightweight detection model corresponding to each trained transition detection model, where the preset lightweight detection model is a detection model obtained by performing lightweight processing on the preset depth detection model; A detection unit, configured to perform key point detection on an image sample in the image sample set by using the candidate trained lightweight detection model; A screening unit, configured to screen out a target trained lightweight detection model from the candidate trained lightweight detection models based on the detection results, and perform key point detection on the image to be detected by using the target trained lightweight detection model.

14. An electronic device, characterized in that, It includes a processor and a memory. The memory stores an application program, and the processor is configured to run the application program in the memory to execute the steps in the key point detection method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps in the key point detection method according to any one of claims 1 to 12 are implemented.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by the processor to execute the steps in the key point detection method according to any one of claims 1 to 12.

Images