Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses a kind of processing method, device, electronic equipment and storage medium of image.The method comprises: input target image into calibrated key point detection model, obtain the target heat map corresponding to target image output by calibrated key point detection model, wherein, target heat map is used to characterize the position of key point in target image;Average brightness of target heat map is obtained, and the coordinate aggregation degree of target heat map is obtained;Based on average brightness and / or coordinate aggregation degree, the confidence of target image detected by calibrated key point detection model is obtained;Target heat map is processed based on confidence.The application carries out key point detection to image by calibrated key point detection model and outputs heat map, improves the accuracy of model to image key point detection, and based on heat map analysis model, the confidence of image key point detection is further processed according to confidence, and the reliability of image processing is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, digital image processing has been widely applied to all aspects of human life, such as industrial inspection, medicine, and intelligent robots. In any field, people prefer to use images to describe and express the characteristics and logical relationships of things; therefore, the development of digital image processing technology and the requirements for it are becoming increasingly important. However, in related technologies, the use of models to detect key points in images cannot determine whether these key points are occluded, leading to unreliable results for determining image processing methods based on the model's key point detection capabilities. Summary of the Invention

[0003] In view of the above problems, this application proposes an image processing method, apparatus, electronic device, and storage medium, which can perform key point detection on an image using a calibrated key point detection model and output a heatmap, thereby improving the accuracy of the model in detecting key points in the image. Furthermore, based on the heatmap analysis, the confidence level of the model in detecting key points in the image is obtained, and the heatmap is then processed according to the confidence level, thereby improving the reliability of image processing.

[0004] In a first aspect, embodiments of this application provide an image processing method, the method comprising: inputting a target image into a calibrated keypoint detection model to obtain a target heatmap output by the calibrated keypoint detection model corresponding to the target image, wherein the target heatmap is used to characterize the positions of keypoints in the target image; obtaining the average brightness of the target heatmap and obtaining the coordinate clustering degree of the target heatmap; obtaining the confidence level of the calibrated keypoint detection model in detecting the target image based on the average brightness and / or the coordinate clustering degree; and processing the target heatmap based on the confidence level.

[0005] Secondly, embodiments of this application provide an image processing apparatus, comprising: a heatmap acquisition module, an average brightness and coordinate clustering acquisition module, a confidence level acquisition module, and an image processing module. The heatmap acquisition module is used to input a target image into a calibrated keypoint detection model to obtain a target heatmap corresponding to the target image output by the calibrated keypoint detection model, wherein the target heatmap is used to characterize the positions of keypoints in the target image; the average brightness and coordinate clustering acquisition module is used to obtain the average brightness of the target heatmap and the coordinate clustering degree of the target heatmap; the confidence level acquisition module is used to obtain the confidence level of the calibrated keypoint detection model in detecting the target image based on the average brightness and / or the coordinate clustering degree; and the image processing module is used to process the target heatmap based on the confidence level.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs the above-described method.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the above-described method.

[0008] The image processing method, apparatus, electronic device, and storage medium provided in this application embodiment obtain a target heatmap corresponding to the target image by inputting a target image into a calibrated keypoint detection model. The target heatmap is used to characterize the positions of key points in the target image. The method involves obtaining the average brightness and coordinate clustering degree of the target heatmap; obtaining the confidence level of the calibrated keypoint detection model in detecting the target image based on the average brightness and / or coordinate clustering degree; and processing the target heatmap based on the confidence level. By using a calibrated keypoint detection model to perform keypoint detection on the image and outputting a heatmap, the accuracy of the model's keypoint detection in the image is improved. Furthermore, by analyzing the confidence level of the model's keypoint detection based on the heatmap and then processing the heatmap according to the confidence level, the reliability of image processing is improved. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0011] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0012] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0013] Figure 4 A schematic diagram of a label corresponding to a sample image provided in an embodiment of this application is shown;

[0014] Figure 5 This diagram illustrates a heatmap output by a trained keypoint detection model according to an embodiment of this application.

[0015] Figure 6 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0016] Figure 7 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0017] Figure 8 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0018] Figure 9 This illustration shows a schematic diagram of a target heatmap output by a calibrated key point detection model according to an embodiment of this application;

[0019] Figure 10 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0020] Figure 11 A block diagram of an image processing apparatus according to an embodiment of this application is shown;

[0021] Figure 12 A block diagram of an electronic device for performing an image processing method according to an embodiment of the present application is shown;

[0022] Figure 13 A storage unit for storing or carrying program code implementing an image processing method according to an embodiment of the present application is shown. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0024] With the development of hardware and software in the information age, video imaging devices such as smartphones and security monitoring are widely used, which also brings a large amount of image data. How to enable machines to understand and utilize this information is a problem that computer vision needs to study and solve.

[0025] In related technologies, keypoint detection techniques have emerged in the field of computer science, which locate key points in a given image dataset. For example, in keypoint detection of facial images, key points can be feature points of areas such as the eyes and eyebrows. In these techniques, keypoint detection is often performed using models, such as deep neural networks. However, due to the uncertainty in keypoint detection results using models—such as key points not present in the model's training set or those that are occluded—the processing of images based on keypoint detection results from models suffers from low reliability.

[0026] To address the aforementioned problems, the inventors, through long-term research, discovered and proposed the image processing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. This method uses a calibrated keypoint detection model to perform keypoint detection on images and outputs a heatmap, thereby improving the accuracy of the model's keypoint detection. Furthermore, based on the confidence level of the image keypoint detection by the heatmap analysis model, and then processing the heatmap according to the confidence level, the reliability of image processing is improved. The specific image processing method will be described in detail in subsequent embodiments.

[0027] Please see Figure 1 , Figure 1 A schematic flowchart of an image processing method according to an embodiment of this application is shown. This image processing method uses a calibrated keypoint detection model to perform keypoint detection on an image and outputs a heatmap, improving the accuracy of the model's keypoint detection. Furthermore, it analyzes the confidence level of the keypoint detection based on the heatmap, and then processes the heatmap according to the confidence level, thereby improving the reliability of image processing. In a specific embodiment, this image processing method can be applied to, for example... Figure 11 The image processing apparatus 200 shown and the electronic device 100 on which the image processing apparatus 200 is disposed. Figure 12 The following will use an electronic device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the electronic device used in this embodiment may include smartphones, tablets, wearable electronic devices, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown is described in detail. The image processing method may specifically include the following steps:

[0028] Step S110: Input the target image into the calibrated key point detection model to obtain the target heat map corresponding to the target image output by the calibrated key point detection model, wherein the target heat map is used to characterize the position of key points in the target image.

[0029] In some implementations, the electronic device may have a pre-set target image, or it may obtain the target image from a related cloud or electronic device via wireless communication technologies (such as WiFi, Bluetooth, Zigbee, etc.), or it may obtain the target image from a related electronic device via a serial communication interface (such as a serial peripheral interface). In the embodiments of this application, the method by which the electronic device obtains the target image is not limited.

[0030] The target image can be understood as an image in which keypoint detection is performed using the calibrated keypoint detection model provided in this application embodiment. For example, the target image can be a face image, an animal image, a building image, etc., and is not limited thereto. For example, the electronic device may include a camera, and the processor in the electronic device can acquire a face image captured by the camera as the target image.

[0031] In some implementations, the electronic device may have a pre-set calibrated key point detection model, or it may obtain the calibrated key point detection model from the associated cloud or electronic device through wireless communication technology, or it may obtain the calibrated key point detection model from the associated electronic device through a serial communication interface, without limitation.

[0032] The process involves inputting an image into a calibrated keypoint detection model to obtain a heatmap corresponding to the image, output by the model. The heatmap may include multiple bright spots, each corresponding to a keypoint in the image. The pixel size corresponding to each bright spot can be a probability value of [0-1], representing the probability of a keypoint at that pixel's location.

[0033] It should be noted that algorithms for keypoint detection based on heatmaps typically take the coordinates of the brightest spot among the bright spots in the heatmap as the result of keypoint detection in the image. This can be understood as follows: after an image is input into a calibrated keypoint detection model, the calibrated model can output a heatmap corresponding to that image. This heatmap represents the detection result of the calibrated keypoint detection model at the corresponding locations of keypoints in the image.

[0034] Furthermore, after inputting the target image into the calibrated keypoint detection model, a target heatmap corresponding to the target image can be obtained from the output of the calibrated keypoint detection model. The target heatmap is used to characterize the location of key points in the target image.

[0035] Step S120: Obtain the average brightness of the target heatmap and the coordinate clustering degree of the target heatmap.

[0036] In some implementations, after the electronic device obtains the target heatmap corresponding to the target image output by the calibrated key point detection model, it can obtain the average brightness of the target heatmap and the degree of coordinate clustering of the target heatmap based on the pixels of the target heatmap.

[0037] The average brightness of the target heatmap can be the average pixel value of the target heatmap. The degree of coordinate clustering of the target heatmap can be the average, mode, arithmetic mean, median, etc., of the distances from each point in the target heatmap to the center point of the target heatmap, and is not limited here.

[0038] Step S130: Based on the average brightness and / or the degree of coordinate clustering, obtain the confidence level of the calibrated key point detection model for the detection of the target image.

[0039] In some implementations, after the electronic device obtains the average brightness and coordinate clustering of the target heatmap, it can obtain the confidence level of the calibrated key point detection model for target image detection based on the average brightness and / or coordinate clustering.

[0040] Optionally, after obtaining the average brightness of the target heatmap, the electronic device can determine this average brightness as the confidence level of the calibrated keypoint detection model for target image detection. Similarly, after obtaining the coordinate clustering degree of the target heatmap, the electronic device can also determine this coordinate clustering degree as the confidence level of the calibrated keypoint detection model for target image detection. Furthermore, after obtaining both the average brightness and coordinate clustering degree of the target heatmap, the electronic device can calculate the quotient of the average brightness divided by the coordinate clustering degree and determine this quotient as the confidence level of the calibrated keypoint detection model for target image detection.

[0041] Step S140: Process the target heatmap based on the confidence level.

[0042] In some implementations, after the electronic device obtains the confidence level of the calibrated key point detection model for target image detection, it can process the target heatmap based on the confidence level.

[0043] In some implementations, the electronic device may have a preset confidence threshold. After obtaining the confidence level of the calibrated keypoint detection model for target image detection, the electronic device can compare this confidence level with the preset confidence threshold. The result of this comparison can serve as the basis for determining whether to retain the target heatmap. For example, if the confidence level is greater than or equal to the preset confidence threshold, the target heatmap is retained; if the confidence level is less than the preset confidence threshold, the target heatmap is deleted.

[0044] In some implementations, the comparison between the confidence level and a preset confidence threshold can also serve as a basis for determining whether to generate a prompt message indicating that the target heatmap is unqualified. For example, if the confidence level is less than the preset confidence threshold, a prompt message indicating that the target heatmap is unqualified is generated, so that the user or downstream task can process the target heatmap based on the prompt message.

[0045] Optionally, the electronic device processes the target heatmap based on confidence level. This could involve deleting the target heatmap based on confidence level, or further extracting information from the target heatmap based on confidence level, such as performing facial recognition, identity recognition, or building recognition based on the target heatmap. No limitation is imposed here.

[0046] It is understood that in this embodiment, the use of a calibrated keypoint detection model to detect keypoints in the target image improves the accuracy of the keypoint locations in the target heatmap output by the calibrated keypoint detection model. Therefore, when obtaining the confidence level of the calibrated keypoint detection model for target image detection based on the target heatmap, an accurate confidence level can be obtained. Furthermore, without changing the calibrated keypoint detection model, the confidence level of the calibrated keypoint detection model for target image detection is obtained simply by calculating the average brightness and coordinate clustering of the target heatmap. This means that it is unnecessary to repeatedly input the target image into the calibrated keypoint detection model for multiple keypoint location predictions; the uncertainty of the calibrated keypoint detection model for keypoint detection in the target image can still be obtained. This reduces the time cost of confidence level measurement, reduces the computational load for obtaining confidence level, and consequently reduces the computational load for measuring the uncertainty of the calibrated keypoint detection model for keypoint detection in the target image, thus reducing device power consumption. Furthermore, processing the target heatmap based on the confidence level of accurate convergence improves the reliability of the target heatmap processing.

[0047] An embodiment of this application provides an image processing method that, by inputting a target image into a calibrated keypoint detection model, obtains a target heatmap corresponding to the target image output by the calibrated keypoint detection model. The target heatmap is used to characterize the positions of keypoints in the target image. The method involves obtaining the average brightness and coordinate clustering degree of the target heatmap; obtaining the confidence level of the calibrated keypoint detection model in detecting the target image based on the average brightness and / or coordinate clustering degree; and processing the target heatmap based on the confidence level. By using a calibrated keypoint detection model to perform keypoint detection on the image and outputting a heatmap, the accuracy of the model's keypoint detection in the image is improved. Furthermore, by analyzing the confidence level of the model's keypoint detection based on the heatmap and then processing the heatmap according to the confidence level, the reliability of image processing is improved.

[0048] Please see Figure 2 , Figure 2 A schematic flowchart illustrating an image processing method according to an embodiment of this application is shown. This method is applied to the aforementioned electronic device, and will be discussed below. Figure 2 The process shown is described in detail, and the image processing method may specifically include the following steps:

[0049] Step S210: Input the target image into the calibrated key point detection model to obtain the target heat map corresponding to the target image output by the calibrated key point detection model, wherein the target heat map is used to characterize the position of key points in the target image.

[0050] In some implementations, the electronic device may have a pre-set calibrated keypoint detection model. The electronic device can input a target image into the calibrated keypoint detection model to obtain a target heatmap corresponding to the target image, output by the calibrated keypoint detection model. Furthermore, the electronic device can analyze the output target heatmap to obtain the confidence level of the calibrated keypoint detection model in detecting the target image. Among these, most keypoint detection algorithms based on heatmap analysis obtain the coordinates of the maximum pixel value of each bright spot in the heatmap as the result of keypoint detection corresponding to that bright spot.

[0051] In some implementations, please refer to Figure 3 The image processing method provided in this application embodiment may include steps S201-S203 before step S210.

[0052] Step S201: Obtain a sample image set, wherein the sample image set includes multiple sample images and labels corresponding to each of the multiple sample images, wherein the labels include heatmaps used to characterize the positions of key points in the sample images.

[0053] In some implementations, the electronic device may have a pre-set sample image set. The electronic device may also acquire the sample image set from a associated cloud or electronic device via wireless communication technology. Alternatively, the electronic device may acquire sample images from a associated electronic device via a serial communication interface. The sample image set may include multiple sample images and corresponding labels for each sample image; the labels may include heatmaps characterizing the locations of key points in the sample images. For an example, please refer to [link to example]. Figure 4 It displays the labels corresponding to key points in the sample image. These labels are heatmaps showing the locations of the key points.

[0054] The sample images can be human face images, animal images, building images, etc., and are not limited here. The labels for the heatmaps corresponding to each sample image, which represent the positions of key points in the sample image, can be, for example, a heatmap of the position of the mouth key point in the sample image corresponding to the mouth key point in the sample image, a heatmap of the position of the building key point in the sample image corresponding to the building key point in the sample image, etc.

[0055] Step S202: Using the multiple sample images as input parameters and the labels corresponding to each of the multiple sample images as output parameters, train the initial model to obtain a trained keypoint detection model.

[0056] In some implementations, the electronic device may have a pre-set initial model, such as a deep learning algorithm or model like Keras, AlexNet, VGG19, ResNet_152, InceptionV4, or DenseNet. It is understood that the electronic device can train the initial model based on a sample image set to obtain a trained keypoint detection model that takes an image as input and outputs a heatmap corresponding to the keypoint locations in that image. For an example, please refer to [link to example]. Figure 5 It shows a heatmap of the output of the trained keypoint detection model.

[0057] Furthermore, to ensure the accuracy of keypoint detection in images by the trained keypoint detection model, the electronic device can use multiple sample images as input parameters and the corresponding labels of each sample image as output parameters to train the initial model and obtain the trained keypoint detection model.

[0058] The process involves using multiple sample images as input parameters and their corresponding labels as output parameters to train an initial model. This training of the keypoint detection model can be achieved by inputting sample images into the initial model, obtaining a heatmap of keypoint detection results from the initial model, and calculating the loss function value of the initial model based on the heatmap and the corresponding labels. If this loss function value is less than or equal to a preset loss function value, the model obtained from this training is considered a trained keypoint detection model. If the loss function value is greater than the preset loss function value, the initial model continues to be trained until, after inputting sample images into the model obtained from training the initial model, the loss function value between the heatmap of keypoint detection results and the corresponding labels of the sample images is less than or equal to the preset loss function value. In this case, the model obtained from this training is considered a trained keypoint detection model.

[0059] Step S203: Calibrate the trained keypoint detection model to obtain the calibrated keypoint detection model.

[0060] In some implementations, considering that deep learning models often exhibit overconfidence, in this embodiment, this can be understood as a significant mismatch between the probability predicted by the trained keypoint detection model and the actual location of the keypoint in the image. Specifically, the pixel value corresponding to a bright spot in the heatmap can be determined as the probability that the keypoint in the image appears at the corresponding bright spot location, with a probability value ranging from 0 to 1. However, based on the heatmap analysis of the sample image, the location with the highest probability value (from 0 to 1) among the bright spots is selected as the model's predicted keypoint location. Therefore, the predicted probability values ​​of the keypoint locations output by the model are generally too high, resulting in low confidence in the model's keypoint location detection.

[0061] Furthermore, in order to improve the accuracy of the probability values ​​of the predicted key point locations in the image output by the trained model and to increase the confidence of the model in detecting key point locations in the image, the trained key point detection model can be calibrated in this embodiment to obtain a calibrated key point detection model.

[0062] In some implementations, please refer to Figure 6 Step S203 may include steps S2031-S2032.

[0063] Step S2031: Determine the parameters to be calibrated from the multiple parameters included in the trained keypoint detection model.

[0064] In some implementations, after the electronic device obtains a trained keypoint detection model, it can determine the parameters to be calibrated from the multiple parameters included in the trained keypoint detection model.

[0065] For example, a trained keypoint detection model may include a logistic regression function (softmax), where the output of the softmax function reflects the probability value of the model's predicted keypoint locations in the image. The softmax function may include a temperature parameter. Furthermore, the electronic device can determine the temperature parameter included in the softmax function as a parameter to be calibrated from among the multiple parameters included in the trained keypoint detection model.

[0066] Step S2032: Based on the output obtained by inputting the sample image into the trained keypoint detection model and the label corresponding to the sample image, calibrate the parameters to be calibrated to obtain the calibrated keypoint detection model.

[0067] In some implementations, after determining the parameters to be calibrated, the electronic device can calibrate the parameters based on the output obtained by inputting sample images into a trained keypoint detection model and the labels corresponding to the sample images, thereby obtaining a calibrated keypoint detection model.

[0068] The electronic device calibrates the parameters to be calibrated based on the output obtained by inputting sample images into the trained keypoint detection model and the corresponding labels of the sample images. This can be understood as optimizing the parameters to be calibrated without changing the accuracy of keypoint detection by the trained keypoint detection model, and converging the probability of the keypoint location predicted by the calibrated keypoint detection model, so as to avoid the probability of the keypoint location predicted by the calibrated keypoint detection model deviating significantly from the probability of the corresponding position of the keypoint in the target image.

[0069] For example, the temperature parameter of the softmax function in a trained keypoint detection model is calibrated. After obtaining the trained keypoint detection model, the temperature parameter is calibrated by minimizing the negative log likelihood loss (NLL loss) function in the trained keypoint detection model. Minimizing the NLL loss function to calibrate the temperature parameter can be achieved by calculating the loss value between the output of the softmax function obtained by inputting a sample image into the trained keypoint detection model and the corresponding label of the sample image. The temperature parameter is then calibrated based on the relationship between this loss value and a preset loss value until the loss value between the output of the softmax function obtained by inputting a sample image into the trained keypoint detection model and the corresponding label of the sample image is less than the preset loss value. The preset loss value can be pre-set in the electronic device.

[0070] Furthermore, after calibrating the parameters to be calibrated, a calibrated keypoint detection model can be obtained based on the calibrated parameters.

[0071] For example, the temperature parameter in the softmax function of the trained keypoint detection model is identified as the parameter to be calibrated. By minimizing the negative log-likelihood loss function in the trained keypoint detection model, the loss value between the output of the softmax function of the trained keypoint detection model and the corresponding label of the sample image is calculated, thus obtaining the calibrated temperature parameter (T). l The process involves obtaining a calibrated keypoint detection model. This calibrated keypoint detection model can be based on the calibrated temperature parameter (T). l This involves calibrating the probabilities of keypoint locations in the heatmap output by the trained keypoint detection model. For example, a calibrated keypoint detection model can be based on the calibration formula: Obtaining the probability values ​​of keypoint locations in the calibrated heatmap, i.e., when the calibrated keypoint detection model predicts the locations of keypoints in the image, can directly output the calibrated probability values ​​of keypoints at their corresponding locations. This avoids the model's output probability value predicting the location of keypoints in the image deviating from the actual probability of the keypoint at that location by more than a preset deviation value. The preset deviation value can be pre-set in the electronic device.

[0072] Where l represents the key point l in the heatmap output by the calibrated key point detection model; σ l(i, j) represents the probability value obtained after calibrating the probability of key point l at position (i, j) in the heatmap output by the trained key point detection model; e represents the natural constant, with a value of approximately 2.7; c l (i, j) represents the conversion of the brightness value of the key point l in the heatmap at the corresponding position (i, j) into a probability value in the range [0~1]; T l This represents the parameter to be calibrated after calibration; s=1 represents the position where the width of the heatmap is 1, w represents the width of the heatmap; t=1 represents the position where the height of the heatmap is 1, h represents the height of the heatmap; c l (s, t) represents the conversion of the brightness value of the key point l in the heat map at the corresponding position (s, t) into a probability value in the range of [0~1].

[0073] In some implementations, a wider range of model calibration methods can be used during the calibration of the trained keypoint detection model, such as sigmoid calibration and order-preserving regression, and are not limited to the use of the temperature scaling parameters provided in the embodiments of this application.

[0074] Step S220: Obtain the average brightness of the target heatmap and the coordinate clustering degree of the target heatmap.

[0075] It is understandable that in algorithms based on heatmap analysis of key points, when the model is uncertain about the key point detection result, such as when the key point has not appeared in the model's training set or when there are occluded key points, using the coordinates of the brightest point in the heatmap as the detection result will result in a large error. In this embodiment, after obtaining the calibrated key point detection model, considering that if the brightness value of the heatmap output by the calibrated key point detection model for a certain key point in the target image is low, or the shape of the heatmap is scattered, that is, the brightest point is scattered in various positions of the heatmap, it indicates that the calibrated key point detection model has high uncertainty in predicting the key point. In this case, directly taking the brightest point in the heatmap as the prediction result for the corresponding position of the key point will often result in a large error. In this embodiment, the confidence of the calibrated key point detection model in detecting key points in the target image can be obtained by analyzing and calculating the average brightness and the degree of coordinate clustering of the heatmap.

[0076] In some implementations, please refer to Figure 7 The electronic device may acquire the average brightness of the target heat map in steps S221-S222.

[0077] Step S221: Obtain the pixel values ​​of the target heatmap and the number of pixels in the target heatmap.

[0078] In some implementations, after obtaining the target heatmap, the electronic device can acquire the pixel values ​​of the target heatmap and the number of pixels in the target heatmap. In some implementations, the pixel values ​​of the target heatmap can be the target heatmap output by the electronic device based on a calibrated keypoint detection model, converting the brightness values ​​of the target heatmap into probability values ​​within the range of [0 to 1].

[0079] Step S222: Obtain the average brightness of the target heatmap based on the pixel value and the number of pixels.

[0080] In some implementations, after obtaining the pixel values ​​and number of pixels of the target heatmap, the electronic device can calculate the average brightness based on the following formula: Obtain the average brightness of the target heatmap. Here, mean_intensity represents the average brightness of the target heatmap, n represents the number of pixels in the target heatmap, and intensity... i This represents the pixel value at position i in the target heatmap.

[0081] In some implementations, please refer to Figure 8 The degree of coordinate aggregation of the target heat map acquired by the electronic device may include steps S223-S224.

[0082] Step S223: Obtain the distance from the position of each pixel in the target heatmap to the center point of the target heatmap.

[0083] In some implementations, after obtaining the target heatmap, the electronic device can acquire the distance from the position of each pixel in the target heatmap to the center point (centroid) of the target heatmap.

[0084] Step S224: Obtain the coordinate clustering degree of the target heatmap based on the distance.

[0085] In some implementations, after the electronic device obtains the distance from the position of each pixel in the target heatmap to the center point of the target heatmap, it can determine the average, median, arithmetic mean, mode, etc. of the distance from the position of each pixel to the center point of the target heatmap as the degree of coordinate clustering of the target heatmap.

[0086] As one implementation method, the coordinate clustering degree of the target heatmap can be obtained based on the distance from the position of each pixel in the target heatmap to the center point of the target heatmap. This can be achieved by obtaining the average distance from the position of each pixel in the target heatmap to the center point of the target heatmap, and determining the average distance as the coordinate clustering degree of the target heatmap.

[0087] Optionally, before calculating the average distance from the position of each pixel in the target heatmap to the center point of the target heatmap, the electronic device can establish a coordinate system based on the target heatmap, and normalize the coordinate values ​​of each pixel in the target heatmap by dividing them by the height and width of the target image, so that the position coordinate values ​​of each pixel in the target heatmap are within the range of 0-1. Then, based on the coordinate clustering formula: Obtain the average distance from the position of each pixel in the target heatmap to the center point of the target heatmap, and determine this average distance as the coordinate clustering degree of the target heatmap. Wherein, (hx i hy i (cx, cy) represents the coordinates of pixel i in the target heatmap, (cx, cy) represents the coordinates of the center point (centroid) of the target heatmap, n represents the number of pixels in the target heatmap, and cluster_index represents the degree of coordinate clustering in the target heatmap.

[0088] Step S230: Based on the average brightness and / or the degree of coordinate clustering, obtain the confidence level of the calibrated key point detection model for the detection of the target image.

[0089] In some implementations, it is considered that the higher the average brightness value of the target heatmap, the higher the confidence level of the calibrated keypoint detection model in detecting the target image, and vice versa. In the embodiments of this application, after the electronic device obtains the average brightness of the target heatmap, it can determine the average brightness of the target heatmap as the confidence level of the calibrated keypoint detection model in detecting the target image.

[0090] In some implementations, considering that the smaller the coordinate clustering of the target heatmap, the more dispersed the predicted keypoints in the target heatmap, and the lower the confidence of the calibrated keypoint detection model in detecting the target image, and vice versa, the confidence is higher. In the embodiments of this application, after the electronic device obtains the coordinate clustering of the target heatmap, it can determine the coordinate clustering of the target heatmap as the confidence of the calibrated keypoint detection model in detecting the target image.

[0091] In some implementations, after the electronic device obtains the average brightness of the target heatmap and the degree of coordinate clustering of the target heatmap, it can comprehensively analyze the confidence of the calibrated key point detection model in the target image detection from the two perspectives of brightness and coordinate clustering, so as to improve the accuracy of the confidence measurement of the calibrated key point detection model in the target image detection.

[0092] In one implementation, the electronic device can obtain the confidence level of the calibrated keypoint detection model for target image detection based on the average brightness of the target thermal image and the coordinate clustering degree of the target thermal image. Specifically, the electronic device can calculate the quotient of the average brightness divided by the coordinate clustering degree, and determine this quotient as the confidence level of the calibrated keypoint detection model for target image detection. For example, if the average brightness is... Coordinate clustering degree The quotient of average brightness divided by coordinate clustering The average brightness divided by the degree of coordinate clustering is determined as the confidence level of the calibrated keypoint detection model in detecting the target image.

[0093] For example, please refer to Figure 9 This demonstrates the results of a calibrated keypoint detection model for target image detection. After detecting keypoints in the target image, the calibrated keypoint detection model outputs a target heatmap of the corresponding keypoint locations. The electronic device can further obtain the confidence level (mean intensity divided by coordinate clustering, confidence: 1.75) of the calibrated keypoint detection model based on the average brightness (mean_intensity: 0.7) and coordinate clustering degree (cluster_index: 0.4) of the target heatmap.

[0094] For example, please refer to Figure 10 This document illustrates a flowchart of an embodiment of the confidence measurement process of a calibrated keypoint detection model for target image detection using an electronic device. The electronic device trains an initial model based on images from a sample image set to obtain a trained keypoint detection model (i.e., keypoint detection model training). The trained keypoint detection model is then calibrated to obtain a calibrated keypoint detection model (i.e., model calibration). The target image is then input into the calibrated keypoint detection model to obtain a target heatmap output by the calibrated keypoint detection model corresponding to the target image. Based on this target heatmap, the confidence of the calibrated keypoint detection model in detecting the target image is measured, and thus the confidence of the calibrated keypoint detection model in detecting the target image is obtained.

[0095] Step S240: If the confidence level is less than the first confidence level threshold, then delete the target heatmap.

[0096] In some implementations, the electronic device may have a pre-set first confidence threshold. After obtaining the confidence level of the calibrated keypoint detection model for target image detection, the electronic device can compare this confidence level with the first confidence threshold. The relationship between the confidence level of the calibrated keypoint detection model for target image detection and the first confidence threshold can be used as a criterion for determining whether to delete the target heatmap. For example, if the confidence level is less than the first confidence threshold, the target heatmap is deleted.

[0097] In some implementations, the electronic device may have a pre-set second confidence threshold. After obtaining the confidence level of the calibrated keypoint detection model's detection of the target image, the electronic device can compare this confidence level with the second confidence threshold. The relationship between the confidence level of the calibrated keypoint detection model's detection of the target image and the second confidence threshold can serve as a criterion for determining whether to generate target prompt information. For example, if the confidence level is less than the second confidence threshold, target prompt information is generated. This target prompt information indicates that the calibrated keypoint detection model's detection of the target image is unsatisfactory, enabling the user or downstream tasks of the electronic device to determine whether to receive the target heatmap based on the target prompt information, and subsequently perform image information extraction based on the target heatmap, such as face recognition or identity verification.

[0098] In this context, a calibrated keypoint detection model's failure to detect a target image can be understood as the electronic device being unable to extract the necessary information from the target heatmap output by the model. For example, if an electronic device aims to obtain user identity information from a facial image and then control a door, and the confidence level of the calibrated keypoint detection model's detection of the target image is less than the access control confidence threshold, a target notification message is generated. This message indicates that the calibrated keypoint detection model's detection of the target image is unsuccessful, meaning the electronic device cannot identify the user's identity information from the target heatmap. The electronic device can then delete the target heatmap. Similarly, if an electronic device aims to obtain user identity information from a facial image and then make a payment, and the confidence level of the calibrated keypoint detection model's detection of the target image is less than the payment confidence threshold, a target notification message is generated. This message indicates that the calibrated keypoint detection model's detection of the target image is unsuccessful, meaning the electronic device cannot identify the user's identity information from the target heatmap. The electronic device can then delete the target heatmap. Among them, the access control confidence threshold and the payment confidence threshold can be understood as the second confidence threshold. They can be the same or different, and no restriction is made here.

[0099] The image processing method provided in one embodiment of this application is compared to... Figure 1The method shown also deletes the target heatmap if the confidence level of the calibrated keypoint detection model for target image detection is less than a first confidence threshold. Compared to Figure 1 The image processing method shown in this embodiment measures the confidence level of the model's predicted key point locations based on heatmaps. The confidence level of the model's predicted key point locations is obtained through the brightness and coordinate point clustering of the heatmap. Before analyzing the heatmap, the trained key point detection model that outputs the heatmap is calibrated to reduce errors caused by the model's overconfidence in predicting key point locations. By calculating the confidence level of the model's predicted key point locations, the user or downstream tasks are advised to use the heatmap with caution, thereby improving the reliability of image processing and the user experience.

[0100] Please see Figure 11 , Figure 11 A block diagram of an image processing apparatus according to an embodiment of this application is shown. This image processing apparatus 200 is applied to the aforementioned electronic device, and will be discussed below. Figure 11 The process is described in detail below. The image processing device 200 includes: a heatmap acquisition module 210, an average brightness and coordinate clustering acquisition module 220, a confidence acquisition module 230, and an image processing module 240, wherein:

[0101] The heatmap acquisition module 210 is used to input the target image into the calibrated key point detection model and obtain the target heatmap corresponding to the target image output by the calibrated key point detection model, wherein the target heatmap is used to characterize the position of key points in the target image.

[0102] The average brightness and coordinate clustering degree acquisition module 220 is used to acquire the average brightness of the target heat map and the coordinate clustering degree of the target heat map.

[0103] The confidence acquisition module 230 is used to obtain the confidence of the calibrated key point detection model for the detection of the target image based on the average brightness and / or the degree of coordinate clustering.

[0104] Image processing module 240 is used to process the target heatmap based on the confidence level.

[0105] Further, the average brightness and coordinate clustering acquisition module 220 includes: a pixel acquisition module and an average brightness acquisition module, wherein:

[0106] The pixel acquisition module is used to acquire the pixel values ​​of the target heatmap and the number of pixels in the target heatmap.

[0107] The average brightness acquisition module is used to obtain the average brightness of the target heatmap based on the pixel value and the number of pixels.

[0108] Furthermore, the average brightness and coordinate clustering acquisition module 220 includes: a distance acquisition module and a coordinate clustering acquisition module, wherein:

[0109] The distance acquisition module is used to obtain the distance from the position of each pixel in the target heatmap to the center point of the target heatmap.

[0110] The coordinate clustering degree acquisition module is used to obtain the coordinate clustering degree of the target heatmap based on the distance.

[0111] Furthermore, the coordinate clustering degree acquisition module includes: an average distance acquisition unit and a coordinate clustering degree acquisition unit, wherein:

[0112] The average distance acquisition unit is used to acquire the average distance from the position of each pixel in the target heatmap to the center point of the target heatmap based on the distance.

[0113] The coordinate clustering degree acquisition unit is used to determine the average distance as the coordinate clustering degree of the target heatmap.

[0114] Furthermore, the confidence level acquisition module 230 includes: a quotient calculation unit and a confidence level determination unit, wherein:

[0115] A quotient calculation unit is used to calculate the quotient of the average brightness divided by the degree of coordinate aggregation.

[0116] A confidence determination unit is used to determine the confidence of the calibrated key point detection model in detecting the target image by dividing the average brightness by the degree of coordinate clustering.

[0117] Further, the image processing module 240 includes: a first processing unit, wherein:

[0118] The first processing unit is configured to delete the target heatmap if the confidence level is less than a first confidence level threshold.

[0119] Further, the image processing module 240 includes: a second processing unit, wherein:

[0120] The second processing unit is configured to generate target prompt information if the confidence level is less than a second confidence level threshold, wherein the target prompt information is used to indicate that the calibrated key point detection model fails to detect the target image.

[0121] Furthermore, before inputting the target image into the calibrated keypoint detection model to obtain the target heatmap corresponding to the target image output by the calibrated keypoint detection model, the image processing device 200 further includes: a sample image acquisition module, a model training module, and a model calibration module, wherein:

[0122] A sample image acquisition module is used to acquire a sample image set, wherein the sample image set includes multiple sample images and labels corresponding to each of the multiple sample images, wherein the labels include heatmaps used to characterize the positions of key points in the sample images.

[0123] The model training module is used to train the initial model by taking the multiple sample images as input parameters and the labels corresponding to each of the multiple sample images as output parameters, and to obtain the trained key point detection model.

[0124] The model calibration module is used to calibrate the trained keypoint detection model to obtain the calibrated keypoint detection model.

[0125] Furthermore, the model calibration module includes: a parameter determination unit and a parameter calibration unit, wherein:

[0126] The parameter determination unit is used to determine the parameter to be calibrated from multiple parameters included in the trained keypoint detection model.

[0127] The parameter calibration unit is used to calibrate the parameters to be calibrated based on the output obtained by inputting the sample image into the trained keypoint detection model and the label corresponding to the sample image, so as to obtain the calibrated keypoint detection model.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0130] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0131] Please see Figure 12This document illustrates a structural block diagram of an electronic device 100 provided in an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0132] The processor 110 may include one or more processing cores. The processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and by calling data stored in the memory 120. Optionally, the processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 110 and may be implemented separately using a communication chip.

[0133] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0134] Please see Figure 13 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 300 stores program code that can be invoked by a processor to execute the methods described in the above method embodiments.

[0135] The computer-readable storage medium 300 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 310 may be compressed, for example, in a suitable form.

[0136] In summary, the image processing method, apparatus, electronic device, and storage medium provided in this application obtain a target heatmap corresponding to the target image by inputting a target image into a calibrated keypoint detection model. The target heatmap is used to characterize the positions of key points in the target image. The method involves obtaining the average brightness and coordinate clustering degree of the target heatmap; obtaining the confidence level of the calibrated keypoint detection model in detecting the target image based on the average brightness and / or coordinate clustering degree; and processing the target heatmap based on the confidence level. By using a calibrated keypoint detection model to perform keypoint detection on the image and outputting a heatmap, the accuracy of the model's keypoint detection is improved. Furthermore, by analyzing the confidence level of the model's keypoint detection based on the heatmap and then processing the heatmap according to the confidence level, the reliability of image processing is improved.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: The target image is input into a calibrated keypoint detection model to obtain a target heatmap output by the calibrated keypoint detection model corresponding to the target image, wherein the target heatmap is used to characterize the position of keypoints in the target image; The average brightness of the target heatmap is obtained, as well as the distance from the position of each pixel in the target heatmap to the center point of the target heatmap; Based on the distance, obtain the average distance from the position of each pixel in the target heatmap to the center point of the target heatmap; The average distance is determined as the degree of coordinate clustering of the target heatmap; Calculate the quotient of the average brightness divided by the degree of coordinate clustering; The quotient of the average brightness divided by the degree of coordinate clustering is determined as the confidence level of the calibrated key point detection model in detecting the target image; The target heatmap is processed based on the confidence level.

2. The method according to claim 1, characterized in that, The step of obtaining the average brightness of the target heatmap includes: Obtain the pixel values ​​of the target heatmap and the number of pixels in the target heatmap; The average brightness of the target heatmap is obtained based on the pixel value and the number of pixels.

3. The method according to claim 1, characterized in that, The processing of the target heatmap based on the confidence level includes: If the confidence level is less than the first confidence level threshold, the target heatmap is deleted.

4. The method according to claim 1, characterized in that, The processing of the target heatmap based on the confidence level includes: If the confidence level is less than the second confidence level threshold, a target prompt message is generated, wherein the target prompt message is used to indicate that the calibrated key point detection model fails to detect the target image.

5. The method according to any one of claims 1-4, characterized in that, Before inputting the target image into the calibrated keypoint detection model to obtain the target heatmap corresponding to the target image output by the calibrated keypoint detection model, the method further includes: Obtain a sample image set, wherein the sample image set includes multiple sample images and labels corresponding to each of the multiple sample images, wherein the labels include heatmaps used to characterize the positions of key points in the sample images; The initial model is trained by using the multiple sample images as input parameters and the labels corresponding to each of the multiple sample images as output parameters, thereby obtaining a trained key point detection model. The trained keypoint detection model is calibrated to obtain the calibrated keypoint detection model.

6. The method according to claim 5, characterized in that, The step of calibrating the trained keypoint detection model to obtain the calibrated keypoint detection model includes: The parameters to be calibrated are determined from the multiple parameters included in the trained keypoint detection model; Based on the output obtained by inputting the sample image into the trained keypoint detection model, and the label corresponding to the sample image, the parameters to be calibrated are calibrated to obtain the calibrated keypoint detection model.

7. An image processing apparatus, characterized in that, The device includes: A heatmap acquisition module is used to input a target image into a calibrated keypoint detection model and obtain a target heatmap corresponding to the target image output by the calibrated keypoint detection model, wherein the target heatmap is used to characterize the position of keypoints in the target image; An average brightness and coordinate clustering degree acquisition module is used to acquire the average brightness of the target heatmap, and to acquire the distance from the position of each pixel in the target heatmap to the center point of the target heatmap; based on the distance, to acquire the average distance from the position of each pixel in the target heatmap to the center point of the target heatmap; and to determine the average distance as the coordinate clustering degree of the target heatmap. The confidence acquisition module is used to calculate the quotient of the average brightness divided by the degree of coordinate clustering, and to determine the quotient of the average brightness divided by the degree of coordinate clustering as the confidence of the calibrated key point detection model for the detection of the target image. An image processing module is used to process the target heatmap based on the confidence level.

8. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-6.

Images