Image processing method and device and electronic equipment

By using a key point prediction model based on preset probability distribution and standardized flow, the key point position in the image is calculated using the probability distribution map, the problem of inefficiency in the existing technology is solved and more efficient key point positioning is achieved.

CN119090952APending Publication Date: 2024-12-06BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202310659452.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, the efficiency of determining image key points is low, mainly because the Gaussian thermal map requires a higher resolution, resulting in large calculations.

Method used

The key point prediction model based on the preset probability distribution and the error probability distribution determined by the standardized flow is used to calculate the key point positions through the probability distribution map to avoid generating high-resolution Gaussian heat maps.

Benefits of technology

Improve the accuracy and efficiency of key point positions, reduce the amount of calculation, and reduce the need for high-resolution image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090952A_ABST
    Figure CN119090952A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method and device and electronic equipment, and the method comprises the steps: obtaining an image which comprises an object, and enabling the object to be associated with a plurality of key points; the image is processed according to a key point prediction model, a probability distribution diagram corresponding to each key point is obtained, and the key point prediction model is obtained through training based on preset probability distribution and error probability distribution determined by a standardized flow; and determining the position of each key point in the image according to the probability distribution diagram corresponding to each key point. And the accuracy and efficiency of determining the key points are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] The electronic device can recognize the object in the image by detecting the key points in the image. For example, the electronic device can extract the key points in the image, determine the outline of the object based on the key points, and then recognize the object in the image based on the outline.

[0003] At present, electronic devices can determine key points in an image based on Gaussian heat maps. For example, electronic devices can process images based on a pre-trained model to obtain Gaussian heat maps corresponding to each key point in the image, and then determine the maximum point in the Gaussian heat map as the position estimation result of the key point. However, in the above method, the Gaussian heat map needs to have a high resolution, and the electronic device has a large amount of calculation to determine the key points, which leads to low efficiency in determining the key points. Summary of the invention

[0004] The present disclosure provides an image processing method, device and electronic device, which are used to solve the technical problem of low efficiency in determining key points in the prior art.

[0005] In a first aspect, the present disclosure provides an image processing method, the method comprising:

[0006] acquiring an image including an object, the object being associated with a plurality of key points;

[0007] Processing the image according to a key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is trained based on a preset probability distribution and an error probability distribution determined by a normalized flow;

[0008] According to the probability distribution graph corresponding to each key point, the position of each key point in the image is determined.

[0009] In a second aspect, the present disclosure provides an image processing device, the image processing device comprising an acquisition module, a processing module and a determination module, wherein:

[0010] The acquisition module is used to acquire an image including an object, wherein the object is associated with a plurality of key points;

[0011] The processing module is used to process the image according to the key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is obtained by training based on a preset probability distribution and an error probability distribution determined by a normalized flow;

[0012] The determination module is used to determine the position of each key point in the image according to the probability distribution graph corresponding to each key point.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image processing method as described in the first aspect and various possible aspects of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0017] The present disclosure provides an image processing method, device and electronic device, the electronic device can obtain an image including an object, wherein the object is associated with multiple key points, and the image is processed according to a key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is trained based on a preset probability distribution and an error probability distribution determined by a standardized flow, and the position of each key point in the image is determined according to the probability distribution map corresponding to each key point. In the above method, since the error probability distribution determined by the standardized flow can supplement the preset probability distribution, the key point prediction model can learn the complex distribution corresponding to the key point, and the accuracy of the probability distribution map corresponding to each key point output by the key point prediction model is high, which improves the accuracy of determining the position of the key point, and since the electronic device can accurately calculate the position coordinates of the key point based on the probability distribution map, there is no need to render a high-resolution Gaussian heat map, so the amount of calculation to determine the key point can be reduced, and the efficiency of determining the key point can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 2A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of key points provided in an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of the structure of a key point prediction model provided by an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of a probability distribution diagram provided by an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of a process for determining a probability distribution graph provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of a process for determining target coordinates of a key point provided by an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of an image processing method provided by an embodiment of the present disclosure;

[0027] Fig. 9 A schematic diagram of a training method for a key point prediction model provided in an embodiment of the present disclosure;

[0028] Fig.10 A schematic diagram of a key point prediction model provided by an embodiment of the present disclosure;

[0029] Fig.11 A schematic diagram of a training framework of a key point prediction model provided in an embodiment of the present disclosure;

[0030] Fig.12 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0031] Fig.13 A schematic diagram of the structure of another image processing device provided by an embodiment of the present disclosure;

[0032] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0035] Electronic device: is a device with wireless transceiver function. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The electronic device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a vehicle-mounted electronic device, a wireless terminal in self-driving, a wireless electronic device in remote medical care, a wireless electronic device in smart grid, a wireless electronic device in transportation safety, a wireless electronic device in smart city, a wireless electronic device in smart home, a wearable electronic device, etc. The electronic device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access electronic device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote electronic device, a mobile device, a UE electronic device, a wireless communication device, a UE agent or a UE device, etc. The electronic device can also be fixed or mobile.

[0036] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0037] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including: an electronic device. The display screen of the electronic device includes image A, wherein image A may include an object. The electronic device may perform key point extraction processing on image A to obtain image B. Image B may include an object, and image B may include multiple key points associated with the object. For example, after the electronic device performs key point extraction processing on image A, image B may include 14 key points, and the 14 key points may be the head key points, hand key points, leg key points, etc. of the object. In this way, the electronic device can accurately determine the object, the outline of the object, and other information in image B through the 14 key points.

[0038] In the related art, electronic devices can detect key points in an image and identify objects in the image through the key points. For example, electronic devices can detect multiple key points in a face image and then identify the location of the face in the face image. Currently, electronic devices can determine key points in an image based on a Gaussian heat map. For example, the electronic device can process the image based on a pre-trained model to obtain a Gaussian heat map corresponding to each key point in the image, and determine the maximum value point in the Gaussian heat map as the position estimation result of the key point. For example, Figure 1 In the embodiment shown, the electronic device can generate 14 Gaussian heat maps, and the position of the maximum point in each Gaussian heat map is different. In this way, 14 key points associated with the object can be determined through the 14 Gaussian heat maps. However, since the Gaussian heat map can directly display the position of the key point, the Gaussian heat map output by the pre-trained model needs to have a higher resolution, which makes the electronic device calculate more to determine the key point, which leads to lower efficiency in determining the key point.

[0039] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides an image processing method, an electronic device can obtain an image including an object, the object in the image is associated with multiple key points, the electronic device processes the image according to the key point prediction model, obtains multiple feature maps associated with the key points, and normalizes each feature map to obtain a probability distribution map corresponding to each key point, and determines the position of each key point in the image according to the probability distribution map corresponding to each key point. In the above method, since the key point prediction model is trained based on the error probability distribution determined by the preset probability distribution and the normalized flow, and the probability distribution corresponding to the key point is relatively complex, the key point prediction model can learn the accurate probability distribution map corresponding to the key point based on the error probability distribution during the training stage, thereby improving the accuracy of determining the key point position, and since the electronic device can calculate the position of the key point in the image based on the probability distribution map, there is no need to generate a Gaussian heat map with a higher resolution, thereby reducing the amount of calculation for determining the key point position and improving the efficiency of determining the key point.

[0040] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0041] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0042] S201: Acquire an image including an object.

[0043] The execution subject of the embodiment of the present disclosure may be an electronic device, or an image processing device disposed in the electronic device. The image processing device may be implemented based on software, or the image processing device may be implemented based on a combination of software and hardware, which is not limited in the embodiment of the present disclosure.

[0044] The image may include an object. For example, the image may include any object such as a user, a face, or a fruit, which is not limited in the embodiments of the present disclosure. Optionally, the electronic device may receive images sent by other devices. For example, the electronic device may receive an image sent by a server, which may include a face image. For example, the electronic device may receive a request for key point detection sent by a server, which may include an image to be detected.

[0045] It should be noted that the electronic device can acquire an image including an object according to any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0046] Among them, an object in an image may be associated with multiple key points. For example, an object in an image may have multiple key points. For example, a face image may have multiple key points, and a user image may also have multiple key points. It should be noted that in the actual application process, the image obtained by the electronic device is not marked with multiple key points. The electronic device can obtain multiple key points in the image by processing the image.

[0047] Optionally, different objects in the image may have different key points associated with the objects. For example, if the image includes a face image, the face image may be associated with 30 key points (e.g., top of head key points, ear key points, etc.), and if the image includes a user, the image may be associated with 63 key points (e.g., right shoulder key points, right leg key points, etc.).

[0048] Optionally, each part may be associated with multiple key points. For example, if the object in the image includes a user, the top of the user's head may be associated with 1 key point, the user's left arm may be associated with 2 key points, the user's right leg may be associated with 3 key points, and so on.

[0049] It should be noted that the multiple key points associated with the object can be predetermined key points. For example, for a face image, the electronic device can pre-set that the face image can include up to 30 key points, and can pre-set the position of the face indicated by each key point. For example, the first key point in the face image is the key point at the top of the face, and the 30th key point in the face image is the key point at the bottom of the face. For example, for an image including a human body, the electronic device can pre-set that the image including the human body can include up to 63 key points, and the outline of the human body can be determined by the 63 key points, and the electronic device can pre-set the position of the human body indicated by each key point. For example, the 20th key point in the image including the human body is the key point of the left hand, the 40th key point is the key point of the right hand, and so on, which is not limited in the embodiments of the present disclosure.

[0050] Next, combine Figure 3 , describing the key points associated with the object.

[0051] Figure 3 A schematic diagram of key points provided in the embodiment of the present disclosure. Figure 3 , including human body images and face images. Among them, the object in the human body image is the human body, and the object in the face image is the face. The human body image can include multiple key points associated with the human body, and the face image also includes multiple key points associated with the face. Figure 3 As can be seen from the embodiment shown, different objects in the image have different key points associated with them. Figure 3 As shown, the application scenario of the human body image can be human body contour detection, therefore, multiple key points in the human body image can be located at the edge of the human body in the image, and the application scenario of the face image can be face recognition, therefore, multiple key points in the face image can be located at positions with facial features in the image.

[0052] S202: Process the image according to the key point prediction model to obtain a probability distribution map corresponding to each key point.

[0053] Optionally, the key point prediction model can obtain key points in the image. For example, the key point prediction model can process the image and output the image after the key points are marked. Different key point prediction models are targeted at different objects. For example, key point prediction model A can process an image including a human body and obtain multiple key points in the image, and key point prediction model B can process an image including an animal and obtain multiple key points in the image.

[0054] For example, if the key point prediction model can process images including human bodies, then any image including human bodies is input to the key point prediction model, and the key point prediction model can output an image with 63 key points marked. For example, if the key point prediction model can process images including faces, then any image including faces is input to the key point prediction model, and the key point prediction model can output an image with 30 key points marked.

[0055] It should be noted that if the key point prediction model can process images including human bodies, then the key point prediction model can also process images including human faces and limbs. For example, the key point prediction model can determine 63 key points in an image including human bodies, where if the human face can include 20 key points, then when the key point prediction model processes a facial image, it can annotate the facial image including 20 key points.

[0056] Optionally, the key point prediction model is trained based on a preset probability distribution and an error probability distribution determined by a standardized flow. The preset probability distribution may be a Gaussian distribution or a Laplace distribution. For example, during the training phase of the key point prediction model, the key point prediction model may learn the Gaussian distribution or the Laplace distribution corresponding to the key point. Since the actual distribution of the key point is relatively complex, an error probability distribution may be determined by the standardized flow, and the Gaussian distribution or the Laplace distribution may be supplemented based on the error probability distribution, so that the key point prediction model may fit a real distribution with a high degree of match with the key point.

[0057] Next, combine Figure 4 , the structure of the key point prediction model is explained.

[0058] Figure 4 This is a schematic diagram of the structure of a key point prediction model provided by an embodiment of the present disclosure. Figure 4 , including: a key point prediction model. Among them, the key point prediction model may include a convolutional neural network (CNN), a 1*1 convolutional layer (1*1conv) and a normalization layer (softmax layer). Among them, the convolutional neural network CNN can downsample the input image to obtain multiple feature maps. If the number of feature maps output by the convolutional neural network CNN (that is, the number of channels of the feature map) is greater than the number of key points associated with the objects in the image, the number of feature maps can be made the same as the number of key points based on the 1*1 convolutional layer. The normalization layer can process the feature map, and then the probability distribution map associated with each feature map can be obtained.

[0059] Optionally, the probability distribution graph can be used to indicate the distribution probability of key points in the image. For example, multiple key points associated with an object are distributed at different positions in the image. Therefore, for any key point in the image, the closer a certain position in the image is to the actual position of the key point, the higher the probability value of the position is, and the farther a certain position in the image is from the actual position of the key point, the lower the probability value of the position is.

[0060] Next, combine Figure 5 , explain the probability distribution diagram.

[0061] Figure 5 A schematic diagram of a probability distribution diagram provided by an embodiment of the present disclosure. Figure 5 , including: probability distribution graph. The probability distribution graph may include 25 points, each of which may correspond to a probability value, which may be the probability of the key point appearing at the position. Figure 5 In the embodiment shown, the sum of the probability values ​​corresponding to the 25 points is 1, wherein the closer the point is to the key point position, the higher the probability value is, and the farther the point is from the key point position, the lower the probability value is. Figure 5 It can be seen from the illustrated embodiment that the position of the key point corresponding to the probability distribution graph is close to the position of the coordinate (4, 3).

[0062] The electronic device may obtain the probability distribution map corresponding to each key point according to the following feasible implementation method: the image is processed according to the key point prediction model to obtain multiple feature maps associated with the key point, and each feature map is normalized to obtain the probability distribution map corresponding to each key point. For example, the electronic device performs downsampling processing on the image based on the CNN network in the key point prediction model, and the CNN network can output multiple feature maps corresponding to the image.

[0063] It should be noted that the key point prediction model is a pre-trained model, and the number of key points and the types of key points (such as head key points, hand key points or leg key points, etc.) predicted by the key point prediction model are also determined during pre-training.

[0064] Next, combine Figure 6 , the process of determining the probability distribution map corresponding to each key point is explained.

[0065] Figure 6 A schematic diagram of a process for determining a probability distribution graph provided by an embodiment of the present disclosure. Figure 6 , including a key point prediction model and an image. The key point prediction model may include a convolutional neural network, a 1*1 convolutional layer, and a normalization layer ( Figure 6The image size may be 192*144, and the downsampling multiple of the convolutional neural network in the key point prediction model is 16 times.

[0066] See also Figure 6 , electronic equipment ( Figure 6 (not shown) The image can be input into the key point prediction model. After the convolutional neural network in the key point prediction model processes the image, multiple feature maps can be obtained, wherein the size of the feature map is 12*9. After multiple feature maps are input into the 1*1 convolutional layer, 63 feature maps can be obtained, wherein the number of key points can be 63.

[0067] See also Figure 6 , each feature map is normalized through the normalization layer to obtain the probability distribution Figure 1 , probability distribution Figure 2 , ..., probability distribution graph 63. Each probability distribution graph may correspond to a probability distribution of a key point. Since the key point prediction model can learn a complex distribution that is closer to the true probability distribution of the key point during the training phase, the probability distribution graph of the key point obtained based on the key point prediction model has a higher accuracy.

[0068] S203: Determine the position of each key point in the image according to the probability distribution graph corresponding to each key point.

[0069] Among them, the electronic device can determine the position of each key point in the image according to the following feasible implementation method: for any key point, obtain the first coordinate of the point corresponding to each probability value in the probability distribution map corresponding to the key point, and determine the position of the key point in the image according to each probability value and the first coordinate corresponding to each probability value.

[0070] The probability distribution graph may include multiple points corresponding to probability values, and the first coordinate may be the coordinate of the point in the probability distribution graph. Figure 5 In the illustrated embodiment, the probability distribution graph includes 25 probability values, and the probability distribution graph may include 25 points. Therefore, the horizontal coordinates of the points in the probability distribution graph may range from 1 to 5, and the vertical coordinates may range from 1 to 5. Figure 5 In the illustrated embodiment, if the lower left corner of the probability distribution graph is the coordinate origin, the first coordinate corresponding to the point with a probability value of 0.6 may be (4, 3), and the first coordinate corresponding to the point with a probability value of 0.1 may be (4, 2), (3, 3), (5, 3) and (4, 4) respectively.

[0071] Optionally, the electronic device may obtain the first coordinates of the point corresponding to each pixel value in the probability distribution diagram, and the electronic device may also determine the first coordinates according to any other feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0072] Among them, the position of the key point in the image is determined according to each probability value and the first coordinate corresponding to each probability value. Specifically, each probability value is multiplied by the first coordinate corresponding to each probability value to obtain the second coordinate corresponding to each probability value, multiple second coordinates are added to obtain the target coordinates of the key point, and the position of the key point in the image is determined based on the target coordinates.

[0073] The second coordinate corresponding to the probability value may be the coordinate obtained by multiplying the probability value by the first coordinate. Figure 5 In the illustrated embodiment, taking the probability value of 0.6 as an example, the first coordinate corresponding to the probability value is (4, 3). Therefore, after multiplying the first coordinate by the probability value, the coordinate (2.4, 1.8) can be obtained. The electronic device can determine the coordinate (2.4, 1.8) as the second coordinate corresponding to the probability value. Taking the probability value of 0.1 as an example, the first coordinates corresponding to the four probability values ​​of 0.1 can be (4, 2), (3, 3), (5, 3) and (4, 4), respectively. Therefore, the second coordinates corresponding to the four probability values ​​of 0.1 can be (0.4, 0.2), (0.3, 0.3), (0.5, 0.3) and (0.4, 0.4), respectively.

[0074] Optionally, the target coordinate of the key point may be the sum of multiple second coordinates corresponding to the key point. For example, the electronic device may add the horizontal coordinates of the multiple second coordinates corresponding to the key point to obtain the horizontal coordinate of the target coordinate corresponding to the key point, and add the vertical coordinates of the multiple second coordinates corresponding to the key point to obtain the vertical coordinate of the target coordinate corresponding to the key point.

[0075] Next, combine Figure 7 , the process of determining the target coordinates of the key points is explained.

[0076] Figure 7 A schematic diagram of a process for determining target coordinates of key points provided by an embodiment of the present disclosure. Figure 7 , including: a probability distribution diagram corresponding to a key point. The probability distribution diagram includes 16 probability values, each probability value corresponds to a point in the probability distribution diagram. Electronic device ( Figure 7(not shown) can determine the second coordinate based on each probability value and the first coordinate of the point corresponding to the probability value, where the multiple second coordinates may include (0.2, 0.2), (0.09, 0.09), (0.4, 0.3), (0.06, 0.06), (0.3, 0.2), (2.4, 1.2), (0.06, 0), and (0.28, 0).

[0077] See also Figure 7 , the electronic device can add the abscissas of the multiple second coordinates to get 3.79, and add the ordinates of the multiple second coordinates to get 2.05, and then determine that the target coordinates corresponding to the key point are (3.79, 2.05). In this way, the electronic device can determine the accurate coordinates corresponding to each key point, and because the decimal point of the coordinates can be retained for multiple digits (such as the probability value in the probability distribution diagram can be retained for multiple digits after the decimal point), the electronic device can obtain more accurate target coordinates, thereby improving the accuracy of determining the position of the key point.

[0078] Optionally, the electronic device determines the position of the key point in the image based on the target coordinates, specifically by obtaining a downsampling multiple when the key point prediction model processes the image, and determining the position of the key point in the image based on the downsampling multiple and the target coordinates.

[0079] The downsampling multiple may be a downsampling multiple of a convolutional neural network in the key point prediction model. For example, the CNN network in the key point prediction model may downsample an image. If the downsampling multiple of the CNN network is 16 (4*4) times and the image size is 192*144, the size of the downsampled image is 12*9.

[0080] Optionally, the electronic device may determine the position of the key point in the image based on the target coordinates and the downsampling multiple. For example, if the target coordinates corresponding to the key point are (1, 1) and the downsampling multiple is 16 (4*4), then the position coordinates of the key point in the image may be (4, 4), that is, the horizontal coordinate of the target coordinates is multiplied by 4, and the vertical coordinate of the target coordinates is multiplied by 4; if the coordinates corresponding to the key point are (2, 3) and the downsampling multiple is 25 (5*5), then the position coordinates of the key point in the image may be (10, 15), that is, the horizontal coordinate of the target coordinates is multiplied by 5, and the vertical coordinate of the target coordinates is multiplied by 5.

[0081] The disclosed embodiment provides an image processing method, in which an electronic device can acquire an image including an object, and the electronic device processes the image according to a key point prediction model to obtain a plurality of feature maps associated with the key points, and normalizes each feature map to obtain a probability distribution map corresponding to each key point, and determines the position of each key point in the image according to the probability distribution map corresponding to each key point. In the above method, since the key point prediction model is obtained by training based on a preset probability distribution and an error probability distribution determined by a normalized flow, the key point prediction model can learn the complex distribution associated with the key point, thereby improving the accuracy of the probability distribution map corresponding to the key point, and improving the accuracy of determining the position of the key point, and since the electronic device can calculate the position of the key point in the image based on the probability distribution map, there is no need to generate a Gaussian heat map with a higher resolution, thereby reducing the amount of calculation for determining the position of the key point and improving the efficiency of determining the key point.

[0082] exist Figure 2 Based on the embodiment shown below, combined with Figure 8 , the process of the above-mentioned image processing method is explained.

[0083] Figure 8 A schematic diagram of an image processing method provided by an embodiment of the present disclosure. Figure 8 , including: including image A and a key point prediction model, image A can be a human body image. Electronic device ( Figure 8 (not shown) can input image A to the key point prediction model, the CNN network in the key point prediction model can downsample image A by 4*4 times, and the key point prediction model can output a probability distribution Figure 1 , probability distribution Figure 2 , ..., probability distribution Fig.14 , where 14 probability distribution graphs correspond one-to-one to the key points.

[0084] See also Figure 8 , the electronic device can determine the second coordinate corresponding to each probability value of the probability distribution graph, and determine the target coordinate associated with the key point based on the multiple second coordinates. Figure 1 The target coordinate 1 corresponding to the key point 1 can be obtained, and the electronic device is distributed according to the probability Figure 2 The target coordinate 2 corresponding to the key point 2 can be obtained, etc. The electronic device can multiply the horizontal coordinate and the vertical coordinate of each target coordinate by 4 respectively, and 14 positions can be obtained.

[0085] See also Figure 8 The electronic device can determine the 14 positions in image A and mark key points at each position, thereby obtaining image B, wherein image B can include a human body and 14 key points associated with the human body.

[0086] In this way, the electronic device can accurately determine the probability distribution map associated with each key point based on the key point prediction model, and since the target coordinates are determined based on all parameters in the probability distribution map (probability values, coordinates of probability values), the accuracy of the target coordinates is relatively high, which can in turn improve the accuracy of determining the key points. Moreover, since the key points are determined based on the position coordinates, the key points have a high accuracy and there is no need to render a Gaussian heat map, which reduces the computational complexity and improves the efficiency of determining the key points.

[0087] Based on any of the above embodiments, the present disclosure also includes a training method for a key point prediction model. Fig. 9 , the training method of the above key point prediction model is explained.

[0088] Fig. 9 A schematic diagram of a training method for a key point prediction model provided in an embodiment of the present disclosure. Fig. 9 The method flow includes:

[0089] S901: Obtain a sample image.

[0090] The sample image may include a sample object. For example, the sample image may include a human body image, a human face image, etc. It should be noted that during the key point prediction model training process, the sample object in the sample image is the same as the object in the image during the key point prediction model use process. For example, if the key point prediction model can detect key points in a human face image, then during the key point prediction model training process, the sample object in the sample image may be a human face; if the key point prediction model can detect key points in a human body, then during the key point prediction model training process, the sample object in the sample image may be a human body.

[0091] It should be noted that the electronic device can obtain the sample image according to any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0092] S902: Obtain the sample position of each sample key point in the sample image.

[0093] Optionally, the sample key points may be key points pre-set for the sample object. For example, if the sample object is a face, the sample object may be associated with 30 key points, and if the sample object is a human body image, the sample object may be associated with 63 key points.

[0094] It should be noted that the sample key points associated with the sample objects in the sample image may be pre-set key points or may be sample key points determined by the electronic device according to any other feasible implementation method. The embodiments of the present disclosure are not limited to this. Furthermore, the embodiments of the present disclosure do not limit information such as the number of key points associated with the sample objects, the locations of the key points, and the types of the key points.

[0095] Optionally, the sample position may be the position of the sample key point in the sample image. For example, when making a sample image for training, the user may accurately mark each sample key point in the sample image, and then determine the sample position of the sample key point in the sample image. The sample position may be represented based on coordinates (e.g., a coordinate system in pixels), or may be represented based on any other feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0096] S903: Update the model parameters of the key point prediction model according to the sample image and the sample position.

[0097] Specifically, the electronic device can update the model parameters of the key point prediction model according to the following feasible implementation method: process the sample image according to the key point prediction model to obtain the predicted position corresponding to each sample key point and the variance associated with the predicted position, determine the error parameter according to the predicted position, variance and sample position corresponding to each sample key point, and update the model parameters of the key point prediction model according to the error parameter.

[0098] Optionally, the predicted position may be the position of the sample key point predicted by the key point prediction model based on the sample image. For example, the electronic device may process the sample image to obtain a sample probability distribution map associated with each sample key point, and determine the position of each sample key point in the sample image based on the sample probability distribution map. It should be noted that the method for the electronic device to determine the predicted position of each sample key point is similar to Figure 2 The method for the electronic device in the illustrated embodiment to determine the position of each key point in the image is the same, and the embodiment of the present disclosure will not be described in detail here.

[0099] The variance associated with the predicted position may be the variance of the predicted distribution of the key points. For example, the variance associated with the predicted position may indicate the degree of dispersion of the predicted value.

[0100] Optionally, the electronic device may process the sample image based on the key point prediction model, and then obtain the variance associated with the predicted position. For example, the key point prediction model may set a branch for training variance during the training phase, and the structure of the branch may include an adaptive average pooling layer and a fully connected layer, so that the variance associated with the predicted position can be obtained based on the branch.

[0101] Next, combine Fig.10 , the structure of the key point prediction model in the training phase is explained.

[0102] Fig.10 A schematic diagram of a key point prediction model provided by an embodiment of the present disclosure. Fig.10 , including a key point prediction model. The key point prediction model includes a convolutional neural network CNN, and two branches can be connected after the convolutional neural network, one of which includes a 1*1 convolution layer and a normalization layer, and the branch can output a probability distribution map associated with each key point, and the other branch includes an adaptive average pooling layer and a fully connected layer, and the branch can output the variance associated with the predicted position of each key point.

[0103] For example, Fig.10 In the illustrated embodiment, after the convolutional neural network in the key point prediction model processes the image, multiple feature maps can be obtained. After passing through the adaptive average pooling layer and the fully connected layer, the output result can be transformed into a three-dimensional vector of (N, K, 2), wherein the three-dimensional vector can be the variance of the key point prediction distribution, N represents the batch size (data size) of the convolutional neural network, K represents the number of key points, and 2 can be expressed as the variance of the x-coordinate and the variance of the y-coordinate.

[0104] Optionally, the error parameter may be a variable parameter in the key point prediction model training process. Optionally, the error parameter may be determined based on the predicted position, variance and sample position corresponding to the sample key point, specifically, the position difference between the sample position and the predicted position may be determined, and the ratio of the position difference to the variance may be determined as the error parameter.

[0105] For example, the electronic device may determine the error parameter based on the following formula:

[0106]

[0107] Among them, A can be an error parameter, B can be a sample position, C can be a predicted position, and σ can be a variance. In this way, the key point prediction model can update the parameters in the key point prediction model through the error parameter, avoid the dispersion of training samples, and improve the accuracy of key point prediction model training.

[0108] Among them, the electronic device updates the model parameters of the key point prediction model according to the error parameters, which can be specifically: processing the sample image according to the key point prediction model to obtain the preset probability distribution corresponding to the sample key points, transforming the preset probability distribution according to the normalized flow model to obtain the error probability distribution corresponding to the sample key points, constructing a loss function according to the preset probability distribution, error probability distribution, error parameters and variance, and updating the model parameters of the key point prediction model according to the loss function.

[0109] The preset probability distribution may be a simple distribution such as a Gaussian distribution or a Laplace distribution, which is not limited in the embodiments of the present disclosure.

[0110] Among them, the standardized flow model can perform a reversible transformation on the probability distribution. For example, the standardized flow model can have the function of standardized flow, which can transform any complex distribution into a simple distribution (e.g., transform a complex distribution into a Gaussian distribution), and can also transform any simple distribution into a complex distribution, and the transformation process of the probability distribution by the standardized flow is reversible.

[0111] Optionally, during the training of the key point prediction model, a standardized flow model can be introduced, wherein the key point prediction model is used to learn a preset probability distribution corresponding to the key point, which has a difference with the true distribution corresponding to the key point, and the standardized flow model is used to learn an error probability distribution, which is a supplement to the preset probability distribution, so that the preset probability distribution is close to the true distribution of the key point.

[0112] Next, combine Fig.11 , the training framework of the key point prediction model is explained.

[0113] Fig.11 A schematic diagram of a training framework for a key point prediction model provided in an embodiment of the present disclosure. Fig.11 , including: sample image, key point prediction model and standardized flow model. The sample image is processed according to the key point prediction model to obtain the predicted position and variance corresponding to the key point. The preset probability distribution is transformed according to the standardized flow model to obtain the error probability distribution, and the probability distribution corresponding to the key point is obtained by predicting the position, variance and error probability distribution. For example, the predicted position is μ, the variance is σ, and the error probability distribution fitted by the standardized flow model is Then the probability distribution corresponding to the key point can be:

[0114] Optionally, a loss function is constructed according to a preset probability distribution, an error probability distribution, an error parameter and a variance, which may be the following formula:

[0115] L=-logQ(A)-logG θ (A)-log(s)+logσ

[0116] Wherein, L may be a loss function, A may be an error parameter, Q may be a preset probability distribution (such as Gaussian distribution or Laplace distribution, etc.), G may be a normalized flow, s may be a constant (s is used for correction and may be set arbitrarily, and the embodiments of the present disclosure do not limit this), and σ may be a variance. The loss function may be determined according to the above formula, and the model parameters may be updated according to the loss function.

[0117] The disclosed embodiment provides a training method for a key point prediction model, which includes obtaining a sample image, obtaining a sample position of each sample key point in the sample image, processing the sample image according to the key point prediction model, obtaining a predicted position corresponding to each sample key point and a variance associated with the predicted position, determining an error parameter according to the predicted position, variance and sample position corresponding to each sample key point, and updating the model parameters of the key point prediction model according to the error parameter. In this way, the key point prediction model can learn the complex distribution associated with the key points, so the training accuracy of the key point prediction model is high, improving the accuracy of the probability distribution map corresponding to the key points generated by the key point prediction model, and further improving the accuracy of determining the key point positions.

[0118] Fig.12 This is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. Fig.12 The image processing device 120 includes an acquisition module 121, a processing module 122 and a determination module 123, wherein:

[0119] The acquisition module 121 is used to acquire an image including an object, wherein the object is associated with a plurality of key points;

[0120] The processing module 122 is used to process the image according to the key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is obtained by training based on a preset probability distribution and an error probability distribution determined by a standardized flow;

[0121] The determination module 123 is used to determine the position of each key point in the image according to the probability distribution graph corresponding to each key point.

[0122] According to one or more embodiments of the present disclosure, the processing module 122 is specifically used for:

[0123] Processing the image according to the key point prediction model to obtain a plurality of feature maps associated with the key points;

[0124] Each feature map is normalized to obtain a probability distribution map corresponding to each key point.

[0125] According to one or more embodiments of the present disclosure, the determining module 123 is specifically configured to:

[0126] For any key point, obtain the first coordinate of the point corresponding to each probability value in the probability distribution map corresponding to the key point in the probability distribution map;

[0127] The position of the key point in the image is determined according to each probability value and the first coordinate corresponding to each probability value.

[0128] According to one or more embodiments of the present disclosure, the determining module 123 is specifically configured to:

[0129] Multiply each probability value by the first coordinate corresponding to each probability value to obtain a second coordinate corresponding to each probability value;

[0130] Adding the multiple second coordinates to obtain the target coordinates of the key point;

[0131] Based on the target coordinates, the position of the key point in the image is determined.

[0132] The image processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0133] Fig.13 A schematic diagram of the structure of another image processing device provided by an embodiment of the present disclosure. Fig.12 Based on the examples shown, see Fig.13 The image processing device 120 further includes a training module 124, wherein the training module 124 is used to:

[0134] Acquire a sample image, wherein the sample image includes a sample object;

[0135] Obtaining a sample position of each sample key point in the sample object;

[0136] The model parameters of the key point prediction model are updated according to the sample image and the sample position.

[0137] According to one or more embodiments of the present disclosure, the training module 124 is specifically used for:

[0138] Processing the sample image according to the key point prediction model to obtain a predicted position corresponding to each sample key point and a variance associated with the predicted position;

[0139] Determining an error parameter according to the predicted position corresponding to the sample key point, the variance and the sample position;

[0140] The model parameters of the key point prediction model are updated according to the error parameter and the variance.

[0141] According to one or more embodiments of the present disclosure, the training module 124 is specifically used for:

[0142] Processing the sample image according to the key point prediction model to obtain a preset probability distribution corresponding to the sample key point;

[0143] Transforming the preset probability distribution according to the standardized flow model to obtain the error probability distribution corresponding to the sample key point;

[0144] A loss function is constructed according to the preset probability distribution, the error probability distribution, the error parameter and the variance, and the model parameters of the key point prediction model are updated according to the loss function.

[0145] The image processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0146] Fig.14 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.14 , which shows a structural schematic diagram of an electronic device 1400 suitable for implementing an embodiment of the present disclosure. The electronic device 1400 can be any device with end-to-end computing capabilities. Fig.14 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0147] like Fig.14 As shown, the electronic device 1400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage device 1408 to a random access memory (RAM) 1403. Various programs and data required for the operation of the electronic device 1400 are also stored in the RAM 1403. The processing device 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0148] Typically, the following devices may be connected to the I / O interface 1405: input devices 1406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1409. The communication device 1409 may allow the electronic device 1400 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.14 The electronic device 1400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0149] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1409, or installed from the storage device 1408, or installed from the ROM 1402. When the computer program is executed by the processing device 1401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0150] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0151] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0152] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0153] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0154] An embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and various image processing methods that may be involved in the first aspect.

[0155] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0158] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0161] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0162] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0163] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as a terminal device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0164] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the terminal device.

[0165] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0166] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0167] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0168] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0169] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0170] In a first aspect, one or more embodiments of the present disclosure provide an image processing method, the method comprising:

[0171] acquiring an image including an object, the object being associated with a plurality of key points;

[0172] Processing the image according to a key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is trained based on a preset probability distribution and an error probability distribution determined by a normalized flow;

[0173] According to the probability distribution graph corresponding to each key point, the position of each key point in the image is determined.

[0174] According to one or more embodiments of the present disclosure, the image is processed according to the key point prediction model to obtain a probability distribution map corresponding to each key point, including:

[0175] Processing the image according to the key point prediction model to obtain a plurality of feature maps associated with the key points;

[0176] Each feature map is normalized to obtain a probability distribution map corresponding to each key point.

[0177] According to one or more embodiments of the present disclosure, determining the position of each key point in the image according to the probability distribution corresponding to each key point includes:

[0178] For any key point, obtain the first coordinate of the point corresponding to each probability value in the probability distribution map corresponding to the key point in the probability distribution map;

[0179] The position of the key point in the image is determined according to each probability value and the first coordinate corresponding to each probability value.

[0180] According to one or more embodiments of the present disclosure, determining the position of the key point in the image according to each probability value and the first coordinate corresponding to each probability value includes:

[0181] Multiply each probability value by the first coordinate corresponding to each probability value to obtain a second coordinate corresponding to each probability value;

[0182] Adding the multiple second coordinates to obtain the target coordinates of the key point;

[0183] Based on the target coordinates, the position of the key point in the image is determined.

[0184] According to one or more embodiments of the present disclosure, the key point prediction model is trained according to the following steps:

[0185] Acquire a sample image, wherein the sample image includes a sample object;

[0186] Obtaining a sample position of each sample key point in the sample object;

[0187] The model parameters of the key point prediction model are updated according to the sample image and the sample position.

[0188] According to one or more embodiments of the present disclosure, updating the model parameters of the key point prediction model according to the sample image and the sample position includes:

[0189] Processing the sample image according to the key point prediction model to obtain a predicted position corresponding to each sample key point and a variance associated with the predicted position;

[0190] Determining an error parameter according to the predicted position corresponding to the sample key point, the variance and the sample position;

[0191] The model parameters of the key point prediction model are updated according to the error parameter and the variance.

[0192] According to one or more embodiments of the present disclosure, updating the model parameters of the key point prediction model based on the error parameter and the variance includes:

[0193] Processing the sample image according to the key point prediction model to obtain a preset probability distribution corresponding to the sample key point;

[0194] Transforming the preset probability distribution according to the standardized flow model to obtain the error probability distribution corresponding to the sample key point;

[0195] A loss function is constructed according to the preset probability distribution, the error probability distribution, the error parameter and the variance, and the model parameters of the key point prediction model are updated according to the loss function.

[0196] In a second aspect, one or more embodiments of the present disclosure provide an image processing device, the image processing device comprising an acquisition module, a processing module and a determination module, wherein:

[0197] The acquisition module is used to acquire an image including an object, wherein the object is associated with a plurality of key points;

[0198] The processing module is used to process the image according to the key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is obtained by training based on a preset probability distribution and an error probability distribution determined by a normalized flow;

[0199] The determination module is used to determine the position of each key point in the image according to the probability distribution graph corresponding to each key point.

[0200] According to one or more embodiments of the present disclosure, the processing module is specifically used to:

[0201] Processing the image according to the key point prediction model to obtain a plurality of feature maps associated with the key points;

[0202] Each feature map is normalized to obtain a probability distribution map corresponding to each key point.

[0203] According to one or more embodiments of the present disclosure, the determination module is specifically used to:

[0204] For any key point, obtain the first coordinate of the point corresponding to each probability value in the probability distribution map corresponding to the key point in the probability distribution map;

[0205] The position of the key point in the image is determined according to each probability value and the first coordinate corresponding to each probability value.

[0206] According to one or more embodiments of the present disclosure, the determination module is specifically used to:

[0207] Multiply each probability value by the first coordinate corresponding to each probability value to obtain a second coordinate corresponding to each probability value;

[0208] Adding the multiple second coordinates to obtain the target coordinates of the key point;

[0209] Based on the target coordinates, the position of the key point in the image is determined.

[0210] According to one or more embodiments of the present disclosure, the image processing device also includes a training module 1, wherein the training module is used to: obtain a sample image, wherein the sample image includes a sample object; obtain the sample position of each sample key point in the sample object; and update the model parameters of the key point prediction model according to the sample image and the sample position.

[0211] According to one or more embodiments of the present disclosure, the training module is specifically used for:

[0212] Processing the sample image according to the key point prediction model to obtain a predicted position corresponding to each sample key point and a variance associated with the predicted position;

[0213] An error parameter is determined according to the predicted position corresponding to the sample key point, the variance and the sample position; and a model parameter of the key point prediction model is updated according to the error parameter and the variance.

[0214] According to one or more embodiments of the present disclosure, the training module is specifically used for:

[0215] Processing the sample image according to the key point prediction model to obtain a preset probability distribution corresponding to the sample key point;

[0216] Transforming the preset probability distribution according to the standardized flow model to obtain the error probability distribution corresponding to the sample key point;

[0217] A loss function is constructed according to the preset probability distribution, the error probability distribution, the error parameter and the variance, and the model parameters of the key point prediction model are updated according to the loss function.

[0218] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0219] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method as described in the first aspect and various possible aspects of the first aspect.

[0220] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0221] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0222] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: acquiring an image including an object, the object being associated with a plurality of key points; Processing the image according to a key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is trained based on a preset probability distribution and an error probability distribution determined by a normalized flow; According to the probability distribution graph corresponding to each key point, the position of each key point in the image is determined.

2. The method according to claim 1, characterized in that The image is processed according to the key point prediction model to obtain a probability distribution map corresponding to each key point, including: Processing the image according to the key point prediction model to obtain a plurality of feature maps associated with the key points; Each feature map is normalized to obtain a probability distribution map corresponding to each key point.

3. The method according to claim 1, characterized in that Determining the position of each key point in the image according to the probability distribution corresponding to each key point includes: For any key point, obtain the first coordinate of the point corresponding to each probability value in the probability distribution map corresponding to the key point in the probability distribution map; The position of the key point in the image is determined according to each probability value and the first coordinate corresponding to each probability value.

4. The method according to claim 3, characterized in that Determining the position of the key point in the image according to each probability value and the first coordinate corresponding to each probability value includes: Multiply each probability value by the first coordinate corresponding to each probability value to obtain a second coordinate corresponding to each probability value; Adding the multiple second coordinates to obtain the target coordinates of the key point; Based on the target coordinates, the position of the key point in the image is determined.

5. The method according to any one of claims 2 to 4, characterized in that: The key point prediction model is trained according to the following steps: Acquire a sample image, wherein the sample image includes a sample object; Obtaining a sample position of each sample key point in the sample object; The model parameters of the key point prediction model are updated according to the sample image and the sample position.

6. The method according to claim 5, characterized in that Updating the model parameters of the key point prediction model according to the sample image and the sample position, including: Processing the sample image according to the key point prediction model to obtain a predicted position corresponding to each sample key point and a variance associated with the predicted position; Determining an error parameter according to the predicted position corresponding to the sample key point, the variance and the sample position; The model parameters of the key point prediction model are updated according to the error parameter and the variance.

7. The method according to claim 6, characterized in that Based on the error parameter and the variance, updating the model parameters of the key point prediction model includes: Processing the sample image according to the key point prediction model to obtain a preset probability distribution corresponding to the sample key point; Transforming the preset probability distribution according to the standardized flow model to obtain the error probability distribution corresponding to the sample key point; A loss function is constructed according to the preset probability distribution, the error probability distribution, the error parameter and the variance, and the model parameters of the key point prediction model are updated according to the loss function.

8. An image processing device, characterized in that: It includes an acquisition module, a processing module and a determination module, wherein: The acquisition module is used to acquire an image including an object, wherein the object is associated with a plurality of key points; The processing module is used to process the image according to the key point prediction model to obtain a probability distribution map corresponding to each key point, wherein the key point prediction model is obtained by training based on a preset probability distribution and an error probability distribution determined by a normalized flow; The determination module is used to determine the position of each key point in the image according to the probability distribution graph corresponding to each key point.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Human body 3D key point detection method, model training method and related device

    CN112270669A

  • Hand posture estimation method, device and equipment and computer storage medium

    CN114586078A

  • Real-time human body 2D / 3D skeleton key point identification method based on monocular RGB image regression

    CN115171149A

  • Key point and joint key point detection, model training, equipment and storage medium

    CN115240267A

  • Real-time detection method and system based on key points in distribution network overhead line inspection

    CN115937542A