A method, device, equipment and storage medium for generating true value of key points

Abstract

The embodiment of the present invention provides a method, device, equipment and storage medium for generating true values ​​of key points. The method includes: obtaining an original image in which key points have been identified; initializing a zero matrix corresponding to the key points; assigning values ​​to the zero matrix according to the position information of the key points in the original image, the size ratio of the zero matrix to the original image, and the type of true value feature map to be generated; generating a true value feature map of the type corresponding to the key points according to the assigned zero matrix, so as to train the key point recognition model using the true value feature map; the pixel value range of the true value feature map of this type is the same as that of the true value feature map of other types of key points. The present invention initializes the zero matrix corresponding to the key points; assigns values ​​to the zero matrix according to the type of true value feature map to be generated, combined with the position of the key points in the original image and the size ratio of the zero matrix to the original image, so that the pixel value ranges of multiple true value feature maps corresponding to the key points are the same.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a key point true value generation method, device, equipment and storage medium. Background Art

[0002] With the improvement of computer computing power and the development of deep learning technology, a key point recognition algorithm based on deep learning has been proposed. By using a large amount of labeled data to train the deep learning network, the deep learning network can recognize the key points of the human body in various postures with strong generalization ability, such as Figure 1 The figure shows a schematic diagram of the key points of the human body identified using a deep learning network.

[0003] However, most key point recognition algorithms based on deep learning have the problem of incompatibility between computing power and accuracy. If you want to reduce the computational complexity, you will inevitably reduce the accuracy, which makes it impossible to deploy key point recognition algorithms on mobile terminals. In order to solve the problem of incompatibility between computing power and accuracy, a method of indirectly identifying key points through heat maps and displacement maps is proposed. Compared with the traditional key point recognition method based on a single heat map, the algorithm for locating key points in the heat map through the displacement map is simpler. Therefore, the method of identifying key points through heat maps and displacement maps can reduce the complexity of the deep learning network and has a higher recognition accuracy.

[0004] Specifically, a heat map and two displacement maps (x-direction and y-direction) can be used to represent a key point. The heat map is a Gaussian-like distribution, and the displacement map represents the displacement between the current position and the true value point. The pixel value ranges of the heat map and the displacement map are different, and the pixel value range of the heat map is much smaller than the pixel value range of the displacement map. Since the pixel value range of the heat map is much smaller than the pixel value range of the displacement map, new technical problems have arisen. The key point recognition model is a multi-task learning network. When training the key point recognition model, the key point recognition model needs to learn the heat map and the displacement map at the same time. The pixel value range of the displacement map is much larger than the pixel value range of the heat map, resulting in the loss value of the displacement map being much larger than the loss value of the heat map, making the adjusted key point recognition model pay more attention to the displacement map, which affects the learning effect of the heat map. Moreover, since the pixel value range of the displacement map is different from that of the heat map, the key point recognition model cannot use the commonly used ReLU (Rectified Linear Unit) activation function and Sigmoid (S-shaped growth curve) activation function, which affects the training effect of the key point recognition model. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, equipment and storage medium for generating true values ​​of key points to solve the problem that the pixel value ranges of heat maps and displacement maps are different.

[0006] In view of the above technical problems, the specific technical solutions of the embodiments of the present invention are as follows:

[0007] In the first aspect of the implementation of the present invention, a method for generating true values ​​of key points is first provided, including: acquiring an original image in which key points have been identified; initializing a zero matrix corresponding to the key points; assigning values ​​to the zero matrix according to position information of the key points in the original image, a size ratio between the zero matrix and the original image, and a type of true value feature map to be generated; generating a true value feature map of the type corresponding to the key points according to the assigned zero matrix, so as to train a preset key point recognition model using the true value feature map generated according to the zero matrix; wherein the true value feature map of the type has the same pixel value range as the true value feature map of other types of the key points.

[0008] Among them, when the true value feature image is a true value heat map, the zero matrix is ​​assigned according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image and the type of the true value feature map to be generated, including: determining the coordinates of the key point in the zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image; determining whether the pixel point to be assigned is within a preset range of the key point according to the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned in the zero matrix; if the pixel point to be assigned is within the preset range, assigning the pixel point to be assigned a first pixel value; otherwise, assigning the pixel point to be assigned a second pixel value.

[0009] Wherein, the first pixel value is:

[0010]

[0011] e is the natural base, d 2 is the square of the distance between the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned in the zero matrix; α represents the preset parameter value; r is the radius of the preset range centered on the key point.

[0012] Wherein, the true value feature image includes: a mask displacement map; the mask displacement map includes: a mask horizontal axis displacement map and a mask vertical axis displacement map; the initialization of the zero matrix corresponding to the key point includes: initializing the horizontal axis zero matrix corresponding to the key point, and initializing the vertical axis zero matrix corresponding to the key point; wherein, the horizontal axis zero matrix is ​​used to generate the mask horizontal axis displacement map, and the vertical axis zero matrix is ​​used to generate the mask vertical axis displacement map.

[0013] Among them, assigning values ​​to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image and the type of the true value feature map to be generated includes: determining the coordinates of the key point in the horizontal axis zero matrix and the coordinates of the key point in the vertical axis zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image; assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the horizontal coordinates of the key point in the original image, the size ratio of the horizontal axis zero matrix to the original image and the horizontal coordinates of the pixel points to be assigned in the horizontal axis zero matrix; assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the vertical coordinates of the key point in the original image, the size ratio of the vertical axis zero matrix to the original image and the vertical coordinates of the pixel points to be assigned in the vertical axis zero matrix.

[0014] The step of assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix includes: using the following formula to determine the values ​​of the pixel points to be assigned in the horizontal axis zero matrix: xab =a×sx k ; Among them, w xab is represented by the value of the pixel to be assigned in the horizontal axis zero matrix; a is the horizontal coordinate of the pixel to be assigned in the horizontal axis zero matrix; s represents that the size of the original image is s times the size of the horizontal axis zero matrix; x k is the horizontal coordinate of the key point in the original image; and / or, assigning values ​​to the pixel points to be assigned in the vertical axis zero matrix, including: using the following formula to determine the value of the pixel points to be assigned in the vertical axis zero matrix: yab =b×sy k ; Among them, w yab represents the value of the pixel to be assigned in the vertical axis zero matrix; b is the vertical coordinate of the pixel to be assigned in the vertical axis zero matrix; s represents that the size of the original image is s times the size of the vertical axis zero matrix; y k is the vertical coordinate of the key point in the original image.

[0015] Among them, the true value feature map corresponding to the key point is generated according to the zero matrix after the pixel points are assigned, including: generating the original horizontal axis displacement map corresponding to the key point according to the assigned horizontal axis zero matrix; generating the original vertical axis displacement map corresponding to the key point according to the assigned vertical axis zero matrix; generating a true value mask map corresponding to the key point; wherein the true value mask map, the horizontal axis zero matrix and the vertical axis zero matrix have the same size, the pixel points in the true value mask map within the preset range of the key point are assigned the third pixel value, and the remaining pixel points are assigned the fourth pixel value; according to the original horizontal axis displacement map and the true value mask map, a mask horizontal axis displacement map corresponding to the key point is generated; according to the original vertical axis displacement map and the true value mask map, a mask vertical axis displacement map corresponding to the key point is generated.

[0016] The step of generating the mask horizontal axis displacement map corresponding to the key point according to the original horizontal axis displacement map and the true value mask map includes: generating the mask horizontal axis displacement map by using the following formula:

[0017]

[0018] M mx Represents the horizontal axis displacement map of the mask, M h represents the true value mask map; M x Represents the original horizontal axis displacement diagram; max(M h ·M x ) represents the maximum pixel value within the preset range of the key point; min(M h ·M x ) represents the minimum pixel value within the preset range of the key point;

[0019] And / or, generating the masked longitudinal axis displacement map corresponding to the key point according to the original longitudinal axis displacement map and the true value mask map, comprises:

[0020] The mask vertical axis displacement map is generated using the following formula:

[0021]

[0022] M my Represents the vertical axis displacement map of the mask, M h represents the true value mask map; M y Represents the original vertical axis displacement diagram; max(M h ·M y ) represents the maximum pixel value within the preset range of the key point; min(M h ·M y ) represents the minimum pixel value within the preset range of the key point.

[0023] Before initializing the zero matrix corresponding to the key point, the method further includes: reading the step parameter value in the key point recognition model, and taking the inverse of the step parameter value as a preset ratio; and determining the size of the zero matrix according to the preset ratio and the size of the original image.

[0024] In the second aspect of the implementation of the present invention, a key point true value generation device is also provided, including: an acquisition module, used to acquire an original image in which key points have been identified; an initialization module, used to initialize a zero matrix corresponding to the key points; an assignment module, used to assign values ​​to the zero matrix according to the position information of the key points in the original image, the size ratio of the zero matrix to the original image and the type of true value feature map to be generated; a generation module, used to generate a true value feature map of the type corresponding to the key point according to the assigned zero matrix, so as to use the true value feature map generated according to the zero matrix to train a preset key point recognition model; wherein the true value feature map of the type has the same pixel value range as the true value feature map of other types of the key points.

[0025] In the third aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any of the method steps described above when executing the program stored in the memory.

[0026] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-described methods.

[0027] In a fifth aspect of the implementation of the present invention, there is also provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described methods.

[0028] The embodiment of the present invention provides a method, device, equipment and storage medium for generating true values ​​of key points, by initializing the zero matrix corresponding to the key points; assigning values ​​to the zero matrix according to the type of true value feature map to be generated, combined with the position information of the key points in the original image and the size ratio of the zero matrix to the original image, so that the pixel value ranges of the multiple true value feature maps corresponding to the key points are the same. When the true value feature map generated by the embodiment of the present invention is used to train the key point recognition model, since the pixel value ranges of different types of true value feature maps are the same, the loss ranges of different types of true value feature maps are also the same, which makes the key point recognition model pay equal attention to each true value feature map, and the predicted feature map output by the key point recognition model will be closer to the true value feature map, and the accuracy of key point recognition will be higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0030] Figure 1 A schematic diagram of key points of a human body identified by a deep learning network in the prior art;

[0031] Figure 2 is a flow chart of a key point true value generating method according to an embodiment of the present invention;

[0032] Figure 3 is a flow chart of steps for generating a true value heat map according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of a true value heat map according to an embodiment of the present invention;

[0034] Figure 5 is a flow chart of steps for generating a true value heat map according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of an original displacement map according to an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of a true value mask map according to an embodiment of the present invention;

[0037] Figure 8 is a schematic diagram of a mask displacement map according to an embodiment of the present invention;

[0038] Fig. 9 is a structural diagram of a training system for a key point recognition model according to an embodiment of the present invention;

[0039] Fig.10 is a structural diagram of a key point recognition model according to an embodiment of the present invention;

[0040] Fig.11 is a structural diagram of a key point true value generating device according to an embodiment of the present invention;

[0041] Fig.12 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0043] The embodiment of the present invention provides a method for generating a key point true value. Figure 2 FIG. 1 is a flowchart of a method for generating true values ​​of key points according to an embodiment of the present invention.

[0044] Step S210, obtaining the original image with key points identified.

[0045] Key points are used to locate key parts of a human body, such as the head, neck, shoulders, hands, legs and / or feet. For any key part of the body, when locating the key part of the body, one or more key points may be required.

[0046] Before obtaining the original image with the key points identified, the key points in the original image may be identified by a preset key point identification algorithm, or the key points in the original image may be identified manually.

[0047] Step S220, initializing the zero matrix corresponding to the key point.

[0048] The zero matrix refers to an image matrix whose pixel values ​​are all 0. Furthermore, an image matrix with a preset size is generated for the key points, and each pixel value in the image matrix is ​​initialized to 0.

[0049] If the original image includes multiple key points, a zero matrix is ​​initialized for each key point.

[0050] Specifically, when initializing (each) of the zero matrix, the size of the zero matrix accounts for a ratio of the size of the original image to a preset ratio, wherein the preset ratio is less than 1. In other words, the size of the zero matrix is ​​smaller than the size of the original image.

[0051] Furthermore, before initializing the zero matrix corresponding to (each) the key point, it also includes: reading the step parameter value (stride parameter value) in the preset key point recognition model, and taking the inverse of the step parameter value as the preset ratio; wherein the preset ratio is the ratio of the size of the zero matrix to the size of the original image; according to the preset ratio and the size of the original image, the size of the zero matrix is ​​determined. The key point recognition model is a deep learning network for identifying key points. The types of true value feature maps include but are not limited to: true value heat map and mask displacement map. The true value feature map generated according to the zero matrix can be used as a training sample of the key point recognition model, so that the size of the original image is s times the size of the training sample, that is, the length of the training sample is 1 / s of the length of the original image, and the width of the training sample is 1 / s of the width of the original image, and s is the stride parameter value. The steps of how to generate multiple true value feature maps according to the zero matrix will be introduced later, so I will not repeat them here.

[0052] Step S230, assigning values ​​to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image, and the type of the true value feature map to be generated.

[0053] The position information may be position coordinates. Further, the position coordinates may be represented by the row (horizontal axis) and column (vertical axis) of the pixel point in the original image and the zero matrix. For example, if the pixel point is in the 2nd row and the 3rd column in the original image, the coordinates of the pixel point are (2,3).

[0054] The types of true value feature maps include, but are not limited to: true value heat maps and mask displacement maps. Among them, the types of mask displacement maps include: mask horizontal axis displacement maps and mask vertical axis displacement maps.

[0055] The true value heat map refers to the actual heat map of the key points.

[0056] The mask displacement map refers to the mask of the displacement map of the key point.

[0057] The masked horizontal axis displacement map represents the mask of the displacement map corresponding to the horizontal axis direction (x direction).

[0058] The mask vertical axis displacement map represents the mask of the displacement map corresponding to the vertical axis direction (y direction).

[0059] Assigning a value to the zero matrix includes: assigning a value to each pixel in the zero matrix. Assigning a value to each pixel refers to updating the pixel value of each pixel in the zero matrix.

[0060] Specifically, multiple zero matrices corresponding to the key points can be initialized; the multiple zero matrices have the same size, and each zero matrix corresponds to a true value feature map to be generated; according to the position information of the key points in the original image and the size ratio of the zero matrix to the original image, the position information of the key points in the zero matrix is ​​determined; according to the position information of the key points in the zero matrix and the type of each true value feature map to be generated, the zero matrix corresponding to each true value feature map to be generated is assigned a value.

[0061] Furthermore, when assigning values ​​to the zero matrix corresponding to each true value feature map to be generated, the pixel value ranges of the zero matrices corresponding to the multiple true value feature maps to be generated are made the same.

[0062] Step S240, generating a true value feature map of the type corresponding to the key point according to the assigned zero matrix, so as to train a preset key point recognition model using the true value feature map generated according to the zero matrix; wherein the true value feature map of the type has the same pixel value range as the true value feature maps of other types of the key point.

[0063] In other words, the pixel value ranges of the multiple types of true value feature maps corresponding to the key points are the same. The multiple types of true value feature maps include but are not limited to: true value heat map and mask displacement map. Furthermore, the pixel value ranges corresponding to the true value heat map and the mask displacement map are both between 0 and 1.

[0064] If the true value feature map to be generated corresponding to the assigned zero matrix is ​​a true value heat map, then the true value heat map corresponding to the key point is directly generated according to the assigned zero matrix.

[0065] If the true value feature map to be generated corresponding to the assigned zero matrix is ​​a mask displacement map, the original displacement map is first generated according to the assigned zero matrix, and then the mask displacement map corresponding to the key point is generated according to the original displacement map and the true value heat map.

[0066] Furthermore, the types of original displacement maps include original horizontal axis displacement maps and original vertical axis displacement maps. Among them, the original horizontal axis displacement map represents the displacement map corresponding to the horizontal axis direction. The original vertical axis displacement map represents the displacement map corresponding to the vertical axis direction. According to the assigned horizontal axis zero matrix, the original horizontal axis displacement map is generated, and then according to the original horizontal axis displacement map and the true value heat map, the mask horizontal axis displacement map corresponding to the key point is generated. According to the assigned vertical axis zero matrix, the original vertical axis displacement map is generated, and then according to the original vertical axis displacement map and the true value heat map, the mask vertical axis displacement map corresponding to the key point is generated. The specific generation steps of the mask horizontal axis displacement map and the mask vertical axis displacement map will be described later, so they will not be repeated here.

[0067] In this embodiment, the generated true value feature map is used as a true value sample for training the key point recognition model.

[0068] In an embodiment of the present invention, a zero matrix corresponding to a key point is initialized; according to the type of true value feature map to be generated, combined with the position information of the key point in the original image and the size ratio of the zero matrix to the original image, the zero matrix is ​​assigned a value so that the pixel value ranges of the multiple true value feature maps corresponding to the key point are the same.

[0069] Furthermore, when the key point recognition model learns the true value feature map generated by the present embodiment, the trained key point recognition model will not pay more attention to one true value feature map and ignore the learning of other types of true value feature maps because the pixel value ranges of different types of true value feature maps are different. When the key point recognition model is trained using the true value feature map generated by the present embodiment, since the pixel value ranges of different types of true value feature maps are the same, the loss ranges of different types of true value feature maps are also the same, which makes the key point recognition model pay equal attention to each true value feature map, and the predicted feature map output by the key point recognition model will be closer to the true value feature map, and the accuracy of key point recognition will be higher.

[0070] Furthermore, the size of the zero matrix initialized in this embodiment is smaller than the size of the original image, so that the size of the true value feature map is smaller than the size of the original image. This can reduce the overhead of the key point recognition model when learning the true value feature map, and can also reduce the complexity of the key point recognition model.

[0071] Furthermore, the pixel value ranges of different types of true value feature maps generated by this embodiment are the same, so that the key point recognition model can apply the ReLU activation function and the Sigmoid activation function to increase the accuracy of key point recognition.

[0072] The following further describes the steps of how to generate the true value heat map corresponding to the key points after assigning values ​​to the zero matrix.

[0073] like Figure 3 , which is a flow chart of steps for generating a true value heat map according to an embodiment of the present invention.

[0074] Step S310, determining the coordinates of the key point in the zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix and the original image.

[0075] The size of the original image is w i ×h i , the size of the zero matrix is ​​w h ×h h ; Among them, w h =w i / s、hh =h i / s, the step size parameter value of the key point recognition model is s; the coordinates of the key point in the original image are (x, y); the coordinates of the key point in the true value heat map are (x h ,y h )for:

[0076] x h = int(round(x / s));

[0077] y h = int(round(y / s));

[0078] Among them, round(z) means rounding z, and int(z) means rounding z.

[0079] Step S320, determining whether the pixel to be assigned is within a preset range of the key point based on the coordinates of the key point in the zero matrix and the coordinates of the pixel to be assigned in the zero matrix; if yes, executing step S330; if not, executing step S340.

[0080] The preset range may be: a preset range centered at the key point. The radius r of the preset range may be an empirical value or a value obtained through experiments.

[0081] Traverse each position (pixel point) in the zero matrix, and for each pixel point (a, b) position to be assigned, that is, the pixel value of the ath row and bth column, calculate the key point (x h ,y h ) and the square of the distance between the pixel to be assigned (a, b) 2 :

[0082] d 2 =(x h -a) 2 +(y h -b) 2

[0083] The square of this distance d 2 The square of the radius of the preset range centered at the key point r 2 Compare; if d 2 ≤r 2 , it means that the pixel point is within the preset range of the key point, and step S330 is executed; if d 2 >r 2 , it means that the pixel point is not within the preset range of the key point, and step S340 is executed.

[0084] Step S330: If the pixel point to be assigned a value is within the preset range, assigning the pixel point to be assigned a value as a first pixel value.

[0085] The first pixel value may be a value determined according to the distance between the coordinates of the pixel point to be assigned a value and the coordinates of the key point in the zero matrix and the radius of the preset range.

[0086] The first pixel value can be calculated using the following formula:

[0087]

[0088] Among them, e is the natural base, d 2 is the square of the distance between the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned in the zero matrix; α represents the preset parameter value; r is the radius of the preset range centered on the key point. α can be an empirical value or a value obtained through experiments.

[0089] In this way, if there are multiple pixel points to be assigned values ​​within the preset range of the key point, the first pixel values ​​corresponding to the multiple pixel points to be assigned values ​​may be different in size.

[0090] Step S340: If the pixel point to be assigned a value is not within the preset range, assigning the pixel point to be assigned a value as a second pixel value.

[0091] The second pixel value may be a preset constant value. In this embodiment, the second pixel value may be set to 0.

[0092] Step S350, determine whether all the pixel points to be assigned in the zero matrix have been assigned; if yes, execute step S360; if not, determine the next pixel point to be assigned, and jump to step S320.

[0093] Step S360: Generate a true value heat map corresponding to the key point according to the assigned zero matrix.

[0094] like Figure 4 , which is a schematic diagram of a true value heat map according to an embodiment of the present invention. Figure 4 These are the true value heat maps corresponding to the 14 key points in the original image, where the empty map indicates that the true value of the key point is null.

[0095] The following further describes the steps of how to generate a mask displacement map corresponding to the key points after obtaining the original image in which the key points have been identified.

[0096] like Figure 5 , which is a flow chart of steps for generating a true value heat map according to an embodiment of the present invention.

[0097] Step S510, initializing the horizontal axis zero matrix corresponding to the key point, and initializing the vertical axis zero matrix corresponding to the key point; wherein the horizontal axis zero matrix is ​​used to generate the mask horizontal axis displacement map, and the vertical axis zero matrix is ​​used to generate the mask vertical axis displacement map.

[0098] The size of the horizontal axis zero matrix and the vertical axis zero matrix initialized for the key point are the same. Further, the size of the horizontal axis zero matrix and the vertical axis zero matrix initialized for the key point and the size of the zero matrix initialized for the key point for generating the true value heat map are all the same.

[0099] Step S520, determining the coordinates of the key point in the horizontal axis zero matrix and the coordinates of the key point in the vertical axis zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image.

[0100] Since the horizontal axis zero matrix, the vertical axis zero matrix, and the zero matrix used to generate the true value heat map have the same size, the horizontal coordinate x of the key point in the horizontal axis zero matrix can be determined by referring to the determination method in step S310. k And the vertical coordinate y of the key point in the vertical axis zero matrix k .

[0101] Step S530, assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the horizontal coordinates of the key points in the original image, the size ratio of the horizontal axis zero matrix and the original image, and the horizontal coordinates of the pixel points to be assigned in the horizontal axis zero matrix.

[0102] The value of the pixel point (a, b) to be assigned in the horizontal axis zero matrix can be determined by the following formula:

[0103] w xab =a×sx k ;

[0104] Among them, w xab is represented by the value of the pixel to be assigned in the horizontal axis zero matrix; a is the horizontal coordinate of the pixel to be assigned in the horizontal axis zero matrix; d represents that the size of the original image is s times that of the horizontal axis zero matrix; x k is the horizontal coordinate of the key point in the original image.

[0105] Step S540, assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the vertical coordinate of the key point in the original image, the ratio of the size of the vertical axis zero matrix to the size of the original image, and the vertical coordinate of the pixel points to be assigned in the vertical axis zero matrix.

[0106] The following formula can be used to determine the value of the pixel point to be assigned in the vertical axis zero matrix:

[0107] w yab =b×sy k ;

[0108] Among them, w yab y is the value of the pixel to be assigned in the vertical axis zero matrix; b is the vertical coordinate of the pixel to be assigned in the vertical axis zero matrix; s means that the size of the original image is s times that of the vertical axis zero matrix; k is the vertical coordinate of the key point in the original image.

[0109] Step S550, generating an original horizontal axis displacement map corresponding to the key point according to the assigned horizontal axis zero matrix; generating an original vertical axis displacement map corresponding to the key point according to the assigned vertical axis zero matrix.

[0110] like Figure 6 FIG. 1 is a schematic diagram of an original displacement map according to an embodiment of the present invention. Figure 6 In , the first 14 images are the original horizontal axis displacement maps corresponding to the 14 key points in the original image, and the last 14 images are the original vertical axis displacement maps corresponding to the 14 key points in the original image. Figure 6 In the 28 images in , the empty image indicates that the true value of the key point is null.

[0111] Step S560, generating a true value mask map corresponding to the key point; wherein the true value mask map, the horizontal axis zero matrix and the vertical axis zero matrix have the same size, and the pixel points in the true value mask map within the preset range of the key point are assigned a third pixel value, and the remaining pixel points are assigned a fourth pixel value.

[0112] The true value mask map refers to the mask of the true value heat map.

[0113] The third pixel value and the fourth pixel value can both be preset fixed values. In this embodiment, the third pixel value can be 1. The fourth pixel value can be 0. Further, all pixel values ​​greater than zero in the true value heat map are set to 1 to form a true value mask map corresponding to the true value heat map. The steps of generating the true value heat map corresponding to the key point can refer to Figure 3 Steps shown.

[0114] like Figure 7 FIG. 1 is a schematic diagram of a true value mask according to an embodiment of the present invention. Figure 7 There are two identical sets of true mask maps, each set of which contains 14 true mask maps, that is, Figure 7 The nth image and the n+14th image in are the same. Each truth mask in each group is Figure 4 The mask of the ground truth heat map at the corresponding position in .

[0115] Step S570, generating a mask horizontal axis displacement map corresponding to the key point according to the original horizontal axis displacement map and the true value mask map; generating a mask vertical axis displacement map corresponding to the key point according to the original vertical axis displacement map and the true value mask map.

[0116] The mask horizontal axis displacement map is generated using the following formula:

[0117]

[0118] Among them, M mx Represents the horizontal axis displacement map of the mask, M h represents the true value mask map; M x Represents the original horizontal axis displacement diagram; max(M h ·M x ) represents the maximum pixel value within the preset range of the key point; min(M h ·M x ) represents the minimum pixel value within the preset range of the key point.

[0119] The mask vertical axis displacement map is generated using the following formula:

[0120]

[0121] Among them, M my Represents the vertical axis displacement map of the mask, M h represents the true value mask map; M y Represents the original vertical axis displacement diagram; max(M h ·M y ) represents the maximum pixel value within the preset range of the key point; min(M h ·M y ) represents the minimum pixel value within the preset range of the key point.

[0122] like Figure 8 FIG. 1 is a schematic diagram of a mask displacement map according to an embodiment of the present invention. Figure 8 In the figure, the first 14 images are the mask horizontal axis displacement maps corresponding to the 14 key points in the original image, and the last 14 images are the mask vertical axis displacement maps corresponding to the 14 key points in the original image. Figure 8 The empty graph in indicates that the true value of the keypoint is null.

[0123] For example, if the original image has 100×100 pixels, the coordinates of the key point P in the original image are (80,60), and the pixels of the initialized zero matrix are 10×10, then the size of the original image is 10 times the size of the zero matrix, that is, s = 10. The pixel coordinates in the zero matrix are represented as (a, b).

[0124] Determine the horizontal coordinate x of the key point P in the true value heat map h = int(round(x / s)) = int(round(80 / 10)) = 8, the vertical coordinate y of the key point P in the true value heat map h = int(round(y / s)) = int(round(60 / 8)) = 8, so the coordinates of the key point P in the true value heat map are (8, 8).

[0125] The zero matrix is ​​represented by the following Table 1, where the position of P in Table 1 is the position of the key point P. The pixel points in the zero matrix that are within the preset range of the key point P are assigned the first pixel value, and the pixel points that are not within the preset range of the key point P in the zero matrix are assigned 0. The preset range is a range with the key point P as the center and a radius of 1.

[0126] Among them, α=1.

[0127]

[0128]

[0129] Table 1

[0130] The values ​​of non-zero pixels in the true value heat map represented by Table 1 are set to 1 to obtain a mask heat map corresponding to the true value heat map. The mask heat map can be represented by Table 2.

[0131] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1(P) 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0

[0132] Table 2

[0133] Initialize the horizontal axis zero matrix corresponding to the key point P. The size of the horizontal axis zero matrix is ​​the same as the size of the zero matrix corresponding to the true value heat map. Assign values ​​to each pixel in the horizontal axis zero matrix. The value of each pixel can be obtained by formula w xab =a×sx k Determine, and then obtain the original displacement diagram of the horizontal axis shown in Table 3.

[0134]

[0135]

[0136] Table 3

[0137] By performing a dot product of the mask heat map in Table 2 and the horizontal axis original displacement map in Table 3, all pixel values ​​that are not within the preset range of the key points can be set to 0, and only the pixel values ​​within the preset range of the key points are retained. The maximum pixel value within the preset range of the key points is determined to be 10, and the minimum pixel value is -10. In this way, the mask horizontal axis displacement map shown in Table 4 can be obtained.

[0138] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0.5(P) 0.5 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0

[0139] Table 4

[0140] Initialize the vertical axis zero matrix corresponding to the key point P. The size of the vertical axis zero matrix is ​​the same as the size of the zero matrix corresponding to the true value heat map. Assign values ​​to each pixel in the vertical axis zero matrix. The value of each pixel can be obtained by formula w yab =b×sy k Determine, and then obtain the original displacement diagram of the vertical axis shown in Table 5.

[0141] -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0(P) 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20 -70 -60 -50 -40 -30 -20 -10 0 10 20

[0142] Table 5

[0143] By performing a dot product of the mask heat map in Table 2 and the original vertical axis displacement map in Table 5, all pixel values ​​that are not within the preset range of the key points can be set to 0, and only the pixel values ​​within the preset range of the key points are retained. The maximum pixel value within the preset range of the key points is determined to be 10, and the minimum pixel value is -10. In this way, the mask vertical axis displacement map shown in Table 6 can be obtained.

[0144]

[0145]

[0146] Table 6

[0147] The present embodiment adopts the synchronous execution mode to generate the step of the mask horizontal axis displacement map and the mask vertical axis displacement map. Those skilled in the art should know that the step of generating the mask horizontal axis displacement map and the step of generating the mask vertical axis displacement map can also be performed separately. In terms of execution order, step S530 and step S540 can be executed in parallel or in a one-after-the-other manner, and the present embodiment does not limit the execution order.

[0148] In an embodiment of the present invention, the pixel value range of the true value heat map can be between 0 and 1, and the pixel value range of the mask displacement map can also be between 0 and 1, and the pixel value range of the true value heat map and the mask displacement map is the same. For example: if the pixel value range of the heat map can be between 0 and 1, and the pixels of the original image are 1000×800, then the pixel value of the displacement map can be between -1000 and 1000, then the maximum loss value of the heat map is 1 (such as: the pixel value of the true value heat map is 1, and the pixel value of the predicted heat map is 0), and the maximum loss value of the displacement map is 2000 (the pixel value of the true value displacement map is 1000, and the pixel value of the predicted displacement map is -1000), so the key point recognition model will inevitably adjust the parameters due to the excessive loss of the displacement map, and so on, the key point recognition model will pay more attention to the learning of the true value displacement map (corresponding to the mask displacement map of the embodiment of the present invention), and ignore the learning of the true value heat map, which also avoids reducing the accuracy of key point recognition. However, in this embodiment, the pixel value range of the true heat map and the mask displacement map is the same, and the key point recognition model can learn the true heat map and the mask displacement map in a balanced manner, so that the predicted heat map and predicted displacement map output by the key point recognition model are closer to the true value.

[0149] The true value feature map generated by the embodiment of the present invention can be applied to the training of the key point recognition model. Fig. 9 , which is a structural diagram of a training system for a key point recognition model according to an embodiment of the present invention.

[0150] The training system includes: a human body detection network 910, a key point correction unit 920, a truth value generation unit 930, a key point recognition model 940 and a loss determination unit 950. The output end of the key point correction unit 920 is respectively connected to the input end of the truth value generation unit 930 and the output end of the human body detection network 910, the input end of the key point recognition model 940 is connected to the output end of the human body detection network 910, and the output end of the truth value generation unit 930 and the output end of the key point recognition model 940 are respectively connected to the input end of the loss determination unit 950.

[0151] The human body detection network 910 is used to receive an image, perform human body recognition on the image, and output the human body image and the bounding box (BBox) information in the human body image.

[0152] The key point correction unit 920 is used to receive the coordinates of the key points in the image, and obtain the human body image and BBox information output by the human body detection network 910; according to the coordinates of the key points in the image, each key point is mapped to the human body image, and the coordinates of each key point mapped to the human body image are corrected according to the BBox information.

[0153] The truth value generating unit 930 is used to receive the human body image with corrected key point coordinates, and generate a truth value feature map corresponding to each corrected key point.

[0154] The key point recognition model 940 is used to receive the human body image output by the human body detection network 910, perform key point recognition based on the human body image, and output a prediction feature map corresponding to each key point.

[0155] The loss determination unit 950 is used to receive the true value feature map corresponding to each key point output by the true value generation unit 930 and the predicted feature map corresponding to each key point output by the key point recognition model 940, and determine the feature map loss according to the true value feature map and the predicted feature map corresponding to the same key point. The feature map loss is used to determine whether the key point recognition model 940 needs to adjust its parameters.

[0156] The following is a further description of each of the above components.

[0157] The image includes multiple key points, and the coordinates of the multiple key points are expressed as P = {P1, P2, P3, ..., P n}; where n is the number of key points, P i =(x i ,y i ) is the coordinate of the i-th key point, x i is the horizontal axis, y i Is the vertical axis.

[0158] In the process of training the keypoint recognition model, each of the above components performs the following steps:

[0159] Step 1: The human body detection network 910 identifies the human body image in the image and determines the BBox information of the human body image. The BBox information can be expressed as (x bbox ,y bbox ,w bbox ,h bbox ), x bbox and bbox Respectively represent the horizontal and vertical coordinates of the vertex in the upper left corner of the human body border, w bbox Indicates the width of the human body image, h bbox Indicates the height of the human image.

[0160] Step 2: The key point correction unit 920 calculates the key point coordinates P = {P1, P2, P3, ..., P n} Map the n key points to the human body image respectively, and correct the coordinates of the n key points according to the BBox information of the human body image. Further, the corrected key point coordinates are P′={P′1,P′2,P′3,...,P′ n}. P′ i =(x′ i ,y′ i ) is the coordinate of the i-th key point, x′ i =x i -x bbox is the corrected horizontal coordinate, y′ i =y i -y bbox The original image used by the truth value generating unit 930 can be generated according to the human body image with key points mapped and the key point coordinates corrected. The number of original images is the same as the number of key points, and each original image includes a key point, and the coordinates of the key point are the corrected coordinates.

[0161] Step 3: The truth value generating unit 930 generates a truth value feature map corresponding to each key point according to the key point truth value generating method of the embodiment of the present invention. The truth value feature map includes a truth value heat map and a mask displacement map.

[0162] Step 4: The key point recognition model 940 receives the human body image output by the human body detection network 910, and predicts the predicted feature map corresponding to each key point according to the human body image. The predicted feature map includes: a predicted heat map and a predicted displacement map.

[0163] Specifically, the structure of the key point recognition model 940 is as follows: Fig.10 The key point recognition model 940 includes: a feature extraction network 1010, a first convolution group 1020 and a second convolution group 1030. The output end of the feature extraction network 1010 is connected to the input end of the first convolution group 1020 and the input end of the second convolution group 1030 respectively.

[0164] The feature extraction network 1010 performs feature extraction on the human body image to obtain a feature matrix, and inputs the feature matrix into the first convolution group 1020 and the second convolution group 1030 respectively, so that the first convolution group 1020 performs convolution processing on the feature matrix to obtain a predicted heat map corresponding to each key point in the human body image. At the same time, the second convolution group 1030 performs convolution processing on the feature matrix to obtain a predicted displacement map corresponding to each key point in the human body image.

[0165] Furthermore, the feature extraction network 1010 includes but is not limited to the following network types: LeNet, AlexNet, VGG (Visual Geometry Group Network), GoogLeNet, Residual Network (ResNet) and MobileNet.

[0166] You can use w f ×hf ×3 human body image is input to the feature extraction network 1010, and the output (w f / σ)×(h f / σ)×512 feature matrix, where w f ×h f is the size of the human image, 3 is the number of channels of the human image, 512 is the number of feature matrices, σ is the preset reduction factor, the smaller σ is, the higher the network accuracy is. For example, σ can be 16. f / σ)×(h f / σ)×512 feature matrix is ​​input to the first convolution group 1020, and the first convolution group 1020 outputs n predicted heat maps; (w f / σ)×(h f / σ)×512 feature matrix is ​​input to the second convolution group 1030, and the second convolution group 1030 outputs n predicted displacement maps in the x-axis direction (predicted horizontal axis displacement map) and n predicted displacement maps in the y-axis direction (predicted vertical axis displacement map).

[0167] Step 5: The loss determination model receives the true heat map, mask horizontal axis displacement map and mask vertical axis displacement map corresponding to the n key points, and receives the predicted heat map, predicted horizontal axis displacement map and predicted vertical axis displacement map corresponding to the n key points output by the key point recognition model 940; for the same key point, a preset loss function is used to calculate the first loss value between the true heat map and the predicted heat map, the second loss value between the mask horizontal axis displacement map and the predicted horizontal axis displacement map, and the third loss value between the mask horizontal axis displacement map and the predicted horizontal axis displacement map. The weighted sum of the first loss value, the second loss value and the third loss value is determined as the comprehensive loss value of the image.

[0168] In this embodiment, the parameters in the key point recognition model 940 can be adjusted according to the image comprehensive loss value so that the key point recognition model 940 can output a more accurate prediction feature map. If it is necessary to continue training the key point recognition model 940, steps 1 to 5 can be repeated.

[0169] The embodiment of the present invention also provides a key point true value generating device. Fig.11 , which is a structural diagram of a key point true value generating device according to an embodiment of the present invention.

[0170] The key point true value generating device includes: an acquisition module 1110, an initialization module 1120, an assignment module 1130 and a generation module 1140.

[0171] The acquisition module 1110 is used to acquire the original image in which key points have been identified.

[0172] The initialization module 1120 is used to initialize the zero matrix corresponding to the key point.

[0173] The assignment module 1130 is used to assign values ​​to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix and the original image, and the type of the true value feature map to be generated.

[0174] The generation module 1140 is used to generate a true value feature map of the type corresponding to the key point according to the assigned zero matrix, so as to train a preset key point recognition model using the true value feature map generated according to the zero matrix; wherein the true value feature map of the type has the same pixel value range as the true value feature map of other types of the key point.

[0175] The functions of the device described in the embodiment of the present invention have been described in the above method embodiment. Therefore, for details not provided in the description of this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.

[0176] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a processor 1210 , a communication interface 1220 , a memory 1230 and a communication bus 1240 , wherein the processor 1210 , the communication interface 1220 , and the memory 1230 communicate with each other via the communication bus 1240 .

[0177] The memory 1230 is used to store computer programs.

[0178] The processor 1210, when used to execute the program stored in the memory 1230, implements the following steps: obtaining an original image in which key points have been identified; initializing a zero matrix corresponding to the key points; assigning values ​​to the zero matrix according to the position information of the key points in the original image, the size ratio of the zero matrix to the original image, and the type of true value feature map to be generated; generating a true value feature map of the type corresponding to the key points according to the assigned zero matrix, so as to train a preset key point recognition model using the true value feature map generated according to the zero matrix; wherein the true value feature map of the type has the same pixel value range as the true value feature map of other types of the key points.

[0179] Among them, when the true value feature image is a true value heat map, the zero matrix is ​​assigned according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image and the type of the true value feature map to be generated, including: determining the coordinates of the key point in the zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image; determining whether the pixel point to be assigned is within a preset range of the key point according to the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned in the zero matrix; if the pixel point to be assigned is within the preset range, assigning the pixel point to be assigned a first pixel value; otherwise, assigning the pixel point to be assigned a second pixel value.

[0180] Wherein, the first pixel value is:

[0181]

[0182] e is the natural base, d 2 is the square of the distance between the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned in the zero matrix; α represents the preset parameter value; r is the radius of the preset range centered on the key point.

[0183] Wherein, the true value feature image includes: a mask displacement map; the mask displacement map includes: a mask horizontal axis displacement map and a mask vertical axis displacement map; the initialization of the zero matrix corresponding to the key point includes: initializing the horizontal axis zero matrix corresponding to the key point, and initializing the vertical axis zero matrix corresponding to the key point; wherein, the horizontal axis zero matrix is ​​used to generate the mask horizontal axis displacement map, and the vertical axis zero matrix is ​​used to generate the mask vertical axis displacement map.

[0184] Among them, assigning values ​​to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image and the type of the true value feature map to be generated includes: determining the coordinates of the key point in the horizontal axis zero matrix and the coordinates of the key point in the vertical axis zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image; assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the horizontal coordinates of the key point in the original image, the size ratio of the horizontal axis zero matrix to the original image and the horizontal coordinates of the pixel points to be assigned in the horizontal axis zero matrix; assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the vertical coordinates of the key point in the original image, the size ratio of the vertical axis zero matrix to the original image and the vertical coordinates of the pixel points to be assigned in the vertical axis zero matrix.

[0185] The step of assigning values ​​to the pixel points to be assigned in the horizontal axis zero matrix includes: using the following formula to determine the values ​​of the pixel points to be assigned in the horizontal axis zero matrix: xab =a×sx k ; Among them, w xab is represented by the value of the pixel to be assigned in the horizontal axis zero matrix; a is the horizontal coordinate of the pixel to be assigned in the horizontal axis zero matrix; s represents that the size of the original image is s times the size of the horizontal axis zero matrix; x k is the horizontal coordinate of the key point in the original image; and / or, assigning values ​​to the pixel points to be assigned in the vertical axis zero matrix, including: using the following formula to determine the value of the pixel points to be assigned in the vertical axis zero matrix: yab =b×sy k ; Among them, w yab represents the value of the pixel to be assigned in the vertical axis zero matrix; b is the vertical coordinate of the pixel to be assigned in the vertical axis zero matrix; s represents that the size of the original image is s times the size of the vertical axis zero matrix; y k is the vertical coordinate of the key point in the original image.

[0186] Among them, the true value feature map corresponding to the key point is generated according to the zero matrix after the pixel points are assigned, including: generating the original horizontal axis displacement map corresponding to the key point according to the assigned horizontal axis zero matrix; generating the original vertical axis displacement map corresponding to the key point according to the assigned vertical axis zero matrix; generating a true value mask map corresponding to the key point; wherein the true value mask map, the horizontal axis zero matrix and the vertical axis zero matrix have the same size, the pixel points in the true value mask map within the preset range of the key point are assigned the third pixel value, and the remaining pixel points are assigned the fourth pixel value; according to the original horizontal axis displacement map and the true value mask map, a mask horizontal axis displacement map corresponding to the key point is generated; according to the original vertical axis displacement map and the true value mask map, a mask vertical axis displacement map corresponding to the key point is generated.

[0187] The step of generating the mask horizontal axis displacement map corresponding to the key point according to the original horizontal axis displacement map and the true value mask map includes: generating the mask horizontal axis displacement map by using the following formula:

[0188]

[0189] M mx Represents the horizontal axis displacement map of the mask, M h represents the true value mask map; M x Represents the original horizontal axis displacement diagram; max(M h ·M x) represents the maximum pixel value within the preset range of the key point; min(M h ·M x ) represents the minimum pixel value within the preset range of the key point;

[0190] And / or, generating the masked longitudinal axis displacement map corresponding to the key point according to the original longitudinal axis displacement map and the true value mask map, comprises:

[0191] The mask vertical axis displacement map is generated using the following formula:

[0192]

[0193] M my Represents the vertical axis displacement map of the mask, M h represents the true value mask map; M y Represents the original vertical axis displacement diagram; max(M h ·M y ) represents the maximum pixel value within the preset range of the key point; min(M h ·M y ) represents the minimum pixel value within the preset range of the key point.

[0194] The initializing the zero matrix corresponding to the key point includes: when initializing the zero matrix, making the ratio of the size of the zero matrix to the size of the original image a preset ratio; wherein the preset ratio is less than 1.

[0195] Wherein, before initializing the zero matrix corresponding to the key point, it also includes: reading the step parameter value in the key point recognition model, and taking the inverse of the step parameter value as the preset ratio; wherein the preset ratio is the ratio of the size of the zero matrix to the size of the original image; and determining the size of the zero matrix according to the preset ratio and the size of the original image.

[0196] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0197] The communication interface is used for communication between the above terminal and other devices.

[0198] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0199] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the key point true value generation method described in any of the above embodiments.

[0201] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the key point true value generation method described in any of the above embodiments.

[0202] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0203] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0204] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0205] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating true values ​​of key points, characterized in that: include: Get the original image with key points identified; Initialize the zero matrix corresponding to the key point; Assigning a value to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image, and the type of the true value feature map to be generated, wherein the true value feature map includes: a true value heat map and a mask displacement map; According to the assigned zero matrix, a true value feature map of the type corresponding to the key point is generated, so as to train a preset key point recognition model using the true value feature map generated according to the zero matrix; wherein the true value feature map of the type has the same pixel value range as the true value feature map of other types of the key point.

2. The method according to claim 1, characterized in that When the true value feature image is a true value heat map, assigning a value to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image, and the type of the true value feature map to be generated includes: Determine the coordinates of the key point in the zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image; Determine whether the pixel point to be assigned a value is within a preset range of the key point according to the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned a value in the zero matrix; If the pixel point to be assigned a value is within the preset range, the pixel point to be assigned a value is assigned a first pixel value; otherwise, the pixel point to be assigned a value is assigned a second pixel value.

3. The method according to claim 2, characterized in that The first pixel value is: Among them, e is the natural base, d 2 is the square of the distance between the coordinates of the key point in the zero matrix and the coordinates of the pixel point to be assigned in the zero matrix; α represents the preset parameter value; r is the radius of the preset range centered on the key point.

4. The method according to claim 1, characterized in that: The mask displacement map includes: a mask horizontal axis displacement map and a mask vertical axis displacement map; The initializing the zero matrix corresponding to the key point includes: Initialize the horizontal axis zero matrix corresponding to the key point, and initialize the vertical axis zero matrix corresponding to the key point; wherein the horizontal axis zero matrix is ​​used to generate a mask horizontal axis displacement map, and the vertical axis zero matrix is ​​used to generate a mask vertical axis displacement map.

5. The method according to claim 4, characterized in that The assigning of values ​​to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image, and the type of the true value feature map to be generated includes: Determine the coordinates of the key point in the horizontal axis zero matrix and the coordinates of the key point in the vertical axis zero matrix according to the coordinates of the key point in the original image and the size ratio of the zero matrix to the original image; Assign values ​​to the pixel points to be assigned in the horizontal axis zero matrix according to the horizontal coordinates of the key points in the original image, the size ratio of the horizontal axis zero matrix to the original image, and the horizontal coordinates of the pixel points to be assigned in the horizontal axis zero matrix; According to the vertical coordinate of the key point in the original image, the size ratio of the vertical axis zero matrix and the original image, and the vertical coordinate of the pixel point to be assigned in the vertical axis zero matrix, the pixel point to be assigned in the horizontal axis zero matrix is ​​assigned a value.

6. The method according to claim 5, characterized in that The step of assigning values ​​to the pixel points to be assigned values ​​in the horizontal axis zero matrix includes: The following formula is used to determine the value of the pixel to be assigned in the horizontal axis zero matrix: w xab =a×s-x k ; Among them, w xab is represented by the value of the pixel to be assigned in the horizontal axis zero matrix; a is the horizontal coordinate of the pixel to be assigned in the horizontal axis zero matrix; s represents that the size of the original image is s times the size of the horizontal axis zero matrix; x k is the horizontal coordinate of the key point in the original image; And / or, assigning values ​​to the pixel points to be assigned values ​​in the vertical axis zero matrix includes: The following formula is used to determine the value of the pixel to be assigned in the vertical axis zero matrix: w yab =b×s-y k ; Among them, w yab represents the value of the pixel to be assigned in the vertical axis zero matrix; b is the vertical coordinate of the pixel to be assigned in the vertical axis zero matrix; s represents that the size of the original image is s times the size of the vertical axis zero matrix; y k is the vertical coordinate of the key point in the original image.

7. The method according to claim 5, characterized in that The step of generating a true value feature map corresponding to the key point according to the zero matrix after the pixel points are assigned values ​​comprises: Generate an original horizontal axis displacement map corresponding to the key point according to the assigned horizontal axis zero matrix; generate an original vertical axis displacement map corresponding to the key point according to the assigned vertical axis zero matrix; Generate a true value mask map corresponding to the key point; wherein the true value mask map, the horizontal axis zero matrix and the vertical axis zero matrix have the same size, and the pixel points in the true value mask map within a preset range of the key point are assigned a third pixel value, and the remaining pixel points are assigned a fourth pixel value; According to the original horizontal axis displacement map and the true value mask map, a mask horizontal axis displacement map corresponding to the key point is generated; according to the original vertical axis displacement map and the true value mask map, a mask vertical axis displacement map corresponding to the key point is generated.

8. The method according to claim 7, characterized in that The step of generating a masked horizontal axis displacement map corresponding to the key point according to the original horizontal axis displacement map and the true value mask map comprises: The mask horizontal axis displacement map is generated using the following formula: Among them, M mx Represents the horizontal axis displacement map of the mask, M h represents the true value mask map; M x Represents the original horizontal axis displacement diagram; max(M h ·M x ) represents the maximum pixel value within the preset range of the key point; min(M h ·M x ) represents the minimum pixel value within the preset range of the key point; And / or, generating the masked longitudinal axis displacement map corresponding to the key point according to the original longitudinal axis displacement map and the true value mask map, comprises: The mask vertical axis displacement map is generated using the following formula: Among them, M my Represents the vertical axis displacement map of the mask, M h represents the true value mask map; M y Represents the original vertical axis displacement diagram; max(M h ·M y ) represents the maximum pixel value within the preset range of the key point; min(M h ·M y ) represents the minimum pixel value within the preset range of the key point.

9. The method according to any one of claims 1 to 8, characterized in that: Before initializing the zero matrix corresponding to the key point, the method further includes: Reading a step parameter value in the key point recognition model, and taking the reciprocal of the step parameter value as a preset ratio; wherein the preset ratio is a ratio of the size of the zero matrix to the size of the original image; The size of the zero matrix is ​​determined according to the preset ratio and the size of the original image.

10. A key point true value generating device, characterized in that: include: An acquisition module is used to acquire the original image with key points identified; An initialization module, used for initializing the zero matrix corresponding to the key point; An assignment module, used for assigning values ​​to the zero matrix according to the position information of the key point in the original image, the size ratio of the zero matrix to the original image, and the type of the true value feature map to be generated, wherein the true value feature map includes: a true value heat map and a mask displacement map; A generation module is used to generate a true value feature map of the type corresponding to the key point according to the assigned zero matrix, so as to train a preset key point recognition model using the true value feature map generated according to the zero matrix; wherein the true value feature map of the type has the same pixel value range as the true value feature map of other types of the key point.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images