Key point recognition model generation method and key point recognition method
By generating heat map sets and converting them into prediction vectors, combined with supervised vector training model, the problems of negative sample proportion imbalance and data exchange burden in CNN method are solved, the accuracy and efficiency of key point recognition are improved, and the image is achieved Identification of key points outside.
Patent Information
- Application Number
- CN202010836005.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-08-19
AI Technical Summary
The existing CNN method based on heat map regression has problems in the detection of key point, the large data exchange burden, the unbalanced negative sample proportion leads to limited training convergence speed and effect, and cannot handle key points outside the input image.
By generating a heat map set and converting it into a prediction vector, training the model with a supervised vector, equalizing the positive and negative sample ratios, and decoding using the prediction vector to improve the efficiency of key point recognition.
It improves the convergence speed and recognition accuracy of model training, reduces the computational burden of post-processing, and can effectively identify key points outside the image.
Smart Images

Figure CN111985556B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for generating a key point recognition model, a key point recognition method and device, a computer-readable storage medium, and an electronic device. Background Art
[0002] Key point detection technology refers to the technology of determining points with certain key features (such as the corners of the eyes, the tip of the nose, etc.) from various images such as face images and human body images. The current key point detection methods can be divided into: traditional methods, CNN (Convolutional Neural Networks) methods based on coordinate regression, CNN methods based on heatmap regression, etc. Summary of the invention
[0003] Embodiments of the present disclosure provide a method and device for generating a key point recognition model, a key point recognition method and device, a computer-readable storage medium, and an electronic device.
[0004] An embodiment of the present disclosure provides a method for generating a key point recognition model, the method comprising: generating a heat map set corresponding to a sample image based on a heat map generation module included in an initial model, wherein the heat map in the heat map set represents the position of a predicted key point on a target object, and the sample image has pre-labeled position information of key points in the key point set representing the target object; converting the heat map in the heat map set into a predicted vector based on a predicted vector conversion module included in the initial model; determining a supervision vector corresponding to the key point in the key point set based on the position information of the key point in the key point set; and training the initial model based on the predicted vector and the supervision vector to obtain a key point recognition model.
[0005] According to another aspect of an embodiment of the present disclosure, a key point recognition method is provided, the method comprising: acquiring an image to be recognized including a target object; inputting the image to be recognized into a heat map generation module included in a pre-trained key point recognition model to obtain a heat map set, wherein the heat maps in the heat map set represent the positions of predicted key points on the target object, and the key point recognition model is trained based on the generation method of the above-mentioned key point recognition model; based on a prediction vector conversion module included in the key point recognition model, converting the heat map in the heat map set into a prediction vector; based on a decoding module included in the key point recognition model, decoding the prediction vector to obtain the position information of the key points in the key point set corresponding to the image to be recognized.
[0006] According to another aspect of an embodiment of the present disclosure, a device for generating a key point recognition model is provided, the device comprising: a first generation module, for generating a set of heat maps corresponding to a sample image based on a heat map generation module included in an initial model, wherein the heat maps in the heat map set represent the positions of predicted key points on a target object, and the sample image has pre-labeled position information of key points in the set of key points representing the target object; a first conversion module, for converting the heat maps in the heat map set into prediction vectors based on a prediction vector conversion module included in the initial model; a determination module, for determining supervision vectors corresponding to key points in the set of key points based on the position information of the key points in the set of key points; and a training module, for training the initial model based on the prediction vectors and supervision vectors to obtain a key point recognition model.
[0007] According to another aspect of an embodiment of the present disclosure, a key point recognition device is provided, which includes: a second acquisition module, used to acquire an image to be recognized including a target object; a second generation module, used to input the image to be recognized into a heat map generation module included in a pre-trained key point recognition model to obtain a heat map set, wherein the heat maps in the heat map set represent the positions of predicted key points on the target object, and the key point recognition model is trained based on the generation method of the above-mentioned key point recognition model; a second conversion module, used to convert the heat maps in the heat map set into prediction vectors based on a prediction vector conversion module included in the key point recognition model; and a decoding module, used to decode the prediction vectors based on the decoding module included in the key point recognition model to obtain the position information of the key points in the key point set corresponding to the image to be recognized.
[0008] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores a computer program for executing the above-mentioned key point recognition model generation method or key point recognition method.
[0009] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; a processor, configured to read executable instructions from the memory and execute the instructions to implement the above-mentioned key point recognition model generation method or key point recognition method.
[0010] Based on the key point recognition model generation method and device, key point recognition method and device, computer-readable storage medium and electronic device provided by the above embodiments of the present disclosure, a heat map set is generated based on the heat map generation module included in the initial model, and then the heat map in the heat map set is converted into a prediction vector based on the prediction vector conversion module included in the initial model, and then the supervision vector corresponding to each key point is determined based on the position information of the key point in the key point set, and finally the initial model is trained based on the prediction vector and the supervision vector to obtain the key point recognition model, so that in the model training stage, the supervision vector is used to avoid the excessively high proportion of negative samples caused by the inclusion of a large number of background pixels in the heat map, so that the proportion of positive and negative samples is balanced, and the influence of negative samples on model training is reduced, thereby improving the convergence speed of model training and improving the recognition accuracy of the trained model. The key point recognition model outputs a vector. When the key point recognition model is used for key point recognition, the post-processing process of determining the key point coordinates is more efficient because the amount of data included in the vector is small.
[0011] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.
[0013] Figure 1 is a system diagram to which the present disclosure is applicable.
[0014] Figure 2 It is a flowchart of a method for generating a key point recognition model provided by an exemplary embodiment of the present disclosure.
[0015] Figure 3 It is a flowchart of a method for generating a key point recognition model provided by another exemplary embodiment of the present disclosure.
[0016] Figure 4 It is an exemplary schematic diagram of the dimensionality reduction operation of the method for generating a key point recognition model provided by an exemplary embodiment of the present disclosure.
[0017] Figure 5 It is a flowchart of a key point identification method provided by an exemplary embodiment of the present disclosure.
[0018] Figure 6It is a flowchart of a key point identification method provided by another exemplary embodiment of the present disclosure.
[0019] Figure 7 It is an exemplary schematic diagram of determining key points outside the range of an image to be recognized in a key point recognition method provided by an exemplary embodiment of the present disclosure.
[0020] Figure 8 It is a structural diagram of a device for generating a key point recognition model provided by an exemplary embodiment of the present disclosure.
[0021] Fig. 9 It is a structural schematic diagram of a device for generating a key point recognition model provided by another exemplary embodiment of the present disclosure.
[0022] Fig.10 It is a structural schematic diagram of a key point identification device provided by an exemplary embodiment of the present disclosure.
[0023] Fig.11 It is a structural schematic diagram of a key point identification device provided by another exemplary embodiment of the present disclosure.
[0024] Fig.12 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0026] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0027] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0028] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0029] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0030] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.
[0031] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0032] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0034] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0035] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0036] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0037] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.
[0038] Application Overview
[0039] The CNN method based on heatmap regression has the following problems:
[0040] 1) The network outputs a two-dimensional matrix (heatmap) for each key point. In embedded systems, the data exchange of a large number of heatmaps and the post-processing process of parsing the key point coordinates from the heatmap will cause a huge burden on the processor.
[0041] 2) The positive and negative samples in the heatmap labels are extremely unbalanced, with negative samples (background pixels) accounting for the vast majority. Therefore, the optimization of negative samples dominates during training, which restricts the convergence speed and effect of training.
[0042] 3) The final result obtained by this method is limited to the image range of the input network and cannot process key points outside the input image.
[0043] To solve the above problems, for problem 1), operations such as soft-argmax and integral regression can be used to convert the heatmap output into coordinate point output, simplify post-processing, and reduce the burden of data transmission. However, other complex operations (such as element-by-element exponential operations) are usually introduced in the above, which will also increase the computational burden of the embedded system. Moreover, it does not solve the problems of imbalance of positive and negative samples in the heatmap and inability to process key points outside the input image.
[0044] Exemplary Systems
[0045] Figure 1 An exemplary system architecture 100 is shown to which a method and apparatus for generating a key point recognition model or a method and apparatus for recognizing a key point according to an embodiment of the present disclosure can be applied.
[0046] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0047] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as shooting applications, video applications, search applications, web browser applications, instant messaging tools, etc.
[0048] The terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.
[0049] The server 103 may be a server that provides various services, such as a background model training server that performs model training using sample images uploaded by the terminal device 101. The background model training server may process the received sample images to obtain heat maps, prediction vectors, supervision vectors, etc., and train a key point recognition model using the prediction vectors and supervision vectors. The server 103 may also be a background key point recognition server that performs key point recognition on the image to be recognized uploaded by the terminal device 101. The background key point recognition server may use the key point recognition model to recognize the image to be recognized and obtain the location information of each key point.
[0050] It should be noted that the key point recognition model generation method or key point recognition method provided in the embodiments of the present disclosure can be executed by the server 103 or by the terminal device 101. Accordingly, the key point recognition model generation device or key point recognition device can be set in the server 103 or in the terminal device 101.
[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. When sample images, images to be identified, etc. do not need to be acquired remotely, the above system architecture may not include a network, but only include a server or a terminal device.
[0052] Exemplary Methods
[0053] Figure 2is a flow chart of a method for generating a key point recognition model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices (such as Figure 1 On the terminal device 101 or server 103 as shown, Figure 2 As shown, the method comprises the following steps:
[0054] Step 201: Generate a set of heat maps corresponding to the sample image based on the heat map generation module included in the initial model.
[0055] In this embodiment, the electronic device can generate a set of heat maps corresponding to the sample image based on the heat map generation module included in the initial model. The heat maps in the heat map set represent the positions of predicted key points on the target object, and the sample image has pre-annotated position information of key points in the set of key points representing the target object.
[0056] The above-mentioned heat map generation module can be implemented using various existing deep neural network structures. For example, the heat map generation module may include a downsampling submodule and an upsampling submodule. The downsampling submodule may include but is not limited to at least one of the networks such as ResNet and MobileNet. The upsampling submodule may include but is not limited to at least one of the networks such as FPN (Feature Pyramid Networks) and Deconvolution. The downsampling module and the upsampling module are used together to extract the spatial features of the input image, and a heat map can be obtained based on the spatial features. It should be noted that the method for generating a heat map is the current prior art and will not be repeated here.
[0057] Each heat map in the above heat map set corresponds to a predicted key point, each pixel in the heat map corresponds to a position in the input sample image, and one heat map corresponds to one key point. The size of the heat map and the input sample image can be the same or different, for example, the width ratio and height ratio of the heat map to the sample image are in a preset ratio (e.g., 1:4).
[0058] The above-mentioned target object can be an image of various objects, such as images of human faces, human bodies, animals, etc. The above-mentioned key point set is a set of key points related to the target object, and the key points are used to characterize special points related to the target object. The key points can be points representing a specific part of the target object, such as points at the corners of the eyes, corners of the mouth, and tip of the nose on the face, and points at the shoulder joints, elbow joints, and wrist joints of the human body. The key points can also be points at specific positions outside the target object, for example, the key points can be the upper left corner and lower right corner of the detection frame of the face. Each key point corresponds to a position information characterizing its position in the sample image. Usually, the position information includes two coordinate values, namely, the coordinate values in the width direction (x direction) and the height direction (y direction).
[0059] Step 202: based on the prediction vector conversion module included in the initial model, convert the heat map in the heat map set into a prediction vector.
[0060] In this embodiment, the electronic device may convert the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the initial model.
[0061] Among them, the above-mentioned prediction vector conversion module is used to convert each heat map into a corresponding prediction vector. As an example, the prediction vector conversion module may include a preset formula, through which the data of each pixel point included in the heat map can be integrated (for example, averaging each column or row of data) to obtain a prediction vector. Alternatively, the prediction vector conversion module may include a correspondence table that characterizes the correspondence between the heat map and the prediction vector. The correspondence table can be obtained in advance by counting a large number of heat maps and corresponding prediction vectors. The electronic device can search the corresponding table for the prediction vector corresponding to each heat map in the heat map set.
[0062] The above prediction vector is a vector (also called vector) including multiple elements, each element corresponding to an area in the input sample image. For example, the prediction vector may include a first prediction vector and a second prediction vector, each element in the first prediction vector corresponds to a column (or multiple columns) of pixels in the sample image, and the value of the element indicates the probability of the presence of a key point in the corresponding column (or multiple columns). Correspondingly, each element in the second prediction vector corresponds to a row (or multiple rows) of pixels in the sample image, and the value of the element indicates the probability of the presence of a key point in the corresponding row (or multiple rows).
[0063] Step 203: Determine the supervision vector corresponding to the key point in the key point set based on the position information of the key point in the key point set.
[0064] In this embodiment, the electronic device may determine the supervision vector corresponding to the key point in the key point set based on the position information of the key point in the key point set.
[0065] Among them, the supervision vector has the same dimension as the prediction vector, and the supervision vector is used to represent the position of the real key point in the sample image.
[0066] As an example, the supervision vector may be manually set by a technician using an electronic device, or may be converted by the electronic device from the position information of the key point according to a preset formula.
[0067] Step 204: Based on the prediction vector and the supervision vector, the initial model is trained to obtain a key point recognition model.
[0068] In this embodiment, the electronic device can train the initial model based on the prediction vector and the supervision vector to obtain a key point recognition model.
[0069] Specifically, the electronic device can use machine learning methods to train the initial model by taking the prediction vectors corresponding to each heat map as the actual output and the supervision vectors corresponding to each prediction vector as the expected output. During training, the electronic device can use the gradient descent method and the back propagation method to adjust the parameters of the initial model based on the actual output and the expected output, so as to reduce the gap between the actual output and the expected output, and use the model obtained after each parameter adjustment as the initial model for the next training, and end the training when the preset training end conditions are met, so as to obtain the key point recognition model through training.
[0070] It should be noted that the number of sample images in this embodiment may be at least one, and each time a sample image is used to optimize the parameters of the initial model, it is considered as one training, and multiple trainings are performed until the training end condition is met. The preset training end condition here may include but is not limited to at least one of the following: the training time exceeds the preset duration; the number of training times exceeds the preset number of times; the loss value calculated using the preset loss function (such as the regression loss function) is less than the preset loss value threshold.
[0071] In practical applications, an image containing a target object can be input into a trained key point recognition model to obtain a prediction vector, which is then decoded by a decoding module included in the key point recognition model to obtain a predicted key point set.
[0072] The method provided by the above-mentioned embodiment of the present disclosure generates a heat map set based on a heat map generation module included in the initial model, and then converts the heat maps in the heat map set into prediction vectors based on the prediction vector conversion module included in the initial model, and then determines the supervision vector corresponding to each key point based on the position information of the key points in the key point set, and finally trains the initial model based on the prediction vector and the supervision vector to obtain a key point recognition model. Therefore, in the model training stage, the supervision vector is used to avoid the excessively high proportion of negative samples in the heat map due to the presence of a large number of background pixels, so that the ratio of positive and negative samples is balanced. Since the influence of negative samples on model training is reduced, the convergence speed of model training is improved, and the recognition accuracy of the trained model is improved.
[0073] In some optional implementations, before the above step 201, the electronic device may further perform the following steps:
[0074] First, an original sample image is obtained. The original sample image may be an image that includes the target object and has not been processed. For example, the original sample image may be an image directly obtained by photographing the target object.
[0075] Then, the area including the target object is extracted from the original sample image as a sample image. Specifically, the electronic device can use the existing target detection method to determine the area where the target object is located from the original sample image, and extract the area as a sample image. For example, an existing target detection network (such as SSD (Single Shot MultiBox Detector), DPM (Deformable Part Model), RPN (Region Proposal Networks), etc.) can be used to perform target detection on the original sample image. The target detection network can determine the position of the target object from the image input therein. Typically, the target detection network outputs coordinate information, which can characterize the area in the original sample image. For example, the coordinate information may include two diagonal coordinates of a rectangular box. Through the two diagonal coordinates, a rectangular box can be determined in the original sample image, and the area covered by the rectangular box is the sample image.
[0076] This implementation method first performs target detection on the original sample image, so that the obtained sample image mainly includes the target object, thereby reducing the influence of the background image on key point recognition and helping to improve the accuracy of model training.
[0077] Further references Figure 3 , shows a flow chart of another embodiment of a method for generating a key point recognition model. Figure 3 As shown, the method for generating a key point recognition model may include the following steps:
[0078] Step 301: Generate a set of heat maps corresponding to the sample image based on the heat map generation module included in the initial model.
[0079] In this embodiment, step 301 and Figure 2 Step 201 in the corresponding embodiment is basically the same and will not be described again here.
[0080] Step 302: for each heat map in the heat map set, perform dimensionality reduction operations on the heat map in a first direction and a second direction to obtain a first prediction vector and a second prediction vector corresponding to the heat map, respectively.
[0081] The first direction and the second direction are only used to determine the directions of the position definition in the sample image and the heat map, and the actual directions are not clearly defined. As an example, the first direction and the second direction can be the width direction (x direction) and the height direction (y direction) of the sample image, respectively. The dimensionality reduction operation is used to convert the matrix into a vector, and various methods can be used to perform the dimensionality reduction operation, such as through pooling operations or convolution operations.
[0082] Specifically, as an example, assume that the size of the heat map is H×W, H represents the pixel height of the heat map in the vertical direction, and W represents the pixel width of the heat map in the horizontal direction. Use an average pooling kernel of size H×b, and use an average pooling operation with stride=1 to convert the heat map into a 1×W vector as the first prediction vector; similarly, use an average pooling kernel of size b×W to generate an H×1 vector as the second prediction vector. Among them, b is a parameter, which is usually an odd number much smaller than H and W in order to ensure the symmetry of the pooling operation conversion.
[0083] In some optional implementations, step 302 may be performed as follows:
[0084] First, a preset number of group dimensionality reduction kernels are obtained.
[0085] Each set of dimensionality reduction cores in the preset number of groups of dimensionality reduction cores includes a first dimensionality reduction core corresponding to the first direction and a second dimensionality reduction core corresponding to the second direction. As an example, when dimensionality reduction is performed by pooling operation, the dimensionality reduction core is a pooling core, and when dimensionality reduction is performed by convolution, the dimensionality reduction core is a convolution core. Figure 4 As shown, the preset number is 2, the size of the first dimensionality reduction kernel in the first group (the pooling kernel shown by K11 in the figure) can be H×3, and the size of the second dimensionality reduction kernel in the first group (as shown by K12 in the figure) can be 3×W; the size of the first dimensionality reduction kernel in the second group (as shown by K21 in the figure) can be H×5, and the size of the second dimensionality reduction kernel in the second group (as shown by K22 in the figure) can be 5×W.
[0086] Then, based on a preset number of groups of dimensionality reduction kernels, a dimensionality reduction operation is performed on the heat map to obtain a first sub-prediction vector and a second sub-prediction vector corresponding to each group of dimensionality reduction kernels in the preset number of groups of dimensionality reduction kernels.
[0087] like Figure 4 As shown, K11 and K21 slide in the horizontal direction to perform pooling operations in the horizontal direction to obtain first sub-prediction vectors A11 and A12 in the horizontal direction; K12 and K22 slide in the vertical direction to perform pooling operations in the vertical direction to obtain second sub-prediction vectors A21 and A22 in the vertical direction.
[0088] Finally, based on the preset weights corresponding to each group of dimensionality reduction kernels, weighted sums are performed on each first sub-prediction vector and each second sub-prediction vector to obtain the first prediction vector and the second prediction vector corresponding to the heat map.
[0089] Among them, the sum of the preset weights corresponding to each group of dimensionality reduction kernels is 1, such as Figure 4 As shown, the preset weight corresponding to the first set of dimensionality reduction kernels is α, and the preset weight corresponding to the second set of dimensionality reduction kernels is 1-α. After the weighted sum of the elements in A11 and the corresponding elements in A12, the first prediction vector A1 is obtained; after the weighted sum of the elements in A21 and the corresponding elements in A22, the first prediction vector A2 is obtained.
[0090] This implementation method performs dimensionality reduction operations by using a preset number of groups of dimensionality reduction kernels, and performs weighted summation on the generated first sub-prediction vectors and second sub-prediction vectors respectively. The preset weights can be used to specifically concentrate the pixel range of the dimensionality reduction operation to a certain size, and take into account pixel ranges of other sizes, thereby improving the accuracy and flexibility of determining the prediction vector.
[0091] Step 303: Determine a first supervisory vector and a second supervisory vector corresponding to the key point in the key point set based on the coordinates of the first direction and the second direction included in the position information of the key point in the key point set.
[0092] As an example, the first supervisory vector and the second supervisory vector may be obtained by converting the coordinates of each key point in the first direction and the second direction into corresponding preset formulas by the electronic device.
[0093] In some optional implementations, step 303 may be performed as follows:
[0094] For each keypoint in the keypoint set, perform the following steps:
[0095] Step 1: Determine the coordinates on the heat map corresponding to the key point based on the coordinates in the first direction and the coordinates in the second direction included in the position information of the key point.
[0096] Specifically, as an example, the coordinates on the heat map can be determined according to the following formula (1):
[0097]
[0098] Among them, x′, y′ are the coordinates of the first direction and the second direction on the heat map, respectively, x, y are the coordinates of the first direction and the second direction of the marked key point, respectively, λ is the ratio of the size of the input sample image to the heat map (for example, 4), and int() means rounding the values in the brackets.
[0099] Step 2: Determine the first distance between each element in the first supervision vector corresponding to the key point and the first coordinate on the heat map corresponding to the key point, and determine the second distance between each element in the second supervision vector corresponding to the key point and the second coordinate on the heat map corresponding to the key point.
[0100] As an example, taking the first supervision vector as an example, for the coordinate x of the element at a certain position i in the first supervision vector i (The length of the first supervision vector is the same as the width of the heat map, where x i It can be considered as x on the heat map i ), the distance between the element and the first coordinate (i.e., x′) corresponding to the key point on the heat map for:
[0101]
[0102] The distance calculated by formula (2) is the first distance. For the second supervision vector, the same method is used to calculate the second distance between each element and the first coordinate (ie, the above y′) corresponding to the key point on the heat map.
[0103] It should be noted that the elements in the first supervisory vector and the second supervisory vector at this time are not the final values. Since the distance is determined only by the position in the supervisory vector, the elements in the first supervisory vector and the second supervisory vector in this step can be any initial values.
[0104] Step three: Based on the first distance and the second distance, determine the first supervision vector and the second supervision vector corresponding to the key point.
[0105] As an example, the first distance and the second distance may be processed using a Gaussian function, and the first supervision vector corresponding to the x direction may be obtained by the following formula (3):
[0106]
[0107] in, is the Gaussian formula, σ is the standard deviation. For the second supervisory vector g corresponding to the y direction y , can be obtained using the same method.
[0108] By executing the above steps 1 to 3 for each key point in the key point set, a first supervision vector and a second supervision vector corresponding to each key point are finally obtained.
[0109] This implementation method determines the distance between each element in the first supervision vector and the second supervision vector and the first coordinate and the second coordinate corresponding to the key point on the heat map, and determines the first supervision vector and the second supervision vector based on the distance. This can accurately map the annotated key points to the first supervision vector and the second supervision vector, so that the generated first supervision vector and the second supervision vector accurately represent the actual key points, which helps to improve the accuracy of model training.
[0110] In some optional implementations, the above step three may include the following sub-steps:
[0111] The supervisory vector elements corresponding to the shortest distance among the obtained first distances and second distances are enhanced to obtain a first supervisory vector and a second supervisory vector.
[0112] Specifically, taking the above formula (3) as an example, the supervision vector elements can be enhanced according to the following formula (4):
[0113]
[0114] Among them, θ>0, the maximum value element in the supervision vector can be enhanced, which can make the model more focused on learning the correct key point position, thereby improving the recognition accuracy of the trained model.
[0115] Step 304: Based on the prediction vector and the supervision vector, the initial model is trained to obtain a key point recognition model.
[0116] In this embodiment, step 304 and Figure 2 Step 204 in the corresponding embodiment is basically the same and will not be described again here.
[0117] Above Figure 3The method provided in the corresponding embodiment can obtain the first prediction vector, the second prediction vector, the first supervision vector, and the second supervision vector by performing dimensionality reduction operations on each heat map in the first direction and the second direction, as well as the coordinates of each marked key point in the first direction and the second direction. Therefore, during training, the corresponding model parameters can be optimized for different directions to improve the recognition accuracy of the trained model.
[0118] Figure 5 is a flowchart of a key point recognition method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices (such as Figure 1 On the terminal device 101 or server 103 shown in FIG. Figure 5 As shown, the method comprises the following steps:
[0119] Step 501: Acquire an image to be recognized including a target object.
[0120] In this embodiment, the electronic device can obtain the image to be identified including the target object locally or remotely. The target object here can be the same as the above Figure 2 The target objects described in the corresponding embodiments are the same. For the description of the target objects, please refer to the above Figure 2 The corresponding embodiments are not described in detail here.
[0121] Step 502: input the image to be recognized into a heat map generation module included in a pre-trained key point recognition model to obtain a heat map set.
[0122] In this embodiment, the electronic device can input the image to be recognized into the heat map generation module included in the pre-trained key point recognition model to obtain a heat map set. The heat map in the heat map set represents the position of the predicted key point on the target object. The key point recognition model is based on the above Figure 2 The method described in the corresponding embodiment is trained. For the description of the heat map generation module, please refer to the above Figure 2 The corresponding embodiments are not described in detail here.
[0123] Step 503: based on the prediction vector conversion module included in the key point recognition model, the heat map in the heat map set is converted into a prediction vector.
[0124] In this embodiment, the electronic device can convert the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the key point recognition model. For the description of the prediction vector conversion module, please refer to the above Figure 2 The corresponding embodiments are not described in detail here.
[0125] Step 504: Based on the decoding module included in the key point recognition model, the prediction vector is decoded to obtain the position information of the key points in the key point set corresponding to the image to be recognized.
[0126] In this embodiment, the electronic device can decode the prediction vector based on the decoding module included in the key point recognition model to obtain the position information of the key points in the key point set corresponding to the image to be recognized. The decoding module is used to perform calculations using the prediction vector to obtain the position information of the key points.
[0127] The decoding module can perform reverse operations on the prediction vector to restore the elements of the key points marked in the prediction vector to the coordinates in the image to be recognized.
[0128] The method provided by the above embodiment of the present disclosure is to utilize the above Figure 2 The key point recognition model described in the corresponding embodiment recognizes the image to be recognized and obtains the position information of each key point on the target object in the image to be recognized, thereby locating the key points using the prediction vector output by the key point recognition model. Since the prediction vector is a vector, when the key point recognition model is used for key point recognition, the amount of data included in the vector is small. Therefore, compared with the two-dimensional matrix, the post-processing process of determining the key point coordinates is more efficient.
[0129] In some optional implementations, before the above step 501, the electronic device may further perform the following steps:
[0130] First, an original image is obtained. The original image may be an image that includes the target object and has not been processed. For example, the original image may be an image directly obtained by photographing the target object.
[0131] Then, the area including the target object is extracted from the original image as the image to be identified. Specifically, the electronic device can use the existing target detection method to determine the area where the target object is located from the original image, and extract the area as the image to be identified. Usually, the target detection network outputs coordinate information, and the coordinate information can characterize the area in the original image. For example, the coordinate information may include two diagonal coordinates of a rectangular box. Through the two diagonal coordinates, a rectangular box can be determined in the original image, and the area covered by the rectangular box is the image to be identified.
[0132] This implementation method first performs target detection on the original image, so that the obtained image to be identified mainly includes the target object, thereby reducing the influence of the background image on the key point recognition, which helps to improve the accuracy of identifying the key points.
[0133] Further references Figure 6 , shows a flow chart of another embodiment of a key point identification method. Figure 6 As shown in the above Figure 5 Based on the corresponding embodiment, step 504 can be performed as follows:
[0134] For each of the obtained prediction vectors, perform the following steps:
[0135] Step 5041, in the heat map corresponding to the prediction vector, determine the coordinates corresponding to the maximum value of the elements in the prediction vector.
[0136] Specifically, as an example, for the prediction vector, an argmax operation can be performed to obtain the coordinates corresponding to the maximum value of the elements in the prediction vector. For example, the coordinates corresponding to the maximum value are (pos x ,pos y ).
[0137] Step 5042, determine whether the position indicated by the coordinates is on the boundary of the prediction vector.
[0138] The boundary of the prediction vector is the first and last elements in the prediction vector. Usually, the prediction vector includes a first prediction vector in the x direction and a second prediction vector in the y direction. If the size of the first prediction vector is 1×W, the boundary of the first prediction vector is 0 and W-1. If the size of the second prediction vector is H×1, the boundary of the first prediction vector is 0 and H-1.
[0139] Step 5043: If it is on the boundary, perform peak estimation based on the coordinates to obtain the peak coordinates.
[0140] Specifically, if the coordinates corresponding to the above maximum value are on the boundary, the peak position of the predicted vector distribution may be outside the range of the predicted vector, and the actual position of the key point may be outside the range of the image of the above input model. It should be noted that the peak value here and the above maximum value have different meanings. The above maximum value refers to the maximum value of the actual output in the predicted vector, and the peak value refers to the maximum value of the distribution obeyed by the predicted vector.
[0141] As an example, assume that the output prediction vector approximately obeys the Gaussian distribution (because the supervision vector is approximately Gaussian), and the distribution is truncated at the boundary position of the prediction vector, that is, the position corresponding to the maximum value s = 0 or s = W-1, so that the maximum value of the prediction vector is not the maximum value of the Gaussian distribution. So assume that the value of the s position approximately obeys the following formula:
[0142]
[0143] Where τ is a scale parameter and σ is the standard deviation of the Gaussian distribution. These two parameters can be selected according to the actual situation of the model. Generally, σ can be consistent with formula (3). And d is the distance between the current position s and the peak value of the ideal distribution (the maximum value of the Gaussian distribution). Therefore, the peak value of the ideal distribution is calculated according to formula (4):
[0144]
[0145] For the second prediction vector in the y direction, the peak point in the y direction may be determined in the same way.
[0146] Step 5044: Based on the peak coordinates, determine the position information of the key point corresponding to the prediction vector in the image to be identified.
[0147] Specifically, the position information of the key point can be determined according to the proportional relationship between the heat map corresponding to the prediction vector and the image to be identified. For example, the coordinates obtained in step 5043 can be mapped back to the image to be identified according to the following formula (6):
[0148] coord x =loc*λ (6)
[0149] Among them, λ is the ratio of the size of the image to be identified to the size of the heat map.
[0150] It should be noted that when the image to be identified is extracted from the original image, the coordinates of the key points in the original image can be determined according to the following formula (7):
[0151] cOord x =loc*λ+offset x (7)
[0152] Among them, offset x is the offset of the image to be recognized relative to the original image.
[0153] For the coordinate coord in the y direction y , can be obtained in the same way.
[0154] like Figure 7 As shown, the range of the image to be identified is shown as 701 in the figure. By using the above steps 5041 to 5044, key points 702 can be determined outside the range of the image to be identified.
[0155] By executing steps 5041 to 5044 for each of the prediction vectors, the position information of the key point corresponding to each prediction vector can be obtained.
[0156] Above Figure 6According to the method provided in the corresponding embodiment, when the maximum value of the elements in the prediction vector is on the boundary, the actual position of the key point can be determined by peak estimation, so that the key point outside the range of the input image can be determined.
[0157] In some optional implementations, after step 5042, if the coordinates corresponding to the maximum value of the elements in the prediction vector are not on the boundary, the electronic device can determine the position information of the key point corresponding to the prediction vector in the image to be identified based on the coordinates corresponding to the maximum value. Specifically, the above coordinates can be mapped to the image to be identified based on the proportional relationship between the heat map and the size of the image to be identified to obtain the position information of the key point. This implementation method can efficiently determine the position information of the key point in the image to be identified by directly performing coordinate mapping when judging that the coordinate corresponding to the maximum value is on the boundary of the step.
[0158] In some optional implementations, if the coordinates corresponding to the maximum value of the elements in the prediction vector are not on the boundary, the electronic device may determine the position information of the key point corresponding to the prediction vector in the image to be recognized according to the following steps:
[0159] First, the second largest value is determined from among the elements included in the prediction vector.
[0160] Then, based on the positional relationship between the coordinates corresponding to the second largest value and the coordinates corresponding to the maximum value, an offset correction is performed to obtain the position information of the key point corresponding to the prediction vector in the image to be identified.
[0161] Generally, in the process of key point detection, sub-pixel technology can be used to detect key points with sub-pixel accuracy. As an example, based on the existing heatmapencode method, offset correction can be performed according to the following formula (8):
[0162]
[0163] Among them, 0.25 is a common value for offset correction using sub-pixel accuracy, snd x Indicates the second largest value. For the offset correction in the y direction, loc can be obtained in the same way. y .
[0164] This implementation method can more accurately determine the position of the key point by performing an offset correction on the coordinates when the coordinates of the maximum value are not on the boundary, thereby improving the accuracy of key point positioning using the key point recognition model.
[0165] Exemplary Devices
[0166] Figure 8is a schematic diagram of a structure of a device for generating a key point recognition model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 8 As shown, the generation device of the key point recognition model includes: a first generation module 801, which is used to generate a heat map set corresponding to the sample image based on the heat map generation module included in the initial model, wherein the heat map in the heat map set represents the position of the predicted key point on the target object, and the sample image has pre-labeled position information of the key point in the key point set representing the target object; a first conversion module 802, which is used to convert the heat map in the heat map set into a predicted vector based on the predicted vector conversion module included in the initial model; a determination module 803, which is used to determine the supervision vector corresponding to the key point in the key point set based on the position information of the key point in the key point set; a training module 804, which is used to train the initial model based on the predicted vector and the supervision vector to obtain the key point recognition model.
[0167] In this embodiment, the first generation module 801 can generate a set of heat maps corresponding to the sample image based on the heat map generation module included in the initial model. The heat maps in the heat map set represent the positions of predicted key points on the target object, and the sample image has pre-annotated position information of key points in the set of key points representing the target object.
[0168] The above-mentioned heat map generation module can be implemented using various existing deep neural network structures. For example, the heat map generation module may include a downsampling submodule and an upsampling submodule. The downsampling submodule may include but is not limited to at least one of the networks such as ResNet and MobileNet. The upsampling submodule may include but is not limited to at least one of the networks such as FPN (Feature Pyramid Networks) and Deconvolution. The downsampling module and the upsampling module are used together to extract the spatial features of the input image, and a heat map can be obtained based on the spatial features. It should be noted that the method for generating a heat map is the current prior art and will not be repeated here.
[0169] Each heat map in the above heat map set corresponds to a predicted key point, each pixel in the heat map corresponds to a position in the input sample image, and a heat map usually corresponds to a key point. The size of the heat map and the input sample image can be the same or different, for example, the width ratio and height ratio of the heat map to the sample image are in a preset ratio (e.g., 1:4).
[0170] The above-mentioned target object can be an image of various objects, such as images of human faces, human bodies, animals, etc. The above-mentioned key point set is a set of key points related to the target object, and the key points are used to characterize special points related to the target object. The key points can be points representing a specific part of the target object, such as points at the corners of the eyes, corners of the mouth, and tip of the nose on the face, and points at the shoulder joints, elbow joints, and wrist joints of the human body. The key points can also be points at specific positions outside the target object, for example, the key points can be the upper left corner and lower right corner of the detection frame of the face. Each key point corresponds to a position information characterizing its position in the sample image. Usually, the position information includes two coordinate values, namely, the coordinate values in the width direction (x direction) and the height direction (y direction).
[0171] In this embodiment, the first conversion module 802 may convert the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the initial model.
[0172] Among them, the above-mentioned prediction vector conversion module is used to convert each heat map into a corresponding prediction vector. As an example, the prediction vector conversion module may include a preset formula, through which the data of each pixel point included in the heat map can be integrated to obtain a prediction vector. Alternatively, the prediction vector conversion module may include a correspondence table that characterizes the correspondence between the heat map and the prediction vector. The correspondence table can be obtained in advance by counting a large number of heat maps and corresponding prediction vectors. The first conversion module 802 can search the corresponding relationship table for the prediction vector corresponding to each heat map in the heat map set.
[0173] The above prediction vector is a vector (also called vector) including multiple elements, each element corresponding to an area in the input sample image. For example, the prediction vector may include a first prediction vector and a second prediction vector, each element in the first prediction vector corresponds to a column (or multiple columns) of pixels in the sample image, and the value of the element indicates the probability of the presence of a key point in the corresponding column (or multiple columns). Correspondingly, each element in the second prediction vector corresponds to a row (or multiple rows) of pixels in the sample image, and the value of the element indicates the probability of the presence of a key point in the corresponding row (or multiple rows).
[0174] In this embodiment, the determination module 803 may determine the supervision vector corresponding to the key point in the key point set based on the position information of the key point in the key point set.
[0175] Among them, the supervision vector has the same dimension as the prediction vector, and the supervision vector is used to represent the position of the real key point in the sample image.
[0176] As an example, the supervision vector may be manually set by a technician using an electronic device, or may be converted by the electronic device from the position information of the key point according to a preset formula.
[0177] In this embodiment, the training module 804 can train the initial model based on the prediction vector and the supervision vector to obtain a key point recognition model.
[0178] Specifically, the training module 804 can use machine learning methods to train the initial model by taking the prediction vectors corresponding to each heat map as the actual output and the supervision vectors corresponding to each prediction vector as the expected output. During training, the training module 804 can use the gradient descent method and the back propagation method to adjust the parameters of the initial model based on the actual output and the expected output, so that the gap between the actual output and the expected output is reduced, and the model obtained after each parameter adjustment is used as the initial model for the next training, and the training is terminated when the preset training termination conditions are met, so as to obtain the key point recognition model through training.
[0179] It should be noted that the number of sample images in this embodiment may be at least one, and each time a sample image is used to optimize the parameters of the initial model, it is considered as one training, and multiple trainings are performed until the training end condition is met. The preset training end condition here may include but is not limited to at least one of the following: the training time exceeds the preset duration; the number of training times exceeds the preset number of times; the loss value calculated using the preset loss function (such as the regression loss function) is less than the preset loss value threshold.
[0180] In practical applications, an image containing a target object can be input into a trained key point recognition model to obtain a prediction vector, which is then decoded by a decoding module included in the key point recognition model to obtain a predicted key point set.
[0181] Reference Fig. 9 , Fig. 9 It is a structural schematic diagram of a device for generating a key point recognition model provided by another exemplary embodiment of the present disclosure.
[0182] In some optional implementations, the first conversion module 802 can be further used to: for each heat map in the heat map set, perform dimensionality reduction operations on the heat map in the first direction and the second direction to obtain the first prediction vector and the second prediction vector corresponding to the heat map, respectively; the determination module is further used to: determine the first supervision vector and the second supervision vector corresponding to the key point in the key point set based on the coordinates in the first direction and the second direction included in the position information of the key point in the key point set.
[0183] In some optional implementations, the first conversion module 802 may include: a first acquisition unit 8021, used to acquire a preset number of groups of dimensionality reduction kernels, wherein each group of dimensionality reduction kernels in the preset number of groups of dimensionality reduction kernels includes a first dimensionality reduction kernel corresponding to a first direction and a second dimensionality reduction kernel corresponding to a second direction; a dimensionality reduction unit 8022, used to perform a dimensionality reduction operation on the heat map based on the preset number of groups of dimensionality reduction kernels, and obtain a first sub-prediction vector and a second sub-prediction vector corresponding to each group of dimensionality reduction kernels in the preset number of groups of dimensionality reduction kernels; a calculation unit 8023, used to perform a weighted summation of each first sub-prediction vector and each second sub-prediction vector based on a preset weight corresponding to each group of dimensionality reduction kernels, and obtain a first prediction vector and a second prediction vector corresponding to the heat map.
[0184] In some optional implementations, the determination module 803 may include: a first determination unit 8031, used to determine, for each key point in the key point set, the coordinates on the heat map corresponding to the key point based on the coordinates in the first direction and the coordinates in the second direction included in the position information of the key point; a second determination unit 8032, used to determine the first distance between each element in the first supervision vector corresponding to the key point and the first coordinate on the heat map corresponding to the key point, and to determine the second distance between each element in the second supervision vector corresponding to the key point and the second coordinate on the heat map corresponding to the key point; a third determination unit 8033, used to determine the first supervision vector and the second supervision vector corresponding to the key point based on the first distance and the second distance.
[0185] In some optional implementations, the third determining unit 8033 may be further configured to: enhance the supervisory vector element corresponding to the shortest distance among the obtained first distances and second distances to obtain the first supervisory vector and the second supervisory vector.
[0186] In some optional implementations, the device further includes: a first acquisition module 805, configured to acquire an original sample image; and an extraction module 806, configured to extract a region including the target object from the original sample image as a sample image.
[0187] The key point recognition model generation device provided by the above-mentioned embodiment of the present disclosure generates a heat map set based on the heat map generation module included in the initial model, and then converts the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the initial model, and then determines the supervision vector corresponding to each key point based on the position information of the key point in the key point set. Finally, the initial model is trained based on the prediction vector and the supervision vector to obtain the key point recognition model. Therefore, in the model training stage, the supervision vector is used to avoid the excessively high proportion of negative samples caused by the presence of a large number of background pixels in the heat map, so that the ratio of positive and negative samples is balanced. Since the influence of negative samples on model training is reduced, the convergence speed of model training is improved, and the recognition accuracy of the trained model is improved.
[0188] Fig.10 is a schematic diagram of the structure of a key point recognition device provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Fig.10 As shown, the key point recognition device includes: a second acquisition module 1001, used to acquire an image to be recognized including a target object; a second generation module 1002, used to input the image to be recognized into a heat map generation module included in a pre-trained key point recognition model to obtain a heat map set, wherein the heat map in the heat map set represents the position of the predicted key point on the target object, and the key point recognition model is trained based on the generation method of the above-mentioned key point recognition model; a second conversion module 1003, used to convert the heat map in the heat map set into a predicted vector based on a predicted vector conversion module included in the key point recognition model; a decoding module 1004, used to decode the predicted vector based on a decoding module included in the key point recognition model to obtain the position information of the key point in the key point set corresponding to the image to be recognized.
[0189] In this embodiment, the second acquisition module 1001 can acquire the image to be recognized including the target object locally or remotely. The target object here can be the same as the above Figure 2 The target objects described in the corresponding embodiments are the same. For the description of the target objects, please refer to the above Figure 2 The corresponding embodiments are not described in detail here.
[0190] In this embodiment, the second generation module 1002 can input the image to be recognized into the heat map generation module included in the pre-trained key point recognition model to obtain a heat map set. The heat map in the heat map set represents the position of the predicted key points on the target object. The key point recognition model is based on the above Figure 2 The method described in the corresponding embodiment is trained. For the description of the heat map generation module, please refer to the above Figure 2 The corresponding embodiments are not described in detail here.
[0191] In this embodiment, the second conversion module 1003 can convert the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the key point recognition model. For the description of the prediction vector conversion module, please refer to the above Figure 2 The corresponding embodiments are not described in detail here.
[0192] In this embodiment, the decoding module 1004 can decode the prediction vector based on the decoding module included in the key point recognition model to obtain the position information of the key points in the key point set corresponding to the image to be recognized. The decoding module is used to perform calculations using the prediction vector to obtain the position information of the key points.
[0193] The decoding module can perform reverse operations on the prediction vector to restore the elements of the key points marked in the prediction vector to the coordinates in the image to be recognized.
[0194] Reference Fig.11 , Fig.11 It is a structural schematic diagram of a key point identification device provided by another exemplary embodiment of the present disclosure.
[0195] In some optional implementations, the decoding module 1004 may include: a fourth determination unit 10041, used to determine, for each of the obtained prediction vectors, in the heat map corresponding to the prediction vector, the coordinates corresponding to the maximum value of the elements in the prediction vector; a fifth determination unit 10042, used to determine whether the position indicated by the coordinates is on the boundary of the prediction vector; an estimation unit 10043, used to perform peak estimation based on the coordinates to obtain peak coordinates if on the boundary; and a sixth determination unit 10044, used to determine, based on the peak coordinates, the position information of the key point corresponding to the prediction vector in the image to be identified.
[0196] In some optional implementations, the decoding module further includes: a seventh determination unit 10045, configured to determine, if not on the boundary, the position information of the key point corresponding to the prediction vector in the image to be identified based on the coordinates corresponding to the maximum value.
[0197] In some optional implementations, the seventh determination unit 10045 may include: a determination subunit 100451, used to determine the second largest value from the elements included in the prediction vector; a correction subunit 100452, used to perform offset correction based on the positional relationship between the coordinates corresponding to the second largest value and the coordinates corresponding to the maximum value, so as to obtain the position information of the key point corresponding to the prediction vector in the image to be identified.
[0198] In some optional implementations, the second acquisition module 1001 may include: a second acquisition unit 10011, configured to acquire an original image; and an extraction unit 10012, configured to extract a region including a target object from the original image as an image to be identified.
[0199] The key point recognition model generation device provided in the above embodiment of the present disclosure utilizes the above Figure 2 The key point recognition model described in the corresponding embodiment recognizes the image to be recognized and obtains the position information of each key point on the target object in the image to be recognized, thereby locating the key points using the prediction vector output by the key point recognition model. Since the prediction vector is a vector, when the key point recognition model is used for key point recognition, the amount of data included in the vector is small. Therefore, compared with the two-dimensional matrix, the post-processing process of determining the key point coordinates is more efficient.
[0200] Exemplary Electronic Devices
[0201] Below, reference Fig.12 To describe an electronic device according to an embodiment of the present disclosure. The electronic device may be as follows Figure 1 Any one or both of the terminal device 101 and the server 103 shown, or a stand-alone device independent of them, the stand-alone device can communicate with the terminal device 101 and the server 103 to receive the collected input signals from them.
[0202] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.
[0203] like Fig.12 As shown, the electronic device 1200 includes one or more processors 1201 and a memory 1202 .
[0204] The processor 1201 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1200 to perform desired functions.
[0205] The memory 1202 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 1201 may run the program instructions to implement the key point recognition model generation method or key point recognition method of each embodiment of the present disclosure above and / or other desired functions. Various contents such as sample images, images to be recognized, key point recognition models, etc. may also be stored in the computer-readable storage medium.
[0206] In one example, the electronic device 1200 may further include: an input device 1203 and an output device 1204 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0207] For example, when the electronic device is the terminal device 101 or the server 103, the input device 1203 may be a device such as a mouse, a keyboard, a camera, etc., for inputting images. When the electronic device is a stand-alone device, the input device 1203 may be a communication network connector for receiving input images from the terminal device 101 and the server 103.
[0208] The output device 1204 can output various information to the outside, including the position information of the determined key points, etc. The output device 1204 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0209] Of course, to simplify, Fig.12 Only some of the components related to the present disclosure in the electronic device 1200 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 1200 may also include any other appropriate components.
[0210] Exemplary computer program products and computer-readable storage media
[0211] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the key point recognition model generation method or the key point recognition method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.
[0212] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0213] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps in the method for generating a key point recognition model or the key point recognition method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.
[0214] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0215] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0216] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0217] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0218] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0219] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0220] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0221] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for generating a key point recognition model, include: Based on the heat map generation module included in the initial model, a heat map set corresponding to the sample image is generated, wherein the heat map in the heat map set represents the position of the predicted key point on the target object, and the sample image has pre-labeled position information of the key point in the key point set representing the target object; Based on the prediction vector conversion module included in the initial model, the heat map in the heat map set is converted into a prediction vector, where the prediction vector is a vector including a plurality of elements, where the element corresponds to an area in the sample image, and the value of the element is used to represent the probability that a key point exists in the corresponding area; Based on the position information of the key points in the key point set, determine the supervision vector corresponding to the key points in the key point set, the supervision vector is used to characterize the position of the real key points in the sample image, and the supervision vector has the same dimension as the prediction vector; the position information of the key points includes the coordinates of the key points in the first direction and the coordinates of the key points in the second direction, the supervision vector includes a first supervision vector and a second supervision vector, the first supervision vector is obtained based on the distance between each element in the heat map and the key point in the first direction, and the second supervision vector is obtained based on the distance between each element in the heat map and the key point in the second direction; Based on the prediction vector and the supervision vector, the initial model is trained to obtain a key point recognition model.
2. The method according to claim 1, in, The prediction vector conversion module included in the initial model converts the heat map in the heat map set into a prediction vector, including: For each heat map in the heat map set, performing a dimensionality reduction operation on the heat map in a first direction and a second direction to obtain a first prediction vector and a second prediction vector corresponding to the heat map, respectively; The determining, based on the position information of the key points in the key point set, the supervision vector corresponding to the key points in the key point set comprises: Based on the coordinates of the first direction and the second direction included in the position information of the key points in the key point set, the first supervision vector and the second supervision vector corresponding to the key points in the key point set are determined.
3. The method according to claim 2, in, The performing of dimensionality reduction operations on the heat map in the first direction and the second direction to obtain a first prediction vector and a second prediction vector corresponding to the heat map respectively includes: Obtaining a preset number of groups of dimensionality reduction cores, wherein each group of dimensionality reduction cores in the preset number of groups of dimensionality reduction cores includes a first dimensionality reduction core corresponding to the first direction and a second dimensionality reduction core corresponding to the second direction; Based on the preset number of groups of dimensionality reduction kernels, a dimensionality reduction operation is performed on the heat map to obtain a first sub-prediction vector and a second sub-prediction vector corresponding to each group of dimensionality reduction kernels in the preset number of groups of dimensionality reduction kernels; Based on the preset weights corresponding to each group of dimensionality reduction kernels, weighted sums are performed on each first sub-prediction vector and each second sub-prediction vector to obtain the first prediction vector and the second prediction vector corresponding to the heat map.
4. The method according to claim 2, in, The determining, based on the coordinates of the first direction and the second direction included in the position information of the key points in the key point set, the first supervision vector and the second supervision vector corresponding to the key points in the key point set comprises: For each key point in the key point set, based on the coordinates in the first direction and the coordinates in the second direction included in the position information of the key point, determine the coordinates on the heat map corresponding to the key point; Determine a first distance between each element in a first supervision vector corresponding to the key point and a first coordinate on a heat map corresponding to the key point, and determine a second distance between each element in a second supervision vector corresponding to the key point and a second coordinate on a heat map corresponding to the key point; Based on the first distance and the second distance, a first supervision vector and a second supervision vector corresponding to the key point are determined.
5. A key point identification method, include: Acquire an image to be identified including a target object; Inputting the image to be identified into a heat map generation module included in a pre-trained key point recognition model to obtain a heat map set, wherein the heat maps in the heat map set represent the positions of the predicted key points on the target object, and the key point recognition model is trained based on the method according to any one of claims 1 to 4; Based on the prediction vector conversion module included in the key point recognition model, the heat map in the heat map set is converted into a prediction vector, where the prediction vector is a vector including a plurality of elements, where the element corresponds to an area in the sample image, and the value of the element is used to represent the probability that a key point exists in the corresponding area; Based on the decoding module included in the key point recognition model, the prediction vector is decoded to obtain the position information of the key points in the key point set corresponding to the image to be recognized; The decoding module included in the key point recognition model decodes the prediction vector to obtain the position information of the key points in the key point set corresponding to the image to be recognized, including: For each of the obtained prediction vectors, in the heat map corresponding to the prediction vector, determine the coordinates corresponding to the maximum value of the elements in the prediction vector; Determining whether the position indicated by the coordinates is on a boundary of the prediction vector; If on the boundary, perform peak estimation based on the coordinates to obtain peak coordinates; Based on the peak coordinates, position information of the key point corresponding to the prediction vector in the image to be identified is determined.
6. A device for generating a key point recognition model, include: A first generation module is used to generate a set of heat maps corresponding to the sample image based on the heat map generation module included in the initial model, wherein the heat maps in the heat map set represent the positions of predicted key points on the target object, and the sample image has pre-labeled position information of key points in the set of key points representing the target object; A first conversion module, configured to convert the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the initial model, wherein the prediction vector is a vector including a plurality of elements, wherein the element corresponds to an area in the sample image, and the value of the element is used to indicate the probability that a key point exists in the corresponding area; A determination module, used to determine the supervision vector corresponding to the key point in the key point set based on the position information of the key point in the key point set, the supervision vector is used to characterize the position of the real key point in the sample image, and the supervision vector has the same dimension as the prediction vector; the position information of the key point includes the coordinates of the key point in the first direction and the coordinates of the key point in the second direction, the supervision vector includes a first supervision vector and a second supervision vector, the first supervision vector is obtained based on the distance between each element in the heat map and the key point in the first direction, and the second supervision vector is obtained based on the distance between each element in the heat map and the key point in the second direction; A training module is used to train the initial model based on the prediction vector and the supervision vector to obtain a key point recognition model.
7. A key point recognition device, include: A second acquisition module is used to acquire an image to be identified including a target object; A second generation module, configured to input the image to be identified into a heat map generation module included in a pre-trained key point recognition model to obtain a heat map set, wherein the heat maps in the heat map set represent the positions of the predicted key points on the target object, and the key point recognition model is trained based on the method according to any one of claims 1 to 4; A second conversion module, configured to convert the heat map in the heat map set into a prediction vector based on the prediction vector conversion module included in the key point recognition model, wherein the prediction vector is a vector including a plurality of elements, wherein the element corresponds to an area in the sample image, and the value of the element is used to represent the probability that a key point exists in the corresponding area; A decoding module, configured to decode the prediction vector based on the decoding module included in the key point recognition model to obtain the position information of the key points in the key point set corresponding to the image to be recognized; the decoding module includes: a functional unit configured to perform a reverse operation on the prediction vector to restore the elements marked with key points in the prediction vector to the coordinates in the image to be recognized; Wherein, the decoding module comprises: The fourth determination unit is used to determine, for each of the obtained prediction vectors, the coordinates corresponding to the maximum value of the elements in the prediction vector in the heat map corresponding to the prediction vector; the fifth determination unit is used to determine whether the position indicated by the coordinates is on the boundary of the prediction vector; the estimation unit is used to perform peak estimation based on the coordinates to obtain the peak coordinates if it is on the boundary; the sixth determination unit is used to determine the position information of the key point corresponding to the prediction vector in the image to be identified based on the peak coordinates.
8. A computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 5.
9. An electronic device, the electronic device include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 5.
