Key point detection model training method and device

By performing motion fuzzing processing and key point weight adjustment on the training dataset, the accuracy of the human key point detection model under motion fuzzy conditions is solved, the detection accuracy and robustness are improved, and the hardware and data set costs are reduced.

CN111353325BActive Publication Date: 2024-12-06BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201811560699.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-12-20
Publication Date
2024-12-06
Estimated Expiration
2038-12-20

AI Technical Summary

Technical Problem

The existing human key point detection model is difficult to accurately identify human key points under fuzzy motion conditions, resulting in a decrease in detection accuracy, and the weight adjustment of different key points during training is difficult to balance, resulting in poor recognition effect.

Method used

By performing motion fuzzing on the training data set, the second training data set is generated, and the neural network model is iteratively trained using the loss function of key point weighted Euclidean distance, adjusting the weight of each key point to highlight the identification of key points with poor results , improve the accuracy of the detection model.

Benefits of technology

Accurate key point detection under motion fuzzy conditions is achieved, the accuracy and robustness of the detection model are improved, and the hardware cost and training data set collection cost are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111353325B_ABST
    Figure CN111353325B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a key point detection model training method and device, which obtains a second training data set by performing motion blur processing on a first training data set, inputs each image in the second training data set into a pre-built neural network model, and iteratively trains the neural network using the loss function of the key point weighted Euclidean distance until a converged neural network model is obtained, and uses the converged neural network model as the final key point inspection model. In this process, by performing motion blur processing on normal images with clear subjects and targets contained in the first training data set, the key point detection model is trained using these images with random motion blur processing, thereby training a key point detection model that can accurately detect key points of the human body in images containing motion blur.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of visual detection technology, and in particular to a key point detection model training method and device. Background Art

[0002] With the development of science and technology and the advent of the big data era, how to conveniently and accurately realize video analysis and human-computer interaction has become a hot topic. Among them, human key point detection technology, as an intelligent visual detection technology, can easily give machines the ability to recognize human postures. It has the advantages of being fast and convenient, saving manpower, and reducing operating costs. It is widely used in human-computer interaction, abnormal behavior detection, autonomous driving and other fields.

[0003] At present, in the common human key point recognition technology, the public human key point data set on the network or the self-collected data set is used as the training data set, and the neural network is trained on the training data set to obtain the key point detection model; and the Euclidean distance between the detection value and the annotation value of different human key points is used as the loss function, and the key point detection model is iterated using the loss function until a converged key point detection model is obtained. Then, the image is input into the key point detection model, and the converged key point detection model is used to analyze the image, and the coordinate values ​​of each key point of the human body in the image are output; or, the video frame is input into the key point detection model, and the converged key point detection model is used to analyze the video frame, and the coordinate values ​​of each key point of the human body in the video frame are output.

[0004] In the process of training the key point detection model mentioned above, the human key point dataset or self-collected dataset publicly available on the Internet contains normal images with clear subjects and targets. When the image to be detected, such as a picture or video frame, is a clear image, the detection accuracy is high when the converged key point detection model is used for key point detection. However, in scenarios such as human-computer interaction and video analysis, due to the limited capabilities of the imaging device and the relative motion between the target and the imaging device, the image obtained by the imaging device has motion blur. When the key point detection model mentioned above is used to detect the key points of blurred pictures or video frames, it is difficult to accurately identify the key points. Summary of the invention

[0005] The embodiment of the present invention provides a key point detection model training method and device, which adds random motion blur processing to the key points of the human body, thereby training a key point detection model that can detect the key points of the human body in a motion scene, thereby improving the accuracy of the key point detection of the human body.

[0006] In a first aspect, an embodiment of the present invention provides a key point detection model training method, comprising:

[0007] Performing motion blur processing on each image in the first training data set to obtain a second training data set;

[0008] Each image in the second training data set is input into a neural network model, and the neural network model is iterated using a loss function until the neural network model converges, and the converged neural network model is used as a key point detection model. The neural network model is a pre-built model, and the key point detection model is used to detect key points of characters in character images.

[0009] In a feasible design, inputting each image in the second training data set into a neural network model and iterating the neural network model using a loss function includes:

[0010] Inputting a first group of images in the second training data set into the neural network model, the first group of images comprising a preset number of images;

[0011] Determine the weight of each type of key points in the first group of images, where the same type of key points have the same human body position in different person images in each image included in the first group of images;

[0012] Update the loss function according to the weights of various key points;

[0013] Updating the neural network model using the updated loss function;

[0014] The second group of images in the second training data set is input into the updated neural network model, and the above steps are repeated until a converged neural network model is obtained.

[0015] In a feasible design, determining the weight of each type of key point in the first group of images includes:

[0016] For the first type of key points, using the neural network model, predicting the first coordinate value of the first key point of each person image contained in each image of the first group of images, each image in the first group of images includes at least one person image, each person image includes the first key point, and the first type of key points includes each of the first key points;

[0017] Determining the similarity of each of the first key points according to a first coordinate value and a second coordinate value of each of the first key points, wherein the second coordinate value is a coordinate pre-marked on the character image for the first key point;

[0018] The weights of the first-category key points are determined according to the similarities of the first key points.

[0019] In a feasible design, determining the weight of the first type of key points according to the similarity of each of the first key points includes:

[0020] Determining, according to the similarities of the first key points, a first number of first key points having a similarity greater than a first threshold from among the first key points included in the first category of key points;

[0021] Determining an average accuracy of the first type of key points according to a first number and a second number, wherein the second number is a total number of the first key points;

[0022] The weight of the first type of key points is determined according to the average accuracy.

[0023] In a feasible design, determining the weight of the first type of key points according to the average accuracy rate includes:

[0024] Determine the average accuracy rate according to the accuracy rate formula;

[0025] Determine the reciprocal of the average accuracy rate, and use the reciprocal as the weight of the first type of key points; wherein the average accuracy rate formula is:

[0026]

[0027] Among them, AP represents the average accuracy, i represents the key point of the i-th category, OKS represents the similarity, p represents the number of the person image in the second training data set, and thres represents the first threshold.

[0028] In a feasible design, determining the similarity of each first key point according to the first coordinate value and the second coordinate value of each first key point includes:

[0029] The similarity calculation formula is used to determine the similarity of each of the first key points according to the first coordinate value and the second coordinate value of each of the first key points. The similarity calculation formula is:

[0030]

[0031] Wherein, OKS represents similarity, p represents the number of the person image in the second training data set, represents the key point of the i-th category, and d pi represents the Euclidean distance between the first coordinate value and the second coordinate value, s p and σ i are scale factors, δ(v pi =1) is the Kronecker function.

[0032] In a feasible design, performing motion blur processing on each image in the first training data set to obtain the second training data set includes:

[0033] Randomly generate motion rate, angular velocity and noise function;

[0034] Selecting a preset number of key points on a first image, where the first image is any image in the first training data set;

[0035] Determine a first area according to a position of a key point to be moved, wherein the key point to be moved is any key point among the preset number of key points;

[0036] According to the movement speed, the angular velocity and the noise function, each point in the first area is moved to obtain a second area;

[0037] The second region is superimposed on the first image to obtain the second training data set.

[0038] In a feasible design, moving each point in the first area according to the movement speed, the angular velocity and the noise parameter to obtain the second area includes:

[0039] According to the movement formula, the movement speed, the angular velocity and the noise parameter, each point in the first area is moved to obtain the second area, and the movement formula is:

[0040]

[0041] Wherein, dst is the coordinate value of the first point after movement, src represents the first image, x and y represent the x coordinate and y coordinate of the first point before movement, respectively, the movement speed is v, and the angular velocity is Said Said noisy represents a noise function, and the first point is any point in the first area.

[0042] In a feasible design, after inputting each image in the second training data set into the neural network model and iterating the neural network model using the loss function, the method further includes:

[0043] Convert the image to be detected into a red, green and blue (RGB) image, input the RGB image into the key point detection model, and obtain a probability confidence map of each key point in the RGB image;

[0044] According to the position of the maximum probability point of each probability confidence map, the position of the maximum probability point is used as the coordinate position of the key point corresponding to the probability confidence map.

[0045] In a second aspect, an embodiment of the present invention provides a key point detection model training device, comprising:

[0046] A processing module, used for performing motion blur processing on each image in the first training data set to obtain a second training data set;

[0047] A training module is used to input each image in the second training data set into a neural network model, and iterate the neural network model using a loss function until the neural network model converges, and use the converged neural network model as a key point detection model. The neural network model is a pre-built model, and the key point detection model is used to detect key points of characters in character images.

[0048] In a feasible design, the training module is specifically used to input the first group of images in the second training data set into the neural network model, the first group of images includes a preset number of images; determine the weight of each type of key points in the first group of images, and the human body positions of the same type of key points on different character images in each image included in the first group of images are the same; update the loss function according to the weight of each type of key point; update the neural network model using the updated loss function; input the second group of images in the second training data set into the updated neural network model, and repeat the above steps until a converged neural network model is obtained.

[0049] In a feasible design, the training module, when determining the weight of each type of key points in the first group of images, is used to predict, for the first type of key points, the first coordinate value of the first key point of each character image contained in each image in the first group of images using the neural network model, each image in the first group of images contains at least one character image, each character image contains the first key point, and the first type of key points contains each of the first key points; based on the first coordinate value and the second coordinate value of each of the first key points, the similarity of each of the first key points is determined, the second coordinate value is the coordinate pre-marked for the first key point on the character image; based on the similarity of each of the first key points, the weight of the first type of key points is determined.

[0050] In a feasible design, when determining the weight of the first category of key points based on the similarity of each of the first key points, the training module is used to determine, from the first key points included in the first category of key points, a first number of first key points with a similarity greater than a first threshold based on the similarity of each of the first key points; determine the average accuracy of the first category of key points based on the first number and the second number, the second number being the total number of the first key points; and determine the weight of the first category of key points based on the average accuracy.

[0051] In a feasible design, the training module, when determining the weight of the first type of key points according to the average accuracy, is used to determine the average accuracy according to an accuracy formula;

[0052] Determine the reciprocal of the average accuracy rate, and use the reciprocal as the weight of the first type of key points; wherein the average accuracy rate formula is:

[0053]

[0054] Among them, AP represents the average accuracy, i represents the key point of the i-th category, OKS represents the similarity, p represents the number of the person image in the second training data set, and thres represents the first threshold.

[0055] In a feasible design, when the training module determines the similarity of each first key point according to the first coordinate value and the second coordinate value of each first key point, it uses a similarity calculation formula to determine the similarity of each first key point according to the first coordinate value and the second coordinate value of each first key point, and the similarity calculation formula is:

[0056]

[0057] Wherein, OKS represents similarity, p represents the number of the person image in the second training data set, represents the key point of the i-th category, and d pi represents the Euclidean distance between the first coordinate value and the second coordinate value, s p and σ i are scale factors, δ(v pi =1) is the Kronecker function.

[0058] In a feasible design, the processing module is specifically used to randomly generate motion rate, angular velocity and noise function;

[0059] A preset number of key points are selected on a first image, where the first image is any image in the first training data set; a first area is determined according to the position of the key point to be moved, where the key point to be moved is any key point in the preset number of key points; each point in the first area is moved according to the motion speed, the angular velocity and the noise function to obtain a second area; and the second area is superimposed on the first image to obtain the second training data set.

[0060] In a feasible design, when the processing module moves each point in the first area according to the movement speed, the angular velocity and the noise parameter to obtain the second area, the processing module is used to move each point in the first area according to a movement formula, as well as the movement speed, the angular velocity and the noise parameter to obtain the second area, and the movement formula is:

[0061]

[0062] Wherein, dst is the coordinate value of the first point after movement, src represents the first image, x and y represent the x coordinate and y coordinate of the first point before movement, respectively, the movement speed is v, and the angular velocity is Said Said noisy represents a noise function, and the first point is any point in the first area.

[0063] In a feasible design, the above device also includes:

[0064] A detection module is used to input each image in the second training data set into a neural network model in the training module, and after the neural network model is iterated using a loss function, convert the image to be detected into a red, green and blue (RGB) image, input the RGB image into the key point detection model, and obtain a probability confidence map of each key point in the RGB image; according to the position of the maximum probability point of each probability confidence map, use the position of the maximum probability point as the coordinate position of the key point corresponding to the probability confidence map.

[0065] In a third aspect, an embodiment of the present invention provides a key point detection module training device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the device implements the method described in the first aspect or various feasible implementation methods of the first aspect.

[0066] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the method described in the first aspect or various feasible implementations of the first aspect.

[0067] In a fifth aspect, an embodiment of the present invention provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the method described in the first aspect or various feasible implementations of the first aspect.

[0068] The key point detection model training method and device provided by the embodiment of the present invention obtains a second training data set by performing motion blur processing on the first training data set, inputs each image in the second training data set into a pre-built neural network model, and iteratively trains the neural network using the loss function of the key point weighted Euclidean distance until a converged neural network model is obtained, and the converged neural network model is used as the final key point detection model. In this process, the normal images with clear subjects and clear targets contained in the first training data set are subjected to motion blur processing, and the key point detection model is trained using these images with random motion blur processing, thereby training a key point detection model that can accurately detect the key points of the human body in the images containing motion blur. In addition, in the process of training the key point detection model, the loss function of the key point weighted Euclidean distance is used to adjust the weights of each key point, thereby highlighting the key points with poor recognition effect, and further improving the accuracy of the key point detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0070] Figure 1 is a flow chart of a key point detection model training method provided by an embodiment of the present invention;

[0071] Figure 2 is a flow chart of motion blur processing in a key point detection model training method provided by an embodiment of the present invention;

[0072] Figure 3 is a schematic diagram of an image subjected to motion blur processing in a key point detection model training method provided by an embodiment of the present invention;

[0073] Figure 4 It is a flowchart of key point adaptive weight adjustment in a key point detection model training method provided by an embodiment of the present invention;

[0074] Figure 5 It is a flow chart of determining key point weights in a key point detection model training method provided by an embodiment of the present invention;

[0075] Figure 6 It is a flowchart of detecting key points in a key point detection model training method provided by an embodiment of the present invention;

[0076] Figure 7A structural schematic diagram of a key point detection model training device provided by an embodiment of the present invention;

[0077] Figure 8 A schematic diagram of the structure of another key point detection model training device provided by an embodiment of the present invention;

[0078] Fig. 9 A schematic structural diagram of another electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] At present, in the process of training the key point detection model, the human key point data set or self-collected data set publicly available on the Internet is used. These training data sets contain normal images with clear subjects and clear targets. The key point detection model trained with the training data set can accurately identify the key points of people in clear images. However, in scenes such as human-computer interaction and video analysis, due to the limited capabilities of the imaging device and the relative motion between the target and the imaging device, the image collected by the imaging device contains a blurred image. In this scenario, if the above-mentioned stable key point detection model is used to detect the human key points in the image containing motion blur, it will be difficult to identify the target key points or the positioning deviation will be large, resulting in poor recognition of human key points. In order to eliminate motion blur, if the hardware of the imaging device is simply upgraded, the hardware cost will be increased. In addition, in the process of training the key point detection model, the sum of the Euclidean distances of the detection values ​​and the annotation values ​​of different human key points is used as the loss function. Since multiple key points exist at the same time, the key points that are easier to detect will cover other key points. For example, there are 10 key points of a human body in an image, 8 of which are easy to detect, and the other two are not easy to detect. The loss function obtained based on the 10 key points is small, which makes the key point detection model mistakenly believe that it can accurately identify the 10 key points. In fact, the key point detection model cannot accurately identify the two key points that are not easy to detect.

[0081] In view of this, an embodiment of the present invention provides a key point detection model training method. In the method, firstly, based on a training data set consisting of normal images with clear subjects and targets (hereinafter referred to as the first training data set), the relative movement between the imaging device and the target is simulated, and random motion blur processing is added to the key points of the human body to obtain a second training data set; secondly, during the model training process, according to the recognition accuracy of different key points, the weights of the key points in the loss function are adjusted, so as to highlight the key points with poor recognition, thereby ensuring that all categories of key points are balanced during the entire training process, and achieving the effect of optimizing the key point detection model.

[0082] The key point detection model training method provided in the embodiment of the present invention can be applied to any scenario where key points of the human body need to be detected. The key point detection model training device that executes the key point detection model training method can be set on an electronic device of the client or on a server. The electronic device can be, for example, a computer device, a mobile terminal device, an autonomous driving vehicle device, a human-computer interaction device, and other artificial intelligence devices. The key point detection model training method described in the embodiment of the present invention is described in detail below. For example, please refer to Figure 1 .

[0083] Figure 1 It is a flow chart of a key point detection model training method provided by an embodiment of the present invention. The execution subject of this embodiment is a key point detection model training device, which can be implemented by software, hardware, or a combination of software and hardware. The device can be part or all of an electronic device, or part or all of a server. This implementation is described by taking the application of the key point detection model training method to an electronic device as an example. This embodiment includes:

[0084] 101. Perform motion blur processing on each image in a first training data set to obtain a second training data set, where the second training data set includes at least one image.

[0085] In the embodiment of the present invention, the key point detection model training method includes two stages: the first stage is the training data preparation stage; the second stage is the key point detection model training stage using the training data.

[0086] In this step, the electronic device performs motion blur processing on each normal image in the first training data set in the first stage to obtain a second training data set. The first training data set contains normal images with clear subjects and targets.

[0087] 102. Input each image in the second training data set into a neural network model, and iterate the neural network model using a loss function until the neural network model converges, and use the converged neural network model as a key point detection model, wherein the neural network model is a pre-built model, and the key point detection model is used to detect key points of a person in a person image.

[0088] In this step, each image in the second training data set is input into a pre-built neural network model, and the neural network model is iteratively trained using a loss function until a converged neural network model is trained, and the converged neural network model is used as the final key point detection model.

[0089] The key point detection model training method provided by the embodiment of the present invention obtains a second training data set by performing motion blur processing on the first training data set, inputs each image in the second training data set into a pre-constructed neural network model, and iteratively trains the neural network using the loss function of the key point weighted Euclidean distance until a converged neural network model is obtained, and the converged neural network model is used as the final key point detection model. In this process, the normal images with clear subjects and clear targets contained in the first training data set are subjected to motion blur processing, and the key point detection model is trained using these images with random motion blur processing, thereby training a key point detection model that can accurately detect the key points of the human body in the images containing motion blur. In addition, in the process of training the key point detection model, the loss function of the key point weighted Euclidean distance is used to adjust the weights of each key point, thereby highlighting the key points with poor recognition effect, and further improving the accuracy of the key point detection model.

[0090] The following is a detailed description of the motion blur processing in the first stage and the key point adaptive weight adjustment in the second stage in the above embodiment.

[0091] First, the first stage of motion blur processing.

[0092] For example, see Figure 2 , Figure 2 : is a flowchart of motion blur processing in a key point detection model training method provided by an embodiment of the present invention, and this embodiment includes:

[0093] 201. Obtain a first training data set.

[0094] In this step, a number of images are collected as a first training data set, and the first training data set contains human images with clear subjects and targets. Then, for each first image in the first training data set, the position of the human and the key point positions are marked on the first figure. The position of the human is represented by a four-tuple (x, y, w, h), in which x represents the horizontal coordinate of the upper left corner, y represents the vertical coordinate of the upper left corner, w represents the width of the frame, and h represents the height of the frame. According to the four-tuple, a human rectangular frame can be determined; the position of the human can also be represented by a four-tuple (x1, y1, x2, y2), in which x1 represents the horizontal coordinate of the upper left corner, y1 represents the vertical coordinate of the upper left corner, x2 represents the horizontal coordinate of the lower right corner, and y2 represents the vertical coordinate of the lower right corner. According to the four-tuple, a human rectangular frame can be determined. The position of the key points of the human body is represented by a triple (x, y, c), where x represents the horizontal coordinate, y represents the vertical coordinate, and c represents the state, which includes visible, occluded and invisible, in the image, or unpredictable.

[0095] 202. Randomly generate motion speed, angular velocity and noise function.

[0096] In this step, according to the preset speed [v low ,v up ] and angle Size range, randomly generated movement speed v and movement angle And the noise function noisy(x,y). We get v x and v y Represent the horizontal and vertical components of v respectively.

[0097] 203. Select a preset number of key points on a first image, where the first image is any image in the first training data set.

[0098] In this step, for each image in the first training data set, hereinafter referred to as the first image, the first image is used as the source image src, and zero or more visible human key points are randomly selected in the first image to prepare for motion blur processing.

[0099] 204. Determine a first area according to a position of a key point to be moved, wherein the key point to be moved is any key point among the preset number of key points;

[0100] In this step, for each key point among a preset number of key points, i.e., the key point to be moved, a first area is determined according to the position of the key point to be moved. The first area is, for example, a circular area with the key point to be moved as the center and a preset length as the radius, or the first area is a square with the key point to be moved as the center, etc., and the embodiments of the present invention are not limited.

[0101] 205. Move each point in the first area according to the movement speed, the angular velocity, and the noise function to obtain a second area.

[0102] In this step, each point in the first area is translated according to the motion speed, angular velocity, and noise function obtained in the above step 201 to obtain the second area. During the translation process, the image within a certain range around the key point to be moved, that is, the image in the first area is moved according to the motion rate and angular velocity. For any coordinate point in the first area, hereinafter referred to as the first coordinate point, the coordinates of the first coordinate point are (x, y), and the coordinates can be moved according to the movement formula. The movement formula is:

[0103]

[0104] Wherein, dst is the coordinate value of the first point after movement, src represents the first image, x and y represent the x coordinate and y coordinate of the first point before movement, respectively, the movement speed is v, and the angular velocity is Said Said noisy represents the noise function, and the first point is any point in the first area.

[0105] According to this step, it can be known that during the motion blur processing, for a key point, a small piece of the image around the key point is moved.

[0106] 206. Superimpose the second area and the first image.

[0107] In this step, the second region and the first image are superimposed by integration processing, etc. The integration processing is simulated by discrete superposition, for example.

[0108] It should be noted that, since there are multiple first images in the first training data set, and each first image contains at least one person image, multiple first images with superimposed second regions can be obtained after processing in steps 204 and 205 .

[0109] In addition, it should be noted that in order to enable the key point detection model to be used to detect key points in images containing motion blur, and also to be used to detect key points in images containing clear subjects and clear targets, in the above step 203, for some first images in the first training data set, zero key points can also be selected, that is, some character images in the first training data set are not motion blurred, and the character images can be character images contained in multiple images, or can be multiple character images in one image.

[0110] 207. Perform normalization processing on the first image superimposed with the second region.

[0111] In this step, each first image superimposed with the second region is normalized, so that the image in the second region is normalized to the original pixel range of the first region to ensure the correctness of the color. After normalizing each first image superimposed with the second region, a second training data set is obtained, and each image in the second training data set is an image that has been motion blurred.

[0112] 208. Determine whether the iteration limit is reached. If the iteration limit is reached, execute 209; if the iteration limit is not reached, execute 204.

[0113] In this step, the electronic device determines whether the preset upper limit of the number of iterations is reached. If the upper limit of the iterations is reached, 209 is executed; if the upper limit of the iterations is not reached, the process returns to step 204, randomly selects an image again, and randomly selects key points for motion blur processing.

[0114] 209. Obtain a second training data set based on the normalized image.

[0115] In the process of preparing the training data set in the first stage, random motion blur processing is performed on the key points of each image in the first training data set, which can make the key point detection model more robust to motion blur recognition in actual scenes without increasing the hardware cost of the imaging device. At the same time, the cost of collecting the training data set is reduced. For example, see Figure 3 , Figure 3 is a schematic diagram of an image processed by motion blur in a key point detection model training method provided by an embodiment of the present invention. Figure 3 , Figure 3 (a) in the figure represents an image with a clear theme and a clear target in the first training data set, and the image is a partial image of the left wrist of a person; Figure 3 (b) in FIG. 1 represents an image in the second training data set that has been motion blurred. Compared with the image in (b), the image is blurred. Figure 3 In (a) and (b), the white dots represent the left wrist key points.

[0116] After adding motion blur processing to the first training data set, the motion blur data set, i.e., the second training data set, is used to train the key point detection model, and adaptive weight adjustment of different key points is completed. For example, see the second stage below.

[0117] Secondly, the key points in the second stage are adaptively weighted.

[0118] For example, see Figure 4 , Figure 4 : is a flowchart of key point adaptive weight adjustment in a key point detection model training method provided by an embodiment of the present invention, and this embodiment includes:

[0119] 301. Build a neural network model.

[0120] In this step, a neural network model is pre-built, such as a residual neural network (Resnet) model, a catastrophe progression method (CPM) neural network model, etc. The parameters of the neural network model mainly include the convolution weights and convolution kernel bias of the convolution layer, etc. The initial values ​​of the parameters can be obtained by random initialization or pre-training model.

[0121] 302. Input a first group of images in a second training data set into the neural network model, wherein the first group of images includes a preset number of images.

[0122] Generally speaking, the second training data set contains tens of thousands or even more images. During the key point detection model training process, all images in the second training data set are grouped and input into the neural network model according to the hardware capabilities of the electronic device. Each group of images may be, for example, 32, 64, or 128 images.

[0123] 303. Determine the weight of each type of key points in the first group of images, where the same type of key points have the same human body position in different person images in each image included in the first group of images.

[0124] In an embodiment of the present invention, the second training data set includes multiple images, each image includes at least one person, each person has a preset number of human body key points, such as 14 or 28, etc., and each key point is located at a different position of the human body. Taking 14 key points as an example, these key points include right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, top of the head and neck, etc. Since the first group of images includes multiple person images, theoretically, each person has these 14 key points without occlusion. Therefore, for all person images, there are 14 types of key points, and the same type of key points have the same human body position on different person images in each image included in the first group of images, for example, right shoulder key points and right elbow key points. In this step, the weight of each type of key point is determined, for example, the weight of the right shoulder key point, the weight of the right elbow key point, and the weight of the right knee key point are determined.

[0125] 304. Update the loss function according to the weights of various key points;

[0126] In this embodiment, it is assumed that the initial loss function is obtained by using the sum of the Euclidean distances of the detection values ​​and the annotation values ​​of different key points, and the initial loss function does not consider the weights of key points of different categories. In this step, according to the weights of various key points obtained in the above step 303, the initial loss function is updated to obtain the loss function of the weighted Euclidean distance of the key points.

[0127] 305. Update the neural network model using the updated loss function.

[0128] 306. Determine whether the updated neural network model converges. If the updated neural network model does not converge, return to execute 307. If the updated neural network model converges, execute 308.

[0129] 307. Input the second group of images in the second training data set into the updated neural network model, and repeat the above steps until a converged neural network model is obtained.

[0130] 308. A key point detection model is obtained according to the converged neural network model.

[0131] Next, how to determine the weight of each type of key point in the first group of images in the above step 303 is described in detail. Figure 5 , Figure 5 : is a flowchart of determining key point weights in a key point detection model training method provided by an embodiment of the present invention, and this embodiment includes:

[0132] 401. For the first type of key points, use the neural network model to predict the first coordinate value of the first key point of each person image contained in each image in the first group of images.

[0133] In an embodiment of the present invention, in each image in the first group of images, the sum of the first key points of each character is called the first type of key points. Taking the 14 types of key points in the human body as an example, the first type of key points is any one of the 14 key points, for example, right shoulder key points, right elbow key points, etc.

[0134] In this step, the neural network model is used to predict the first coordinate value of each first key point included in the first type of key points.

[0135] 402. Determine the similarity of each of the first key points according to the first coordinate value and the second coordinate value of each of the first key points.

[0136] The second coordinate value is the coordinate pre-marked for the first key point on the character image.

[0137] In this step, after obtaining the first coordinate value and the second coordinate value of the first key point, that is, the marked coordinates and the predicted values ​​through the neural network model, the Euclidean distance of the first key point is obtained according to the first coordinate value and the second coordinate value. Then, the similarity of the first key point is determined according to the Euclidean distance, etc. In the determination process, the similarity is determined using the similarity calculation formula, wherein the similarity calculation formula is:

[0138]

[0139] Wherein, OKS represents similarity, p represents the number of the person image in the second training data set, represents the key point of the i-th category, and d pi represents the Euclidean distance between the first coordinate value and the second coordinate value, s p and σ i are scale factors, δ(v pi =1) is the Kronecker function.

[0140] Assuming that the first group of images has 32 images in total, each image has a person, and assuming that the first key point is the right elbow key point, then in this step, the similarities of 32 right elbow key points are calculated without occlusion.

[0141] 403. Determine, according to the similarity of each of the first key points, a first number of first key points having a similarity greater than a first threshold from among the first key points included in the first category of key points.

[0142] In this step, the similarity of each first key point is compared with the preset first threshold value, so as to determine the number of first key points whose similarity is greater than the first threshold value. For example, assuming that the first threshold value is 0.5, among the 32 right elbow key points, there are 20 key points whose similarity is greater than 0.5, and 12 key points whose similarity is less than 0.5; for another example, assuming that the first threshold value is 0.6, among the 32 left shoulder key points, there are 8 key points whose similarity is greater than 0.5, and 24 key points whose similarity is less than 0.5.

[0143] 404. Determine an average accuracy rate of the first category of key points according to a first number and a second number, where the second number is the total number of the first key points.

[0144] In this step, the accuracy formula can be used to determine the average accuracy. The accuracy formula is:

[0145]

[0146] Wherein, AP represents the average accuracy, i represents the i-th key point, OKS represents the similarity, p represents the number of the person image in the second training data set, and thres represents the first threshold. Continuing with the example in step 403, when the first key point is the right elbow key point, the average accuracy is When the first type of key point is the left shoulder key point, the average accuracy is In other words, the neural network model has a higher recognition accuracy for right elbow key points, but a lower recognition accuracy for left shoulder key points.

[0147] 405. Determine the weight of the first type of key points according to the average accuracy.

[0148] In the embodiment of the present invention, the weight of the first type of key points is determined according to the average accuracy. For example, the reciprocal of the average accuracy is determined and the reciprocal is used as the weight of the first type of key points. Continuing with the example in step 403 above, the weight of the right elbow key point is The weight of the key point of the left shoulder class is 4. It can be seen that after the weight is adaptively adjusted, the weight of the key point of the left shoulder class with low recognition accuracy is automatically increased, that is, the weight of the key point with poor recognition is highlighted, and the accuracy of the key point with poor recognition is improved.

[0149] In this embodiment, after calculating the weights of each key point, the loss function is updated according to the weights of each key point. The formula of the loss function is as follows:

[0150]

[0151] Wherein, P represents the number of the person image in the second training data set, J represents the total number of human key point categories, and z represents the coordinate in the feature map range Z. and They represent the output feature map and target feature map of key point j of the person image numbered p, respectively. The inconsistency between the two is measured by the Euclidean distance. j Represents the weight of the j-th key point. The output feature map is obtained by, for example, predicting the coordinate values ​​of each key point using a neural network model, and the target feature map is obtained by, for example, marking the coordinates of each key point.

[0152] In the second stage of training the key point detection model, the weighted Euclidean distance of the key points is used as the loss function, which can highlight and optimize the key points with poor recognition, thereby improving the detection accuracy of the key point detection model for each key point.

[0153] In the embodiment of the present invention, after the above-mentioned first stage and second stage, a converged neural network model, i.e., a key point detection model, is obtained. Afterwards, when it is necessary to detect key points, the image to be detected is converted into a red, green, blue (RGB) image, and the RGB image is input into the key point detection model, so that the key point detection model outputs the coordinate values ​​of each key point of the human body on the RGB image. The following is a detailed description of how to use the key point detection model to detect key points of the human body. For example, see Figure 6 , Figure 6 : is a flowchart of detecting key points in a key point detection model training method provided by an embodiment of the present invention, and this embodiment includes:

[0154] 501. Obtain an image to be detected.

[0155] In this step, the image captured by the imaging device is obtained.

[0156] 502. Determine whether the image to be detected is an RGB image. If the image to be detected is an RGB image, execute 504; if the image to be detected is not an RGB image, execute 503;

[0157] 503. Convert the image to be detected into an RGB image and execute step 504;

[0158] 504. Input the RGB image into a key point detection model to obtain a probability confidence map of each key point in the RGB image.

[0159] 505. According to the position of the maximum probability point of each probability confidence map, use the position of the maximum probability point as the coordinate position of the key point corresponding to the probability confidence map.

[0160] In the above steps 504 and 505, the color RGB image is input into the trained key point detection model, and the probability confidence map of each human key point is calculated using a forward algorithm, and the position of the maximum probability point in each probability confidence map is taken as the coordinate value of the key point, thereby outputting the position coordinates of all key points.

[0161] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0162] Figure 7 The present invention provides a key point detection model training device according to an embodiment of the present invention. The key point detection model training device can be implemented by software and / or hardware. Figure 7 As shown, the key point detection model training device 100 includes:

[0163] The processing module 11 is used to perform motion blur processing on each image in the first training data set to obtain a second training data set;

[0164] The training module 12 is used to input each image in the second training data set into the neural network model, and iterate the neural network model using the loss function until the neural network model converges, and use the converged neural network model as the key point detection model. The neural network model is a pre-built model, and the key point detection model is used to detect the key points of the characters in the character images.

[0165] In a feasible design, the training module 12 is specifically used to input the first group of images in the second training data set into the neural network model, the first group of images includes a preset number of images; determine the weight of each type of key point in the first group of images, and the human body position of the same type of key points on different character images in each image included in the first group of images is the same; update the loss function according to the weight of each type of key point; update the neural network model using the updated loss function; input the second group of images in the second training data set into the updated neural network model, and repeat the above steps until a converged neural network model is obtained.

[0166] In a feasible design, the training module 12, when determining the weight of each type of key point in the first group of images, is used to predict, for the first type of key point, the first coordinate value of the first key point of each character image contained in each image in the first group of images using the neural network model, each image in the first group of images contains at least one character image, each character image contains the first key point, and the first type of key point contains each of the first key points; determine the similarity of each of the first key points based on the first coordinate value and the second coordinate value of each of the first key points, the second coordinate value being the coordinate pre-marked for the first key point on the character image; and determine the weight of the first type of key point based on the similarity of each of the first key points.

[0167] In a feasible design, the training module 12, when determining the weight of the first category of key points according to the similarity of each of the first key points, is used to determine, from the first key points included in the first category of key points, a first number of first key points whose similarity is greater than a first threshold according to the similarity of each of the first key points; determine the average accuracy of the first category of key points according to the first number and the second number, the second number being the total number of the first key points; and determine the weight of the first category of key points according to the average accuracy.

[0168] In a feasible design, the training module 12, when determining the weight of the first type of key points according to the average accuracy, is used to determine the average accuracy according to an accuracy formula;

[0169] Determine the reciprocal of the average accuracy rate, and use the reciprocal as the weight of the first type of key points; wherein the average accuracy rate formula is:

[0170]

[0171] Among them, AP represents the average accuracy, i represents the key point of the i-th category, OKS represents the similarity, p represents the number of the person image in the second training data set, and thres represents the first threshold.

[0172] In a feasible design, when the training module 12 determines the similarity of each first key point according to the first coordinate value and the second coordinate value of each first key point, it uses a similarity calculation formula to determine the similarity of each first key point according to the first coordinate value and the second coordinate value of each first key point. The similarity calculation formula is:

[0173]

[0174] Wherein, OKS represents similarity, p represents the number of the person image in the second training data set, represents the key point of the i-th category, and d pi represents the Euclidean distance between the first coordinate value and the second coordinate value, s p and σ i are scale factors, δ(v pi =1) is the Kronecker function.

[0175] In a feasible design, the processing module 11 is specifically used to randomly generate motion rate, angular velocity and noise function;

[0176] A preset number of key points are selected on a first image, where the first image is any image in the first training data set; a first area is determined according to the position of the key point to be moved, where the key point to be moved is any key point in the preset number of key points; each point in the first area is moved according to the motion speed, the angular velocity and the noise function to obtain a second area; and the second area is superimposed on the first image to obtain the second training data set.

[0177] In a feasible design, the processing module 11, when moving each point in the first area according to the movement speed, the angular velocity and the noise parameter to obtain the second area, is used to move each point in the first area according to a movement formula, as well as the movement speed, the angular velocity and the noise parameter to obtain the second area, and the movement formula is:

[0178]

[0179] Wherein, dst is the coordinate value of the first point after movement, src represents the first image, x and y represent the x coordinate and y coordinate of the first point before movement, respectively, the movement speed is v, and the angular velocity is Said Said noisy represents a noise function, and the first point is any point in the first area.

[0180] Figure 8 A schematic diagram of the structure of another key point detection model training device provided by an embodiment of the present invention, wherein the key point detection model training device is Figure 7 On this basis, further, it also includes:

[0181] The detection module 13 is used to input each image in the second training data set into the neural network model in the training module 12, and after iterating the neural network model using the loss function, convert the image to be detected into a red, green and blue (RGB) image, input the RGB image into the key point detection model, and obtain a probability confidence map of each key point in the RGB image; according to the position of the maximum probability point of each probability confidence map, use the position of the maximum probability point as the coordinate position of the key point corresponding to the probability confidence map.

[0182] The key point detection model training device provided in the embodiment of the present invention, its implementation principle and technology can be found in the above method embodiment, and will not be repeated here.

[0183] Fig. 9 A schematic diagram of the structure of another electronic device provided by an embodiment of the present invention. Fig. 9 As shown, the electronic device 200 includes:

[0184] At least one processor 21 and a memory 22;

[0185] The memory 22 stores computer-executable instructions;

[0186] The at least one processor 21 executes the computer-executable instructions stored in the memory 22, so that the at least one processor 21 performs the key point detection model training method as described above.

[0187] The specific implementation process of the processor 21 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0188] Optionally, the electronic device 200 further includes a communication component 23 , wherein the processor 21 , the memory 22 and the communication component 23 may be connected via a bus 24 .

[0189] An embodiment of the present invention further provides a storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the key point detection model training method as described above.

[0190] An embodiment of the present invention also provides a computer program product. When the computer program product is run on a computer, the computer executes the key point detection model training method as described above.

[0191] In the above embodiments, it should be understood that the described devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0192] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0194] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention.

[0195] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0196] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0197] The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present invention is not limited to only one bus or one type of bus.

[0198] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0199] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in a terminal or server.

[0200] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A key point detection model training method, characterized in that: include: Performing motion blur processing on each image in the first training data set to obtain a second training data set; Inputting each image in the second training data set into a neural network model, and iterating the neural network model using a loss function until the neural network model converges, and using the converged neural network model as a key point detection model, wherein the neural network model is a pre-built model, and the key point detection model is used to detect key points of a person in a person image; The step of performing motion blur processing on each image in the first training data set to obtain a second training data set includes: Randomly generate motion speed, angular velocity and noise function; Selecting a preset number of key points on a first image, where the first image is any image in the first training data set; Determine a first area according to a position of a key point to be moved, wherein the key point to be moved is any key point among the preset number of key points; According to the movement speed, the angular velocity and the noise function, each point in the first area is moved to obtain a second area; The second region is superimposed on the first image to obtain the second training data set.

2. The method according to claim 1, characterized in that The step of inputting each image in the second training data set into a neural network model and iterating the neural network model using a loss function comprises: Inputting a first group of images in the second training data set into the neural network model, the first group of images comprising a preset number of images; Determine the weight of each type of key points in the first group of images, where the same type of key points have the same human body position in different person images in each image included in the first group of images; Update the loss function according to the weights of various key points; Updating the neural network model using the updated loss function; The second group of images in the second training data set is input into the updated neural network model, and the above steps are repeated until a converged neural network model is obtained.

3. The method according to claim 2, characterized in that Determining the weight of each type of key point in the first group of images includes: For the first type of key points, using the neural network model, predicting the first coordinate value of the first key point of each person image contained in each image of the first group of images, each image in the first group of images includes at least one person image, each person image includes the first key point, and the first type of key points includes each of the first key points; Determining the similarity of each of the first key points according to a first coordinate value and a second coordinate value of each of the first key points, wherein the second coordinate value is a coordinate pre-marked on the character image for the first key point; The weights of the first-category key points are determined according to the similarities of the first key points.

4. The method according to claim 3, characterized in that Determining the weight of the first-category key points according to the similarities of the first key points includes: Determining, according to the similarities of the first key points, a first number of first key points having a similarity greater than a first threshold from among the first key points included in the first category of key points; Determining an average accuracy of the first type of key points according to a first number and a second number, wherein the second number is a total number of the first key points; The weight of the first type of key points is determined according to the average accuracy.

5. The method according to claim 4, characterized in that Determining the weight of the first type of key points according to the average accuracy rate includes: Determine the average accuracy rate according to the accuracy rate formula; Determine the reciprocal of the average accuracy rate, and use the reciprocal as the weight of the first type of key points; wherein the average accuracy rate formula is: Among them, AP represents the average accuracy, i represents the key point of the i-th category, OKS represents the similarity, p represents the number of the person image in the second training data set, and thres represents the first threshold.

6. The method according to any one of claims 2 to 5, characterized in that: The determining the similarity of each of the first key points according to the first coordinate value and the second coordinate value of each of the first key points includes: The similarity calculation formula is used to determine the similarity of each of the first key points according to the first coordinate value and the second coordinate value of each of the first key points. The similarity calculation formula is: Wherein, OKS represents similarity, p represents the number of the person image in the second training data set, represents the key point of the i-th category, and d pi represents the Euclidean distance between the first coordinate value and the second coordinate value, s p and σ i are the scale factors, δ(v pi =1) is the Kronecker function.

7. The method according to claim 1, characterized in that The step of moving each point in the first area according to the movement speed, the angular velocity and the noise function to obtain a second area comprises: According to the movement formula, the movement speed, the angular velocity and the noise function, each point in the first area is moved to obtain the second area, and the movement formula is: Wherein, dst is the coordinate value of the first point after movement, src represents the first image, x and y represent the x coordinate and y coordinate of the first point before movement, respectively, the movement speed is v, and the angular velocity is Said Said noisy represents a noise function, and the first point is any point in the first area.

8. The method according to any one of claims 1 to 5, characterized in that: After inputting each image in the second training data set into the neural network model and iterating the neural network model using the loss function, the method further includes: Convert the image to be detected into a red, green and blue (RGB) image, input the RGB image into the key point detection model, and obtain a probability confidence map of each key point in the RGB image; According to the position of the maximum probability point of each probability confidence map, the position of the maximum probability point is used as the coordinate position of the key point corresponding to the probability confidence map.

9. A key point detection model training device, characterized in that: include: A processing module, used for performing motion blur processing on each image in the first training data set to obtain a second training data set; A training module, used for inputting each image in the second training data set into a neural network model, and iterating the neural network model using a loss function until the neural network model converges, and using the converged neural network model as a key point detection model, wherein the neural network model is a pre-built model, and the key point detection model is used to detect key points of a person in a person image; The processing module is specifically used for: Randomly generate motion speed, angular velocity and noise function; Selecting a preset number of key points on a first image, where the first image is any image in the first training data set; Determine a first area according to a position of a key point to be moved, wherein the key point to be moved is any key point among the preset number of key points; According to the movement speed, the angular velocity and the noise function, each point in the first area is moved to obtain a second area; The second region is superimposed on the first image to obtain the second training data set.

10. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 8 is implemented when the processor executes the program.

11. A storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The program product stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.