Data set generation method, electronic equipment and computer readable medium
By generating target pose images with different poses, the problem of insufficient sample data during training of key point detection model is solved, the detection accuracy and generalization performance are improved, and the diversity of key point data sets is achieved.
Patent Information
- Application Number
- CN202310652023.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, due to insufficient sample data during training of key point detection models, the detection accuracy is not high and the diversity requirements cannot be met.
By obtaining the initial key point data set, target pose images with different poses are generated based on the coordinate information of the key point, so that a new key point data set is generated quickly, large and accurate, meeting the diversity requirements of the key point data set.
The accuracy and generalization performance of the key point detection model are improved, and various possible pose combinations of key points can be traversed without manual annotation.
Smart Images

Figure CN119068281A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a data set generation method, an electronic device, and a computer-readable medium. Background Art
[0002] Key point detection is the basic algorithm for many computer vision tasks. After obtaining the key points, it can be further used for tasks such as posture estimation and behavior recognition. These technologies have a wide range of application value in real life, such as intelligent security monitoring in public places, human-machine interaction, and can also be migrated to pet behavior detection. Taking the target object as a human as an example, due to the flexibility of the human body, various postures will appear. Changes in any part of the human body will produce new postures. At the same time, the visibility of key points is greatly affected by posture, clothing, and perspective, and is also affected by occlusion, lighting and other environmental factors. Therefore, when training the key point detection model, the training sample data needs to be diverse. However, the number of training sample data collected during model training is limited, which cannot meet the various situations faced in daily life, resulting in low detection accuracy of the trained key point detection model. At present, it is not possible to obtain training sample data with sufficient diversity. Summary of the invention
[0003] Embodiments of the present application provide a data set generation method, an electronic device, and a computer-readable medium.
[0004] In a first aspect, an embodiment of the present application provides a data set generation method, comprising: obtaining an initial key point data set; wherein the initial key point data set comprises: at least one first image and first coordinate information of key points of a target object in each of the first images; determining second coordinate information of the key points based on the first coordinate information of the key points in the first image to form a target posture different from the initial posture of the target object in the first image; generating a second image with the target posture based on the second coordinate information of the key points to obtain a target key point data set.
[0005] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; a memory, wherein at least one program is stored in the memory, and when the at least one program is executed by the at least one processor, any one of the above-mentioned data set generation methods is implemented.
[0006] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, any one of the above-mentioned data set generation methods is implemented.
[0007] The dataset generation method provided in the embodiment of the present application obtains a target key point dataset based on an initial key point dataset, does not require manual re-labeling, and can generate new key point datasets quickly, in large quantities, and accurately. The target key point dataset can traverse various possible posture combinations of the key points, meeting the diversity requirements of the key point dataset. From the perspective of optimizing the key point dataset, the accuracy and generalization performance of the key point detection model are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 A flowchart of a method for generating a data set provided in one embodiment of the present application;
[0009] Figure 2 A schematic diagram of key points in an embodiment of the present application;
[0010] Figure 3 This is a schematic diagram of the third circular area in the embodiment of the present application;
[0011] Figure 4 A schematic block diagram of the composition of an electronic device provided in accordance with another embodiment of the present application. DETAILED DESCRIPTION
[0012] In order to enable those skilled in the art to better understand the technical solution of the present application, the data set generation method, electronic device, and computer-readable medium provided by the present application are described in detail below in conjunction with the accompanying drawings.
[0013] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make this application thorough and complete and to enable those skilled in the art to fully understand the scope of this application.
[0014] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.
[0015] As used herein, the term "and / or" includes any and all combinations of at least one of the associated listed items.
[0016] The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of at least one other feature, whole, step, operation, element, component and / or its group is not excluded.
[0017] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.
[0018] Figure 1 A flowchart of a method for generating a data set provided in one embodiment of the present application.
[0019] First, refer to Figure 1 , the present application embodiment provides a data set generation method, including:
[0020] Step 100, obtaining an initial key point data set; wherein the initial key point data set includes: at least one first image and first coordinate information of key points of a target object in each first image.
[0021] In some exemplary embodiments, the initial key point dataset may be any one of public datasets such as Human3.6M, 3DHP, Surreal, Leeds Sports Pose (LSP), Microsoft Common Objects in Context (MSCOCO), etc.
[0022] In some exemplary embodiments, the target object in the first image may be any one of a person, an animal, a human face, or other movable objects.
[0023] In some exemplary embodiments, key points include movable key points and non-movable key points.
[0024] In some exemplary embodiments, Figure 2 As shown, the key points include at least one of the following: nose key point 0, neck key point 1, right shoulder key point 2, right elbow key point 3, right wrist key point 4, left shoulder key point 5, left elbow key point 6, left wrist key point 7, right hip key point 8, right knee key point 9, right ankle key point 10, left hip key point 11, left knee key point 12, left ankle key point 13, right eye key point 14, left eye key point 15, right ear key point 16, left ear key point 17.
[0025] In some exemplary embodiments, the nose key point can rotate around the neck key point and is a movable key point.
[0026] In some exemplary embodiments, the right elbow key point can rotate around the right shoulder key point and is a movable key point.
[0027] In some exemplary embodiments, the right wrist key point can rotate around the right elbow key point and is a movable key point.
[0028] In some exemplary embodiments, the left elbow key point can rotate around the left shoulder key point and is a movable key point.
[0029] In some exemplary embodiments, the left wrist key point can rotate around the left elbow key point and is a movable key point.
[0030] In some exemplary embodiments, the right knee key point can rotate around the right hip key point and is a movable key point.
[0031] In some exemplary embodiments, the right ankle key point can rotate around the right knee key point and is a movable key point.
[0032] In some exemplary embodiments, the left knee key point can rotate around the left hip key point and is a movable key point.
[0033] In some exemplary embodiments, the left ankle key point can rotate around the left knee key point and is a movable key point.
[0034] In some exemplary embodiments, the right eye key point, the left eye key point, the right ear key point, and the left ear key point may move along with the rotation of the nose key point, and are movable key points.
[0035] In some exemplary embodiments, the portion consisting of the neck key point, the left shoulder key point, the left hip key point, the right hip key point, and the right shoulder key point is considered to be a rigid body, and these key points are all non-movable key points.
[0036] In some exemplary embodiments, the key points include at least one of the following: left eyeball key point, right eyeball key point, left eye inner corner key point, left eye outer corner key point, right eye inner corner key point, right eye outer corner key point, left eyebrow inner end key point, left eyebrow outer end key point, right eyebrow inner end key point, right eyebrow outer end key point, nose tip key point, left mouth corner key point, right mouth corner key point, upper lip center key point, lower lip center key point.
[0037] In some exemplary embodiments, the left eyeball key point can be moved within the left eye and is a movable key point.
[0038] In some exemplary embodiments, the right eyeball key point can be moved within the right eye and is a movable key point.
[0039] In some exemplary embodiments, the left eyebrow inner end key point, the left eyebrow outer end key point, the right eyebrow inner end key point, the right eyebrow outer end key point, the left corner of the mouth key point, the right corner of the mouth key point, the upper lip center key point, and the lower lip center key point are all movable key points.
[0040] In some exemplary embodiments, the left eye inner corner keypoint, the left eye outer corner keypoint, the right eye inner corner keypoint, the right eye outer corner keypoint, and the nose tip keypoint are all non-movable keypoints.
[0041] Step 101, determining second coordinate information of key points according to first coordinate information of key points in a first image to form a target posture different from an initial posture of the target object in the first image; generating a second image with the target posture according to the second coordinate information of the key points to obtain a target key point data set.
[0042] In some exemplary embodiments, after obtaining the target key point data set, the method further includes: performing model training on the key point detection model according to the target key point data set.
[0043] In some exemplary embodiments, when determining the second coordinate information of a key point based on the first coordinate information of the key point in the first image, all key points in the first image should be traversed, that is, for each key point in the first image, the second coordinate information of the key point should be determined based on the first coordinate information of the key point in the first image.
[0044] In some exemplary embodiments, the number of key points in the first image is the same as the number of key points in the second image, and the key points in the first image correspond one-to-one to the key points in the second image.
[0045] In some exemplary embodiments, when determining the second coordinate information of the key point according to the first coordinate information of the key point in the first image, at least one different second coordinate information of the key point may be determined according to the first coordinate information of the same key point in the first image.
[0046] In some exemplary embodiments, determining the second coordinate information of a key point based on the first coordinate information of the key point in the first image includes: determining the second coordinate information of the movable key point based on the first coordinate information of the movable key point in the first image; wherein the second coordinate information of the movable key point is different from the first coordinate information of the movable key point in the first image; and using the first coordinate information of the non-movable key point in the first image as the second coordinate information of the non-movable key point.
[0047] In some exemplary embodiments, determining the second coordinate information of the movable key point according to the first coordinate information of the movable key point in the first image includes: determining the second coordinate information of the key point that rotates according to the first coordinate information of the key point that rotates among the movable key points in the first image, and the radius and angle of rotation of the key point that rotates; and using the first coordinate information of the key point that does not rotate among the movable key points in the first image as the second coordinate information of the key point that rotates. This implementation is for the case where there is no key point that moves with the key point that rotates.
[0048] In some exemplary embodiments, determining the second coordinate information of the movable key point according to the first coordinate information of the movable key point in the first image includes: determining the second coordinate information of the key point that rotates according to the first coordinate information of the key point that rotates among the movable key points in the first image, and the radius and angle of rotation of the key point that rotates; determining the second coordinate information of the key point that moves with the key point that rotates according to the first coordinate information of the key point that moves with the key point that rotates among the movable key points in the first image; using the first coordinate information of the key point that does not move with the key point that rotates among the movable key points in the first image as the second coordinate information of the key point that does not move with the key point that rotates. This implementation is for the case where there is a key point that moves with the key point that rotates.
[0049] In some exemplary embodiments, the radius of rotation refers to the distance between the key point where the rotation occurs and the origin of the rotation. For example, when the right elbow rotates around the right shoulder, the right shoulder is the origin of the rotation, and the right elbow is the key point where the rotation occurs, then the distance between the key point of the right shoulder and the key point of the right elbow is the radius of rotation. Other situations are similar and will not be described here.
[0050] In some exemplary embodiments, the rotation angle can be any angle between 0 and 360°. In order to traverse from 0 to 360, the second coordinate information of the key point corresponding to each angle with a fixed angle as the interval can be calculated with a fixed angle as the step size, and different angles correspond to different second images. Take the rotation of the right elbow around the right shoulder as an example, take the direction of the right elbow and the right shoulder in the first image as the initial direction, and start from the initial direction, and obtain the second coordinate information of the right elbow key point every 5 degrees. For example, after the right elbow rotates around the right shoulder by 5 degrees, a second coordinate information of the right elbow key point is obtained. After the right elbow rotates around the right shoulder by 5 degrees, the right wrist will move with the rotation of the right elbow. Therefore, it is necessary to calculate a second coordinate information of the right wrist key point. Except for the change of the coordinate information of the right elbow key point and the right wrist key point, the coordinate information of other key points does not change. A second coordinate information of the right elbow key point, a second coordinate information of the right wrist key point, and the second coordinate information of other key points correspond to a second image.
[0051] After the right elbow rotates 10 degrees around the right shoulder, another second coordinate information of the right elbow key point is obtained. After the right elbow rotates 10 degrees around the right shoulder, the right wrist will move with the rotation of the right elbow. Therefore, it is necessary to calculate another second coordinate information of the right wrist key point. Except for the changes in the coordinate information of the right elbow key point and the right wrist key point, the coordinate information of other key points does not change. The other second coordinate information of the right elbow key point, the other second coordinate information of the right wrist key point and the second coordinate information of other key points correspond to a second image.
[0052] By analogy, a total of 72 second images can be obtained.
[0053] In some exemplary embodiments, assuming that the first coordinate information of a key point that rotates in the first image is (x0, y0), the radius of rotation is r, the angle of rotation is a0, and the second coordinate information of the key point that rotates is (x1, y1), then, x1=x0+r×cos(a0×3.14 / 180), y1=x0+r×sin(a0×3.14 / 180).
[0054] In some exemplary embodiments, determining second coordinate information of a key point that moves along with the key point that rotates among movable key points based on first coordinate information of the key point that moves along with the key point that rotates in the first image includes: determining second coordinate information of the key point that moves along with the key point that rotates based on the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves along with the key point that rotates in the first image.
[0055] In some exemplary embodiments, determining the second coordinate information of a key point that moves along with the key point that rotates based on the first coordinate information of the key point that rotates in the first image and the first coordinate information of a key point that moves along with the key point that rotates in the first image includes: determining the relative position relationship information between the key point that rotates and the key point that moves along with the key point that rotates based on the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves along with the key point that rotates in the first image; determining the second coordinate information of the key point that moves along with the key point that rotates based on the second coordinate information of the key point that rotates and the relative position relationship information.
[0056] In some exemplary embodiments, determining the relative position relationship information between the key point that rotates and the key point that moves with the key point that rotates based on the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves with the key point that rotates in the first image includes: determining the relative position information as the difference between the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves with the key point that rotates in the first image.
[0057] Determining the second coordinate information of the key point that moves with the key point that rotates based on the second coordinate information and relative position relationship information of the key point that rotates includes: determining the second coordinate information of the key point that moves with the key point that rotates as the difference between the second coordinate information of the key point that rotates and the relative position relationship information.
[0058] In some exemplary embodiments, determining the relative position relationship information between the key point that rotates and the key point that moves with the key point that rotates based on the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves with the key point that rotates in the first image includes: determining the relative position information as the difference between the first coordinate information of the key point that moves with the key point that rotates in the first image and the first coordinate information of the key point that rotates in the first image.
[0059] Determining the second coordinate information of the key point that moves with the key point that rotates based on the second coordinate information and the relative position relationship information of the key point that rotates includes: determining the second coordinate information of the key point that moves with the key point that rotates as the sum of the second coordinate information of the key point that rotates and the relative position relationship information.
[0060] In some exemplary embodiments, when the key point that is rotated is the nose key point, the key points that move with the key point that is rotated include: a right eye key point, a left eye key point, a right ear key point, and a left ear key point.
[0061] In some exemplary embodiments, when the key point at which the rotation occurs is the right elbow key point, the key points that move with the key point at which the rotation occurs include the right wrist key point.
[0062] In some exemplary embodiments, when the key point where the rotation occurs is the right wrist key point, there is no key point that moves with the key point where the rotation occurs, that is, when the right wrist key point rotates around the right elbow key point, there is no key point that moves with the right wrist key point.
[0063] In some exemplary embodiments, when the key point at which the rotation occurs is the left elbow key point, the key points that move with the key point at which the rotation occurs include the left wrist key point.
[0064] In some exemplary embodiments, when the key point where the rotation occurs is the left wrist key point, there is no key point that moves with the key point where the rotation occurs, that is, when the left wrist key point rotates around the left elbow key point, there is no key point that moves with the left wrist key point.
[0065] In some exemplary embodiments, when the key point at which the rotation occurs is the right knee key point, the key points that move with the key point at which the rotation occurs include the right ankle key point.
[0066] In some exemplary embodiments, when the key point where the rotation occurs is the right ankle key point, there is no key point that moves with the key point where the rotation occurs, that is, when the right ankle key point rotates around the right knee key point, there is no key point that moves with the right ankle key point.
[0067] In some exemplary embodiments, when the key point at which the rotation occurs is the left knee key point, the key points that move with the key point at which the rotation occurs include the left ankle key point.
[0068] In some exemplary embodiments, when the key point where the rotation occurs is the left ankle key point, there is no key point that moves with the key point where the rotation occurs, that is, when the left ankle key point rotates around the left knee key point, there is no key point that moves with the left ankle key point.
[0069] In some exemplary embodiments, determining the second coordinate information of the movable key point based on the first coordinate information of the movable key point in the first image includes: determining the movable area of the key point that has moved based on the first coordinate information of the key point that has moved among the movable key points in the first image; determining the second coordinate information of the key point that has moved as the first coordinate information of the point in the movable area of the key point that has moved in the first image; and using the first coordinate information of the key point that has not moved among the movable key points in the first image as the second coordinate information of the key point that has not moved.
[0070] In some exemplary embodiments, the movable area of the left eye key point is a circular area with the midpoint of the left eye inner corner key point and the left eye outer corner key point as the center and a quarter of the distance between the left eye inner corner key point and the left eye outer corner key point as the radius. Assuming that the first coordinate information of the left eye inner corner key point in the first image is (x2, y2), and the first coordinate information of the left eye outer corner key point in the first image is (x3, y3), then the movable area of the left eye key point is As the center of the circle, is the first circular area with radius.
[0071] In some exemplary embodiments, the movable area of the right eye key point is a circular area with the midpoint of the right eye inner corner key point and the right eye outer corner key point as the center and a quarter of the distance between the right eye inner corner key point and the right eye outer corner key point as the radius. Assuming that the first coordinate information of the right eye inner corner key point in the first image is (x4, y4), and the first coordinate information of the right eye outer corner key point in the first image is (x5, y5), then the movable area of the right eye key point is As the center of the circle, The second circular area with radius .
[0072] In some exemplary embodiments, when the eyebrow is raised, the eyebrow moves up and down, and the first coordinate information of the key point of the inner end of the left eyebrow in the first image changes in the vertical coordinate. The range of change of the vertical coordinate of the first coordinate information of the key point of the inner end of the left eyebrow in the first image is half of the distance between the key point of the inner end of the left eyebrow and the midpoint of the key point of the inner corner of the left eye and the key point of the inner corner of the right eye in the vertical coordinate direction. The movable area of the key point of the inner end of the left eyebrow is the range of change of the vertical coordinate of the first coordinate information of the key point of the inner end of the left eyebrow. Assume that the first coordinate information of the key point of the inner corner of the left eye in the first image is (x2, y2), the first coordinate information of the key point of the inner corner of the right eye in the first image is (x4, y4), the first coordinate information of the key point of the inner end of the left eyebrow in the first image is (x6, y6), and the first coordinate information of the midpoint of the key point of the inner corner of the left eye and the key point of the inner corner of the right eye in the first image is The range of variation of the ordinate of the first coordinate information of the left eyebrow inner end key point in the first image is arrive in the range between.
[0073] In some exemplary embodiments, when the eyebrow is raised, the eyebrow moves up and down, and the first coordinate information of the left eyebrow outer end key point in the first image changes in the vertical coordinate. The vertical coordinate change range of the first coordinate information of the left eyebrow outer end key point in the first image is half of the distance between the left eyebrow outer end key point and the midpoint between the left eye outer corner key point and the right eye outer corner key point in the vertical coordinate direction. The movable area of the left eyebrow outer end key point is the vertical coordinate change range of the first coordinate information of the left eyebrow outer end key point in the first image. Assume that the first coordinate information of the left eye outer corner key point in the first image is (x3, y3), the first coordinate information of the right eye outer corner key point is (x5, y5), the first coordinate information of the left eyebrow outer end key point in the first image is (x7, y7), and the first coordinate information of the midpoint between the left eye outer corner key point and the right eye outer corner key point in the first image is The range of variation of the ordinate of the first coordinate information of the left eyebrow outer end key point in the first image is arrive in the range between.
[0074] In some exemplary embodiments, when the eyebrow is raised, the eyebrow moves up and down, and the first coordinate information of the key point of the inner end of the right eyebrow in the first image changes in the vertical coordinate. The range of change of the vertical coordinate of the first coordinate information of the key point of the inner end of the right eyebrow in the first image is half of the distance between the key point of the inner end of the right eyebrow and the midpoint between the key point of the inner corner of the left eye and the key point of the inner corner of the right eye in the vertical coordinate direction. The movable area of the key point of the inner end of the right eyebrow is the range of change of the vertical coordinate of the first coordinate information of the key point of the inner end of the right eyebrow in the first image. Assume that the first coordinate information of the key point of the inner corner of the left eye in the first image is (x2, y2), the first coordinate information of the key point of the inner corner of the right eye in the first image is (x4, y4), the first coordinate information of the key point of the inner end of the right eyebrow in the first image is (x8, y8), and the first coordinate information of the midpoint of the key point of the inner corner of the left eye and the key point of the inner corner of the right eye in the first image is The range of variation of the ordinate of the first coordinate information of the key point at the inner end of the right eyebrow in the first image is arrive in the range between.
[0075] In some exemplary embodiments, when the eyebrows are raised, the eyebrows move up and down, and the first coordinate information of the right eyebrow outer end key point in the first image changes in the vertical coordinate. The vertical coordinate change range of the first coordinate information of the right eyebrow outer end key point in the first image is half of the distance between the right eyebrow outer end key point and the midpoint between the left eye outer corner key point and the right eye outer corner key point in the vertical coordinate direction. The movable area of the right eyebrow outer end key point is the vertical coordinate change range of the first coordinate information of the right eyebrow outer end key point in the first image. Assume that the first coordinate information of the left eye outer corner key point in the first image is (x3, y3), the first coordinate information of the right eye outer corner key point in the first image is (x5, y5), the first coordinate information of the right eyebrow outer end key point in the first image is (x9, y9), and the first coordinate information of the midpoint between the left eye outer corner key point and the right eye outer corner key point in the first image is The range of the vertical coordinate of the first coordinate information of the key point at the outer end of the right eyebrow in the first image is arrive in the range between.
[0076] In some exemplary embodiments, the eyebrows move closer to the center when frowning. The movable area of the left eyebrow inner end key point is the product of the distance from the left eyebrow inner end key point to the midpoint of the left eye inner corner key point and the right eye inner corner key point and the distance coefficient; wherein the distance coefficient is a range of values. Assume that the first coordinate information of the left eye inner corner key point in the first image is (x2, y2), the first coordinate information of the right eye inner corner key point in the first image is (x4, y4), the first coordinate information of the left eyebrow inner end key point in the first image is (x6, y6), and the first coordinate information of the midpoint of the left eye inner corner key point and the right eye inner corner key point in the first image is The distance between the left eyebrow inner end key point to the left eye inner corner key point and the right eye inner corner key point in the first image is
[0077] In some exemplary embodiments, the distance factor is between 0.8 and 1.2.
[0078] In some exemplary embodiments, when frowning, the eyebrows move closer to the center of the eyebrows. The movable area of the left eyebrow outer end key point is the product of the distance from the left eyebrow outer end key point to the midpoint of the left eye outer corner key point and the right eye outer corner key point and the distance coefficient; wherein the distance coefficient is a range of values. Assume that the first coordinate information of the left eye outer corner key point in the first image is (x3, y3), the first coordinate information of the right eye outer corner key point is (x5, y5), the first coordinate information of the left eyebrow outer end key point in the first image is (x7, y7), and the first coordinate information of the midpoint of the left eye outer corner key point and the right eye outer corner key point in the first image is The distance between the left eyebrow outer end key point and the midpoint between the left eye outer corner key point and the right eye outer corner key point is
[0079] In some exemplary embodiments, the eyebrows move closer to the center when frowning. The movable area of the right eyebrow inner end key point is the product of the distance from the right eyebrow inner end key point to the midpoint of the left eye inner corner key point and the right eye inner corner key point and the distance coefficient; wherein the distance coefficient is a range of values. Assume that the first coordinate information of the left eye inner corner key point in the first image is (x2, y2), the first coordinate information of the right eye inner corner key point in the first image is (x4, y4), the first coordinate information of the right eyebrow inner end key point in the first image is (x8, y8), and the first coordinate information of the midpoint of the left eye inner corner key point and the right eye inner corner key point in the first image is The distance between the right eyebrow inner end key point to the midpoint of the left eye inner corner key point and the right eye inner corner key point is
[0080] In some exemplary embodiments, the eyebrows move closer to the center when frowning. The movable area of the right eyebrow outer end key point is the product of the distance from the right eyebrow outer end key point to the midpoint of the left eye outer corner key point and the right eye outer corner key point and the distance coefficient; wherein the distance coefficient is a range of values. Assume that the first coordinate information of the left eye outer corner key point in the first image is (x3, y3), the first coordinate information of the right eye outer corner key point in the first image is (x5, y5), the first coordinate information of the right eyebrow outer end key point in the first image is (x9, y9), and the first coordinate information of the midpoint of the left eye outer corner key point and the right eye outer corner key point in the first image is The distance between the right eyebrow outer end key point to the left eye outer corner key point and the midpoint of the right eye outer corner key point is
[0081] In some exemplary embodiments, Figure 3 As shown, the movable area of the left mouth corner key point is a sector-shaped area from 170° to 190° in the third circular area.
[0082] In some exemplary embodiments, Figure 3 As shown, the movable area of the right mouth corner key point is a sector-shaped area from 350° to 10° in the third circular area.
[0083] In some exemplary embodiments, Figure 3 As shown, the movable area of the key point of the upper lip center is a fan-shaped area from 70° to 110° in the third circular area.
[0084] In some exemplary embodiments, Figure 3 As shown, the movable area of the key point of the center of the lower lip is a fan-shaped area from 250° to 290° in the third circular area.
[0085] In some exemplary embodiments, Figure 3 As shown, the third circular area is a circular area with the midpoint of the right corner of mouth key point and the left corner of mouth key point as the center and three quarters of the distance between the right corner of mouth key point and the left corner of mouth key point as the radius.
[0086] In some exemplary embodiments, the angle range given above is given based on the direction of the line connecting the center of the circle to the right corner of the mouth key point being 0°. If 0° is in other directions, it can be derived based on the angle range obtained based on the direction of the line connecting the center of the circle to the right corner of the mouth key point being 0°.
[0087] In some exemplary embodiments, generating a second image with a target posture based on the second coordinate information of the key points to obtain a target key point data set includes: determining the feature information of the key points based on the initial key point data set; and inputting the second coordinate information of the key points and the feature information of the key points into an adversarial generation model to obtain the second image.
[0088] In some exemplary embodiments, determining the feature information of the key point based on the initial key point data set includes: intercepting at least one image block centered on the key point from the first image; performing feature extraction on each image block to obtain feature information of the image block; determining the feature information of the key point as the average value of feature information of the target image block; wherein the target image block includes: all image blocks centered on the key point intercepted from all first images.
[0089] In some exemplary embodiments, the size of the image block intercepted from the first image may be at least one of 32×32, 46×64, and 96×96. For example, the first coordinate information of a key point is (x, y), and the coordinate information of the upper left corner of the image block of size 32×32 intercepted from the first image is (x-16, y-16), the coordinate information of the lower left corner is (x-16, y+16), the coordinate information of the upper right corner is (x+16, y-16), and the coordinate information of the lower right corner is (x+16, y+16).
[0090] In some exemplary embodiments, extracting features from each image block to obtain feature information of the image block includes: inputting each image block into a feature extraction model to obtain feature information of the image block.
[0091] In some exemplary embodiments, the feature extraction module may be, for example, a vggnet pre-trained model.
[0092] In some exemplary embodiments, the target image block used to determine the feature information of a certain key point includes: all image blocks captured from all first images and centered on the certain key point.
[0093] In some exemplary embodiments, inputting the second coordinate information of the key point and the feature information of the key point into the adversarial generation model to obtain the second image includes: inputting the random number z, the second coordinate information of the key point and the feature information of the key point into the adversarial generation model to obtain the second image.
[0094] In some exemplary embodiments, the random number z, the first coordinate information of the key point in the first image, and the feature information of the key point can be used as inputs of the adversarial generative model, and the first image can be used as the output of the adversarial generative model to train the adversarial generative model.
[0095] In some exemplary embodiments, the adversarial generative model includes a generator and a discriminator. The generator generates an image based on a random number z, first coordinate information of the key point in the first image, and feature information of the key point. The discriminator compares the generated image with the first image to determine the authenticity of the generated image.
[0096] In some exemplary embodiments, in the process of training the adversarial generative model, the objective function used is:
[0097]
[0098] Where x is the first image, y is the first coordinate information of the key point in the first image and the feature information of the key point, z is a random number representing random noise, V(D, G) is the objective function value, pdata(x) is the distribution of the initial key point data set, E x~pdata(x) is the expectation of the initial key point dataset, pz(z) is the distribution of the generated image dataset, E z~pz(z) is the expectation of the generated image dataset, D(x|y) is the probability of whether x is real, G(z|y) is the image generated based on z and y, and D(G(z|y)) is the probability that G(z|y) is close to x.
[0099] In some exemplary embodiments, after generating a second image with a target posture according to the second coordinate information of the key points to obtain a target key point data set, the method further includes: detecting the key points in the second image to obtain the third coordinate information of the key points in the second image, and when the third coordinate information of the key points in the second image obtained by detection is significantly different from the second coordinate information of the key points, determining that the adversarial generation model is broken, and regenerating a second image with a target posture according to the second coordinate information of the key points to obtain a target key point data set. Ensure that the adversarial generation model generates the second image strictly in accordance with the constraint conditions.
[0100] In some exemplary embodiments, the training data in the initial key point dataset may also be added to the final key point dataset.
[0101] In some exemplary embodiments, after obtaining the target key point data set, the method further includes: performing style migration on the data in the target key point data set, and adding the data obtained by style migration to the target key point training data set.
[0102] In some exemplary embodiments, after obtaining the target key point data set, the method further includes: filtering out data with artifacts in the target key point data set. This method can improve the overall quality of the target key point data set.
[0103] The dataset generation method provided in the embodiment of the present application obtains a target key point dataset based on an initial key point dataset, does not require manual re-labeling, and can generate new key point datasets quickly, in large quantities, and accurately. The target key point dataset can traverse various possible posture combinations of the key points, meeting the diversity requirements of the key point dataset. From the perspective of optimizing the key point dataset, the accuracy and generalization performance of the key point detection model are improved.
[0104] Second, refer to Figure 4 Another embodiment of the present application provides an electronic device, including: at least one processor 401; a memory 402, wherein at least one program is stored in the memory 402, and when the at least one program is executed by the at least one processor 401, any one of the above-mentioned data set generation methods is implemented.
[0105] In some exemplary embodiments, the electronic device further includes: one or more I / O interfaces 403 connected between the processor 401 and the memory 402 , and configured to implement information interaction between the processor 401 and the memory 402 .
[0106] Among them, the processor 401 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 402 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 403 is connected between the processor 401 and the memory 402, and can realize information interaction between the processor 401 and the memory 402, including but not limited to a data bus (Bus), etc.
[0107] In some embodiments, the processor 401 , the memory 402 , and the I / O interface 403 are connected to each other via a bus 404 , and further connected to other components of the computing device.
[0108] In a third aspect, another embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein the computer program implements any of the above-mentioned data set generation methods when executed by a processor.
[0109] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0110] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.
Claims
1. A method for generating a data set, comprising: Acquire an initial key point data set; wherein the initial key point data set includes: at least one first image and first coordinate information of key points of a target object in each of the first images; determining second coordinate information of the key point according to first coordinate information of the key point in the first image to form a target posture different from an initial posture of the target object in the first image; A second image having the target posture is generated according to the second coordinate information of the key point to obtain a target key point data set.
2. The method for generating a data set according to claim 1, wherein: The key points include: movable key points and non-movable key points; Determining the second coordinate information of the key point according to the first coordinate information of the key point in the first image includes: Determining second coordinate information of the movable key point according to first coordinate information of the movable key point in the first image; wherein the second coordinate information of the movable key point is different from the first coordinate information of the movable key point in the first image; The first coordinate information of the non-movable key point in the first image is used as the second coordinate information of the non-movable key point.
3. The method for generating a data set according to claim 2, wherein: Determining the second coordinate information of the movable key point according to the first coordinate information of the movable key point in the first image comprises: Determine, according to the first coordinate information of the key point that rotates among the movable key points in the first image, and the radius and angle of rotation of the key point that rotates, the second coordinate information of the key point that rotates after the key point rotates by the angle; Determine second coordinate information of a key point that moves with the key point that rotates according to first coordinate information of the key point that rotates in the first image and first coordinate information of a key point that moves with the key point that rotates among the movable key points in the first image; The first coordinate information of the key point that does not move with the key point that rotates among the movable key points in the first image is used as the second coordinate information of the key point that does not move with the key point that rotates.
4. The method for generating a data set according to claim 3, wherein: The determining, based on the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves with the key point that rotates among the movable key points in the first image, second coordinate information of the key point that moves with the key point that rotates includes: Determine relative position relationship information between the key point that rotates and the key point that moves with the key point that rotates according to first coordinate information of the key point that rotates in the first image and first coordinate information of the key point that moves with the key point that rotates in the first image; The second coordinate information of the key point moving with the key point that rotates is determined according to the second coordinate information of the key point that rotates and the relative position relationship information.
5. The method for generating a data set according to claim 4, wherein: Determining the relative position relationship information between the key point that rotates and the key point that moves with the key point that rotates according to the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves with the key point that rotates in the first image comprises: determining the relative position information as the difference between the first coordinate information of the key point that rotates in the first image and the first coordinate information of the key point that moves with the key point that rotates in the first image; The method of determining the second coordinate information of the key point that moves with the key point that rotates based on the second coordinate information of the key point that rotates and the relative position relationship information includes: determining the second coordinate information of the key point that moves with the key point that rotates as the difference between the second coordinate information of the key point that rotates and the relative position relationship information.
6. The method for generating a data set according to claim 2, wherein: Determining the second coordinate information of the movable key point according to the first coordinate information of the movable key point in the first image comprises: Determine a movable area of the key point that moves according to first coordinate information of the key point that moves among the movable key points in the first image; Determine the second coordinate information of the key point where the movement occurs as the first coordinate information of a point in the movable area of the key point where the movement occurs in the first image; The first coordinate information of the key point that has not moved among the movable key points in the first image is used as the second coordinate information of the key point that has not moved.
7. The method for generating a data set according to any one of claims 1 to 6, wherein: Generating a second image having the target posture according to the second coordinate information of the key point to obtain a target key point data set includes: Determine the feature information of the key points according to the initial key point data set; The second coordinate information of the key point and the feature information of the key point are input into the adversarial generation model to obtain the second image.
8. The method for generating a data set according to claim 7, wherein: Determining the feature information of the key point according to the initial key point data set includes: Cutting out at least one image block centered on the key point from the first image; performing feature extraction on each of the image blocks to obtain feature information of the image block; The characteristic information of the key point is determined as an average value of characteristic information of a target image block; wherein the target image block includes: all image blocks centered on the key point and intercepted from all the first images.
9. An electronic device, comprising: at least one processor; A memory, wherein at least one program is stored in the memory, and when the at least one program is executed by the at least one processor, the data set generation method according to any one of claims 1 to 8 is implemented.
10. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data set generation method according to any one of claims 1 to 8.
