Object key point recognition model generation method and object key point recognition method
By cropping boundary points to generate sample out-of-bounds images and training the model, the problem of low key point recognition accuracy caused by manual annotation is solved, and efficient and accurate object key point recognition is achieved.
Patent Information
- Application Number
- CN202210513485.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-11
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2042-05-11
AI Technical Summary
In existing technologies, key point annotation by manually collecting images of objects that have crossed boundaries is costly, has low annotation quality, and is highly subjective, which leads to a decrease in the recognition accuracy of key point recognition models in non-boundary situations.
By acquiring multiple original images of the target object and their key points, pre-defined cropping boundary points are determined, sample images that exceed the boundaries are cropped, and these images are combined to train the model, thereby generating an object key point recognition model.
This improved the accuracy and stability of the object key point recognition model in boundary crossing situations, reduced manual costs, and improved the accuracy of key point annotation.
Smart Images
Figure CN116994304B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of image processing, and particularly relates to a method for generating an object key point recognition model and a method for recognizing object key points. BACKGROUND
[0002] The object in the image is out of bounds, that is, part of the object in the image is outside the image. The scenario of object out of bounds is extremely common, and the object out of bounds may exist in mobile AR (Augmented Reality) special effect applications such as face special effects, gesture special effects and body special effects. The application of the above effects all need to be based on the recognition of object key points.
[0003] In the related art, in order to be able to realize key point recognition in a non-out-of-bound situation, the model can be trained by increasing the image in the object out-of-bound situation and the corresponding key points, so that the model learns to enable the model to recognize the key points in the image in the object out-of-bound situation. However, for the object out-of-bound image and the corresponding key points as samples, the object out-of-bound image is currently mainly collected by manual collection. In the above method, the object out-of-bound image needs to consume a lot of manual cost and time, and secondly, due to the randomness of manual labeling and the existence of subjective consciousness, there is a problem that the position of the face key point when the object is out of bounds cannot be accurately labeled, which will lead to a decrease in the quality of key point labeling, a decrease in the accuracy of the key points, and thus will fundamentally affect the key point recognition model. The accuracy of key point recognition in a non-out-of-bound situation. SUMMARY
[0004] In view of the above technical problems, the present disclosure provides a method for generating an object key point recognition model and a method for recognizing object key points.
[0005] According to an aspect of an embodiment of the present disclosure, a method for generating an object key point recognition model is provided, comprising:
[0006] obtaining a plurality of original images of a target object and a plurality of object key points corresponding to the target object in the plurality of original images;
[0007] determining preset cropping boundary points corresponding to the plurality of original images, the preset cropping boundary points being points with a distance from the image center point of the original image less than a preset distance;
[0008] determining a cropping boundary of the plurality of original images according to the preset cropping boundary points;
[0009] cropping the plurality of original images based on the cropping boundary to obtain a plurality of sample out-of-bound images of the target object;
[0010] determine target object key points located in the plurality of sample out-of-bound images from a plurality of object key points corresponding to the plurality of sample out-of-bound images;
[0011] perform object key point identification training on a to-be-trained model based on the plurality of sample out-of-bound images and the target object key points, to obtain an object key point identification model.
[0012] According to another aspect of the embodiments of the present disclosure, an object key point identification method is provided, comprising:
[0013] obtaining a target to-be-identified image;
[0014] inputting the target to-be-identified image into an object key point identification model obtained according to the object key point identification model generation method to perform object key point identification processing, to obtain object key points corresponding to the target to-be-identified image.
[0015] According to another aspect of the embodiments of the present disclosure, an object key point identification model generation apparatus is provided, comprising:
[0016] an information obtaining module configured to obtain a plurality of original images of a target object and a plurality of object key points corresponding to the target object in the plurality of original images;
[0017] a preset clipping boundary point determining module configured to determine preset clipping boundary points corresponding to the plurality of original images, the preset clipping boundary points being points having a distance from a center point of the original image less than a preset distance;
[0018] a clipping boundary determining module configured to determine clipping boundaries of the plurality of original images according to the preset clipping boundary points;
[0019] a sample out-of-bound image obtaining module configured to perform clipping processing on the plurality of original images based on the clipping boundaries, to obtain a plurality of sample out-of-bound images of the target object;
[0020] a target object key point determining module configured to determine target object key points located in the plurality of sample out-of-bound images from a plurality of object key points corresponding to the plurality of sample out-of-bound images;
[0021] a key point identification model generation module configured to perform object key point identification training on a to-be-trained model based on the plurality of sample out-of-bound images and the target object key points, to obtain an object key point identification model.
[0022] According to another aspect of the embodiments of the present disclosure, an object key point identification apparatus is provided, comprising:
[0023] a target to-be-identified image obtaining module configured to obtain a target to-be-identified image;
[0024] An object key point identification module is configured to input the target image to be identified into an object key point identification model generated according to the method for generating an object key point identification model, to perform object key point identification processing, and to obtain object key points corresponding to the target image to be identified.
[0025] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, including a processor, a memory for storing instructions executable by the processor, and wherein the processor is configured to execute the instructions to implement the method for generating an object key point identification model or implement the object key point identification method.
[0026] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for generating an object key point identification model or perform the object key point identification method.
[0027] According to another aspect of the embodiments of the present disclosure, a computer program product containing instructions is provided, when the computer program product is run on a computer, the computer is enabled to perform the method for generating an object key point identification model or perform the object key point identification method.
[0028] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:
[0029] The plurality of original images are cropped by the cropping boundary determined by the preset cropping boundary points to obtain a plurality of sample out-of-bound images, the plurality of sample out-of-bound images can be quickly generated, and the target object key points located in the sample out-of-bound images are determined from the plurality of object key points corresponding to the plurality of original images in combination with the plurality of sample out-of-bound images, which can improve the accuracy of the target object key points of the sample out-of-bound images, and then the object key point identification model is obtained by performing object key point identification training on the to-be-trained model in combination with the plurality of sample out-of-bound images and the target object key points, which can greatly improve the object key point identification precision and stability of the object key point identification model under the object out-of-bound condition.
[0030] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS
[0031] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.
[0032] Figure 1is a schematic diagram of an application system according to an exemplary embodiment;
[0033] Figure 2 is a flowchart of a method for generating an object key point recognition model according to an exemplary embodiment;
[0034] Figure 3 is a flowchart of a method for determining a preset crop boundary point corresponding to a plurality of original images according to an exemplary embodiment;
[0035] FIGS. 4(a)(b) are schematic diagrams of a crop boundary in a method for generating an object key point recognition model according to an exemplary embodiment;
[0036] Figure 5 is a flowchart of a method for cropping a plurality of original images based on a crop boundary to obtain a plurality of sample out-of-bound images of a target object according to an exemplary embodiment;
[0037] Figure 6 is a flowchart of a method for determining a mask region from a first region and a second region according to an exemplary embodiment;
[0038] Figure 7 is a block diagram of a device for generating an object key point recognition model according to an exemplary embodiment;
[0039] Figure 8 is a block diagram of an electronic device for generating an object key point recognition model or recognizing an object key point according to an exemplary embodiment;
[0040] Figure 9 is a block diagram of an electronic device for generating an object key point recognition model or recognizing an object key point according to an exemplary embodiment. DETAILED DESCRIPTION
[0041] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings denote the same or similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.
[0042] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.
[0043] In addition, for a better illustration of the present application, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that the present application can be practiced without certain specific details. In some instances, well-known methods, apparatus, elements and circuits have not been described in detail in order to avoid obscuring the present application.
[0044] Artificial Intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other directions.
[0045] In recent years, with the research and progress of artificial intelligence technology, artificial intelligence technology has been widely applied in many fields. The scheme provided in the embodiments of the present application relates to machine learning / deep learning and other technologies, which are specifically explained as follows:
[0046] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application system according to an exemplary embodiment. The application system can be used for the object key point identification model generation method or object key point identification method of the present application. As shown in Figure 1 , the application system can at least include a server 01 and a terminal 02.
[0047] In the embodiments of the present application, the server 01 can be used to generate an object key point identification model. Specifically, the server 01 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0048] In the embodiments of the present application, the terminal 02 can perform object key point identification processing based on the object key point identification model trained by the server 01. The terminal 02 can include physical devices such as smartphones, desktop computers, tablet computers, notebook computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, and can also include software running in physical devices, such as applications and the like. The operating system running on the terminal 02 in the embodiments of the present application can include but is not limited to Android system, IOS system, Linux, Windows, etc.
[0049] In addition, it should be noted that Figure 1The shown is only one application environment provided by the present disclosure, and other application environments can also be included in actual application, for example, the generation of the object key point recognition model can also be implemented on the terminal 02.
[0050] In the embodiments of the present specification, the terminal 02 and the server 01 can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application.
[0051] It should be noted that the following figures show a possible sequence of steps, which actually does not limit the strict sequence. Some steps can be executed in parallel without mutual dependence.
[0052] Specifically, Figure 2 is a flowchart of a method for generating an object key point recognition model according to an example embodiment. As shown in Figure 2 The method for generating an object key point recognition model is used in electronic devices such as terminals or servers, and includes the following steps:
[0053] In step S201, a plurality of original images of a target object and a plurality of object key points corresponding to the target object in the plurality of original images are obtained.
[0054] In one specific embodiment, the target object can refer to an object whose key points need to be recognized. The target object can include a face or a hand, for example, the target object can be a face, a hand or a human body, etc. The original image can be an image containing all features of the target object and not being masked. Each original image can correspond to a plurality of object key points; the plurality of object key points of each original image can represent the characteristics of the target object.
[0055] In one specific embodiment, the plurality of object key points corresponding to the target object in the plurality of original images are obtained in the following manner:
[0056] For each original image, object key point recognition is performed to obtain a plurality of object key points corresponding to each original image.
[0057] In one specific embodiment, a key point recognition model corresponding to the original image can be set in advance, and each original image can be input into the key point recognition model to obtain a plurality of object key points corresponding to each original image. Specifically, the key point recognition model corresponding to the original image can be obtained by training a preset deep learning model based on sample training data (which can include a plurality of sample images containing all features of the target object and a plurality of sample feature key points corresponding to the sample images).
[0058] In one specific embodiment, the target object in the original image can not be in the center region of the original image, and the method can further include:
[0059] Based on the plurality of object key points corresponding to each original image, subject alignment processing is performed on each original image to obtain a plurality of aligned original images and a plurality of object key points corresponding to the plurality of aligned original images.
[0060] In one specific embodiment, a target position after key point alignment can be preset; based on the plurality of object key points corresponding to each original image and the target position, a general transformation is performed to obtain an alignment matrix corresponding to each original image; and according to the alignment matrix, an alignment operation is performed on each original image to obtain a plurality of aligned original images and a plurality of object key points corresponding to each aligned original image.
[0061] In one specific embodiment, the plurality of original images can be the plurality of aligned original images; and the plurality of object key points corresponding to the target object in each original image can be the plurality of object key points corresponding to each aligned original image.
[0062] In the above embodiment, through the subject alignment processing, the target object in the original image can be in the central region of the original image, which can avoid generating sample images with invalid border crossing due to the target object not being in the central region of the image, and further avoid the increase in computational complexity in the training process caused by the sample images with invalid border crossing, improve the model training efficiency, and ensure the object key point recognition accuracy of the model.
[0063] In step S202, a preset cropping boundary point corresponding to a plurality of original images is determined.
[0064] In one specific embodiment, the preset cropping boundary point can be a point with a distance from the image center point of the original image less than a preset distance.
[0065] In one specific embodiment, the method further includes:
[0066] A preset distance is obtained.
[0067] In one specific embodiment, the preset distance can be a random variable. Specifically, the preset distance h can be determined by the following formula:
[0068]
[0069] wherein H and W can be the length and width of the original image respectively; factor is a preset parameter which can control the size of the preset distance h. Specifically, the value range of factor can be 2-3. It can be understood that the above value range can avoid the situation that the part of the target object cropped by the cropping boundary in the final generated image is too small in the proportion of the target object, which can effectively avoid the invalid boundary crossing, and further can avoid the situation that the calculation amount in the training process is increased due to the invalid boundary crossing sample image, and also can avoid the situation that the randomness of the sample boundary crossing image is affected due to the too small preset distance h, thereby ensuring the object key point recognition accuracy of the model.
[0070] In one specific embodiment, as shown in FIG. 2A, the step S202 can include: Figure 3
[0071] S301. Determine the image center point of each original image.
[0072] In one specific embodiment, the image center point of each original image can be a point located at the center position of the original image. Specifically, the center position of each original image can be determined based on the size (e.g. length and width) of each original image, and the point located at the center position can be taken as the image center point of each original image.
[0073] S302. Determine the cropping boundary point range based on the preset distance and the image center point.
[0074] In one specific embodiment, the cropping boundary point range can be used to determine the preset cropping boundary point. Specifically, a circle can be drawn with the image center point as the center and the preset distance as the radius, and the circular range can be taken as the cropping boundary point range.
[0075] S303. Determine the preset cropping boundary point from the cropping boundary point range.
[0076] In one specific embodiment, a random point can be taken from the cropping boundary point range, and the obtained random point can be taken as the preset cropping boundary point.
[0077] In one specific embodiment, if the target object in the original image is not located in the center region of the original image, the subject alignment processing can be performed on each original image based on the plurality of object key points corresponding to each original image, and a plurality of aligned original images and a plurality of object key points corresponding to the plurality of aligned original images can be obtained. Specifically, after the alignment processing, the preset cropping boundary point corresponding to the plurality of aligned original images can be determined.
[0078] In the above embodiment, the preset distance and the image center point are used to determine the range of the cutting boundary points, and the preset cutting boundary points are determined from the range of the cutting boundary points. This can ensure that the out-of-boundary degree of the sample out-of-boundary image is controllable on the basis of randomness of the preset cutting boundary points, and thus can relatively reduce invalid out-of-boundary situations in the multiple sample out-of-boundary images, thereby ensuring the object key point recognition accuracy of the model.
[0079] In one specific embodiment, the preset cutting boundary point (x c , y c ) can be directly obtained by the following formula:
[0080]
[0081] wherein r and a can be random variables, a = 2p * Uniform (0, 1); H and W can be the length and width of the original image, respectively; factor is a hyperparameter for controlling the size of r, and the value range of factor can be 2-3; Uniform (0, 1) can refer to a random number between 0 and 1.
[0082] In step S203, the cutting boundary of the multiple original images is determined according to the preset cutting boundary point.
[0083] In one specific embodiment, the cutting boundary of each original image can be used to determine the range to be cut of the original image. The cutting boundary can be a boundary line passing through the preset cutting boundary point. As shown in FIG. 4(a), the cutting boundary can be a straight line segment; as shown in FIG. 4(b), the cutting boundary can be a broken line segment with a right angle.
[0084] In one specific embodiment, the above method further comprises:
[0085] obtaining preset boundary direction information.
[0086] In one specific embodiment, the preset boundary direction information corresponding to each original image can represent the direction of the cutting boundary corresponding to the original image on the original image. The preset boundary direction information θ can be obtained by the following formula:
[0087] θ = 2p * Uniform (0, 1)
[0088] In one specific embodiment, the above step S203 can comprise:
[0089] determining the cutting boundary according to the preset cutting boundary point and the preset boundary direction information.
[0090] In one specific embodiment, the straight line equation corresponding to the cutting boundary can be obtained by the following formula:
[0091] y = k * x + b
[0092] wherein, k = tan(0); b = y c -k * x c ; x c is the horizontal coordinate of a preset clipping boundary point, y c is the vertical coordinate of a preset clipping boundary point; x is the horizontal coordinate of an arbitrary point on the straight line corresponding to the clipping boundary, and y is the vertical coordinate of the arbitrary point on the clipping boundary.
[0093] In a specific embodiment, since there are two cases of the target object exceeding the boundary line or exceeding the boundary corner in actual application, the above method further comprises:
[0094] obtaining a preset clipping boundary type.
[0095] In a specific embodiment, the preset clipping boundary type can include a straight line type or a right angle type, etc. The preset clipping boundary type can be determined by random probability. Specifically, the probability range of the preset clipping boundary type being a straight line type can be 90%-95%; correspondingly, the probability range of the preset clipping boundary type being a right angle type can be 5%-10%. For example, the probability of the preset clipping boundary type being a straight line type can be 92%, and the probability of the preset clipping boundary type being a right angle type can be 8%. It can be understood that the case of the target object exceeding the boundary corner is less than the case of exceeding the boundary line in actual situation, and the probability of the preset clipping boundary type being a right angle type can be relatively low.
[0096] In a specific embodiment, the above step S203 can comprise:
[0097] determining the clipping boundary based on the preset clipping boundary point and the preset clipping boundary type.
[0098] In a specific embodiment, in the case of the preset clipping boundary type being a straight line type, the straight line equation corresponding to the clipping boundary can be obtained according to the preset clipping boundary point and the preset boundary direction information. In the case of the preset clipping boundary type being a right angle type, the first boundary line can be obtained according to the preset clipping boundary point and the preset boundary direction information; the second boundary line can be obtained by making a straight line perpendicular to the first boundary line through the preset clipping boundary point; and based on the intersecting first boundary line and second boundary line, the line segment on the upper side of the preset clipping boundary point in the original image can be taken as the clipping boundary.
[0099] In the above embodiment, by setting multiple preset clipping boundary types, multiple sample out-of-boundary images can be simulated to different out-of-boundary situations, thereby improving the recognition accuracy of the object key point of the object key point recognition model.
[0100] In step S204, based on the cropping boundary, the plurality of original images are cropped to obtain a plurality of sample out-of-bound images of the target object.
[0101] In one specific embodiment, as shown in Figure 5 The step S204 can include the following steps.
[0102] S501. Based on the cropping boundary of each original image, the image region corresponding to each original image is segmented to obtain a first region and a second region corresponding to each original image.
[0103] In one specific embodiment, the image region corresponding to each original image can refer to the region within the boundary range of the original image. The first region and the second region corresponding to each original image can be regions located on the two sides of the cropping boundary of the original image. It can be understood that the combination of the first region and the second region can obtain the complete image region.
[0104] In one specific embodiment, taking the cropping boundary of each original image and the corresponding image boundary as the region boundary can obtain two regions, and the above two regions are taken as the first region and the second region respectively.
[0105] S502. Determine the mask region from the first region and the second region.
[0106] In one specific embodiment, the mask region of each original image can refer to the region to be masked in the original image; the mask region of each original image can be used to mask the original image to obtain a sample out-of-bound image.
[0107] In one specific embodiment, as shown in Figure 6 The step S402 can include the following steps.
[0108] S601. Take the region in the first region and the second region located on the upper side of the cropping boundary as a target region.
[0109] It should be noted that since the first region and the second region are located on the two sides of the cropping boundary, the first region and the second region can be located on the upper side of the cropping boundary and the lower side of the cropping boundary respectively. For the special case that the cropping boundary is perpendicular to the bottom boundary of the original image, either the first region or the second region can be taken as the target region.
[0110] S602. If the proportion information of the target region in the corresponding original image is less than or equal to the preset proportion information, take the target region as the mask region.
[0111] In a specific embodiment, the proportion information of the target region in the corresponding original image can be pixel proportion information of the target region in the corresponding original image. The preset proportion information can be set according to actual application requirements. Optionally, the preset proportion information can be in a range of 70%-80%. For example, the preset proportion information can be 75%.
[0112] In S603, if the proportion information of the target region in the corresponding original image is greater than the preset proportion information, the non-target region is taken as the mask region.
[0113] In a specific embodiment, the non-target region can be a region in the image region except the target region.
[0114] In the above embodiment, by comparing the proportion information of the target region in the corresponding original image with the preset proportion information, it is ensured that the proportion information of the mask region is not less than the preset proportion information, which can avoid excessive masking due to the excessive area of the mask region, thereby increasing the difficulty of model training, and further improving the model training efficiency on the basis of ensuring the object key point recognition accuracy of the model.
[0115] In S503, based on the mask region, each original image is subjected to masking processing to obtain a sample out-of-bound image corresponding to each original image.
[0116] In a specific embodiment, by assigning the pixel points in each original image located in the mask region as 0, a sample out-of-bound image after masking processing corresponding to each original image can be obtained.
[0117] In a specific embodiment, if the target object in the original image is not located in the center region of the original image, based on the plurality of object key points corresponding to each original image, a subject alignment processing can be performed on each original image to obtain a plurality of aligned original images and a plurality of object key points corresponding to the plurality of aligned original images. Specifically, after the alignment processing, based on the cropping boundary, a cropping processing can be performed on the plurality of aligned original images to obtain a plurality of sample out-of-bound images of the target object.
[0118] In the above embodiment, by using the cropping boundary determined by the preset cropping boundary point to crop the plurality of original images to obtain the plurality of sample out-of-bound images, the plurality of sample out-of-bound images can be quickly generated, thereby improving the generation efficiency of the object key point recognition model.
[0119] In step S205, target object key points located in the plurality of sample out-of-bound images are determined from the plurality of object key points corresponding to the plurality of sample out-of-bound images.
[0120] In a specific embodiment, the target object key point corresponding to each sample out-of-bound image can be a key point located in the sample out-of-bound image from among the plurality of object key points corresponding to the original image corresponding to the sample out-of-bound image. The target object key point corresponding to each sample out-of-bound image can be at least one of the plurality of object key points of the original image corresponding to the sample out-of-bound image.
[0121] In a specific embodiment, the boundary range of the sample out-of-bound image can be obtained based on the sample out-of-bound image, and the target object key point can be determined as a point located in the boundary range of the sample out-of-bound image from among the plurality of object key points of the original image corresponding to the sample out-of-bound image based on the position information of the plurality of object key points.
[0122] In the above embodiment, by combining the plurality of sample out-of-bound images, the target object key point located in the sample out-of-bound image is determined from the plurality of object key points corresponding to the plurality of original images, which can improve the accuracy of the target object key point of the sample out-of-bound image.
[0123] In step S206, the object key point recognition model is obtained by performing object key point recognition training on the to-be-trained model based on the plurality of sample out-of-bound images and the target object key points.
[0124] In a specific embodiment, the current training data can be determined from the training data set composed of the plurality of sample out-of-bound images and the target object key points, the sample out-of-bound image in the current training data is input into the object key point recognition model for object key point recognition processing to obtain a recognition processing result including a plurality of recognized object key points, the loss information is determined based on the recognition processing result and the plurality of object key points in the current training data, and the parameters in the object key point recognition model are adjusted based on the loss information. The training of the object key point recognition processing based on the current training data can be repeated, and the trained object key point recognition model can be obtained.
[0125] In a specific embodiment, the object key point recognition model can be obtained by performing object key point recognition training on the to-be-trained model based on the plurality of sample out-of-bound images and the target object key points, in combination with the plurality of non-out-of-bound images and the plurality of object key points corresponding to the plurality of non-out-of-bound images.
[0126] In the above embodiment, the object key point recognition model is obtained by performing object key point recognition training on the to-be-trained model in combination with the plurality of non-out-of-bound images, the plurality of object key points corresponding to the plurality of non-out-of-bound images, and the plurality of sample out-of-bound images and the target object key points. On the basis of ensuring that the object key point recognition accuracy is not lost under the non-out-of-bound condition, the object key point recognition accuracy when the object is out-of-bound is improved, and the recognition ability of the object key point recognition model for the object key point is further improved.
[0127] In the above embodiment, the plurality of original images are cropped by the cropping boundary determined by the preset cropping boundary point to obtain a plurality of sample out-of-bound images, so that the plurality of sample out-of-bound images can be quickly generated. In combination with the plurality of sample out-of-bound images, the target object key points located in the sample out-of-bound images are determined from the plurality of object key points corresponding to the plurality of original images, so that the accuracy of the target object key points of the sample out-of-bound images can be improved. In combination with the plurality of sample out-of-bound images and the target object key points, the object key point recognition training is performed on the to-be-trained model to obtain the object key point recognition model, so that the object key point recognition precision and stability of the object key point recognition model under the object out-of-bound condition can be greatly improved.
[0128] Based on the object key point recognition model obtained by the above object key point recognition model generation method, the present disclosure further provides an object key point recognition method, which can be used in terminal, server and other electronic devices, and can include the following steps:
[0129] Obtaining a target to-be-recognized image;
[0130] Inputting the target to-be-recognized image into the object key point recognition model obtained according to the above object key point recognition model generation method for object key point recognition processing to obtain the object key points corresponding to the target to-be-recognized image.
[0131] In one specific embodiment, the above object key point recognition method can be used in, but not limited to, video image processing, segment video editing, video call and other applications requiring re-editing of human face. The plurality of object key points obtained by the above object key point recognition method can be applied to interactive entertainment (such as special effect camera, dynamic sticker, makeup or beauty shaping, etc.), intelligent medical beauty for facial feature analysis, face recognition or facial feature positioning, etc. The above object key point recognition method can be not limited to the key point recognition task of face, but can also be applied to other key point recognition tasks, such as hand key point and body key point, etc. The object recognized by the above object key point recognition method can include hand or face, etc., such as hand, face or body, etc.
[0132] Figure 7 is a block diagram of an object key point recognition model generation device according to an example embodiment. As shown in Figure 7 , the device can include:
[0133] The information acquisition module 710 is configured to acquire a plurality of original images of a target object and a plurality of object key points corresponding to the target object in the plurality of original images.
[0134] The preset clipping boundary point determination module 720 is configured to determine preset clipping boundary points corresponding to the plurality of original images, the preset clipping boundary points being points with a distance from a center point of the original image less than a preset distance;
[0135] The clipping boundary determination module 730 is configured to determine a clipping boundary of the plurality of original images according to the preset clipping boundary points.
[0136] The sample out-of-bound image acquisition module 740 is configured to perform clipping processing on the plurality of original images based on the clipping boundary to obtain a plurality of sample out-of-bound images of the target object.
[0137] The target object key point determination module 750 is configured to determine target object key points located in the plurality of sample out-of-bound images from a plurality of object key points corresponding to the plurality of sample out-of-bound images.
[0138] The key point recognition model generation module 760 is configured to perform object key point recognition training on the to-be-trained model based on the plurality of sample out-of-bound images and the target object key points to obtain an object key point recognition model.
[0139] The present disclosure also provides an object key point recognition device, comprising:
[0140] The target to-be-recognized image acquisition module is configured to acquire a target to-be-recognized image.
[0141] The object key point recognition module is configured to input the target to-be-recognized image into the object key point recognition model obtained according to the object key point recognition model generation method to perform object key point recognition processing, and obtain object key points corresponding to the target to-be-recognized image.
[0142] As to the device in the above embodiments, the specific manners in which various modules and units perform operations have been described in details in the embodiments about the method, and will not be described in details here.
[0143] Figure 8 is a block diagram of an electronic device for generating an object key point recognition model or recognizing an object key point according to an example embodiment. The electronic device can be a server, and its internal structure diagram can be as shown in Figure 8As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for generating an object key point recognition model or an object key point recognition method.
[0144] Figure 9 This is a block diagram illustrating an electronic device for generating an object key point recognition model or recognizing object key points, according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an object key point recognition model method or an object key point recognition method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.
[0145] Those skilled in the art will understand that Figure 8 or Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0146] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the object key point recognition model method or object key point recognition method as described in the embodiments of this disclosure.
[0147] In an example embodiment, a computer readable storage medium is also provided, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the object key point identification model method or the object key point identification method in the embodiments of the present disclosure.
[0148] In an example embodiment, a computer program product containing instructions, which, when run on a computer, enables the computer to perform the object key point identification model method or the object key point identification method in the embodiments of the present disclosure.
[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, databases, or other media used to store data in embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0150] Other embodiments of the present disclosure will be apparent to those skilled in the art with the consideration of the specification and practice of the disclosed application. The present application is intended to cover any variations, uses, or adaptive changes of the present disclosure following the general principles of the present disclosure and including common general knowledge or conventional technical means in the art not disclosed in the present disclosure. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0151] It should be understood that the present disclosure is not limited to the precise structures described above and shown in the drawings and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.
Claims
1. A method for generating an object keypoint recognition model, comprising: The method comprises: obtaining a plurality of original images of a target object, a plurality of object key points corresponding to the target object in the plurality of original images, and a preset distance corresponding to each original image; the plurality of original images are images processed by subject alignment; the preset distance corresponding to each original image is a ratio of the maximum value of the length and width of each original image to a preset parameter; the preset parameter has a value range of 2-3; determining a preset cropping boundary point corresponding to the plurality of original images, comprising: determining a cropping boundary point range of each original image based on the preset distance and the center point of each original image; obtaining the preset cropping boundary point by randomly taking points from the cropping boundary point range; the cropping boundary point range is a circular range obtained by taking the center point of each original image as the center and the preset distance as the radius; the preset cropping boundary point corresponding to each original image is a point with a distance from the image center point of the original image less than the preset distance corresponding to each original image; determining a cropping boundary of the plurality of original images according to the preset cropping boundary point; the cropping boundary is a boundary line passing through the preset cropping boundary point; cropping processing the plurality of original images based on the cropping boundary to obtain a plurality of sample out-of-bound images of the target object; determining a target object key point located in the plurality of sample out-of-bound images from a plurality of object key points corresponding to the plurality of sample out-of-bound images; performing object key point identification training on a to-be-trained model based on the plurality of sample out-of-bound images and the target object key point to obtain an object key point identification model.
2. The method of claim 1, wherein, The method further comprises: obtaining a preset cropping boundary type; determining the cropping boundary of the plurality of original images according to the preset cropping boundary point, comprising: determining the cropping boundary based on the preset cropping boundary point and the preset cropping boundary type.
3. The method of claim 1, wherein, The cropping processing the plurality of original images based on the cropping boundary to obtain a plurality of sample out-of-bound images of the target object, comprising: segmenting the image region corresponding to each original image according to the cropping boundary of each original image to obtain a first region and a second region corresponding to each original image; determining a mask region from the first region and the second region; masking processing each original image based on the mask region to obtain a sample out-of-bound image corresponding to each original image.
4. The method of claim 3, wherein, The determining a mask region from the first region and the second region, comprising: regarding a region in the first region and the second region located on the upper side of the cropping boundary as a target region; if the proportion information of the target region in the corresponding original image is less than or equal to the preset proportion information, regarding the target region as the mask region.
5. The method of claim 4, wherein, The method further comprises: if the proportion information of the target region in the corresponding original image is greater than the preset proportion information, regarding a non-target region as the mask region, the non-target region being a region in the image region other than the target region.
6. The method of claim 1, wherein, The object key points corresponding to the target object in the plurality of original images comprise being obtained in the following manner: performing object key point identification on each original image to obtain a plurality of object key points corresponding to each original image; The method further comprises: performing subject alignment processing on each original image based on the plurality of object key points corresponding to each original image to obtain a plurality of aligned original images and a plurality of object key points corresponding to the plurality of aligned original images; The method further comprises: determining the target object key points located within the plurality of sample out-of-bound images from the plurality of object key points corresponding to the plurality of sample out-of-bound images; determining the target object key points located within the plurality of sample out-of-bound images from the plurality of object key points corresponding to the plurality of aligned original images; The method further comprises:
7. The method of claim 1, wherein, obtaining preset boundary direction information; The method further comprises: determining the cropping boundary of the plurality of original images according to the preset cropping boundary points and the preset boundary direction information. The method comprises:
8. An object keypoint identification method, comprising: obtaining a target to-be-identified image; inputting the target to-be-identified image into an object key point identification model generated according to the method for generating an object key point identification model of any one of claims 1-7 to perform object key point identification processing to obtain object key points corresponding to the target to-be-identified image. The device comprises:
9. A device for generating an object key point recognition model, characterized in that, an information obtaining module configured to obtain a plurality of original images of a target object, a plurality of object key points corresponding to the target object in the plurality of original images, and a preset distance corresponding to each original image; the plurality of original images are images after subject alignment processing; the preset distance corresponding to each original image is a ratio of a maximum value of a length and a width of each original image to a preset parameter; the preset parameter has a value range of 2-3; a preset cropping boundary point determining module configured to determine preset cropping boundary points corresponding to the plurality of original images, including: determining a cropping boundary point range of each original image based on the preset distance and a center point of each original image; and obtaining the preset cropping boundary points by randomly selecting points from the cropping boundary point range; the cropping boundary point range is a circular range obtained by taking the center point of each original image as a center and the preset distance as a radius; the preset cropping boundary point corresponding to each original image is a point having a distance from the image center point of the original image less than the preset distance corresponding to each original image; a cropping boundary determining module configured to determine a cropping boundary of the plurality of original images according to the preset cropping boundary points; the cropping boundary is a boundary line passing through the preset cropping boundary points. The sample out-of-bound image obtaining module is configured to perform cropping processing on the plurality of original images based on the cropping boundary, to obtain a plurality of sample out-of-bound images of the target object. The target object key point determining module is configured to determine target object key points located in the plurality of sample out-of-bound images from a plurality of object key points corresponding to the plurality of sample out-of-bound images. The key point identification model generating module is configured to perform object key point identification training on a to-be-trained model based on the plurality of sample out-of-bound images and the target object key points, to obtain an object key point identification model.
10. The apparatus of claim 9, wherein, The apparatus is further configured to obtain a preset cropping boundary type. The cropping boundary determining module is further configured to determine the cropping boundary based on the preset cropping boundary point and the preset cropping boundary type.
11. The apparatus of claim 9, wherein, The sample out-of-bound image obtaining module is further configured to: perform segmentation processing on an image region corresponding to each original image according to the cropping boundary of the original image, to obtain a first region and a second region corresponding to the original image; determine a mask region from the first region and the second region; perform mask processing on the original image based on the mask region, to obtain a sample out-of-bound image corresponding to the original image.
12. The apparatus of claim 11, wherein, The sample out-of-bound image obtaining module is further configured to: take a region located on an upper side of the cropping boundary in the first region and the second region as a target region; if proportion information of the target region in the corresponding original image is less than or equal to preset proportion information, take the target region as the mask region.
13. The apparatus of claim 12, wherein, The apparatus is further configured to: if the proportion information of the target region in the corresponding original image is greater than the preset proportion information, take a non-target region as the mask region, the non-target region being a region other than the target region in the image region.
14. The apparatus of claim 9, wherein, The apparatus is further configured to: perform object key point identification on each original image, to obtain a plurality of object key points corresponding to the original image; perform subject alignment processing on the original image based on the plurality of object key points corresponding to the original image, to obtain a plurality of aligned original images and a plurality of object key points corresponding to the plurality of aligned original images; The target object key point determining module is further configured to determine target object key points located in the plurality of sample out-of-bound images from the plurality of object key points corresponding to the plurality of aligned original images. The sample out-of-bound image obtaining module is further configured to: perform cropping processing on the plurality of aligned original images based on the cropping boundary, to obtain the plurality of sample out-of-bound images.
15. The apparatus of claim 9, wherein, The apparatus is further configured to obtain preset boundary direction information. The cropping boundary determining module is further configured to determine the cropping boundary according to the preset cropping boundary point and the preset boundary direction information.
16. An object keypoint identification apparatus, comprising: The apparatus comprises: a target to-be-identified image obtaining module configured to obtain a target to-be-identified image; An object key point recognition module is configured to input the target image to be recognized into an object key point recognition model obtained by the method for generating an object key point recognition model according to any one of claims 1-7, to perform object key point recognition processing, and to obtain object key points corresponding to the target image to be recognized.
17. An electronic device, comprising: Comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the method for generating an object key point recognition model according to any one of claims 1-7 or the object key point recognition method according to claim 8.
18. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method for generating an object key point recognition model according to any one of claims 1-7 or the object key point recognition method according to claim 8.
19. A computer program product comprising computer instructions, characterized in that, The computer program instructions are executed by the processor to implement the method for generating an object key point recognition model according to any one of claims 1-7 or the object key point recognition method according to claim 8.
Citation Information
Patent Citations
Single-target human body key point detection method, system and device and medium
CN112232194A