Method and apparatus for generating key points

Abstract

The disclosure discloses a key point generation method and device, relates to the technical field of computers, and particularly relates to the technical field of visual processing. A specific implementation scheme is as follows: first, a target human body model corresponding to a target object is acquired; then, position information of a vertex in the target human body model is determined; finally, the position information of the vertex in the target human body model is input into a key point generation model, a human body key point corresponding to the target object is generated, the target human body model is identified through the constructed key point generation model, the human body key point corresponding to the target human body model is determined, and thus the human body key point corresponding to the target object is determined. The human body key point can be generated based on a model surface vertex, is suitable for various flexible human bodies with various postures and muscle non-rigid changes, and the accuracy and flexibility of the human body key point are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to the technical field of visual processing, and especially to a key point generation method and device. BACKGROUND

[0002] In computer vision, recovering three-dimensional information of an object from image information is a challenging topic. Deep neural networks have achieved good performance on two-dimensional problems such as image classification, object detection, and semantic segmentation, but they encounter difficulties in recovering three-dimensional information due to information loss. One of the biggest bottlenecks is the difficulty of obtaining high-quality labeled data to drive the training of neural networks.

[0003] In the specific field of sports applications, it is necessary to recover three-dimensional key point information representing the actions of athletes from images. Using deep learning methods requires a large amount of three-dimensional key point labeling data, but due to the joint movement and muscle changes of the human body, the change of the three-dimensional skeleton is a kind of non-rigid change, which is difficult to label directly. SUMMARY

[0004] The present disclosure provides a key point generation method, device, electronic equipment, storage medium, and computer program product.

[0005] According to an aspect of the present disclosure, a key point generation method is provided, which includes: obtaining a target human model corresponding to a target object; determining position information of a vertex in the target human model; inputting the position information of the vertex in the target human model into a key point generation model to generate human key points corresponding to the target object, wherein the key point generation model is configured to output human key points corresponding to the target human model.

[0006] According to another aspect of the present disclosure, a key point generation device is provided, which includes: an obtaining module configured to obtain a target human model corresponding to a target object; a determining module configured to determine position information of a vertex in the target human model; and a generating module configured to input the position information of the vertex in the target human model into a key point generation model to generate human key points corresponding to the target object, wherein the key point generation model is configured to output human key points corresponding to the target human model.

[0007] According to another aspect of the present disclosure, an electronic device is provided, which includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned key point generation method.

[0008] According to another aspect of the present disclosure, the present embodiment provides a computer readable medium having stored thereon computer instructions for enabling a computer to perform the key point generation method.

[0009] According to another aspect of the present disclosure, the present embodiment provides a computer program product comprising a computer program which, when executed by a processor, implements the key point generation method.

[0010] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0012] Figure 1 is a flowchart of one embodiment of the key point generation method according to the present disclosure;

[0013] Figure 2 is a schematic diagram of one application scenario of the key point generation method according to the present disclosure;

[0014] Figure 3 is a flowchart of one embodiment of obtaining a target human body model corresponding to a target object according to the present disclosure;

[0015] Figure 4 is a flowchart of one embodiment of obtaining a key point generation model according to the present disclosure;

[0016] Figure 5 is a flowchart of one embodiment of training an initial deep neural network according to the present disclosure;

[0017] Figure 6 is a structural schematic diagram of one embodiment of the key point generation device according to the present disclosure;

[0018] Figure 7 is a block diagram of an electronic device for implementing the key point generation method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0020] Reference Figure 1 , Figure 1 A flowchart 100 illustrating an embodiment of a key point generation method applicable to the present disclosure is shown. The key point generation method comprises the following steps:

[0021] In step 110, a target human model corresponding to a target object is obtained.

[0022] In this embodiment, the execution subject (e.g., a server) of the key point generation method can read the target human model locally or obtain the target human model from other servers or terminal devices through wired or wireless connection. The target human model can be a human model to be determined for human key points. The target human model can be a human model corresponding to a target object. The target object can be an object in a motion state. The target human model has skinning weights from each vertex of the model surface to human key points. The human model surface has a fixed 6890 vertices, and each vertex has a preset binding relationship with human key points.

[0023] It should be noted that the wireless connection mode can include, but is not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.

[0024] In step 120, position information of vertices in the target human model is determined.

[0025] In this embodiment, after the execution subject obtains the target human model corresponding to the target object, the target human model can be processed to obtain the model surface of the target human model. The execution subject can analyze the model surface of the target human model to determine the position information of each vertex of the model surface in a spatial rectangular coordinate system, such as the three-dimensional coordinates of each vertex in the spatial rectangular coordinate system.

[0026] Or, the execution subject obtains the target human body model corresponding to the target object, and further obtains a human body parameter model. The human body parameter model can be a parameterized human body 3D model, such as an SMPL human body three-dimensional model. The human body 3D model is determined by 72 posture parameters (controlling the posture of a person, a total of 24 skeletal points) and 10 shape parameters (controlling the shape of a person, such as height, fat, and thin). The human body parameter model predefines the skinning weight of each vertex of the model surface to the skeletal key point. The generated human body 3D model has fixed 6890 vertices, each vertex has a preset binding relationship with the human body skeletal point. Therefore, the human body parameter model can be regarded as a drivable human body 3D model. That is, by inputting different human body model parameters into the human body parameter model, different drivable human body 3D models can be obtained.

[0027] The execution subject can match the target human body model with the human body parameter model. A set of model parameters can be pre-set. The model parameters are input into the human body parameter model to obtain a corresponding human body 3D model. The posture and shape of the obtained human body 3D model are compared with the posture and shape of the target human body model to determine whether the human body 3D model and the target human body model are consistent in posture and shape. If consistent, the human body 3D model is determined as the parameter model corresponding to the target human body model. If not consistent, the model parameters are adjusted to obtain adjusted model parameters. The adjusted model parameters are input into the human body parameter model to obtain a new human body 3D model. The posture and shape of the new human body 3D model are compared with the posture and shape of the target human body model to determine whether the new human body 3D model and the target human body model are consistent in posture and shape. If not consistent, the model parameters are continuously adjusted to obtain the parameter model corresponding to the target human body model. After obtaining the parameter model corresponding to the target human body model, the execution subject can determine the vertex of the model surface of the parameter model, and determine the position information of each vertex in the spatial rectangular coordinate system.

[0028] In step 130, the position information of the vertex in the target human body model is input into the key point generation model to generate the human body key point corresponding to the target object.

[0029] In the embodiment, after the execution subject determines the position information of the vertex in the target human body model, the key point generation model is further obtained. The key point generation model is used to output the human body key point corresponding to the target human body model, and is used to represent the corresponding relationship between the vertex of the model surface and the human body key point.

[0030] The execution subject inputs the position information of the vertices in the target human body model into the key point generation model. The key point generation model processes the position information of the plurality of vertices and outputs human body key points corresponding to the position information of the vertices. The human body key points can be 24 skeletal key points included in the target human body model, i.e., 24 skeletal key points of the target object.

[0031] The key point generation model can analyze and process the position information of the plurality of vertices on the model surface according to the skinning weights and the binding relationship between the vertices on the model surface and the human body key points, and determine a human body key point according to the position information of the plurality of vertices, i.e., each human body key point can be determined by a plurality of vertices on the model surface.

[0032] With reference to Figure 2 , Figure 2 is a schematic diagram of an application scenario of the key point generation method according to the embodiment. In the Figure 2 application scenario, the terminal 201 records a target video of an athlete and obtains a target human body model corresponding to the athlete, and sends the target human body model to the server 202. After the server 202 obtains the target human body model from the terminal 201, the server 202 processes the target human body model and obtains the position information of the vertices in the target human body model. Then, the server 202 obtains a key point generation model, inputs the position information of the vertices in the target human body model into the key point generation model, and the key point generation model processes the input position information of the vertices to generate human body key points corresponding to the target object, i.e., human body key points corresponding to the athlete.

[0033] The key point generation method provided by the embodiment of the present disclosure includes obtaining a target human body model corresponding to a target object, determining the position information of the vertices in the target human body model, and finally inputting the position information of the vertices in the target human body model into a key point generation model to generate human body key points corresponding to the target object. The key point generation model is constructed to recognize the target human body model and determine the human body key points corresponding to the target human body model, thereby determining the human body key points corresponding to the target object. The human body key points can be generated based on the vertices on the model surface, which is suitable for various flexible human bodies with various postures and muscle non-rigid changes, and improves the accuracy and flexibility of the human body key points.

[0034] With reference to Figure 3 , Figure 3 Fig. 300 shows a flowchart of one embodiment of obtaining a target human body model corresponding to a target object, i.e., the above-mentioned step 110. Obtaining a target human body model corresponding to a target object can include the following steps:

[0035] Step 310: Obtain a target video including a target object.

[0036] In the embodiment, the execution subject can read the target video locally or obtain the target video from other servers or terminal devices through wired or wireless connection. The target video can be a video recorded by a target object in a moving state. The target video can be a monocular video recorded by a monocular camera.

[0037] That is, the terminal device can record a target object in a moving state to obtain a target video of the target object in the moving state, and send the target video to the execution subject. The execution subject can receive the target video including the target object sent by the terminal device.

[0038] In step 320, a target image frame including the target object is obtained from the target video.

[0039] In the embodiment, after the execution subject obtains the target video, the execution subject can analyze the target video to determine each image frame included in the target video, and take the image including the target object as a human body image to be processed. The execution subject can perform image preprocessing on the human body image to be processed through a deep learning network, solve image problems such as human body image distortion, blur, unclear light, and complex background, strengthen human body information of the target object in the human body image, and obtain a plurality of preprocessed human body images.

[0040] The execution subject can sample the plurality of preprocessed human body images according to a preset sampling interval to obtain a plurality of sampled human body images, and group the plurality of sampled human body images into a human body image sequence according to the order of shooting time. The execution subject can sequentially take each human body image in the human body image sequence as a target image frame according to the order of shooting time.

[0041] Alternatively, the execution subject can directly group the plurality of preprocessed human body images into a human body image sequence according to the order of shooting time, and sequentially take each human body image in the human body image sequence as a target image frame according to the order of shooting time.

[0042] In step 330, a target human body model corresponding to the target object is obtained based on the target image frame.

[0043] In the embodiment, after the execution subject obtains the target image frame, the execution subject can perform three-dimensional reconstruction of a human body model based on the target image frame to obtain a target human body model corresponding to the target object. This reconstruction method can also be referred to as unstructured reconstruction.

[0044] As an example, the execution subject described above performs three-dimensional reconstruction of a human body model based on a target image frame and a neural network. The target image frame can be input into a deep neural network branch for three-dimensional reconstruction. The deep neural network branch can include a global feature subnetwork and a fitting subnetwork. The global feature subnetwork can be used to extract high-level image features of the target image frame. The global feature subnetwork can be an HourGlass convolutional network. The high-level image features can be input into the fitting subnetwork. The fitting subnetwork can be used to predict whether each voxel block in a three-dimensional space belongs to the interior of the target object based on the high-level image features. The fitting subnetwork can be a multi-layer perceptron structure. The fitting subnetwork outputs a target human body model of the target object.

[0045] In the present implementation, the target human body model of the target object is obtained based on the target image frame. The three-dimensional reconstruction of the human body model based on the human body image is realized. The determined target human body model is more consistent with the target object in the target image frame. The accuracy of the target human body model is improved.

[0046] Reference Figure 4 , Figure 4 A flowchart 400 of one embodiment of obtaining a key point generation model is shown. The flowchart 400 can include the following steps:

[0047] In step 410, a sample human body image including a sample object in a sample video is obtained, and a sample human body model corresponding to the sample object is obtained based on the sample human body image.

[0048] In the present embodiment, the execution subject described above can read the sample video locally or obtain the sample video from other servers or terminal devices through wired or wireless connection. The sample video can be a video recorded for a sample object in a moving state. After obtaining the sample video, the execution subject described above can analyze the sample video to determine each frame of image included in the sample video, and take the image including the sample object as a sample human body image. The execution subject described above can perform image preprocessing on the sample human body image through a deep learning network to solve image problems such as distortion, blur, unclear light, and complex background of the human body image, and to strengthen the human body information of the sample object in the human body image, thereby obtaining multiple frames of preprocessed sample human body images.

[0049] After obtaining the sample human body image, the execution subject described above can analyze the sample object in the sample human body image to obtain image features of the sample object. The execution subject described above can perform three-dimensional reconstruction of a human body model based on the sample human body image and the image features of the sample object to obtain a sample human body model corresponding to the sample object.

[0050] In step 420, a training sample set is obtained.

[0051] In the embodiment, after the execution subject obtains the sample human body model, the sample human body model can be processed to obtain a model surface of the sample human body model. The execution subject can analyze the model surface of the sample human body model to determine position information of each vertex of the model surface in a spatial rectangular coordinate system, for example, three-dimensional coordinates of each vertex in the spatial rectangular coordinate system. In addition, the execution subject obtains a sample human body key point corresponding to the sample object. The sample human body key point can be a human body key point corresponding to the position information of the vertex, which is determined based on the position information of the plurality of vertices of the model surface, and can be a human body key point determined by an operator labeling the sample human body model.

[0052] After the execution subject obtains the position information of the sample vertex in the sample human body model and the sample human body key point corresponding to the sample object, the position information of the sample vertex in the sample human body model and the sample human body key point corresponding to the sample object are used as training samples for training the initial deep neural network.

[0053] In step 430, the position information of the sample vertex in the training sample set is used as input, and the sample human body key point corresponding to the input position information of the sample vertex is used as expected output. The initial deep neural network is trained by using a machine learning method to obtain a key point generation model.

[0054] In the embodiment, after the execution subject obtains the training sample set, the initial deep neural network is obtained. The execution subject can train the initial deep neural network based on the position information of the sample vertex in the sample human body model and the sample human body key point corresponding to the sample object by using a machine learning method to obtain a key point generation model.

[0055] Specifically, the execution subject can use the position information of the sample vertex in the training sample set as input, and obtain corresponding prediction information through the processing of the initial deep neural network. The initial deep neural network can be various existing neural networks, such as RCNN, Fast RCNN, Faster RCNN, YOLO, SSD, etc. If the prediction information does not satisfy the constraint condition, the network parameters of the initial deep neural network are adjusted, and the position information of the sample vertex in the sample human body model and the sample human body key point corresponding to the sample object are input again for further training. If the prediction information satisfies the constraint condition, the model training is completed, and a key point generation model is obtained. The constraint condition can be that the difference between the prediction information and the sample human body key point satisfies a preset threshold. The preset threshold can be pre-set according to experience, and the present disclosure does not make specific limitations thereto.

[0056] In this implementation, a training sample set is obtained by processing the sample video, and an initial deep neural network is trained based on the training sample set to obtain a key point generation model. This enables the generated key point generation model to generate human body key points based on the human body model, thereby improving the accuracy and determination efficiency of human body key points.

[0057] refer to Figure 5 , Figure 5 A flowchart 500 illustrating an embodiment of training an initial deep neural network is shown, which may include the following steps:

[0058] Step 510: Input the position information of the sample vertices in the training sample set into the initial deep neural network to obtain the human body key points corresponding to the position information of the sample vertices.

[0059] In this embodiment, the execution entity inputs the position information of sample vertices in the training sample set into the initial deep neural network. The initial deep neural network processes the input position information of the sample vertices to obtain the prediction information of the sample object. The prediction information may include the human body key points corresponding to the position information of the sample vertices.

[0060] The initial deep neural network mentioned above can be any existing neural network, such as RCNN, Fast RCNN, Faster RCNN, YOLO, SSD, etc. It should be noted that the aforementioned RCNN, Fast RCNN, Faster RCNN, YOLO, SSD, and other neural networks are well-known technologies that are currently widely researched and applied, and will not be elaborated here.

[0061] Typically, the neural network needs to be trained multiple times using a training sample set and a preset loss function to obtain optimal network parameters. After obtaining the optimal network parameters, these parameters can be saved to complete the training of the keypoint generation network.

[0062] Step 520: Compare the human body key points with the sample human body key points to obtain the comparison results.

[0063] In this embodiment, the execution entity inputs the position information of the sample vertices in the training sample set into the initial deep neural network, obtains the human key points corresponding to the position information of the output sample vertices, compares the human key points with the sample human key points, determines the difference value between the human key points and the sample human key points, and uses the difference value as the comparison result between the human key points and the sample human key points.

[0064] Step 530: Based on the comparison results, determine whether the initial deep neural network has achieved the preset optimization goal.

[0065] In this embodiment, after the execution entity obtains the comparison result between the human body key points and the sample human body key points, it determines whether the initial deep neural network has reached the preset optimization target based on the comparison result. The preset optimization target may be that the difference between the human body key points and the sample human body key points meets the preset threshold. The preset threshold can be preset based on experience, and this disclosure does not make specific limitations on it.

[0066] Specifically, after obtaining the comparison results between the human body key points and the sample human body key points, the aforementioned execution entity compares the difference value between the human body key points and the sample human body key points with a preset threshold, determines the magnitude relationship between the difference value and the preset threshold, and thus determines whether the initial deep neural network has reached the preset optimization goal.

[0067] Step 540: In response to determining that the initial deep neural network has achieved the optimization objective, the initial deep neural network is identified as the key point generation model.

[0068] In this embodiment, the execution entity determines that the initial deep neural network has reached the optimization target based on the comparison results, that is, it determines that the difference between the human body key points and the sample human body key points is not greater than a preset threshold. Then, it determines that the initial deep neural network has obtained the optimal network parameters, determines that the model training is complete, and determines the initial deep neural network corresponding to the optimal network parameters as the key point generation model.

[0069] In this implementation, the difference between human key points and sample human key points is used to determine whether the initial deep neural network has reached the preset optimization target, thereby determining whether the training is complete. This improves the efficiency and accuracy of model training, enabling the generated key point generation model to generate human key points based on the human model, thus improving the accuracy and determination efficiency of human key points.

[0070] As an optional implementation, please refer to [reference]. Figure 5 The flowchart 500 may also include the following steps:

[0071] Step 550: In response to the determination that the initial deep neural network has not reached the optimization objective, a loss function is determined based on human body key points and sample human body key points.

[0072] In this embodiment, the execution entity determines that the initial deep neural network has not reached the optimization target based on the comparison result, that is, it determines that the difference between the human key point and the sample human key point is greater than a preset threshold. Then, it determines the loss function based on the difference between the human key point and the sample human key point. The loss function can be the smoothL1 loss function, and the optimizer is the Adam optimizer.

[0073] Step 560: Based on the loss function, update the network parameters of the initial deep neural network to obtain the updated initial deep neural network.

[0074] In this embodiment, after obtaining the loss function, the execution entity updates the network parameters of the initial deep neural network according to the loss function to obtain the updated initial deep neural network. Specifically, the execution entity can adjust the network parameters of each network layer in the initial deep neural network according to the loss function to obtain the updated initial deep neural network.

[0075] Step 570: Based on the unused position information of sample vertices in the sample human body model and the sample human body key points corresponding to the sample objects in the training sample set, the updated initial deep neural network is trained using machine learning methods to obtain a key point generation model for outputting human body key points.

[0076] In this embodiment, after the execution entity obtains the updated initial deep neural network, it can form a new training sample set by combining the position information of sample vertices in the unused sample human body model in the training sample set with the sample human body key points corresponding to the sample object, and train the updated initial deep neural network based on the new training sample set.

[0077] Specifically, the aforementioned execution entity can use the position information of sample vertices in the new training sample set as input. After processing by the updated initial deep neural network, the corresponding prediction information is obtained. If the prediction information does not meet the constraints, the network parameters are further adjusted, and the position information of sample vertices in the sample human body model and the sample human body key points corresponding to the sample object are input again to continue training. If the prediction information meets the constraints, the model training is completed, and the key point generation model is obtained.

[0078] In this implementation, a loss function is determined based on the difference between human key points and sample human key points to optimize the training model, thereby improving the model training efficiency and accuracy. This enables the generated key point generation model to generate human key points based on the human model, improving the accuracy and determination efficiency of human key points.

[0079] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a keypoint generation device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0080] like Figure 6 As shown, the key point generation device 600 in this embodiment includes: an acquisition module 610, a determination module 620, and a generation module 630.

[0081] The acquisition module 610 is configured to acquire the target human body model corresponding to the target object.

[0082] The determination module 620 is configured to determine the position information of vertices in the target human body model;

[0083] The generation module 630 is configured to input the position information of the vertices in the target human body model into the key point generation model to generate human body key points corresponding to the target object. The key point generation model is used to output the human body key points corresponding to the target human body model.

[0084] In some optional embodiments of this example, the acquisition module 610 is further configured to: acquire a target video including the target object; acquire a target image frame including the target object from the target video; and acquire a target human body model corresponding to the target object based on the target image frame.

[0085] In some optional embodiments of this example, the keypoint generation model is obtained based on the following steps: obtaining a sample human image including a sample object in a sample video, and obtaining a sample human model corresponding to the sample object based on the sample human image; obtaining a training sample set, wherein the training sample set includes the position information of sample vertices in the sample human model and the sample human keypoints corresponding to the sample object; using a machine learning method, taking the position information of sample vertices in the training sample set as input, and taking the sample human keypoints corresponding to the input sample vertex position information as the expected output, training an initial deep neural network to obtain a keypoint generation model.

[0086] In some optional embodiments of this example, a machine learning method is used to train an initial deep neural network by taking the position information of sample vertices in the training sample set as input and the sample human keypoints corresponding to the input sample vertex position information as the expected output, thereby obtaining a keypoint generation model. This includes: inputting the position information of sample vertices in the training sample set into the initial deep neural network to obtain the human keypoints corresponding to the sample vertex position information; comparing the human keypoints with the sample human keypoints to obtain a comparison result; determining whether the initial deep neural network has reached a preset optimization target based on the comparison result; and, in response to determining that the initial deep neural network has reached the optimization target, identifying the initial deep neural network as the keypoint generation model.

[0087] In some optional embodiments of this example, a machine learning method is used to train an initial deep neural network by taking the position information of sample vertices in the training sample set as input and the sample human keypoints corresponding to the input sample vertex position information as the expected output, thereby obtaining a keypoint generation model. The method further includes, in response to determining that the initial deep neural network has not reached the optimization objective, determining a loss function based on human keypoints and sample human keypoints; updating the network parameters of the initial deep neural network based on the loss function to obtain an updated initial deep neural network; and continuing to train the updated initial deep neural network using machine learning methods based on the position information of sample vertices in the sample human model that are not used in the training sample set and the sample human keypoints corresponding to the sample objects, to obtain a keypoint generation model for outputting human keypoints.

[0088] The key point generation device provided in the embodiments of this disclosure obtains a target human body model corresponding to the target object, then determines the position information of the vertices in the target human body model, and finally inputs the position information of the vertices in the target human body model into a key point generation model to generate human body key points corresponding to the target object. By recognizing the target human body model through the constructed key point generation model, the device determines the human body key points corresponding to the target human body model, thereby determining the human body key points corresponding to the target object. It can generate human body key points based on the vertices of the model surface and is applicable to various flexible human bodies with diverse posture changes and non-rigid muscle changes, thus improving the accuracy and flexibility of human body key points.

[0089] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0090] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0091] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0093] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as keypoint generation methods. For example, in some embodiments, the keypoint generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the keypoint generation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the keypoint generation method by any other suitable means (e.g., by means of firmware).

[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0100] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0101] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0102] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A keypoint generation method, comprising: Obtaining a target human body model corresponding to a target object includes: obtaining a target video containing the target object; obtaining a target image frame containing the target object from the target video; and obtaining a target human body model corresponding to the target object based on the target image frame. Determining the position information of each vertex on the model surface of the target human body model includes: matching the target human body model with a human body parameter model, continuously adjusting the model parameters of the human body parameter model to obtain a human body parameter model that is consistent with the posture and shape of the target human body model, using the vertices of the model surface of the human body parameter model as the vertices of the model surface of the target human body model, and determining the position information of each vertex. The human body parameter model predefines the skinning weights and binding relationships from each vertex on the model surface to the skeletal keypoints. The position information of the vertices in the target human body model is input into the key point generation model to generate human body key points corresponding to the target object. The key point generation model is used to analyze and process the position information of multiple vertices on the model surface according to the skin weight and binding relationship between the vertices on the model surface of the target human body model and human body key points, and output the human body key points corresponding to the target human body model according to the position information of the multiple vertices. The keypoint generation model is obtained based on the following steps: acquiring sample human images including sample objects in sample videos, and acquiring sample human models corresponding to the sample objects based on the sample human images; acquiring a training sample set, wherein the training sample set includes the position information of sample vertices in the sample human models and sample human keypoints corresponding to the sample objects; using machine learning methods, taking the position information of sample vertices in the training sample set as input, and taking the sample human keypoints corresponding to the input sample vertex position information as the expected output, training an initial deep neural network to obtain the keypoint generation model.

2. The method according to claim 1, wherein, The method of using machine learning takes the position information of sample vertices in the training sample set as input and the sample human body key points corresponding to the input sample vertex position information as the expected output to train an initial deep neural network, thereby obtaining a key point generation model, including: The position information of sample vertices in the training sample set is input into the initial deep neural network to obtain the human body key points corresponding to the position information of the sample vertices. The key points of the human body are compared with the key points of the sample human body to obtain the comparison results; Based on the comparison results, it is determined whether the initial deep neural network has achieved the preset optimization target; In response to determining that the initial deep neural network has reached the optimization objective, the initial deep neural network is identified as a keypoint generation model.

3. The method according to claim 2, wherein, The method of using machine learning, taking the position information of sample vertices in the training sample set as input and the sample human body key points corresponding to the input sample vertex position information as the expected output, trains an initial deep neural network to obtain a key point generation model, and further includes: In response to the determination that the initial deep neural network has not reached the optimization objective, a loss function is determined based on the human body key points and the sample human body key points; Based on the loss function, the network parameters of the initial deep neural network are updated to obtain the updated initial deep neural network; Based on the unused position information of sample vertices in the sample human body model and the sample human body key points corresponding to the sample objects in the training sample set, the updated initial deep neural network is trained using machine learning methods to obtain a key point generation model for outputting human body key points.

4. A key point generation device, comprising: The acquisition module is configured to acquire a target human body model corresponding to a target object, including: acquiring a target video including the target object; acquiring a target image frame including the target object from the target video; and acquiring a target human body model corresponding to the target object based on the target image frame. The determination module is configured to determine the position information of each vertex on the model surface of the target human body model, including: matching the target human body model with a human body parameter model, continuously adjusting the model parameters of the human body parameter model to obtain a human body parameter model that is consistent with the posture and shape of the target human body model, taking the vertices of the model surface of the human body parameter model as the vertices of the model surface of the target human body model, and determining the position information of each vertex. The human body parameter model predefines the skinning weights and binding relationships from each vertex on the model surface to the skeletal keypoints. The generation module is configured to input the position information of the vertices in the target human body model into the key point generation model to generate human body key points corresponding to the target object. The key point generation model is used to analyze and process the position information of multiple vertices on the model surface according to the skin weight and binding relationship between the vertices on the model surface of the target human body model and the human body key points, and output the human body key points corresponding to the target human body model according to the position information of the multiple vertices. The keypoint generation model is obtained based on the following steps: acquiring sample human images including sample objects in sample videos, and acquiring sample human models corresponding to the sample objects based on the sample human images; acquiring a training sample set, wherein the training sample set includes the position information of sample vertices in the sample human models and sample human keypoints corresponding to the sample objects; using machine learning methods, taking the position information of sample vertices in the training sample set as input, and taking the sample human keypoints corresponding to the input sample vertex position information as the expected output, training an initial deep neural network to obtain the keypoint generation model.

5. The apparatus according to claim 4, wherein, The method of using machine learning takes the position information of sample vertices in the training sample set as input and the sample human body key points corresponding to the input sample vertex position information as the expected output to train an initial deep neural network, thereby obtaining a key point generation model, including: The position information of sample vertices in the training sample set is input into the initial deep neural network to obtain the human body key points corresponding to the position information of the sample vertices. The key points of the human body are compared with the key points of the sample human body to obtain the comparison results; Based on the comparison results, it is determined whether the initial deep neural network has achieved the preset optimization target; In response to determining that the initial deep neural network has reached the optimization objective, the initial deep neural network is identified as a keypoint generation model.

6. The apparatus according to claim 5, wherein, The method of using machine learning, taking the position information of sample vertices in the training sample set as input and the sample human body key points corresponding to the input sample vertex position information as the expected output, trains an initial deep neural network to obtain a key point generation model, and further includes: In response to the determination that the initial deep neural network has not reached the optimization objective, a loss function is determined based on the human body key points and the sample human body key points; Based on the loss function, the network parameters of the initial deep neural network are updated to obtain the updated initial deep neural network; Based on the unused position information of sample vertices in the sample human body model and the sample human body key points corresponding to the sample objects in the training sample set, the updated initial deep neural network is trained using machine learning methods to obtain a key point generation model for outputting human body key points.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images