Keypoint detection model training, virtual character driving method and apparatus, electronic device, storage medium, computer program

Abstract

A method and corresponding apparatus for training a keypoint detection model and driving a virtual character are disclosed. The method for driving a virtual character includes: collecting a target image frame containing an image of a part of a human body; inputting the target image frame into a pre-trained keypoint detection model to obtain coordinate information and viewing probability of human body keypoints in the target image frame output by the keypoint detection model; and driving a corresponding virtual character movement based on the coordinate information and viewing probability of the human body keypoints, where the viewing probability is the probability that the human body keypoint appears within the shooting field of view of the target image frame.

Description

[Technical Field]

[0001] This application is, This is the Japanese national phase of the international application PCT / CN2023 / 116711, filed on September 4, 2023. September 20, 2022 Out It is requested Ta Naka This application claims priority to National Patent Application No. 202211145707.2, the entire contents of which are incorporated into this application by reference.

[0002] This application relates to the field of image processing technology, and more particularly to a method for training a keypoint detection model, a method for driving a virtual character, an apparatus for training a keypoint detection model, an apparatus for driving a virtual character, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0003] With the development of the virtual industry, digitized live broadcast formats are emerging in live streaming content (for example, virtual presenters represented entirely by virtual images). [Overview of the project] [Means for solving the problem]

[0004] This application ,tree - Method for training a keypoint detection model, method for driving a virtual character, apparatus for training a keypoint detection model, apparatus for driving a virtual character electronic devices, computer-readable storage media, and computer programs It will be provided.

[0005] The present invention provides a method for driving a virtual character, the method comprising: collecting a target image frame containing an image of a part of a human body; inputting the target image frame into a pre-trained keypoint detection model to obtain coordinate information and field of view probability of human body keypoints in the target image frame output by the keypoint detection model; and driving corresponding virtual character movements based on the coordinate information and field of view probability of the human body keypoints, wherein the field of view probability is the probability that the human body keypoints appear within the field of view of the target image frame.

[0006] The present invention provides a method for training a keypoint detection model, the method comprising: determining the coordinate information of a plurality of keypoints contained in each sample image frame by performing keypoint detection on a plurality of sample image frames in a sample set; determining the field of view label of each keypoint based on the coordinate information of each keypoint; and training a keypoint detection model using the coordinate information of the plurality of keypoints in the plurality of sample image frames and the field of view label as training signals, wherein the field of view label is used to marker whether or not the keypoint is within the field of view of the sample image frame to which it belongs; and the keypoint detection model is used to perform keypoint detection on a target image frame during the model estimation stage and to output the coordinate information and field of view probability of the keypoints in the target image frame.

[0007] The present invention provides a virtual character driving device, the device comprising an image acquisition module, a human body keypoint detection module, and a virtual character driving module, wherein the image acquisition module is configured to acquire a target image frame containing an image of a part of a human body, the human body keypoint detection module is configured to input the target image frame into a pre-trained keypoint detection model and obtain coordinate information and field of view probability of human body keypoints in the target image frame output by the keypoint detection model, the field of view probability being the probability that the human body keypoint appears within the field of view of the target image frame, and the virtual character driving module is configured to drive corresponding virtual character movements based on the coordinate information and field of view probability of the human body keypoints.

[0008] The present invention provides an apparatus for training a keypoint detection model, the apparatus comprising a keypoint detection module, a field label determination module, and a model training module, wherein the keypoint detection module is configured to determine the coordinate information of multiple keypoints contained in each sample image frame by performing keypoint detection on multiple sample image frames in a sample set, the field label determination module is configured to determine the field label of each keypoint based on the coordinate information of each keypoint, the field label is used to mark whether or not the keypoint is within the field of view of the sample image frame to which it belongs, and the model training module is configured to train a keypoint detection model using the coordinate information of multiple keypoints in the multiple sample image frames and the field label as training signals, the keypoint detection model is used to perform keypoint detection on a target image frame during the model estimation stage and to output the coordinate information of the keypoints in the target image frame and the field probability.

[0009] The present invention provides an electronic device comprising at least one processor and a memory communicated with the at least one processor, the memory storing a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor can be made to execute the virtual character driving method or the keypoint detection model training method described above.

[0010] According to the sixth aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium stores computer instructions, and the computer instructions are executed by a processor as described above. Methods for driving virtual characters Alternatively, it can be used to implement a method for training a keypoint detection model.

[0011] According to the seventh aspect of the present application, a computer program product is provided, the computer program product includes computer-executable instructions, and when the computer-executable instructions are executed, the above-mentioned Methods for driving virtual characters Or it is used to implement the method of training the keypoint detection model.

Brief Description of the Drawings

[0012] Hereinafter, the drawings necessary for the description of the embodiments will be briefly described. [Figure 1] It is a flowchart of the method for training the keypoint detection model provided in Embodiment 1 of the present application. [Figure 2] It is a schematic diagram of the sample image frame provided in Embodiment 1 of the present application. [Figure 3] It is a schematic diagram of the cropped sample image frame provided in Embodiment 1 of the present application. [Figure 4] It is a flowchart of the method for virtual character driving provided in Embodiment 2 of the present application. [Figure 5] It is a schematic diagram of the structure of the device for training the keypoint detection model provided in Embodiment 3 of the present application. [Figure 6] It is a schematic diagram of the structure of the device for virtual character driving provided in Embodiment 4 of the present application. [Figure 7] It is a schematic diagram of the structure of the electronic device provided in Embodiment 5 of the present application.

Modes for Carrying Out the Invention

[0013] Hereinafter, the technical solutions in the embodiments of the present application will be described in combination with the drawings in the embodiments of the present application. However, the described embodiments are only some embodiments of the present application. Based on the embodiments of the present application, for those skilled in the art, all other embodiments obtained without creative labor shall fall within the protection scope of the present application.

[0014] The terms “First,” “Second,” etc., used in the specification and claims of this application and in the drawings above are used to distinguish similar subjects and do not necessarily indicate a specific order or priority. The data used in this manner is interchangeable where appropriate so that the embodiments of this application described herein can be carried out in an order other than those illustrated or described herein. Furthermore, the terms “includes” and “has” and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device including a series of steps or units is not limited to the listed steps or units and may include other steps or units not listed or specific to those processes, methods, products, or devices.

[0015] In some related technologies, virtual casters are realized by employing techniques such as optical motion capture and inertial motion capture. However, implementing such technologies requires the caster to wear specialized equipment for extended periods and typically involves connecting multiple types of cables, resulting in a poor live broadcast experience.

[0016] In another related technology, by generating 3D human body data in a real environment using end-to-end 3D (3 Dimensions) pose data, end-to-end 3D pose estimation methods can significantly enhance the interaction capabilities of virtual characters. When acquiring 3D human body data, related technologies typically use a deep learning network with a standard RGB camera to directly predict 3D joint points of the human body from the input RGB video. This generally processes the entire human body and generally has certain requirements regarding the camera's field of view (FOV) and shooting angle; for example, the field of view needs to cover most of the human body. When the user is close to the camera and only a part of the human body is visible, the device's performance is limited, the field of view is small, and parts of the human body such as hands / elbows tend to move frequently from the field of view, making it very easy to misdetect the position of pose points. Also, due to the limited resources of mobile devices, many forms based on heatmaps suffer from resolution limitations, resulting in insufficient accuracy of the output coordinates and significantly impacting the stability of the output coordinates.

[0017] This application provides a method for training a keypoint detection model and a method for driving a virtual character, in order to solve the problems of related technologies, such as the tendency to misdetect the position of pose points and the poor stability of the output coordinates.

[0018] Embodiment 1

[0019] Figure 1 is a flowchart of a method for training a keypoint detection model provided in Embodiment 1 of the present invention, which is used to detect human body keypoints in an image (e.g., a half-body image) that has features of a part of the human body, and is applied to a human body keypoint detection scene, for example, in a live broadcast scene, in which the detection of human body keypoints drives the movement of a virtual character.

[0020] In virtual live broadcasting scenarios, end-to-end 3D pose estimation methods can significantly enhance the interactivity capabilities of virtual presenters. Research and analysis have revealed that virtual presenters generally use mobile phones for live broadcasting. These scenarios have limitations such as limited device performance, small field of view, and frequent movement of body parts like hands and elbows from the field of view. In such scenarios, many open-source 3D pose point detection methods are highly prone to errors. The field of view is the maximum range that the device's camera can observe; a larger field of view means a wider range.

[0021] This embodiment designs a low-cost model training method used to train a keypoint detection model based on a standard RGB camera without increasing the additional shooting hardware and usage costs of the caster, where the keypoint detection model may be a neural network model. The keypoint detection model may include two prediction heads, each corresponding to a neural network, and the inputs to the two prediction heads are feature diagrams. One prediction head is used to predict the location of a keypoint based on the feature diagram, and the other prediction head is used to predict whether the keypoint is located within the field of view based on the feature diagram. Thus, the keypoint detection model can determine whether a keypoint is located within the field of view, and can improve the accuracy of the keypoint detection model for images in near-field scenes.

[0022] As shown in Figure 1, this embodiment may include the following steps.

[0023] Step 101: Keypoint detection is performed on multiple sample image frames in the sample set to determine the coordinate information of multiple keypoints contained in each sample image frame.

[0024] The sample image frames can include major human body poses, thereby improving the accuracy of the pose network's prediction results.

[0025] In one embodiment, keypoint detection can be performed on a sample image frame using a pre-generated pose detection model, which may be a two-dimensional pose detection model or a three-dimensional pose detection model. To improve detection accuracy, the pose detection model may be a model that combines a two-dimensional pose detection model and a three-dimensional pose detection model.

[0026] The key points may differ depending on the business needs, and this embodiment does not limit them. For example, key points may include, but are not limited to, the left shoulder point, right shoulder point, left elbow point, right elbow point, left wrist point, right wrist point, left palm point, right palm point, hip joint point, nose point, etc.

[0027] For example, keypoint coordinate information can be represented using image coordinates and depth information. In one embodiment, the depth information can be obtained by employing a set depth information calculation algorithm without increasing the number of additional depth sensors, thereby saving hardware and orientation costs.

[0028] In one embodiment, step 101 may include the following steps:

[0029] Step 101-1: The plurality of sample image frames are input into a pre-generated two-dimensional pose network, and the two-dimensional coordinate information of the key points of the plurality of sample image frames output by the two-dimensional pose network is obtained.

[0030] For example, the two-dimensional coordinate information is coordinate information based on an image coordinate system, and includes horizontal and vertical coordinate values.

number

number

[0031]

number

number

[0032] Step 101-2: The plurality of sample image frames are input into a pre-generated 3D pose network, and the 3D coordinate information of the key points of the plurality of sample image frames output by the 3D pose network is obtained.

[0033] For example, the 3D coordinate information is coordinate information based on the world coordinate system and includes X-axis coordinate values, Y-axis coordinate values, and Z-axis coordinate values.

number

number

[0034]

number

number

[0035] Furthermore, both the 2D pose network and the 3D pose network may be existing pose network models. This embodiment assumes that although the key points detected by both are the same, the coordinate system scales of the key points do not match, and therefore it is necessary to perform batch processing of the coordinate data in step 101-3.

[0036] Step 101-3: Based on the obtained two-dimensional coordinate information and three-dimensional coordinate information of the key points, the coordinate information of each key point is determined.

[0037] In this step, the coordinate information of each keypoint may be coordinate information that integrates the 2D coordinate information and 3D coordinate information of that keypoint, thereby enabling the conversion of the different output results of the 2D and 3D pose networks into a single coordinate system. For example, this coordinate information may include depth information, which can be calculated based on the 2D and 3D coordinate information.

[0038] In one embodiment, steps 101-3 may include the following steps:

[0039] Step 101-3-1: Determine the first stable keypoint and the second stable keypoint from the multiple keypoints of the multiple sample image frames.

[0040] In fact, stable keypoints can be keypoints that appear relatively consistently in the field of view, such as the shoulder point, elbow point, or nose point.

[0041] The first and second stable keypoints can appear as keypoint pairs, for example, the first stable keypoint being the left shoulder point and the second stable keypoint being the right shoulder point, or, for example, the first stable keypoint being the left elbow point and the second stable keypoint being the right elbow point.

[0042] In one embodiment, the developer can pre-configure whitelists of stable keypoints for different application scenarios. When it is necessary to determine the first and second stable keypoints, the developer can find the whitelist of stable keypoints for the corresponding scene based on the current application scenario (e.g., a live broadcast scene), match the currently detected keypoints against that whitelist of stable keypoints, and designate the matched keypoints as stable keypoints.

[0043] The first and second stable keypoints have no priority restrictions and are used solely to distinguish between different stable keypoints. When searching for a pair of stable keypoints, one can be designated as the first stable keypoint and the other as the second stable keypoint.

[0044] Step 101-3-2: The adjustment coefficient is determined based on the two-dimensional and three-dimensional coordinate information of the first stable keypoint and the two-dimensional and three-dimensional coordinate information of the second stable keypoint.

[0045] This adjustment coefficient is a parameter used to express the scale difference between two-dimensional coordinate information and a three-dimensional coordinate system, and based on this adjustment coefficient, the depth information of multiple keypoints can be determined.

[0046] In one embodiment, step 101-3-2 may include the following steps:

[0047] The absolute value of the difference between the two-dimensional coordinate information of the first stable keypoint and the two-dimensional coordinate information of the second stable keypoint is determined as the first difference value, the absolute value of the difference between the three-dimensional coordinate information of the first stable keypoint and the three-dimensional coordinate information of the second stable keypoint is determined as the second difference value, and the ratio of the first difference value to the second difference value is taken as the adjustment coefficient.

[0048] For example, assuming that the first stable keypoint is the left shoulder point with keypoint number 0, and the second stable keypoint is the right shoulder point with keypoint number 1,

number

[0049] Step 101-3-3: The Z-axis coordinate values ​​of the multiple keypoints in the multiple sample image frames are adjusted using the adjustment coefficient, and the depth value of each keypoint is obtained.

[0050] In one embodiment, the depth value of each keypoint is:

number

[0051] Step 101-3-4: The horizontal coordinate value, vertical coordinate value, and depth value of each key point are used as the coordinate information of that key point.

[0052] After obtaining the depth value for each keypoint, the image coordinates and depth values ​​of each keypoint can be combined with the final coordinate information of each keypoint, that is,

number

number

[0053] In this embodiment, a two-dimensional pose network is used to obtain two-dimensional coordinate information of keypoints in a sample image frame, and a three-dimensional pose network is used to obtain three-dimensional coordinate information of keypoints in a sample image frame. By integrating this two-dimensional and three-dimensional coordinate information to obtain the final coordinate information of each keypoint, it is possible to obtain keypoint coordinate information that is more accurate and stable at a lower cost, thereby improving the accuracy of keypoint detection.

[0054] In one embodiment, after determining the coordinate information of multiple keypoints contained in each sample image frame, preprocessing such as image augmentation can be performed on multiple sample image frames based on the coordinate information of multiple keypoints contained in each sample image frame. Here, the image augmentation process may include at least one of random disturbance and cropping, or a combination of random disturbance and cropping.

[0055] This embodiment is not limited to the implementation form of random disturbance. For example, random disturbance means that the pixel value of each pixel point changes randomly according to a predetermined form. In one exemplary embodiment, if the disturbance range is a predetermined form in which random disturbance is performed according to the range of the original [-20,20], then if the RGB pixel value of one pixel point is (6,12,230), after going through the random disturbance of the predetermined form it becomes (8,12,226). The range of each dye in the pixel value is [0,255], that is, the maximum value after disturbance is 255 and the minimum value is 0.

[0056] To simulate a near-field scene close to the imaging device, a cropping process can be performed on multiple sample image frames (including sample image frames generated after random disturbances). In one embodiment, the cropping process may include the following steps:

[0057] Based on the coordinate information of multiple keypoints contained in each sample image frame, the center position of the crop frame is determined, the crop frame position is determined based on the center position of the crop frame, and the RGB values ​​of pixel points located outside the crop frame are set to black according to the said crop frame position.

[0058] In one embodiment, the center position of the crop frame may be the center position of the human body in the sample image frame, and the center position of the crop frame can be calculated using the coordinate information of the detected keypoints. For example, the center position of the crop frame may be the average of the coordinate information of three points: the left shoulder point, the right shoulder point, and the hip joint point.

[0059] After obtaining the center position of the crop frame, the crop frame position can be determined based on the center position of the crop frame and the size of the pre-set crop frame (i.e., the width and height of the crop frame). After determining the crop frame position, the RGB values ​​of the pixel points located outside the crop frame are set to black (i.e., the RGB value is 0), and a sample image frame after cropping can be obtained. For example, if the sample image frame is as shown in Figure 2, the sample image frame after cropping will be as shown in Figure 3.

[0060] In practice, we can further introduce random disturbances to the size of the crop frame to obtain sample image frames after cropping with different crop frame sizes.

[0061] In this embodiment, image augmentation processing can expand the training dataset, suppress model overfitting, and improve the model's generalization ability. Simultaneously, low-cost label transfer and data preprocessing methods enable effective simulation of the target scene.

[0062] Step 102: Based on the coordinate information of each keypoint, a field of view label for each keypoint is determined, and the field of view label is used to mark whether or not the keypoint is within the field of view of the sample image frame to which it belongs.

[0063] After detecting keypoint sets from multiple sample image frames, the detected keypoints may not be within the range of the sample image frames, depending on the accuracy of the model detection. For example, if a palm point is not present in a sample image frame, the detected keypoints will include the palm point. Based on this, in this embodiment, for each keypoint, the field of view label of that keypoint can be determined by further determining whether or not the keypoint is within the field of view of the sample image frame to which it belongs, based on the keypoint's coordinate information. For example, if a keypoint is within the field of view of the sample image frame to which it belongs, its field of view label is 1, and if a keypoint is not within the field of view of the sample image frame to which it belongs, its field of view label is 0.

[0064] When implementing this, by comparing the size of the sample image frame with the coordinate information of the keypoint, it is possible to determine whether or not the keypoint is within the field of view of the sample image frame to which it belongs. If it is determined that the keypoint is located within the range of the sample image frame to which it belongs based on the coordinate information of the keypoint, it is determined that the keypoint is within the field of view. If it is determined that the keypoint is not within the range of the sample image frame to which it belongs based on the coordinate information of the keypoint, it is determined that the keypoint is outside the field of view.

[0065] In one embodiment, step 102 may include the following steps:

[0066] The width and height of each sample image frame are obtained, and the horizontal coordinate range is determined based on the width of the multiple sample image frames, starting from the origin of the image coordinate system, and the vertical coordinate range is determined based on the height of the multiple sample image frames. If the horizontal coordinate value of the keypoint is within the horizontal coordinate range, or if the vertical coordinate value of the keypoint is within the vertical coordinate range, it is determined that the field of view label of the keypoint is an in-field label. If the horizontal coordinate value of the keypoint is not within the horizontal coordinate range, and the vertical coordinate value of the keypoint is not within the vertical coordinate range, it is determined that the field of view label of the keypoint is an out-of-field label.

[0067] Starting from the origin of the image coordinate system, the width of the sample image frame can be defined as the length of the horizontal coordinate axis; that is, the horizontal coordinate range is [0, width], where width is the width. The height of the sample image frame can be defined as the length of the vertical coordinate axis; that is, the vertical coordinate range is [0, height], where height is the height.

[0068] In practice, the field of view label can be determined based on the following logical determination formula:

number

[0069] Step 103: A keypoint detection model is trained using the coordinate information of multiple keypoints in multiple sample image frames and the field of view labels as training signals.

[0070] In this step, a keypoint detection model can be trained using the coordinate information and field-of-view labels of multiple keypoints from the obtained sample image frames as training signals.

[0071] When the key point detection model is used to perform key point detection on a target image frame and output the coordinate information and field of view probability of the key points in the target image frame during the model estimation stage. In subsequent applications, for example, the corresponding virtual character actions can be driven by adopting the coordinate information and field of view probability.

[0072] In one embodiment, when training the key point detection model, the loss function used is the heatmap loss function Loss heatmap , the position loss function Loss location and the label loss function Loss label including, for example:

Number

[0073] Note that this embodiment does not limit the implementation forms of the above three loss functions. For example, Loss heatmap can be realized by L2 loss, Loss location can be realized by L1 loss, and Loss label can be realized by cross-entropy loss.

[0074] In this embodiment, during the data preparation stage, coordinate information and field-of-view labels of multiple keypoints contained in each of the multiple sample image frames can be obtained. The field-of-view labels are used to mark whether or not a keypoint is within the field of view of the sample image frame to which it belongs. During the model training stage, the coordinate information and field-of-view labels of the multiple keypoints in the multiple sample image frames are used as training signals to train a keypoint detection model. By introducing an additional prediction head, the keypoint detection model can be equipped with the function to output field-of-view labels for keypoints. This effectively determines whether or not a keypoint is located within the field of view, improving the accuracy of keypoint detection for images in near-field scenes. For example, if a user is close to the camera and only a part of their body is visible within the camera's field of view, the keypoint detection model of this embodiment can output field-of-view labels indicating whether multiple human body keypoints are within or outside the field of view. This avoids situations where out-of-field keypoints are subsequently used in scene processing, which can affect the processing effect.

[0075] Embodiment 2

[0076] Figure 4 is a flowchart of a virtual character driving method provided in Embodiment 2 of the present invention, which belongs to the model estimation stage of the keypoint detection model of Embodiment 1.

[0077] As shown in Figure 4, this embodiment may include the following steps.

[0078] Step 201: A target image frame is collected, and the target image frame includes an image of a part of the human body.

[0079] The target image frame may be an image frame collected in real time, for example, a half-body photograph of a newscaster taken by a device such as a mobile phone during a live broadcast.

[0080] Step 202: The target image frame is input to a pre-trained keypoint detection model, and the coordinate information and field of view probability of the human body keypoints in the target image frame are obtained from the keypoint detection model.

[0081] After collecting a target image frame, the target image frame is input to the keypoint detection model generated in Embodiment 1, and human body keypoint detection is performed by the keypoint detection model to obtain coordinate information and field of view probability for one or more human body keypoints, where the field of view probability is the probability that a human body keypoint appears within the field of view of the target image frame.

[0082] The coordinate information is (u n ,v n , depth n ) can be expressed as such, and the coordinate information output by the keypoint detection model is less costly than the 3D coordinate information output by the 3D posture network. This is because some of the information in the 3D coordinate information does not need to be used, and depending on the virtual driving scene, the depth values ​​that are possible in the actual scene are not needed, and only one relative value is sufficient. Therefore, the training method of Embodiment 1 converts the 3D coordinate information into pseudo-3D coordinate information on an image coordinate scale.

[0083] In one embodiment, after obtaining the coordinate information and field of view probability of the human body keypoints in the target image frame, the following steps may be further included.

[0084] The smoothing weights between the current target image frame and the previous target image frame are determined, and the coordinate information and field of view probability are smoothed using these smoothing weights.

[0085] For example, in a scenario where a virtual live broadcast is conducted using a mobile phone, there is no particular interest in key points of the human body outside the field of view; the goal is simply to ensure smooth jumps and avoid obvious errors. Therefore, a filter is used to smooth the coordinate information and predicted probabilities.

[0086] In one embodiment, the step of determining the smoothing weights between the current target image frame and the previous target image frame may include the following steps:

[0087] For each human body keypoint, the distance between the coordinate information of the human body keypoint in the current target image frame and the smoothed coordinate information in the previous target image frame is determined, the distance is compared with a set distance, a distance weight is determined based on the comparison result, and a smoothing weight is calculated using the distance weight and the field of view probability of the human body keypoint in the current target image frame.

[0088] In one embodiment, for a single human body keypoint, after obtaining the smoothed coordinate information from the previous target image frame and the coordinate information from the current target image frame, the distance between the two sets of coordinate information can be calculated using a distance calculation formula. For example, this distance can be calculated using the following formula.

number

[0089] distance n This is the distance between the coordinate information of the nth human body keypoint in the current target image frame and the smoothed coordinate information in the previous target image frame.

number

number

[0090] When implementing this, one may compare the obtained distance with the set distance and calculate the ratio between the two. The result of this comparison will be the ratio between the two, that is,

number

[0091] In one embodiment, the following formula can be used to calculate the distance weight.

number

[0092] k represents the degree of smoothing applied in advance; a larger value indicates a smaller window and a stronger ability to suppress sharp jumps.

[0093] After obtaining the distance weights, the smoothing weights can be calculated by combining the distance weights with the field of view probability of the current human body keypoint in the current target image frame. For example, the smoothing weights can be calculated using the following formula.

number

[0094] prob n This represents the probability of the nth keypoint in the human body being within the field of view.

[0095] After obtaining smoothing weights, the coordinate information and field of view probability can be smoothed using these smoothing weights. In one embodiment, the smoothing process may include the following steps.

[0096] Based on the smoothing weights, the first weight of the previous target image frame and the second weight of the current target image frame are determined. Based on the first and second weights, the coordinate information of the previous target image frame and the coordinate information of the current target image frame are weighted to obtain smoothed coordinate information. Based on the first and second weights, the field of view probability of the previous target image frame and the field of view probability of the current target image frame are weighted to obtain smoothed field of view probabilities.

[0097] For example, if we assume that the first weight is the smoothing weight, then the second weight is the difference between the numerical value 1 and the smoothing weight, i.e., first weight = ratio n Therefore, the second weight = 1 - ratio n That is the case.

[0098] The process of smoothing coordinate information is as follows:

number

[0099] The process of smoothing the field of view probability is as follows:

number

[0100] Cached_prob n This is the field of view probability after smoothing, and Cached_prob n-1 This represents the field of view probability after the previous smoothing process.

[0101] The filter introduced in this embodiment performs smoothing processing on the degree of keypoint jumps and whether or not the keypoint is within the field of view. By reducing the weight of the current frame in cases of severe keypoint coordinate jumps and when the keypoint is not within the field of view, the output human body keypoint results maintain a relatively stable and continuous state overall.

[0102] Step 203: Drive the corresponding virtual character movement based on the coordinate information of the human body keypoints and the field of view probability.

[0103] After obtaining the final coordinate information and field of view probability for the human body keypoints in the target image frame, it is possible to determine whether the corresponding human body keypoint is within the field of view of the target image frame based on the field of view probability. If it is within the field of view, the corresponding human body part of the virtual character can be moved to the position corresponding to the coordinate information based on that information. If it is outside the field of view, the virtual character will not be moved.

[0104] For example, these key points on the human body can enrich the interaction between the caster and the user, such as waving, making a heart shape with the hands, displaying a 3D gift sent by the user on the 3D image's arm or wrist, or having the virtual images of the caster and the user play interactive games in a virtual 3D space.

[0105] In this embodiment, the keypoint detection model can detect the coordinate information and field of view probability of human body keypoints in the target image frame, where the field of view probability is the probability that a human body keypoint appears within the field of view of the target image frame. By driving the corresponding virtual character movement according to the field of view probability and coordinate information, end-to-end signal output for human body keypoints is achieved, while ensuring a certain level of stability in close-up situations and satisfying the virtual character driving requirements of a mobile phone terminal.

[0106] Embodiment 3

[0107] Figure 5 is a schematic diagram of the structure of a keypoint detection model training apparatus provided in Embodiment 3 of the present invention, and may include the following modules. The keypoint detection module 301 is configured to determine the coordinate information of multiple keypoints contained in each sample image frame by performing keypoint detection on multiple sample image frames in a sample set. The field of view label determination module 302 is configured to determine the field of view label for each keypoint based on the coordinate information of each keypoint, and the field of view label is used to mark whether or not the keypoint is within the field of view of the sample image frame to which it belongs. The model training module 303 is configured to train a keypoint detection model using the coordinate information of multiple keypoints in multiple sample image frames and the field of view label as training signals, and the keypoint detection model is used to perform keypoint detection on a target image frame during the model estimation stage and to output the coordinate information of the keypoints in the target image frame and the field of view probability.

[0108] In one embodiment, the keypoint detection module 301 may include the following modules: A two-dimensional pose prediction module is configured to input the plurality of sample image frames into a pre-generated two-dimensional pose network and obtain two-dimensional coordinate information of the keypoints of the plurality of sample image frames output by the two-dimensional pose network. A three-dimensional pose prediction module is configured to input the plurality of sample image frames into a pre-generated three-dimensional pose network and obtain three-dimensional coordinate information of the keypoints of the plurality of sample image frames output by the three-dimensional pose network. A coordinate determination module is configured to determine the coordinate information of each keypoint based on the obtained two-dimensional coordinate information and three-dimensional coordinate information of the keypoints.

[0109] In one embodiment, the two-dimensional coordinate information includes horizontal and vertical coordinate values, and the three-dimensional coordinate information includes X-axis coordinate values, Y-axis coordinate values, and Z-axis coordinate values. The coordinate determination module may include the following modules: A stable keypoint determination module is set up to determine a first stable keypoint and a second stable keypoint from a plurality of keypoints in the plurality of sample image frames. An adjustment coefficient determination module is set up to determine an adjustment coefficient based on the two-dimensional coordinate information and the three-dimensional coordinate information of the first stable keypoint and the second stable keypoint. An adjustment module is set up to adjust the Z-axis coordinate values ​​of a plurality of keypoints in the plurality of sample image frames using the adjustment coefficient to obtain the depth value of each keypoint. A coordinate generation module is set up to use the horizontal coordinate value, vertical coordinate value, and depth value of each keypoint as the coordinate information of that keypoint.

[0110] In one embodiment, the adjustment coefficient determination module is configured to determine the absolute value of the difference between the two-dimensional coordinate information of the first stable keypoint and the second stable keypoint as the first difference value, determine the absolute value of the difference between the three-dimensional coordinate information of the first stable keypoint and the second stable keypoint as the second difference value, and set the ratio of the first difference value to the second difference value as the adjustment coefficient.

[0111] In one embodiment, the coordinate information includes horizontal coordinate values ​​and vertical coordinate values, and the field of view label includes an in-field of view label and an out-of-field of view label.

[0112] The field-of-view label determination module 302 acquires the width and height of each sample image frame, determines the horizontal coordinate range based on the width of the plurality of sample image frames, and determines the vertical coordinate range based on the height of the plurality of sample image frames, starting from the origin of the image coordinate system. If the horizontal coordinate value of the key point is within the horizontal coordinate range, or if the vertical coordinate value of the key point is within the vertical coordinate range, it is determined that the field-of-view label of the key point is an in-field label. If the horizontal coordinate value of the key point is not within the horizontal coordinate range, and the vertical coordinate value of the key point is not within the vertical coordinate range, it is determined that the field-of-view label of the key point is an out-of-field label.

[0113] In one embodiment, the apparatus may further include the following module: The image augmentation module is configured to determine the coordinate information of a plurality of keypoints contained in each sample image frame, and then perform image augmentation processing on the sample image frame based on the plurality of coordinate information contained in each sample image frame, and the image augmentation processing may include at least one of random disturbance, cropping, or a combination of random disturbance and cropping.

[0114] In one embodiment, the image extension module is configured to determine the center position of the crop frame based on the coordinate information of a plurality of keypoints included in each sample image frame, to determine the crop frame position based on the center position of the crop frame, and to set the RGB values ​​of pixel points located outside the crop frame to black according to the crop frame position.

[0115] In one embodiment, the loss function used when training the keypoint detection model includes a heatmap loss function, a location loss function, and a label loss function.

[0116] The keypoint detection model training apparatus provided in the embodiment of the present invention is capable of performing the keypoint detection model training method provided in Embodiment 1 of the present invention and comprises a corresponding functional module for performing the method.

[0117] Embodiment 4

[0118] Figure 6 is a schematic diagram of the structure of a virtual character-driven device provided in Embodiment 4 of the present invention, and may include the following modules: An image acquisition module 401 is configured to acquire a target image frame, the target image frame containing an image of a part of the human body. A human body keypoint detection module 402 is configured to input the target image frame into a pre-trained keypoint detection model and obtain coordinate information and field of view probability of human body keypoints in the target image frame output by the keypoint detection model, the field of view probability being the probability that the human body keypoint appears within the field of view of the target image frame. A virtual character driving module 403 is configured to drive corresponding virtual character movements based on the coordinate information and field of view probability of the human body keypoints.

[0119] In one embodiment, the apparatus may further include the following modules: A smoothing weight determination module is configured to determine the smoothing weights between the current target image frame and the previous target image frame. A smoothing processing module is configured to take the smoothing weights and perform smoothing on the coordinate information and the field of view probability.

[0120] In one embodiment, the smoothing weight determination module is configured to determine the distance between the coordinate information of the human body keypoint in the current target image frame and the smoothed coordinate information in the previous target image frame for each human body keypoint, compare the distance with a set distance, determine the distance weight based on the comparison result, and calculate the smoothing weight by adopting the distance weight and the field of view probability of the human body keypoint in the current target image frame.

[0121] In one embodiment, the smoothing processing module determines a first weight for the previous target image frame and a second weight for the current target image frame based on the smoothing weights, calculates weighted coordinate information for the previous target image frame and the current target image frame based on the first and second weights to obtain smoothed coordinate information, and calculates weighted field of view probabilities for the previous target image frame and the current target image frame based on the first and second weights to obtain smoothed field of view probabilities.

[0122] A virtual character driving device provided in an embodiment of the present invention can perform the virtual character driving method provided in Embodiment 2 of the present invention and comprises a corresponding functional module for performing the method.

[0123] Embodiment 5

[0124] Figure 7 shows a schematic diagram of the structure of an electronic device 10 that can implement an embodiment of the method of the present invention. As shown in Figure 7, the electronic device 10 may be a server, a mobile phone, or other device, and includes at least one processor 11 and a storage device (e.g., read-only memory (ROM) 12, random access memory (RAM) 13, etc.) that is communicated with at least one processor 11, wherein the storage device stores one or more computer programs that can be executed by at least one processor, and the processor 11 can perform several kinds of appropriate operations and processes depending on the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 into the RAM 13. The RAM 13 can further store several kinds of programs and data necessary for the operation of the electronic device 10.

[0125] In some embodiments, the method in Embodiment 1 or Embodiment 2 can be implemented as a computer program and physically contained in a computer-readable storage medium (e.g., storage unit 18). In some embodiments, part or all of the computer program can be loaded and / or installed into the electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method in Embodiment 1 or Embodiment 2 described above can be performed.

[0126] In some embodiments, the method in Embodiment 1 or Embodiment 2 can be implemented as a computer program product which includes computer-executable instructions used to perform one or more steps of the method in Embodiment 1 or Embodiment 2 described above when the computer-executable instructions are executed. [Explanation of Symbols]

[0127] 301 Keypoint Detection Module 302 Field of View Label Determination Module 303 Model Training Module 401 Image Collection Module 402 Human Body Keypoint Detection Module 403 Virtual Character Driven Module 10 Electronic Devices 11 processors 12. Read-only memory (ROM) 13. Random Access Memory (RAM) 14 bus 15 I / O Interfaces 16 Input Units 17 Output Unit 18 memory units 19. Communication Unit

Claims

1. A method for driving virtual characters, The objective is to collect target image frames containing images of parts of the human body, The aforementioned target image frame is input to a pre-trained keypoint detection model, and the coordinate information and field of view probability of the human body keypoints in the target image frame are obtained from the keypoint detection model. The operation of a virtual character is driven based on the coordinate information of the human body keypoints and the field of view probability. Includes, The aforementioned field of view probability is the probability that the human body keypoint appears within the field of view of the target image frame. Before driving the corresponding virtual character movement based on the coordinate information of the aforementioned human body keypoints and the field of view probability, further, The current smoothing weights between the target image frame and the previous target image frame are determined, The aforementioned smoothing weights are used to perform a smoothing process on the coordinate information and the field of view probability. including method.

2. Determining the smoothing weights between the current target image frame and the previous target image frame is: To determine the distance between the coordinate information of each of the multiple human body keypoints in the current target image frame and the smoothed coordinate information in the previous target image frame, The aforementioned distance is compared with the set distance, and the distance weight is determined based on the comparison result. The distance weights of the aforementioned multiple human body keypoints and the field of view probabilities of the aforementioned multiple human body keypoints in the current target image frame are used to calculate the smoothing weights, including, The method according to claim 1.

3. Applying the aforementioned smoothing weights to the coordinate information and the field of view probability results in: Based on the smoothing weights, the first weight of the previous target image frame and the second weight of the current target image frame are determined. Based on the first and second weights, the coordinate information of the previous target image frame and the coordinate information of the current target image frame are weighted and calculated to obtain smoothed coordinate information. Based on the first and second weights, the field of view probability of the previous target image frame and the field of view probability of the current target image frame are weighted and calculated to obtain the smoothed field of view probability. including, The method according to claim 1.

4. A method for training a keypoint detection model, By performing keypoint detection on multiple sample image frames in the sample set, the coordinate information of multiple keypoints contained in each sample image frame is determined, Based on the coordinate information of each keypoint, the field of view label for each keypoint is determined. The keypoint detection model is trained using the coordinate information of multiple keypoints in the multiple sample image frames and the field of view labels as training signals. Includes, The aforementioned field-of-view label is used to mark whether or not the keypoint is within the field of view of the sample image frame to which it belongs. The aforementioned keypoint detection model is used in the model estimation stage to perform keypoint detection on the target image frame and to output the coordinate information and field of view probability of the keypoints in the target image frame. By performing keypoint detection on multiple sample image frames in a sample set, the coordinate information of the multiple keypoints contained in each sample image frame can be determined. The process involves inputting the aforementioned plurality of sample image frames into a pre-generated two-dimensional pose network, and obtaining the two-dimensional coordinate information of the key points of the plurality of sample image frames output by the two-dimensional pose network. The process involves inputting the aforementioned multiple sample image frames into a pre-generated three-dimensional pose network and obtaining the three-dimensional coordinate information of the key points of the multiple sample image frames output by the three-dimensional pose network, Based on the obtained two-dimensional coordinate information and three-dimensional coordinate information of the key points, the coordinate information of each key point is determined. including method.

5. The two-dimensional coordinate information includes horizontal coordinate values ​​and vertical coordinate values, and the three-dimensional coordinate information includes X-axis coordinate values, Y-axis coordinate values ​​and Z-axis coordinate values, Based on the obtained two-dimensional coordinate information and three-dimensional coordinate information of the key points, determining the coordinate information of each key point is: From the multiple keypoints of the aforementioned multiple sample image frames, the first stable keypoint and the second stable keypoint are determined. The adjustment coefficient is determined based on the two-dimensional and three-dimensional coordinate information of the first stable keypoint and the two-dimensional and three-dimensional coordinate information of the second stable keypoint. The adjustment coefficient is used to adjust the Z-axis coordinate values ​​of multiple keypoints in the multiple sample image frames, and the depth value of each keypoint is obtained. The horizontal coordinate value, vertical coordinate value, and depth value of each key point are to be used as the coordinate information of that key point. including, The method according to claim 4.

6. Determining the adjustment coefficient based on the two-dimensional and three-dimensional coordinate information of the first stable keypoint and the two-dimensional and three-dimensional coordinate information of the second stable keypoint is: The absolute value of the difference between the two-dimensional coordinate information of the first stable keypoint and the two-dimensional coordinate information of the second stable keypoint is determined as the first difference value. The absolute value of the difference between the three-dimensional coordinate information of the first stable keypoint and the three-dimensional coordinate information of the second stable keypoint is determined as the second difference value. The ratio of the first difference value to the second difference value is used as the adjustment coefficient, including, The method according to claim 5.

7. The coordinate information includes horizontal coordinate values ​​and vertical coordinate values, and the field of view label includes in-field label and out-of-field label. Based on the coordinate information of each keypoint, determining the field of view label for each keypoint is: Obtain the width and height of each sample image frame, Starting from the origin of the image coordinate system, the horizontal coordinate range is determined based on the width of the multiple sample image frames, and the vertical coordinate range is determined based on the height of the multiple sample image frames. In response to the horizontal coordinate value of each keypoint being within the horizontal coordinate range, or the vertical coordinate value of each keypoint being within the vertical coordinate range, it is determined that the field of view label of the keypoint is an in-field label. In response to the fact that the horizontal coordinate value of each keypoint is not within the horizontal coordinate range, and the vertical coordinate value of each keypoint is not within the vertical coordinate range, it is determined that the field of view label of the keypoint is an out-of-field label. including, The method according to claim 4.

8. After determining the coordinate information of multiple keypoints contained in each sample image frame, further, Image augmentation processing is performed on the multiple sample image frames based on the coordinate information of multiple keypoints contained in each sample image frame. Includes, The aforementioned image augmentation process includes at least one of random disturbance and cropping. The method according to claim 4.

9. If the image augmentation process includes the cropping process, the cropping process is: Based on the coordinate information of multiple keypoints contained in each sample image frame, the center position of the crop frame is determined, Based on the aforementioned center position of the crop frame, the crop frame position is determined, and the RGB values ​​of pixel points located outside the crop frame are set to black according to the aforementioned crop frame position. including, The method according to claim 8.

10. It includes an image acquisition module, a human body keypoint detection module, a virtual character driving module, a smoothing weight determination module, and a smoothing processing module. The aforementioned image acquisition module is installed to acquire a target image frame containing an image of a part of the human body. The human body keypoint detection module is configured to input the target image frame into a pre-trained keypoint detection model and obtain coordinate information and field of view probability of the human body keypoints in the target image frame output by the keypoint detection model, wherein the field of view probability is the probability that the human body keypoints appear within the field of view of the target image frame. The virtual character driving module is installed to drive the corresponding virtual character movements based on the coordinate information of the human body keypoints and the field of view probability. The smoothing weight determination module is configured to determine the smoothing weights between the current target image frame and the previous target image frame. The smoothing processing module is configured to perform smoothing processing on the coordinate information and the field of view probability by employing the smoothing weights. A device driven by virtual characters.

11. It includes a keypoint detection module, a field-of-view label determination module, a model training module, a 2D pose prediction module, a 3D pose prediction module, and a coordinate determination module. The aforementioned keypoint detection module is installed to determine the coordinate information of multiple keypoints contained in each sample image frame by performing keypoint detection on multiple sample image frames in the sample set. The field-of-view label determination module is configured to determine the field-of-view label for each keypoint based on the coordinate information of each keypoint, and the field-of-view label is used to mark whether or not the keypoint is within the field of view of the sample image frame to which it belongs. The model training module is configured to train a keypoint detection model using the coordinate information of multiple keypoints in the multiple sample image frames and the field of view labels as training signals. The keypoint detection model is used in the model estimation phase to perform keypoint detection on a target image frame and output the coordinate information of the keypoints in the target image frame and the field of view probability. The two-dimensional pose prediction module is configured to input the plurality of sample image frames into a pre-generated two-dimensional pose network and to obtain two-dimensional coordinate information of the key points of the plurality of sample image frames output by the two-dimensional pose network. The three-dimensional pose prediction module is configured to input the plurality of sample image frames into a pre-generated three-dimensional pose network and obtain the three-dimensional coordinate information of the key points of the plurality of sample image frames output by the three-dimensional pose network. The coordinate determination module is installed to determine the coordinate information of each key point based on the obtained two-dimensional coordinate information and three-dimensional coordinate information of the key points. A device for training keypoint detection models.

12. At least one processor, A memory device configured to store at least one program, Includes, When the at least one program is executed by the at least one processor, the at least one processor is made to implement the method according to any one of claims 1 to 9. Electronic devices.

13. Computer programs are stored, When the computer program is executed by the processor, the method according to any one of claims 1 to 9 is realized. A computer-readable storage medium.

14. Includes computer-executable instructions, When the computer-executable instruction is executed, the method according to any one of claims 1 to 9 is realized. Computer program.

Images