Key point detection model training and virtual character driving method and corresponding device

Abstract

The application discloses a key point detection model training and virtual character driving method and a corresponding device. The virtual character driving method comprises the following steps: collecting a target image frame, wherein the target image frame comprises an image of a part of a human body; inputting the target image frame into a pre-trained key point detection model, and obtaining coordinate information of a human body key point of the target image frame output by the key point detection model and a field of view probability, wherein the field of view probability is a probability of the human body key point appearing in a shooting field of view of the target image frame; and driving a corresponding virtual character to move according to the coordinate information of the human body key point and the field of view probability, so as to realize end-to-end signal output of the human body key point, ensure a certain stability in a close-up case, and meet the driving requirements of the virtual character on a mobile phone.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a key point detection model training method, a virtual character driving method, a key point detection model training device, a virtual character driving device, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] With the development of the virtual industry, the content of live broadcast has appeared in the form of digital live broadcast, such as virtual anchors completely presented by virtual images.

[0003] In one related technology, a virtual anchor is realized by using technologies such as optical motion capture and inertial motion capture, but the implementation of such technologies requires the anchor to wear professional equipment for a long time, and usually needs to be connected to multiple cables, resulting in a poor live broadcast experience.

[0004] In another related technology, 3D human body data in a real environment is generated through end-to-end 3D pose data, and the end-to-end 3D pose estimation method can greatly enhance the interaction ability of virtual characters. When obtaining 3D human body data, the related technology uses a deep learning network to directly predict the human body 3D joint points from the input RGB video through an ordinary RGB camera. This scheme generally processes the entire human body, and generally has certain requirements for the FOV of the camera and the shooting angle, for example, the field of view needs to cover most of the human body. In the case where the user is close to the camera and only part of the human body appears, due to the limited terminal performance, small field of view, and the fact that parts of the human body such as hands and elbows are easily removed from the field of view, the pose point is very easy to be misdetected. Moreover, due to the limited resources on the mobile terminal, the resolution of many heatmap-based methods is limited, the accuracy of the output coordinates is not enough, and the stability of the output coordinates is also greatly affected. SUMMARY

[0005] The present application provides a key point detection model training and virtual character driving method and corresponding device to solve the problem of misdetected pose points and poor output coordinate stability due to limited terminal performance, small field of view, and the fact that parts of the human body such as hands and elbows are easily removed from the field of view in the prior art.

[0006] According to a first aspect of the present application, a virtual character driving method is provided, the method comprising:

[0007] acquiring a target image frame, the target image frame comprising an image of a partial human body;

[0008] input the target image frame into a pre-trained key point detection model, and obtain coordinate information of human body key points of the target image frame output by the key point detection model and a field of view probability, the field of view probability being a probability of the human body key points appearing in a shooting field of view of the target image frame;

[0009] drive corresponding virtual role actions according to the coordinate information of the human body key points and the field of view probability.

[0010] According to a second aspect of the present application, a method for training a key point detection model is provided, the method comprising:

[0011] performing key point detection on each sample image frame in a sample set to determine coordinate information of each key point;

[0012] determining a field of view label of the key point based on the coordinate information of the key point, the field of view label being used to mark whether the key point is in a shooting field of view of the sample image frame to which the key point belongs;

[0013] using the coordinate information of each key point of each sample image frame and the field of view label as a supervision signal to train a key point detection model, the key point detection model being used to perform key point detection on each target image frame in a model inference stage and output coordinate information of key points of the target image frame and a field of view probability.

[0014] According to a third aspect of the present application, an apparatus for driving a virtual role is provided, the apparatus comprising:

[0015] an image acquisition module configured to acquire a target image frame, the target image frame comprising an image of part of a human body;

[0016] a human body key point detection module configured to input the target image frame into a pre-trained key point detection model, and obtain coordinate information of human body key points of the target image frame output by the key point detection model and a field of view probability, the field of view probability being a probability of the human body key points appearing in a shooting field of view of the target image frame;

[0017] a virtual role driving module configured to drive corresponding virtual role actions according to the coordinate information of the human body key points and the field of view probability.

[0018] According to a fourth aspect of the present application, an apparatus for training a key point detection model is provided, the apparatus comprising:

[0019] a key point detection module configured to perform key point detection on each sample image frame in a sample set to determine coordinate information of each key point;

[0020] A field-of-view label determination module is configured to determine a field-of-view label of the key point based on the coordinate information of the key point, where the field-of-view label is used to mark whether the key point is within a shooting field of view of the sample image frame to which the key point belongs.

[0021] A model training module is configured to use the coordinate information of the key points of each sample image frame and the field-of-view labels as a supervision signal to train a key point detection model, where the key point detection model is used to perform key point detection on each target image frame in a model inference stage and output coordinate information of key points of the target image frame and a field-of-view probability.

[0022] According to a fifth aspect of the present application, an electronic device is provided, and the electronic device comprises:

[0023] at least one processor; and

[0024] a memory connected to the at least one processor in communication; wherein

[0025] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect.

[0026] According to a sixth aspect of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method of the first aspect or the second aspect when executed.

[0027] According to a seventh aspect of the present application, a computer program product is provided, and the computer program product comprises computer executable instructions, and the computer executable instructions are used to implement the method of the first aspect or the second aspect when executed.

[0028] In the embodiment, in the data preparation stage, the coordinate information of each key point in each sample image frame and the field of view label are obtained, the field of view label is used to mark whether the key point is in the shooting field of view of the sample image frame, and in the model training stage, the coordinate information of each key point in each sample image frame and the field of view label are used as a supervision signal to train the key point detection model. By introducing an additional prediction head, the key point detection model can output the field of view label of the key point, thereby effectively determining whether the key point is in the field of view, and improving the key point detection accuracy of the key point detection model for images in a near field scene. For example, in the case where the user is close to the camera and only part of the human body appears in the shooting field of view of the camera, the key point detection model of the embodiment can output the field of view label of each human body key point in the field of view or outside the field of view, thereby avoiding the case where the subsequent scene processing is affected by the key points outside the field of view.

[0029] In addition, in the model inference stage, the key point detection model can detect the coordinate information of the human body key point of the target image frame and the field of view probability, the field of view probability is the probability of the human body key point appearing in the shooting field of view of the target image frame, and then the corresponding virtual role action is driven in combination with the field of view probability and the coordinate information. In this way, the end-to-end signal output of the human body key point is realized, and in the near scene, a certain stability can be ensured to meet the driving requirements of the virtual role on the mobile phone. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0031] Figure 1 It is a method flowchart of training a key point detection model provided by the first embodiment of the present application;

[0032] Figure 2 It is a schematic diagram of a sample image frame provided by the first embodiment of the present application;

[0033] Figure 3 It is a schematic diagram of a cropped sample image frame provided by the first embodiment of the present application;

[0034] Figure 4 It is a method flowchart of driving a virtual role provided by the second embodiment of the present application;

[0035] Figure 5 It is a structural schematic diagram of a key point detection model training device provided by the third embodiment of the present application;

[0036] Figure 6 FIG. 1 is a structural schematic diagram of a virtual character driven device provided by an embodiment of the present application;

[0037] Figure 7 FIG. 1 is a structural schematic diagram of a virtual character driven device provided by an embodiment of the present application; DETAILED DESCRIPTION

[0038] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0040] Embodiment One

[0041] Figure 1 A method flowchart for training a key point detection model is provided in the first embodiment of the present application. The key point detection model is used to detect human key points in an image with local human features (such as a half-body image), and is suitable for scenes of human key point detection, for example, in a live scene, the action of a virtual character is driven by detecting human key points.

[0042] In a virtual live scene, an end-to-end 3D pose estimation method can greatly enhance the interaction ability of a virtual host. Through research and analysis, it is found that virtual hosts generally use mobile phones for live streaming. This scenario has the following characteristics: limited terminal performance, small field of view, and parts of the human body such as hands and elbows are prone to frequently moving out of the field of view. Many open-source 3D pose point detection solutions are also prone to errors in this scenario. Among them, the field of view refers to the maximum range that the terminal camera can observe. The larger the field of view, the larger the observation range.

[0043] Based on this, the embodiment designs a low-cost model training method for training a key point detection model based on an ordinary RGB camera without increasing the additional shooting hardware and use cost of the anchor, wherein the key point detection model can be a neural network model, the key point detection model can include two prediction heads, each prediction head is equivalent to a layer of neural network, and the inputs of the two prediction heads are feature maps, wherein one prediction head is used to predict the positions of key points based on the feature maps, and the other prediction head is used to predict whether the key points are in the field of view based on the feature maps. Then, whether the key points are in the field of view can be determined through the key point detection model, and the key point detection accuracy of the key point detection model for the image in the near-field scene is improved.

[0044] As shown in Figure 1 The embodiment can include the following steps:

[0045] Step 101, performing key point detection on each sample image frame in the sample set to determine the coordinate information of each key point.

[0046] The sample image frame can include the main postures of the human body, so that the accuracy of the prediction result of the posture network is higher.

[0047] In implementation, the sample image frame can be detected by a pre-generated posture detection model, which can be a two-dimensional posture detection model or a three-dimensional posture detection model. In order to improve the accuracy of detection, the posture detection model can also be a model obtained by combining a two-dimensional posture detection model and a three-dimensional posture detection model.

[0048] The key points can be different according to different business requirements, which is not limited by the embodiment. For example, the key points can include but are not limited to: left shoulder point, right shoulder point, left elbow point, right elbow point, left wrist point, right wrist point, left palm point, right palm point, hip joint point, nose point, etc.

[0049] Exemplarily, the coordinate information of the key points can be represented by image coordinates and depth information. In one implementation, the depth information can be obtained by using a set depth information calculation algorithm, without increasing an additional depth sensor to obtain the depth information, thereby saving the hardware cost and calibration cost.

[0050] In one embodiment, step 101 can further include the following steps:

[0051] Step 101-1, inputting the sample image frame into a pre-generated two-dimensional posture network, and obtaining the two-dimensional coordinate information of the key points of the sample image frame output by the two-dimensional posture network.

[0052] Exemplarily, the two-dimensional coordinate information is based on coordinate information in an image coordinate system, including a horizontal coordinate value and a vertical coordinate value, and can be expressed as wherein, is the two-dimensional coordinate information of a key point numbered n of a sample image frame, u n is a horizontal coordinate value of a key point numbered n of a sample image frame, v n is a vertical coordinate value of a key point numbered n of a sample image frame.

[0053] wherein, R nx2 represents an nx2-dimensional real number set, that is, In each element of the set, each component of the vector is a real number.

[0054] Step 101-2, inputting the sample image frame into a pre-generated three-dimensional pose network, and obtaining three-dimensional coordinate information of a key point of the sample image frame output by the three-dimensional pose network.

[0055] Exemplarily, the three-dimensional coordinate information is based on coordinate information in a world coordinate system, including an X-axis coordinate value, a Y-axis coordinate value and a Z-axis coordinate value, and can be expressed as wherein, is the three-dimensional coordinate information of a key point numbered n of a sample image frame, x n is an X-axis coordinate value of a key point numbered n of a sample image frame, y n is a Y-axis coordinate value of a key point numbered n of a sample image frame, z n is a Z-axis coordinate value of a key point numbered n of a sample image frame.

[0056] wherein, R nx3 represents an nx3-dimensional real number set, that is, In each element of the set, each component of the vector is a real number.

[0057] It should be noted that the two-dimensional pose network and the three-dimensional pose network can both be existing pose network models, and the embodiment assumes that the key points detected by the two are consistent, and only the coordinate system scales of the key points of the two are inconsistent, and the coordinate data needs to be uniformly processed in step 101-3.

[0058] Step 101-3, determining coordinate information of the key points based on the two-dimensional coordinate information and the three-dimensional coordinate information of each key point.

[0059] In this step, the coordinate information of each key point can be coordinate information obtained by fusing the two-dimensional coordinate information and the three-dimensional coordinate information of the key point, so as to transform the different output results of the two-dimensional pose network and the three-dimensional pose network to a unified coordinate system. Exemplarily, the coordinate information can include depth information, which can be calculated according to the two-dimensional coordinate information and the three-dimensional coordinate information.

[0060] In an embodiment, step 101-3 can further include the following steps:

[0061] Step 101-3-1, determining a first stable key point and a second stable key point from the key points.

[0062] In practice, the stable key points can be key points that appear more stably in the field of view, such as shoulder points, elbow points, nose points, etc.

[0063] The first stable key point and the second stable key point can appear in the form of a key point pair, for example, the first stable key point is a left shoulder point and the second stable key point is a right shoulder point, or for example, the first stable key point is a left elbow point and the second stable key point is a right elbow point, etc.

[0064] In an implementation, the developer can pre-configure a stable key point whitelist in different application scenarios. When the first stable key point and the second stable key point need to be determined, the stable key point whitelist in the current application scenario (such as a live broadcast scenario) can be found, and the matching key points in the stable key point whitelist are matched with the current detected key points, and then the matching key points are used as stable key points.

[0065] The first stable key point and the second stable key point have no order restriction, and are only used to distinguish different stable key points. If a stable key point pair is found, one of them can be used as the first stable key point and the other as the second stable key point.

[0066] Step 101-3-2, determining an adjustment coefficient according to the two-dimensional coordinate information and the three-dimensional coordinate information of the first stable key point and the second stable key point.

[0067] The adjustment coefficient is a parameter used to reflect the scale difference between the coordinate system of the two-dimensional coordinate information and the coordinate system of the three-dimensional coordinate information, and the depth information of each key point can be determined according to the adjustment coefficient.

[0068] In an embodiment, step 101-3-2 can further include the following steps:

[0069] The absolute value of the difference between the two-dimensional coordinate information of the first stable key point and the second stable key point is determined as a first difference value, and the absolute value of the difference between the three-dimensional coordinate information of the first stable key point and the second stable key point is determined as a second difference value; and the ratio of the first difference value to the second difference value is taken as the adjustment coefficient.

[0070] For example, assuming that the first stable key point is a left shoulder point with a key point number of 0, and the second stable key point is a right shoulder point with a key point number of 1, then:

[0071]

[0072]

[0073]

[0074] In step 101-3-3, the Z-axis coordinate value of each key point is adjusted by using the adjustment coefficient to obtain the depth value of each key point.

[0075] In one implementation, the depth value depth of each key point is: n = scale * z n .

[0076] In step 101-3-4, the horizontal coordinate value, the vertical coordinate value, and the depth value of each key point are taken as the coordinate information of the key point.

[0077] After obtaining the depth value of each key point, the image coordinates and the depth value of each key point can be organized into the final coordinate information of each key point, that is, wherein,

[0078] In this embodiment, the two-dimensional pose network is used to obtain the two-dimensional coordinate information of the key points of the sample image frame, and the three-dimensional pose network is used to obtain the three-dimensional coordinate information of the key points of the sample image frame, and then the two-dimensional coordinate information and the three-dimensional coordinate information are fused to obtain the final coordinate information of each key point, so that more accurate and stable key point coordinate information with lower cost can be obtained, and the detection accuracy of the key points is improved.

[0079] In one embodiment, after determining the coordinate information of each key point of the sample image frame, the sample image frame can also be subjected to image augmentation processing and other preprocessing based on the coordinate information of each key point, wherein the image augmentation processing can include at least one or a combination of the following: random disturbance, cropping processing.

[0080] The specific implementation of the random disturbance is not limited in the embodiment. For example, if the random disturbance refers to the pixel value of each pixel point, the pixel value is randomly changed according to a preset manner. In an exemplary embodiment, if the preset manner is to randomly disturb in a range of [-20, 20] of the original range, if the RGB pixel value of a certain pixel point is (6, 12, 230), the pixel value becomes (8, 12, 226) after the random disturbance according to the preset manner. The range of each color in the pixel value is [0, 255], that is, the maximum value after the disturbance is 255 and the minimum value is 0.

[0081] In order to simulate a near-field scene close to the shooting device, the sample image frames (including the sample image frames generated after the random disturbance) can also be subjected to cropping processing. In an embodiment, the cropping processing can include the following processes:

[0082] The center position of the cropping frame is determined based on the coordinate information of the key points, the position of the cropping frame is determined according to the center position of the cropping frame, and the RGB values of the pixel points located outside the cropping frame are set to black.

[0083] In an implementation, the center position of the cropping frame can be the center position of the human body in the sample image frame, and the center position of the cropping frame can be calculated according to the coordinate information of the detected key points. For example, the center position of the cropping frame can be the average of the coordinate information of the left shoulder point, the right shoulder point and the hip joint point.

[0084] After obtaining the center position of the cropping frame, the position of the cropping frame can be determined according to the center position of the cropping frame and the preset size of the cropping frame (i.e. the width and height of the cropping frame). After determining the position of the cropping frame, the RGB values of the pixel points located outside the cropping frame can be set to black (i.e. the RGB value is 0), and the cropped sample image frame is obtained. For example, if the sample image frame is as shown in Figure 2 , the cropped sample image frame is as shown in Figure 3 .

[0085] In practice, the size of the cropping frame can also be subjected to random disturbance to obtain the cropped sample image frame with different cropping frame sizes.

[0086] In the embodiment, the image augmentation processing can expand the training data set, suppress the model overfitting, and improve the model generalization ability. At the same time, the low-cost labeling migration method and the data preprocessing method can effectively simulate the target scene.

[0087] In step 102, the field of view label of the key point is determined based on the coordinate information of the key point, and the field of view label is used to mark whether the key point is in the shooting field of view of the sample image frame to which the key point belongs.

[0088] When the key point set of each sample image frame is detected, the detected key points may not be within the range of the sample image frame due to the accuracy of the model detection. For example, if the palm point is not in the image in the sample image frame, but the detected key point contains the palm point. Based on this, in the embodiment, for each key point, it can also be judged whether the key point is within the shooting field of view of the sample image frame to which the key point belongs based on the coordinate information of the key point, so as to determine the field of view label of the key point. For example, if a key point is within the shooting field of view of the sample image frame to which the key point belongs, the field of view label of the key point is 1, and if a key point is not within the shooting field of view of the sample image frame to which the key point belongs, the field of view label of the key point is 0.

[0089] In implementation, the size of the sample image frame and the coordinate information of the key point can be compared to determine whether the key point is within the shooting field of view of the sample image frame to which the key point belongs. If it is determined that the key point is within the range of the sample image frame to which the key point belongs according to the coordinate information of the key point, it is determined that the key point is within the shooting field of view, otherwise it is determined that the key point is outside the shooting field of view.

[0090] In an embodiment, step 102 can further include the following steps:

[0091] The width and height of the sample image frame are obtained; the horizontal coordinate range is determined according to the width of the sample image frame, and the vertical coordinate range is determined according to the height of the sample image frame, taking the origin of the image coordinate system as the starting point; if the horizontal coordinate value of the key point is within the horizontal coordinate range, or the vertical coordinate value of the key point is within the vertical coordinate range, it is determined that the field of view label of the key point is the in-field label; if the horizontal coordinate value of the key point is not within the horizontal coordinate range, and the vertical coordinate value of the key point is not within the vertical coordinate range, it is determined that the field of view label of the key point is the out-field label.

[0092] Specifically, the width of the sample image frame can be taken as the length of the horizontal coordinate axis, that is, the horizontal coordinate range is [0, width], and width is the width; the height of the sample image frame can be taken as the length of the vertical coordinate axis, that is, the vertical coordinate range is [0, height], and height is the height.

[0093] In practice, the field of view label can be determined according to the following logical judgment formula:

[0094]

[0095] Step 103, the coordinate information and the field of view label of each key point of each sample image frame are taken as a supervision signal to train the key point detection model.

[0096] In this step, the coordinate information of each key point of each obtained sample image frame and the field of view label can be used as a supervision signal to train the key point detection model.

[0097] The key point detection model is used to perform key point detection on each target image frame in the model inference stage and output the coordinate information of the key points of each target image frame and the field of view probability. In subsequent applications, the coordinate information and the field of view probability can be used to drive the corresponding virtual character actions.

[0098] In an embodiment, when training the key point detection model, the loss function used includes a heat map loss function Loss heatmap , a position loss function Loss location , and a label loss function Loss label , for example:

[0099] Loss total = Loss heatmap + Loss location + Loss label

[0100] It should be noted that the present embodiment does not limit the implementation of the above three loss functions. For example, Loss heatmap can be implemented using L2 loss, Loss location can be implemented using L1 loss, and Loss label can be implemented using cross-entropy loss.

[0101] In the present embodiment, in the data preparation stage, the coordinate information of each key point in each sample image frame and the field of view label can be obtained, and the field of view label is used to mark whether the key point is within the shooting field of view of the sample image frame to which it belongs. The coordinate information of each key point of each sample image frame and the field of view label are used as a supervision signal to train the key point detection model in the model training stage. By introducing an additional prediction head, the key point detection model can output the field of view label of the key point, thereby effectively determining whether the key point is within the field of view, improving the key point detection accuracy of the key point detection model for images in a near-field scene. For example, in the case where the user is close to the camera and only part of the human body appears within the shooting field of view of the camera, the key point detection model of the present embodiment can output the field of view label of each human body key point within the field of view or outside the field of view, avoiding the situation where the subsequent scene processing using the key points outside the field of view affects the processing effect.

[0102] Embodiment Two

[0103] Figure 4 A method flowchart for driving a virtual character is provided in Embodiment Two of the present application. Embodiment Two belongs to the model inference stage of the key point detection model of Embodiment One. In this embodiment

[0104] As Figure 4 shown, the embodiment can include the following steps:

[0105] Step 201, collecting a target image frame, the target image frame including an image of a part of a human body.

[0106] The target image frame can be a real-time collected image frame, for example, a half-body photo of an anchor collected by a mobile phone or the like in a live broadcast scenario.

[0107] Step 202, inputting the target image frame into a pre-trained key point detection model, and obtaining coordinate information of a human body key point of the target image frame and a field of view probability output by the key point detection model.

[0108] When the target image frame is collected, the target image frame can be input into the key point detection model generated in Embodiment One, and the human body key point detection is performed by the key point detection model to obtain coordinate information of one or more human body key points and a field of view probability, wherein the field of view probability is a probability of the human body key point appearing in a shooting field of view of the target image frame.

[0109] The coordinate information can be represented as (u n ,v n ,depth n ), and the coordinate information output by the key point detection model has a lower cost than the three-dimensional coordinate information output by the three-dimensional pose network. This is because some information in the three-dimensional coordinate information is not needed, and in combination with the virtual driving scene, the actual scene may not need the depth value, but only a relative value, so the three-dimensional coordinate information is converted into pseudo 3D coordinate information in the image coordinate scale by the training scheme of Embodiment One.

[0110] In an embodiment, after obtaining the coordinate information of the human body key point of the target image frame and the field of view probability, the following steps can be further included:

[0111] Determining a smoothing weight between the current target image frame and the previous target image frame; and smoothing the coordinate information and the field of view probability by using the smoothing weight.

[0112] For example, in a virtual live broadcast scenario using a mobile phone, the human body key point outside the field of view is not particularly interested, and it is only necessary to keep it as smooth as possible without jumping and obvious errors. Therefore, the coordinate information and the prediction probability can be smoothed by a filter.

[0113] In an embodiment, the step of determining the smoothing weight between the current target image frame and the previous target image frame can further include the following steps:

[0114] For each human body key point, a distance between the coordinate information of the human body key point in the current target image frame and the smoothed coordinate information in the previous target image frame is determined; the distance is compared with a set distance, and a distance weight is determined according to a comparison result; and the distance weight and the field of view probability of the human body key point in the current target image frame are used to calculate a smoothing weight.

[0115] In an implementation, for a human body key point, after the smoothed coordinate information of the human body key point in the previous target image frame and the coordinate information of the human body key point in the current target image frame are obtained, a distance between the two coordinate information can be calculated by using a distance calculation formula, for example, the distance can be calculated by using the following formula:

[0116]

[0117] wherein distance n is a distance between the coordinate information of the nth human body key point in the current target image frame and the smoothed coordinate information in the previous target image frame, is the coordinate information of the nth human body key point in the current target image frame, is the smoothed coordinate information of the nth human body key point in the previous target image frame.

[0118] In an implementation, the obtained distance is compared with a set distance, and a ratio of the two can be calculated, and the comparison result is the ratio of the two, that is, wherein threshold is the set distance. The distance distance n is lower than the threshold thresold, the weight of the buffered historical data is larger, and the weight of the current key point is smaller; the distance distance n is greater than the threshold, the weight of the historical data is smaller, and the weight of the current key point is larger.

[0119] In an implementation, the distance weight can be calculated by using the following formula:

[0120]

[0121] wherein k represents a preset smoothing intensity, and the greater the k is, the smaller the window is, and the stronger the ability to suppress the sharp change is.

[0122] After the distance weight is obtained, the distance weight and the field of view probability of the current human body key point in the current target image frame can be combined to calculate the smoothing weight. For example, the smoothing weight can be calculated by using the following formula:

[0123]

[0124] wherein, prob n is the field of view probability of the nth human body key point.

[0125] When the smoothing weight is obtained, the coordinate information and the field of view probability can be smoothed by using the smoothing weight. In an embodiment, the smoothing further includes the following steps:

[0126] based on the smoothing weight, determining a first weight of the previous target image frame and a second weight of the current target image frame; based on the first weight and the second weight, performing weighted calculation on the coordinate information of the previous target image frame and the coordinate information of the current target image frame to obtain smoothed coordinate information; based on the first weight and the second weight, performing weighted calculation on the field of view probability of the previous target image frame and the field of view probability of the current target image frame to obtain smoothed field of view probability.

[0127] For example, assuming that the first weight is the smoothing weight, the second weight is the difference between the value 1 and the smoothing weight, i.e., the first weight = ratio n , and the second weight = 1-ratio n .

[0128] The process of smoothing the coordinate information is shown in the following formula:

[0129]

[0130] The process of smoothing the field of view probability is shown in the following formula:

[0131] Cached_prob n = ratio n *Cached-prob n-1 -+(1-ratio n )prob n

[0132] wherein, Cached_prob n is the smoothed field of view probability, and Cached_prob n-1 is the field of view probability after the last smoothing.

[0133] The filter introduced in this embodiment smoothes the key point jump degree and whether it is in the field of view, and by adjusting the weight of the current frame in the case of key point coordinate drastic jump and not in the field of view, the output human body key point result is kept in a relatively stable and continuous state.

[0134] Step 203, driving the corresponding virtual role action according to the coordinate information and the field of view probability of the human body key point.

[0135] Specifically, after obtaining the final coordinate information of the human body key points of the target image frame and the field of view probability, it can be determined whether the corresponding human body key points are in the field of view of the target image frame according to the field of view probability. If they are in the field of view, the corresponding human body parts of the virtual character can be moved to the positions corresponding to the coordinate information according to the coordinate information. If they are not in the field of view, the virtual character is not moved.

[0136] For example, through these human body key points, the interaction between the host and the user can be more rich, such as waving hands, making heart signs, displaying 3D gifts sent by the user on the arms or wrists of the 3D image, and making the host and the user's virtual image interact in the virtual 3D space.

[0137] In this embodiment, the coordinate information of the human body key points of the target image frame and the field of view probability of the human body key points in the shooting field of view of the target image frame can be detected by the key point detection model. Then, the corresponding virtual character is driven in combination with the field of view probability and the coordinate information, so as to realize the end-to-end signal output of the human body key points and ensure a certain stability in the close-up case, and meet the driving requirements of the virtual character on the mobile phone.

[0138] Embodiment Three

[0139] Figure 5 The structure schematic diagram of a key point detection model training device provided in Embodiment Three of the present application can include the following modules:

[0140] The key point detection module 301 is configured to perform key point detection on each sample image frame in the sample set to determine the coordinate information of each key point.

[0141] The field of view label determination module 302 is configured to determine the field of view label of the key point based on the coordinate information of the key point, wherein the field of view label is used to mark whether the key point is in the shooting field of view of the sample image frame.

[0142] The model training module 303 is configured to use the coordinate information and the field of view label of each key point of each sample image frame as a supervision signal to train a key point detection model, wherein the key point detection model is used to perform key point detection on each target image frame in the model inference stage and output the coordinate information and the field of view probability of the key points of the target image frame.

[0143] In an embodiment, the key point detection module 301 can further include the following modules:

[0144] a two-dimensional pose prediction module, configured to input the sample image frame into a pre-generated two-dimensional pose network, and obtain two-dimensional coordinate information of key points of the sample image frame output by the two-dimensional pose network;

[0145] a three-dimensional pose prediction module, configured to input the sample image frame into a pre-generated three-dimensional pose network, and obtain three-dimensional coordinate information of key points of the sample image frame output by the three-dimensional pose network;

[0146] a coordinate determination module, configured to determine coordinate information of the key points based on the two-dimensional coordinate information and the three-dimensional coordinate information of the key points.

[0147] In an embodiment, the two-dimensional coordinate information includes a horizontal coordinate value and a vertical coordinate value, and the three-dimensional coordinate information includes an X-axis coordinate value, a Y-axis coordinate value and a Z-axis coordinate value.

[0148] The coordinate determination module can further include the following modules:

[0149] a stable key point determination module, configured to determine a first stable key point and a second stable key point from the key points;

[0150] an adjustment coefficient determination module, configured to determine an adjustment coefficient according to the two-dimensional coordinate information and the three-dimensional coordinate information of the first stable key point and the second stable key point;

[0151] an adjustment module, configured to adjust the Z-axis coordinate value of each key point by using the adjustment coefficient, to obtain a depth value of each key point;

[0152] a coordinate generation module, configured to take the horizontal coordinate value, the vertical coordinate value and the depth value of each key point as the coordinate information of the key point.

[0153] In an embodiment, the adjustment coefficient determination module is specifically configured to:

[0154] determine an absolute value of a difference of the two-dimensional coordinate information of the first stable key point and the second stable key point as a first difference;

[0155] determine an absolute value of a difference of the three-dimensional coordinate information of the first stable key point and the second stable key point as a second difference;

[0156] take a ratio of the first difference and the second difference as the adjustment coefficient.

[0157] In an embodiment, the coordinate information includes a horizontal coordinate value and a vertical coordinate value, and the field of view label includes an in-field-of-view label and an out-of-field-of-view label.

[0158] The field of view label determination module 302 is specifically configured to:

[0159] obtain a width and a height of the sample image frame;

[0160] determine a horizontal coordinate range according to the width of the sample image frame and a vertical coordinate range according to the height of the sample image frame, with the origin of the image coordinate system as a starting point;

[0161] if the horizontal coordinate value of the key point is within the horizontal coordinate range, or the vertical coordinate value of the key point is within the vertical coordinate range, it is determined that the field of view label of the key point is an in-field label;

[0162] if the horizontal coordinate value of the key point is not within the horizontal coordinate range, and the vertical coordinate value of the key point is not within the vertical coordinate range, it is determined that the field of view label of the key point is an out-of-field label.

[0163] In an embodiment, the device can further include the following modules:

[0164] an image augmentation module, configured to, after determining the coordinate information of each key point, perform image augmentation processing on the sample image frame based on the coordinate information, the image augmentation processing including at least one of or a combination of random disturbance and cropping processing.

[0165] In an embodiment, the image augmentation module is specifically configured to:

[0166] determine a cropping frame center position based on the coordinate information of the key point;

[0167] determine a cropping frame position according to the cropping frame center position, and set the RGB value of a pixel point located outside the cropping frame to black according to the cropping frame position.

[0168] In an embodiment, when training the key point detection model, the loss function used includes a heat map loss function, a position loss function, and a label loss function.

[0169] The device for training a key point detection model provided in the embodiments of the present application can execute the method for training a key point detection model provided in Embodiment One of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0170] Embodiment Four

[0171] Figure 6 A structural schematic diagram of a device for driving a virtual character provided in Embodiment Four of the present application can include the following modules:

[0172] The image acquisition module 401 is configured to acquire a target image frame, wherein the target image frame comprises an image of a partial human body.

[0173] The human key point detection module 402 is configured to input the target image frame into a pre-trained key point detection model, and obtain coordinate information of human key points of the target image frame output by the key point detection model and a field of view probability, wherein the field of view probability is a probability of the human key points appearing in a shooting field of view of the target image frame.

[0174] The virtual character driving module 403 is configured to drive a corresponding virtual character to perform an action according to the coordinate information of the human key points and the field of view probability.

[0175] In an embodiment, the apparatus can further comprise the following modules:

[0176] The smoothing weight determination module is configured to determine a smoothing weight between a current target image frame and a previous target image frame.

[0177] The smoothing processing module is configured to perform smoothing processing on the coordinate information and the field of view probability by using the smoothing weight.

[0178] In an embodiment, the smoothing weight determination module is specifically configured to:

[0179] For each human key point, determine a distance between coordinate information of the human key point in the current target image frame and smoothed coordinate information in the previous target image frame;

[0180] Compare the distance with a set distance, and determine a distance weight according to a comparison result;

[0181] Calculate the smoothing weight by using the distance weight and a field of view probability of the human key point in the current target image frame.

[0182] In an embodiment, the smoothing processing module is specifically configured to:

[0183] Determine a first weight of the previous target image frame and a second weight of the current target image frame based on the smoothing weight;

[0184] Perform weighted calculation on coordinate information of the previous target image frame and coordinate information of the current target image frame based on the first weight and the second weight, to obtain smoothed coordinate information;

[0185] Perform weighted calculation on a field of view probability of the previous target image frame and a field of view probability of the current target image frame based on the first weight and the second weight, to obtain smoothed field of view probability.

[0186] The virtual character driving device provided in this application embodiment can execute the virtual character driving method provided in embodiment two of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0187] Example 5

[0188] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the methods of this application is shown. Figure 7 As shown, the electronic device 10 can be a server, mobile phone, or other similar device, including at least one processor 11 and a storage device, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The storage device stores one or more computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from the storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10.

[0189] In some embodiments, the method in Embodiment 1 or Embodiment 2 may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method in Embodiment 1 or Embodiment 2 described above may be performed.

[0190] In some embodiments, the method in Embodiment 1 or Embodiment 2 may be implemented as a computer program product, which includes computer-executable instructions that, when executed, perform one or more steps of the method in Embodiment 1 or Embodiment 2 described above.

Claims

1. A method for driving virtual characters, characterized in that, The method comprises: collecting a target image frame, the target image frame comprising an image of a partial human body; inputting the target image frame into a pre-trained key point detection model and obtaining coordinate information of a human body key point of the target image frame output by the key point detection model and a field of view probability, the field of view probability being a probability of the human body key point appearing in a shooting field of view of the target image frame; driving a corresponding virtual role action according to the coordinate information of the human body key point and the field of view probability.

2. The method of claim 1, wherein, Before the driving of the corresponding virtual role action according to the coordinate information of the human body key point and the field of view probability, the method further comprises: determining a smoothing weight between a current target image frame and a previous target image frame; smoothing the coordinate information and the field of view probability by using the smoothing weight.

3. The method of claim 2, wherein, The determination of the smoothing weight between the current target image frame and the previous target image frame comprises: for each human body key point, determining a distance between coordinate information of the human body key point in the current target image frame and smoothed coordinate information in the previous target image frame; comparing the distance with a set distance and determining a distance weight according to a comparison result; calculating the smoothing weight by using the distance weight and a field of view probability of the human body key point in the current target image frame.

4. The method according to claim 2 or 3, characterized in that, The smoothing of the coordinate information and the field of view probability by using the smoothing weight comprises: determining a first weight of the previous target image frame and a second weight of the current target image frame based on the smoothing weight; performing weighted calculation on coordinate information of the previous target image frame and coordinate information of the current target image frame based on the first weight and the second weight to obtain smoothed coordinate information; performing weighted calculation on a field of view probability of the previous target image frame and a field of view probability of the current target image frame based on the first weight and the second weight to obtain smoothed field of view probability.

5. A method for training a keypoint detection model, the method comprising: The method comprises: performing key point detection on each sample image frame in a sample set to determine coordinate information of each key point; determining a field of view label of the key point based on the coordinate information of the key point, the field of view label being used to mark whether the key point is in a shooting field of view of the sample image frame; training a key point detection model by taking the coordinate information and the field of view label of each key point of each sample image frame as a supervision signal, the key point detection model being used to perform key point detection on each target image frame in a model inference stage and output coordinate information and a field of view probability of a key point of the target image frame.

6. The method of claim 5, wherein, The key point detection on each sample image frame in the sample set to determine coordinate information of each key point comprises: inputting the sample image frame into a pre-generated two-dimensional pose network and obtaining two-dimensional coordinate information of a key point of the sample image frame output by the two-dimensional pose network; inputting the sample image frame into a pre-generated three-dimensional pose network and obtaining three-dimensional coordinate information of a key point of the sample image frame output by the three-dimensional pose network; Determine coordinate information of the key points based on the two-dimensional coordinate information and the three-dimensional coordinate information of each key point.

7. The method of claim 6, wherein, The two-dimensional coordinate information includes a horizontal coordinate value and a vertical coordinate value, and the three-dimensional coordinate information includes an X-axis coordinate value, a Y-axis coordinate value, and a Z-axis coordinate value. The determination of the coordinate information of the key points based on the two-dimensional coordinate information and the three-dimensional coordinate information of each key point includes: Determine a first stable key point and a second stable key point from the key points; Determine an adjustment coefficient based on the two-dimensional coordinate information and the three-dimensional coordinate information of the first stable key point and the second stable key point; Adjust the Z-axis coordinate value of each key point by using the adjustment coefficient to obtain a depth value of each key point; Take the horizontal coordinate value, the vertical coordinate value, and the depth value of each key point as the coordinate information of the key point.

8. The method of claim 7, wherein, The determination of the adjustment coefficient based on the two-dimensional coordinate information and the three-dimensional coordinate information of the first stable key point and the second stable key point includes: Determine a first difference value as an absolute value of a difference between the two-dimensional coordinate information of the first stable key point and the second stable key point; Determine a second difference value as an absolute value of a difference between the three-dimensional coordinate information of the first stable key point and the second stable key point; Take a ratio of the first difference value to the second difference value as the adjustment coefficient.

9. The method according to any one of claims 5-8, characterized in that, The coordinate information includes a horizontal coordinate value and a vertical coordinate value, and the field-of-view label includes an in-field label and an out-of-field label. The determination of the field-of-view label of the key point based on the coordinate information of the key point includes: Obtain a width and a height of the sample image frame; Determine a horizontal coordinate range based on the width of the sample image frame and a vertical coordinate range based on the height of the sample image frame, with an origin of an image coordinate system as a starting point; If the horizontal coordinate value of the key point is within the horizontal coordinate range, or the vertical coordinate value of the key point is within the vertical coordinate range, determine that the field-of-view label of the key point is the in-field label; If the horizontal coordinate value of the key point is not within the horizontal coordinate range, and the vertical coordinate value of the key point is not within the vertical coordinate range, determine that the field-of-view label of the key point is the out-of-field label.

10. The method according to any one of claims 5-8, characterized in that, After the determination of the coordinate information of each key point, the method further includes: Perform image augmentation processing on the sample image frame based on the coordinate information, and the image augmentation processing includes at least one of the following or a combination thereof: random disturbance, cropping processing; The cropping processing includes: Determine a cropping frame center position based on the coordinate information of the key points; Determine a cropping frame position based on the cropping frame center position, and set the RGB value of a pixel point located outside the cropping frame to black according to the cropping frame position.

11. A virtual character-driven device, characterized in that, The device includes: An image acquisition module configured to acquire a target image frame, the target image frame including an image of part of a human body; The human key point detection module is configured to input the target image frame into a pre-trained key point detection model, and obtain coordinate information of human key points of the target image frame output by the key point detection model and a field of view probability, the field of view probability being a probability of the human key points appearing in a shooting field of view of the target image frame. The virtual character driving module is configured to drive a corresponding virtual character to perform an action according to the coordinate information of the human key points and the field of view probability.

12. An apparatus for training a keypoint detection model, the apparatus comprising: The apparatus includes: The key point detection module is configured to perform key point detection on each sample image frame in a sample set to determine coordinate information of each key point. The field of view label determination module is configured to determine a field of view label of the key point based on the coordinate information of the key point, the field of view label being used to mark whether the key point is in a shooting field of view of the sample image frame. The model training module is configured to use the coordinate information of each key point of each sample image frame and the field of view label as a supervision signal to train a key point detection model, the key point detection model being used to perform key point detection on each target image frame in a model inference stage and output coordinate information of key points of the target image frame and a field of view probability.

13. An electronic device, comprising: The electronic device includes: One or more processors; A storage device configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-10.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-10.

15. A computer program product, comprising computer executable instructions that, when executed, implement the method of any one of claims 1-10.

Images