Live streaming room hand motion capture method, device, medium and computer equipment

Abstract

Embodiments of the present application relate to the field of network live broadcast, and provide a method and device for capturing hands in a live broadcast room, a medium and a computer device. The method comprises: constructing a hand motion capture model, training the constructed hand motion capture model by using hand training samples and a hand reconstruction model, and obtaining a trained hand motion capture model; obtaining a monocular RGB image of a host user in the live broadcast room; inputting the monocular RGB image into a hand detection model to obtain a hand image; inputting the hand image into the trained hand motion capture model to obtain hand motion parameters; inputting the hand motion parameters into the hand reconstruction model to obtain a reconstructed three-dimensional hand mesh image; and obtaining a reconstructed two-dimensional hand image according to the three-dimensional hand mesh image, so that the hand image can be quickly captured by using the hand motion capture model and the hand reconstruction model without inputting depth information, and only using a monocular RGB image, and the accuracy of hand capture can be improved.

Description

Technical Field

[0001] This application relates to the field of live streaming technology, and in particular to a method, apparatus, medium, and computer equipment for capturing hand movements in a live streaming room. Background Technology

[0002] With the advancement of network communication technology, users have access to an increasing number of online entertainment options. Among these, live streaming, with its strong real-time and interactive features, is gaining popularity among a growing number of users.

[0003] Virtual anchors (VAS) have been widely used in online live streaming. VAS live streaming involves a real user selecting a virtual avatar, which is then synthesized in real-time by capturing the real user's appearance and movements. This virtual avatar then acts as the live streamer. VAS live streaming can respond in real-time to the user's language and actions, interacting with the audience. It realistically imitates and reproduces various complex expressions and movements of a real person to achieve the desired virtual live streaming effect.

[0004] In virtual YouTuber live streaming, virtual characters need to mimic the hand gestures of the live streamer. Since human hands contain a wealth of interactive information with viewers, hand motion capture can be further used to identify typical hand movements, triggering related special effects. Therefore, accurate and rapid capture of the streamer's hands is crucial. However, in virtual YouTuber live streaming, hand motion is typically captured using a standard RGB camera to obtain a monocular RGB image of the live streamer. Hand movements are then identified based on this monocular RGB image. Because human hand movements are subtle, varied, and prone to self-occlusion, some technologies use estimated or directly input depth information from the monocular RGB image for motion capture. However, depth information is difficult to estimate accurately and measure, increasing the complexity of hand motion capture and resulting in lower accuracy. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this application provides a method, device, medium and computer equipment for capturing hands in a live broadcast room, which can quickly capture hand images and improve the accuracy of hand capture.

[0006] According to a first aspect of the embodiments of this application, a method for capturing hands in a live broadcast room is provided, comprising the following steps:

[0007] A hand motion capture model is constructed, and the constructed hand motion capture model is trained using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model.

[0008] Obtain the monocular RGB image of the live streamer;

[0009] The monocular RGB image is input into the hand detection model to obtain a hand image;

[0010] The hand image is input into the trained hand motion capture model to obtain hand motion parameters;

[0011] The human hand motion parameters are input into the human hand reconstruction model to obtain a reconstructed three-dimensional human hand mesh image;

[0012] Based on the three-dimensional human hand mesh image, a reconstructed two-dimensional human hand image is obtained.

[0013] According to a second aspect of the embodiments of this application, a method for capturing hands in a live broadcast room is provided, comprising:

[0014] The human hand motion capture model construction module is used to construct a human hand motion capture model, and to train the constructed human hand motion capture model using human hand training samples and a human hand reconstruction model to obtain a trained human hand motion capture model.

[0015] The RGB image acquisition module is used to acquire the monocular RGB image of the live streamer.

[0016] The hand image acquisition module is used to input the monocular RGB image into the hand detection model to obtain a hand image;

[0017] The human hand motion parameter acquisition module is used to input the human hand image into the trained human hand motion capture model to obtain human hand motion parameters;

[0018] The mesh image acquisition module is used to input the human hand motion parameters into the human hand reconstruction model to obtain the reconstructed three-dimensional human hand mesh image;

[0019] The image reconstruction module is used to obtain a reconstructed two-dimensional human hand image based on the three-dimensional human hand mesh image.

[0020] According to a third aspect of the embodiments of this application, a computer device is provided, including a processor and a memory; the memory stores a computer program adapted to be loaded by the processor and executed as described above in the live broadcast room hand capture method.

[0021] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, it implements the hand capture method for live streaming rooms as described above.

[0022] This application embodiment constructs a hand motion capture model and trains it using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model. It then acquires a monocular RGB image of a live streamer; inputs the monocular RGB image into a hand detection model to obtain a hand image; inputs the hand image into the trained hand motion capture model to obtain hand motion parameters; inputs the hand motion parameters into a hand reconstruction model to obtain a reconstructed 3D hand mesh image; and obtains a reconstructed 2D hand image based on the 3D hand mesh image. This eliminates the need for depth information input; only a monocular RGB image is required. The hand motion capture model and hand reconstruction model can quickly capture hand images. Furthermore, using the hand motion parameters output by the trained hand motion capture model as input to the hand reconstruction model improves the accuracy of hand capture.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0024] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic block diagram illustrating the application environment of the live streaming room hand capture method provided in this application embodiment;

[0027] Figure 2 A flowchart of the live streaming room hand capture method provided in the first embodiment of this application;

[0028] Figure 3 A 21-point diagram of manpower provided for embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the structure of the hand motion capture model provided in the embodiments of this application;

[0030] Figure 5 A schematic block diagram of a hand capture device for a live streaming room provided in the second embodiment of this application;

[0031] Figure 6 A schematic block diagram of the structure of a computer device provided in the third embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0034] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The singular forms "a," "the," and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The word "if" as used herein can be interpreted as "when," "when," or "in response to determination."

[0035] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0036] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used in this application include not only devices that are wireless signal transmitters, which are wireless signal transmitters with only transmitting capabilities, but also devices that are wireless signal receivers, which are wireless signal receivers with only receiving capabilities, and devices that have receiving and transmitting hardware capable of bidirectional communication on a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having a single-line display, a multi-line display, or no multi-line display; PCS (Personal Communications Service), which can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0037] The hardware referred to by the names "server," "client," and "service node" in this application is essentially a computer device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0038] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0039] Please see Figure 1 , Figure 1 This is a schematic diagram of an application scenario for the live streaming room hand capture method provided in this application embodiment. The application scenario includes a broadcaster client 110, a viewer client 120, and a server 130.

[0040] The broadcaster client 110 and the viewer client 120 interact through the server 130. Specifically, both the broadcaster client 110 and the viewer client 120 can access the Internet via a network access method to establish a data communication link with the server 130. The network can be any type of communication medium capable of providing a communication link between the sharing and receiving terminals, such as a wired communication link, a wireless communication link, or a fiber optic cable, etc., and this application does not impose any limitations on this.

[0041] The broadcaster client 110 refers to the end that sends live video feeds online, typically the client used by the broadcaster in a live stream. The hardware referred to by the broadcaster client 110 essentially refers to a computer device; specifically, as shown in Figure 1, it can be a smartphone, a smart interactive whiteboard, or a personal computer.

[0042] The viewer client 120 refers to the end that receives and watches live online video, typically the client used by viewers during a live stream. The hardware referred to by the viewer client 120 essentially refers to computer equipment, specifically, such as... Figure 1 As shown, it can be a computer device such as a smartphone, smart interactive whiteboard, or personal computer.

[0043] It should be noted that the distinction between the viewer client 120 and the broadcaster client 110 is merely functional. In practical applications, the same client can execute the functions of both the viewer client 120 and the broadcaster client 110 at different times. Therefore, the same client can act as the viewer client 120 when watching a live stream and as the broadcaster client 110 when publishing a live video.

[0044] The server 130 can function as a service server, responsible for further connecting to related audio data servers, video streaming servers, and other servers providing related support, thereby forming a logically interconnected service cluster to serve related terminal devices, such as… Figure 1 The broadcaster client 110 and the viewer client 120 shown provide services.

[0045] In this embodiment, the broadcaster client 110 and the viewer client 120 can join the same live room (i.e., live channel). The aforementioned live room refers to a chat room implemented using Internet technology and the server 130, which typically has audio and video playback control functions. The broadcaster user conducts live broadcasts in the live room through the broadcaster client 110, and the viewer user of the viewer client 120 can log in to the server 130 to enter the live room to watch the live broadcast.

[0046] Within the live streaming room, the host and viewers can interact through well-known online interaction methods such as voice, video, and text. Generally, the host performs programs for the viewers in the form of audio and video streams, while the viewers can interact with the host by text or by sending virtual gifts. Economic transactions can also occur during the interaction. Of course, the application of live streaming rooms is not limited to online entertainment and can be extended to other related scenarios.

[0047] Specifically, the process of a viewer watching a live stream is as follows: A viewer can click to access the live streaming application installed on the viewer client 120 and select to enter any live stream room. This triggers the viewer client 120 to load the live stream room interface for the viewer. The live stream room interface includes several interactive components, such as video components, virtual gift bar components, and public screen components. By loading these interactive components, viewers can watch the live stream in the live stream room and engage in various online interactions. The online interaction methods include, but are not limited to, sending virtual gifts, participating in live stream activities, and chatting in the public screen.

[0048] Virtual anchors (VAS) have been widely used in online live streaming. VAS live streaming involves a real user selecting a virtual avatar, which is then synthesized in real-time by capturing the real user's appearance and movements. This virtual avatar then acts as the live streamer. VAS live streaming can respond in real-time to the user's language and actions, interacting with the audience. It realistically imitates and reproduces various complex expressions and movements of a real person to achieve the desired virtual live streaming effect.

[0049] In virtual YouTuber live streaming, virtual characters need to mimic the hand gestures of the live streamer. Since human hands contain a wealth of interactive information with viewers, hand motion capture can be further used to identify typical hand movements, triggering related special effects. Therefore, accurate and rapid capture of the streamer's hands is crucial. However, in virtual YouTuber live streaming, hand motion is typically captured using a standard RGB camera to obtain a monocular RGB image of the live streamer. Hand movements are then identified based on this monocular RGB image. Because human hand movements are subtle, varied, and prone to self-occlusion, some technologies use estimated or directly input depth information from the monocular RGB image for motion capture. However, depth information is difficult to estimate accurately and measure, increasing the complexity of hand motion capture and resulting in lower accuracy.

[0050] It is worth mentioning that, Figure 1 The application scenario described above is merely an exemplary one and is not intended to limit the scope of the present invention. The solution of the present invention can also be applied to other forms of online live streaming applications, which will not be elaborated upon here.

[0051] Please see Figure 2 This is a flowchart of the live streaming room hand capture method provided in the first embodiment of this application. The live streaming room hand capture method provided in the first embodiment of this application includes the following steps:

[0052] Step S101: Construct a human hand motion capture model, and train the constructed human hand motion capture model using human hand training samples and a human hand reconstruction model to obtain a trained human hand motion capture model.

[0053] Optionally, the hand motion capture model may include a hand feature extraction model and several connection layers; the connection layers are all connected to the hand feature extraction model, and the hand feature parameters are extracted through the hand feature extraction model, and then the hand feature parameters are input to the connection layers to obtain several hand motion parameters.

[0054] Optionally, the hand feature extraction model can be a lightweight network, such as MobileNetv3. MobileNetv3 is a lightweight attention model that combines the depthwise separable convolutions of MobileNetv1, the inverted residual with a linear bottleneck of MobileNetv2, and the SE (squeeze and excitation) based structure of MnasNet. MobileNetv3 first uses MnasNet to search for a coarse structure, and then uses reinforcement learning to select the optimal configuration from a set of discrete choices. Afterwards, MobileNetv3 uses NetAdapt to fine-tune the architecture to adjust underutilized channels with a small reduction. In one embodiment, the number of channels in MobileNetv3 can be adjusted according to actual performance requirements. Increasing the number of channels increases computational cost, thereby increasing the accuracy of hand capture; decreasing the number of channels compresses the network size, thereby reducing computational cost.

[0055] Optionally, the connection layers may include a hand type judgment connection layer, a left hand shape connection layer, a left hand pose connection layer, a left hand camera parameter connection layer, a right hand shape connection layer, a right hand pose connection layer, and a right hand camera parameter connection layer, to separately determine the types of the two hands and obtain the motion parameters of the two hands. The motion parameters may include camera parameters, shape parameters, and pose parameters, etc.

[0056] Optionally, the hand training samples include hand training sample images and corresponding real hand joint coordinates. These hand training samples can be obtained from an existing hand database or can be specifically collected from actual hands. It should be noted that the real hand joint coordinates can include real two-dimensional coordinates or real three-dimensional coordinates of the hand joints. Optionally, the hand training images can be flipped, translated, or rotated before being input into the hand motion capture model to train it, thereby expanding the hand training samples and improving the hand motion capture model's resistance to disturbances and its stability.

[0057] Optionally, the hand reconstruction model can be a parametric hand model, MANO. MANO controls the deformation of the hand relative to a standard pose based on shape and pose parameters, thereby obtaining a deformed 3D mesh image. The shape parameters control the individual-specific hand shape, while the pose parameters control the movement of the fingers and palm.

[0058] Step S102: Obtain the monocular RGB image of the live stream host user.

[0059] A monocular RGB image is an image captured using a regular camera; for example, when a broadcaster uses a mobile phone for live streaming, the monocular RGB image is an image captured using the phone's built-in camera or a camera connected to the phone.

[0060] Step S103: Input the monocular RGB image into the hand detection model to obtain a hand image.

[0061] A monocular RGB image may contain other images besides a human hand. Therefore, to improve the accuracy and efficiency of hand capture, the monocular RGB image needs to be input into a hand detection model to detect the hand region from the monocular RGB image, thereby extracting the hand image. The hand detection model can be any existing model capable of hand detection, such as a YOLOv3-based hand detection model or an RFSong-based hand detection model.

[0062] Step S104: Input the hand image into the trained hand motion capture model to obtain hand motion parameters.

[0063] Human hand motion parameters can include hand type, hand shape parameters, hand posture parameters, and hand camera parameters. For example... Figure 3 As shown, this is a 21-point diagram of a hand, including 16 joints and 5 fingertips. It also illustrates the parent-child relationships between the 21 points, starting from the root node 0, with each finger forming a subtree, and each point and its parent node forming a skeleton. The hand type includes left or right hand; the hand shape parameters and hand posture parameters are the hand shape and posture data obtained from the transformation of the 21-point hand positions.

[0064] Step S105: Input the human hand motion parameters into the human hand reconstruction model to obtain the reconstructed three-dimensional human hand mesh image.

[0065] Optionally, the hand motion parameters can be input into the parametric hand model MANO, and a reconstructed three-dimensional hand mesh image can be obtained through forward motion and linear blending skinning.

[0066] Step S106: Obtain the reconstructed two-dimensional human hand image based on the three-dimensional human hand mesh image.

[0067] Optionally, based on a preset mapping relationship between three-dimensional image coordinates and two-dimensional image coordinates, the three-dimensional human hand mesh image can be converted to obtain a reconstructed two-dimensional human hand image.

[0068] This application embodiment constructs a hand motion capture model and trains it using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model. It then acquires a monocular RGB image of a live streamer; inputs the monocular RGB image into a hand detection model to obtain a hand image; inputs the hand image into the trained hand motion capture model to obtain hand motion parameters; inputs the hand motion parameters into a hand reconstruction model to obtain a reconstructed 3D hand mesh image; and obtains a reconstructed 2D hand image based on the 3D hand mesh image. This eliminates the need for depth information input; only a monocular RGB image is required. The hand motion capture model and hand reconstruction model can quickly capture hand images. Furthermore, using the hand motion parameters output by the trained hand motion capture model as input to the hand reconstruction model improves the accuracy of hand capture.

[0069] In one embodiment, the hand training samples include hand training sample images and corresponding real hand joint coordinates; the step S101 of constructing a hand motion capture model and training the constructed hand motion capture model using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model includes:

[0070] Step S1011: Input the training sample image of the human hand into the constructed human hand motion capture model to obtain the human hand motion parameters.

[0071] Step S1012: Input the human hand motion parameters into the human hand reconstruction model to obtain a reconstructed three-dimensional human hand mesh image, wherein the three-dimensional human hand mesh image includes several predicted three-dimensional coordinates of human hand joints.

[0072] Step S1013: Based on the predicted three-dimensional coordinates of several human hand joints and the corresponding real human hand joint coordinates, obtain the loss function value of the human hand motion capture model.

[0073] Step S1014: If the loss function value is greater than a preset threshold, adjust the hand motion capture model, and retrain the adjusted hand motion capture model using the hand training sample images until the loss function value is less than the preset threshold, thereby obtaining the trained hand motion capture model.

[0074] This embodiment of the application inputs the hand training sample image into the constructed hand motion capture model to obtain hand motion parameters, and then inputs the hand motion parameters into the hand reconstruction model to obtain a reconstructed three-dimensional hand mesh image. The three-dimensional hand mesh image includes several predicted three-dimensional coordinates of hand joints. Based on the several predicted three-dimensional coordinates of hand joints and the corresponding real hand joint coordinates, the loss function value of the hand motion capture model is obtained. Compared with the method of hand reconstruction by iteratively training the hand reconstruction model, the accuracy and efficiency of hand capture can be improved.

[0075] In one embodiment, such as Figure 4 As shown, the hand motion capture model includes a hand feature extraction model, a hand type discrimination connection layer, a left hand shape connection layer, a left hand pose connection layer, a left hand camera parameter connection layer, a right hand shape connection layer, a right hand pose connection layer, and a right hand camera parameter connection layer; the hand feature extraction model is connected to the hand type discrimination connection layer, the left hand shape connection layer, the left hand pose connection layer, the left hand camera parameter connection layer, the right hand shape connection layer, the right hand pose connection layer, and the right hand camera parameter connection layer; step 1011, which involves inputting the hand training sample image into the constructed hand motion capture model to obtain hand motion parameters, includes:

[0076] Step S10111: Input the training sample image of the human hand into the human hand feature extraction model to obtain the human hand feature vector.

[0077] Step S10112: Input the hand feature vector into the hand type discrimination connection layer, and obtain the hand type based on the left and right hand occurrence scores and the preset hand type threshold.

[0078] Step S10113: If the hand type indicates that the hand training sample image includes the left hand, input the hand feature vector to the left hand shape connection layer, the left hand pose connection layer and the left hand camera parameter connection layer to obtain hand motion parameters including left hand shape parameters, left hand pose parameters and left hand camera parameters.

[0079] Step S10114: If the hand type indicates that the hand training sample image includes the right hand, input the hand feature vector to the right hand shape connection layer, the right hand pose connection layer and the right hand camera parameter connection layer to obtain hand motion parameters including right hand shape parameters, right hand pose parameters and right hand camera parameters.

[0080] It is understood that if the hand type indicates that the hand training sample image includes both the left and right hands, then the hand feature vector is input to the left hand shape connection layer, the left hand pose connection layer, the left hand camera parameter connection layer, the right hand shape connection layer, the right hand pose connection layer, and the right hand camera parameter connection layer to obtain hand motion parameters including left hand shape parameters, left hand pose parameters, left hand camera parameters, right hand shape parameters, right hand pose parameters, and right hand camera parameters.

[0081] In this embodiment, based on the predicted hand camera parameters and preset camera parameters, several predicted three-dimensional coordinates of the hand joints are converted into predicted two-dimensional coordinates of the hand joints. Then, based on the actual two-dimensional coordinates of the hand joints and the corresponding predicted two-dimensional coordinates of the hand joints, or the actual three-dimensional coordinates of the hand joints and the corresponding predicted three-dimensional coordinates of the hand joints, the loss function value of the hand motion capture model can be obtained, thereby enabling hand capture of both two-dimensional and three-dimensional monocular RGB images.

[0082] In one embodiment, the real hand joint coordinates include real two-dimensional coordinates or real three-dimensional coordinates of the real hand joints; the hand motion parameters include predicted hand camera parameters; the step S1013, which involves obtaining the loss function value of the hand motion capture model based on several predicted three-dimensional coordinates of the hand joints and the corresponding real hand joint coordinates, includes:

[0083] Step S10131: Based on the predicted human hand camera parameters and preset camera parameters, convert the predicted three-dimensional coordinates of several human hand joints into predicted two-dimensional coordinates of human hand joints.

[0084] Step S10132: Obtain the loss function value of the hand motion capture model based on the actual two-dimensional coordinates of the human hand joints and the corresponding predicted two-dimensional coordinates of the human hand joints, or the actual three-dimensional coordinates of the human hand joints and the corresponding predicted three-dimensional coordinates of the human hand joints.

[0085] It is understood that if the actual hand joint coordinates are actual two-dimensional coordinates of the actual hand joints, then the loss function value of the hand motion capture model is calculated based on the actual two-dimensional coordinates of the actual hand joints and the corresponding predicted two-dimensional coordinates of the hand joints; if the actual hand joint coordinates are actual three-dimensional coordinates of the actual hand joints, then the loss function value of the hand motion capture model is calculated based on the actual three-dimensional coordinates of the actual hand joints and the corresponding predicted three-dimensional coordinates of the hand joints.

[0086] In this embodiment, based on the predicted hand camera parameters and preset camera parameters, several predicted three-dimensional coordinates of the hand joints are converted into predicted two-dimensional coordinates of the hand joints. Then, based on the actual two-dimensional coordinates of the hand joints and the corresponding predicted two-dimensional coordinates of the hand joints, or the actual three-dimensional coordinates of the hand joints and the corresponding predicted three-dimensional coordinates of the hand joints, the loss function value of the hand motion capture model can be obtained, thereby enabling hand capture of both two-dimensional and three-dimensional monocular RGB images.

[0087] In one embodiment, due to physiological limitations, the joints of the human hand do not move very freely; the fingers can only rotate in the palm and back of the hand, and with limited range. Therefore, to restrict the degrees of freedom of the hand rotation parameters and ensure that the model outputs a normal rather than deformed hand, this invention performs principal component analysis (PCA) on the reasonable joint rotation parameters in the dataset. Instead of directly outputting 45-dimensional rotation parameters, the network outputs the coefficients of the first 30 principal components.

[0088]

[0089] in For the average human hand movement posture, θ i Let ω be the principal component of the attitude parameters. This formula indirectly yields the 45-dimensional attitude parameters. During training, ω... i By adding a regularization term and constraining its size, the degrees of freedom of the attitude parameters can be constrained.

[0090] In one embodiment, the loss function value is calculated based on the actual coordinates of the human hand joints and the corresponding predicted hand joint coordinates in the following manner:

[0091] L = w 2d *L 2d +w 3d *L 3d +w 2d-bone *L 2d-bone + w 3d-bone *L 3d-bone +w hand *L hand +w con *L con ;

[0092] in,

[0093]

[0094]

[0095]

[0096]

[0097] L hand =-[y·log(σ(x)+(1-y)·log(1-σ(x))]

[0098]

[0099] In the above formula, L 2d The loss function is the 2D coordinate loss of the hand, where w is the weight of each joint, and k is the predicted 2D coordinate of the hand. * Represents the 2D coordinates of a real human hand; L 3d Let t be the predicted 3D coordinates of the hand, and t be the loss for the hand's 3D coordinates. * Represents the 3D coordinates of a real human hand; L 2d-bone Let L be the 2D length loss of the human hand skeleton, where (i, j) represents the two end indices of the bone joints. 3d-bone 3D orientation loss of human hand skeleton; L hand The loss is used for predicting hand type; σ(x) is the occurrence fraction of both hands, σ is the sigmoid function, y is the truth value, 1 indicates occurrence, 0 indicates non-occurrence; f(I) is the input sample image after a preset transformation, model(f(I)) is the hand motion parameter value obtained after inputting the image after the preset transformation into the hand motion model, and g(model(I)) is the hand motion parameter value obtained after inputting the sample image before transformation into the hand motion model; w 2d w 3d w 2d-bone w 3d-bone w con and w hand The weights for each type of loss.

[0100] Please see Figure 5 This is a schematic diagram of the hand capture device for a live streaming room provided in the second embodiment of this application. The device 200 includes:

[0101] The hand motion capture model construction module 201 is used to construct a hand motion capture model, and to train the constructed hand motion capture model using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model.

[0102] The RGB image acquisition module 202 is used to acquire the monocular RGB image of the live streamer user.

[0103] The hand image acquisition module 203 is used to input the monocular RGB image into the hand detection model to obtain a hand image;

[0104] The human hand motion parameter acquisition module 204 is used to input the human hand image into the trained human hand motion capture model to obtain human hand motion parameters;

[0105] The mesh image acquisition module 205 is used to input the human hand motion parameters into the human hand reconstruction model to obtain the reconstructed three-dimensional human hand mesh image;

[0106] The image reconstruction module 206 is used to obtain a reconstructed two-dimensional human hand image based on the three-dimensional human hand mesh image.

[0107] It should be noted that the hand capture device for live streaming rooms provided in the second embodiment of this application is only illustrated by the above-described division of functional modules when executing the hand capture method in live streaming. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the hand capture device for live streaming rooms provided in the second embodiment of this application and the hand capture method for live streaming rooms in the first embodiment of this application belong to the same concept, and its implementation process is detailed in the method embodiment, which will not be repeated here.

[0108] The embodiment of the live-stream hand capture device in the second embodiment of this application can be applied to computer devices, such as servers. This device embodiment can be implemented by software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by a processor that processes files to read the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, the computer device it resides in may include a processor, network interface, memory, and non-volatile memory, interconnected through a data bus or other known methods.

[0109] Please see Figure 6 This is a hardware structure diagram of a computer device according to a third embodiment of this application. The computer device 300 includes a processor 310, a network interface 320, a memory 330, and a non-volatile memory 340, which are interconnected via a data bus or other known methods. (Except for...) Figure 6 In addition to the processor 310, network interface 320, memory 330, and non-volatile memory 340 shown, the actual functions of the computer device described in this application may also include other hardware, which will not be elaborated further.

[0110] The memory 330 or the non-volatile memory 340 runs a computer program, such as a method for capturing hands in a live broadcast room; when the processor 310 executes the computer program, it implements the steps of the first embodiment described above. The computer device can also serve as a carrier for the hand-capturing device in the live broadcast room according to the second embodiment.

[0111] The processor 310 may include one or more processing cores. The processor 310 connects to various parts of the computer device 300 via various interfaces and lines. It executes instructions, programs, code sets, or instruction sets stored in the memory 340 and / or the main memory 330, and calls data from the non-volatile memory 340 and / or the main memory 330 to perform various functions of the computer device 300 and process data. Optionally, the processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 310 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 310.

[0112] The non-volatile memory 340 may include random access memory (RAM) or read-only memory (ROM). Optionally, the non-volatile memory 340 may be used to store instructions, programs, code, code sets, or instruction sets. The non-volatile memory 340 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above.

[0113] The fourth embodiment of this application also provides a computer storage medium that can store multiple instructions. These instructions are applicable to being loaded by a processor and executed by the method steps of the first embodiment. For a detailed description of the execution process, please refer to the first embodiment; it will not be repeated here. The device containing the storage medium can be an electronic device such as a personal computer, laptop computer, smartphone, or tablet computer.

[0114] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer storage media does not include transient media, such as modulated data signals and carrier waves.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function selected in one or more boxes.

[0118] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0119] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for capturing hands in a live streaming room, characterized in that, Includes the following steps: A hand motion capture model is constructed, and the constructed hand motion capture model is trained using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model. The hand motion capture model includes a hand feature extraction model, a hand type discrimination connection layer, a left hand shape connection layer, a left hand pose connection layer, a left hand camera parameter connection layer, a right hand shape connection layer, a right hand pose connection layer, and a right hand camera parameter connection layer; the hand feature extraction model is connected to the hand type discrimination connection layer, the left hand shape connection layer, the left hand pose connection layer, the left hand camera parameter connection layer, the right hand shape connection layer, the right hand pose connection layer, and the right hand camera parameter connection layer. Obtain the monocular RGB image of the live streamer; The monocular RGB image is input into the hand detection model to obtain a hand image; The hand image is input into the hand feature extraction model to obtain the hand feature vector; The hand feature vector is input into the hand type discrimination connection layer, and the hand type is obtained based on the occurrence score of the left and right hands and the preset hand type threshold. If the hand type indicates that the hand image includes a left hand, the hand feature vector is input to the left hand shape connection layer, the left hand pose connection layer and the left hand camera parameter connection layer to obtain hand motion parameters including left hand shape parameters, left hand pose parameters and left hand camera parameters; If the hand type indicates that the hand image includes the right hand, the hand feature vector is input to the right hand shape connection layer, the right hand pose connection layer and the right hand camera parameter connection layer to obtain hand motion parameters including right hand shape parameters, right hand pose parameters and right hand camera parameters; The human hand motion parameters are input into the human hand reconstruction model to obtain a reconstructed three-dimensional human hand mesh image; Based on the three-dimensional human hand mesh image, a reconstructed two-dimensional human hand image is obtained; based on the reconstructed two-dimensional human hand image, the virtual anchor is controlled to perform the same hand movements as the anchor user.

2. The method for capturing hands in a live streaming room according to claim 1, characterized in that: The hand training samples include hand training sample images and corresponding real hand joint coordinates; The step of constructing a hand motion capture model and training the constructed hand motion capture model using hand training samples and a hand reconstruction model to obtain a trained hand motion capture model includes: The training sample images of the human hand are input into the constructed human hand motion capture model to obtain human hand motion parameters; The human hand motion parameters are input into the human hand reconstruction model to obtain a reconstructed three-dimensional human hand mesh image, which includes several predicted three-dimensional coordinates of human hand joints; Based on several predicted three-dimensional coordinates of human hand joints and the corresponding real human hand joint coordinates, the loss function value of the human hand motion capture model is obtained; If the loss function value is greater than a preset threshold, the hand motion capture model is adjusted, and the adjusted hand motion capture model is retrained using the hand training sample images until the loss function value is less than the preset threshold, thus obtaining the trained hand motion capture model.

3. The method for capturing hands in a live streaming room according to claim 2, characterized in that: The real hand joint coordinates include real hand joint two-dimensional coordinates or real hand joint three-dimensional coordinates; the hand motion parameters include predicted hand camera parameters; The step of obtaining the loss function value of the hand motion capture model based on several predicted three-dimensional coordinates of the hand joints and the corresponding real hand joint coordinates includes: Based on the predicted human hand camera parameters and preset camera parameters, the predicted three-dimensional coordinates of several human hand joints are converted into predicted two-dimensional coordinates of human hand joints. The loss function value of the hand motion capture model is obtained based on the actual two-dimensional coordinates of the human hand joints and the corresponding predicted two-dimensional coordinates of the human hand joints, or the actual three-dimensional coordinates of the human hand joints and the corresponding predicted three-dimensional coordinates of the human hand joints.

4. The method for capturing hands in a live streaming room according to claim 1, characterized in that: The hand reconstruction model includes a left-hand reconstruction model and a right-hand reconstruction model; The step of inputting the human hand motion parameters into the human hand reconstruction model to obtain the reconstructed three-dimensional human hand mesh image includes: If the hand type indicates that the hand image includes the left hand, the left hand camera parameters, the left hand shape parameters, and the left hand pose parameters are input into the left hand reconstruction model to obtain a three-dimensional hand mesh image including the left hand; If the hand type indicates that the hand image includes the right hand, the right hand camera parameters, the right hand shape parameters, and the right hand pose parameters are input into the right hand reconstruction model to obtain a three-dimensional hand mesh image including the right hand.

5. The method for capturing hands in a live streaming room according to any one of claims 2 or 3, characterized in that: Based on the predicted three-dimensional coordinates of several hand joints and the corresponding real hand joint coordinates, the loss function value of the hand motion capture model is obtained in the following way: ； in, In the above formula, For human hand 2D coordinate loss, For the weight corresponding to each joint, For the predicted 2D coordinates of the human hand, Represents the 2D coordinates of a real human hand; For hand-based 3D coordinate loss, For the predicted 3D coordinates of the human hand, Represents the 3D coordinates of a real human hand; 2D length loss of the human hand skeleton Represents the indexes of the two ends of a skeletal joint. 3D orientation loss of the human hand skeleton; Predicting loss based on hand type; For fractions to appear in both hands, For the sigmoid function, The value is true, 1 indicates the presence of the element, and 0 indicates its absence. The input sample image is the image after undergoing a preset transformation. The image, after undergoing a preset transformation, is input into the human hand motion model to obtain the human hand motion parameter values. The hand motion parameter values ​​are obtained by inputting the sample image before transformation into the hand motion model; , , , , and The weights for each type of loss.

6. A hand-capturing device for live streaming, characterized in that, include: The human hand motion capture model construction module is used to construct a human hand motion capture model, and to train the constructed human hand motion capture model using human hand training samples and a human hand reconstruction model to obtain a trained human hand motion capture model. The hand motion capture model includes a hand feature extraction model, a hand type discrimination connection layer, a left hand shape connection layer, a left hand pose connection layer, a left hand camera parameter connection layer, a right hand shape connection layer, a right hand pose connection layer, and a right hand camera parameter connection layer; the hand feature extraction model is connected to the hand type discrimination connection layer, the left hand shape connection layer, the left hand pose connection layer, the left hand camera parameter connection layer, the right hand shape connection layer, the right hand pose connection layer, and the right hand camera parameter connection layer. The RGB image acquisition module is used to acquire the monocular RGB image of the live streamer. The hand image acquisition module is used to input the monocular RGB image into the hand detection model to obtain a hand image; The human hand motion parameter acquisition module is used to input the human hand image into the human hand feature extraction model to obtain the human hand feature vector; input the human hand feature vector into the human hand type discrimination connection layer, and obtain the human hand type based on the left and right hand occurrence scores and a preset human hand type threshold; If the hand type indicates that the hand image includes a left hand, the hand feature vector is input to the left hand shape connection layer, the left hand pose connection layer and the left hand camera parameter connection layer to obtain hand motion parameters including left hand shape parameters, left hand pose parameters and left hand camera parameters; If the hand type indicates that the hand image includes the right hand, the hand feature vector is input to the right hand shape connection layer, the right hand pose connection layer and the right hand camera parameter connection layer to obtain hand motion parameters including right hand shape parameters, right hand pose parameters and right hand camera parameters; The mesh image acquisition module is used to input the human hand motion parameters into the human hand reconstruction model to obtain the reconstructed three-dimensional human hand mesh image; The image reconstruction module is used to obtain a reconstructed two-dimensional human hand image based on the three-dimensional human hand mesh image; and to control the virtual anchor to make the same hand movements as the anchor user based on the reconstructed two-dimensional human hand image.

7. A computer device, comprising a processor and a memory; characterized in that, The memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1 to 5 for capturing human hands in a live broadcast room.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the live streaming hand capture method as described in any one of claims 1 to 5.

Images