Facial and limb fusion capture method, virtual live streaming system, device and medium

Abstract

The application relates to a face and limb fusion capturing method, a virtual live broadcast system, equipment and a medium. A first virtual camera is used in a back end to collect and process a first image shot by a first camera to obtain a first video stream, and a second virtual camera is used to collect and process a second image shot by a second camera to obtain a second video stream. The back end extracts face key point coordinate data from the first video stream, extracts skeleton node coordinate data from the second video stream, integrates the face key point coordinate data and the skeleton node coordinate data, obtains a single message body and sends the single message body to a front end. The front end receives and analyzes the single message body to obtain face two-dimensional absolute pixel coordinates and skeleton three-dimensional absolute pixel coordinates, and performs space registration on the two-dimensional absolute pixel coordinates and the three-dimensional absolute pixel coordinates to obtain face three-dimensional absolute pixel coordinates. The front end fuses the face three-dimensional absolute pixel coordinates and the skeleton three-dimensional absolute pixel coordinates to drive a virtual character to live broadcast. The key frame loss and the skeleton-expression misplacement phenomenon are improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and human-computer interaction technology, and in particular to a method for capturing facial and limb fusion, a virtual live streaming system, equipment, and media. Background Technology

[0002] With the development of the live streaming industry, virtual live streaming technology has emerged. The core principle of virtual live streaming technology is to achieve synchronized movement of virtual avatars through sensor data collection, data-driven modeling, and real-time rendering.

[0003] The relevant technology uses a monocular camera to simultaneously capture facial and limb movements of a person. In the data-driven modeling stage, the AI ​​model is prone to data conflicts when recognizing images. For example, when a person turns their head, it is easy to cause the loss of facial key points, which in turn can cause bone-expression misalignment (average error > 5mm) when sensor data is integrated in Unity (3D development engine).

[0004] Currently, no effective solutions have been proposed for the problems of keyframe loss and skeleton-facial expression misalignment in virtual live streaming. Summary of the Invention

[0005] Therefore, it is necessary to provide a facial and limb fusion capture method, virtual live streaming system, device, and medium that can reduce keyframe loss and skeletal-expression misalignment in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for facial and limb fusion capture, applied to a virtual live streaming system, the virtual live streaming system including a backend and a frontend, the method comprising:

[0007] The backend uses a first virtual camera to capture and process a first image captured by the first camera to obtain a first video stream, and uses a second virtual camera to capture and process a second image captured by the second camera to obtain a second video stream; wherein, the first video stream contains the user's facial image, and the second video stream contains the user's body image;

[0008] The backend extracts facial key point coordinate data from the first video stream and bone node coordinate data from the second video stream. It integrates the facial key point coordinate data and the bone node coordinate data to obtain a single message body and sends the single message body to the frontend.

[0009] The front end receives and parses the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face.

[0010] The front end fuses the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live streaming of the virtual character.

[0011] In some embodiments, the backend uses a first virtual camera to acquire and process a first image captured by the first camera to obtain a first video stream, and uses a second virtual camera to acquire and process a second image captured by the second camera to obtain a second video stream, including:

[0012] The first image is cropped based on the user's facial region, and the cropped first image is then subjected to color correction, resolution matching, and distortion compensation to obtain the first video stream.

[0013] The second image is subjected to color correction, resolution matching, and distortion compensation to obtain the second video stream;

[0014] Align the first video stream and the second video stream according to their timestamps.

[0015] In some embodiments, the backend extracts facial key point coordinate data from the first video stream, extracts skeletal node coordinate data from the second video stream, integrates the facial key point coordinate data and the skeletal node coordinate data to obtain a single message body, and sends the single message body to the frontend, including:

[0016] Multiple facial key points, including eyes, nose, and mouth, are extracted from the first video stream to obtain the facial key point coordinate data, which includes two-dimensional coordinates and confidence scores.

[0017] Extract multiple bone nodes containing the shoulder, hip, and knee from the second video stream to obtain bone node coordinate data containing three-dimensional coordinates and visibility;

[0018] Define a single message body containing multiple feature points, wherein each feature point is defined as an independent sub-message containing three-dimensional coordinates and confidence level;

[0019] The facial key point coordinate data and the bone node coordinate data are integrated into the single message body through the repeated field, and the high-frequency field is set as a low-number field.

[0020] In some embodiments, the front end receives and parses the single message body, including:

[0021] The front end adds an incrementing sequence number to each received single message body and detects whether there is a key frame loss based on the sequence number; wherein, the key frame includes skeletal node data frames cut according to a preset time interval, or facial reference point change frames.

[0022] When a keyframe loss is detected, the front end triggers a retransmission request;

[0023] In response to the retransmission request, the backend will dynamically reduce the sampling accuracy of the skeletal nodes based on the bandwidth and adjust the data transmission interval within a preset time interval to meet the transmission frequency of the key frames.

[0024] In some embodiments, the front end is deployed with a normalized-pixel coordinate bidirectional conversion engine; the front end receives and parses the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face, including:

[0025] The normalized-pixel coordinate bidirectional conversion engine converts the facial key point coordinate data and the bone node coordinate data according to the image resolution of the front end, to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the bones, respectively.

[0026] Based on the calibration parameters of the first camera and the second camera, the two-dimensional absolute pixel coordinates of the face are projected into three-dimensional space to obtain the three-dimensional absolute pixel coordinates of the face; or, the Z-axis coordinate value of the two-dimensional absolute pixel coordinates of the face is supplemented based on depth estimation to obtain the three-dimensional absolute pixel coordinates of the face.

[0027] In some embodiments, the front end fuses the facial three-dimensional absolute pixel coordinates and the skeletal three-dimensional absolute pixel coordinates, including:

[0028] Assign the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to the target data object;

[0029] The target data object is serialized to generate a target sequence.

[0030] Secondly, this application provides a virtual live streaming system, including: a acquisition end, a backend, and a frontend; wherein the acquisition end, the backend, and the frontend are sequentially connected in communication.

[0031] The acquisition device includes a first camera and a second camera;

[0032] The backend creates a first virtual camera and a second virtual camera. The first virtual camera is used to capture and process a first image captured by the first camera to obtain a first video stream. The second virtual camera is used to capture and process a second image captured by the second camera to obtain a second video stream. The first video stream contains the user's facial image, and the second video stream contains the user's body image.

[0033] The backend is also used to extract facial key point coordinate data from the first video stream, extract skeletal node coordinate data from the second video stream, integrate the facial key point coordinate data and the skeletal node coordinate data to obtain a single message body, and send the single message body to the frontend.

[0034] The front end is used to receive and parse the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and to perform spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face.

[0035] The front end is also used to fuse the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live streaming of the virtual character.

[0036] In some embodiments, the first camera and the second camera are mounted at a predetermined angle;

[0037] The focal length of the first camera is within a first preset focal length range, and the focal length of the second camera is within a second preset focal length range, wherein the first preset focal length range is smaller than the second preset focal length range;

[0038] The first camera is equipped with a near-focus RGB camera with a frame rate of not less than a preset frame rate;

[0039] The second camera is equipped with a wide-angle RGB-D camera and supports SLAM spatial positioning.

[0040] Thirdly, this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0041] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0042] The aforementioned facial and limb fusion capture method, virtual live streaming system, equipment, and medium utilize two physical cameras (a first camera and a second camera) to capture images of a person's face and limbs respectively, obtaining two video streams. Two virtual cameras are then created to acquire and process these two video streams, mitigating conflicts between facial and limb data, reducing keyframe loss, and thus minimizing skeletal-expression misalignment. By connecting the virtual cameras to the physical cameras and the backend, a frame buffering mechanism is established, further reducing frame loss caused by delays in feature point extraction. Furthermore, the virtual cameras support color correction, resolution matching, and distortion compensation, improving the accuracy of subsequent feature point extraction. Therefore, this embodiment solves the problems of keyframe loss and skeletal-expression misalignment in related technologies, improving image processing accuracy. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the hardware structure of a virtual live streaming system in one embodiment;

[0044] Figure 2 This is a schematic diagram of the camera layout in one embodiment;

[0045] Figure 3 This is a schematic diagram of the hardware structure of a virtual live streaming system in another embodiment;

[0046] Figure 4 This is a flowchart illustrating a face and limb fusion capture method in one embodiment;

[0047] Figure 5 This is a flowchart of feature extraction and processing for dual video streams in one embodiment;

[0048] Figure 6 This is a flowchart of a front-end receiving and parsing a single message body in one embodiment;

[0049] Figure 7 This is a flowchart of the coordinate transformation process of a normalized-pixel coordinate bidirectional transformation engine in one embodiment;

[0050] Figure 8 This is a diagram of the internal structure of an electronic device in one embodiment;

[0051] Figure 9 This is a diagram of the internal structure of an electronic device in another embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0054] In one embodiment, a virtual live streaming system is provided. Figure 1 This is a schematic diagram of the hardware structure of the virtual live streaming system, such as... Figure 1 As shown, this virtual live streaming system includes a capture terminal, a backend, and a frontend; the capture terminal, backend, and frontend are sequentially connected for communication. The capture terminal, backend, and frontend will be described below.

[0055] The acquisition unit includes a first camera and a second camera. The first camera is used to capture the face of a person, obtaining a first image. The second camera is used to capture the person's body movements, obtaining a second image.

[0056] In some embodiments, Figure 2 A schematic diagram of a camera layout is provided. (For example...) Figure 2As shown, the first and second cameras are installed at a preset angle (e.g., 60°). The focal length of the first camera is within a first preset focal length range (e.g., 0.5~1m), and the focal length of the second camera is within a second preset focal length range (e.g., 3~5m), where the first preset focal length range is shorter than the second preset focal length range. The first camera is horizontally facing the front of the display screen and is equipped with a close-focus RGB camera with a frame rate no lower than a preset range (no lower than 120fps). Optionally, the first camera can also be equipped with a ring light. The second camera is installed at a preset downward angle (e.g., 30°), equipped with a wide-angle RGB-D camera, and supports SLAM spatial positioning. This setup facilitates coverage of the user's entire range of motion, allowing for accurate capture of facial and limb movements using ordinary cameras without the need for special equipment, thus reducing costs.

[0057] The backend can be an electronic device that internally creates a first virtual camera and a second virtual camera. The first virtual camera is used to capture and process a first image captured by the first camera to obtain a first video stream, and the second virtual camera is used to capture and process a second image captured by the second camera to obtain a second video stream. The first video stream contains the user's facial image, and the second video stream contains the user's body image.

[0058] The backend is also used to extract facial key point coordinate data from the first video stream and skeletal node coordinate data from the second video stream. The facial key point coordinate data and skeletal node coordinate data are integrated to obtain a single message body, which is then sent to the frontend.

[0059] The front end can be another electronic device that receives and parses a single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton. It then performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face. Finally, it fuses the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live streaming of the virtual character.

[0060] In this embodiment, the backend is deployed with OBS Studio video software, NVIDIA Broadcast software, and MediaPipe software. Two virtual cameras are created using OBS Studio to acquire video streams from two physical cameras. These video streams are then captured, processed, and transmitted to the frontend. Specifically, NVIDIA Broadcast extracts 468 facial key points (error < 0.5 pixels), and MediaPipe extracts skeletal key nodes. The fusion ratio of facial and motion data is automatically adjusted based on the amplitude of movement (e.g., reducing facial weight during vigorous movement). The frontend is equipped with an Nvidia 20 series or higher graphics card.

[0061] This embodiment uses two physical cameras (a first camera and a second camera) to capture images of a person's face and limbs respectively, obtaining two video feeds. Two virtual cameras are then created to acquire and process these two video feeds, resulting in two video feeds. This alleviates conflicts between facial and limb data, reduces keyframe loss, and thus minimizes skeletal-expression misalignment. By connecting the virtual cameras to the physical cameras and the backend, they act as frame buffers, further reducing frame loss caused by delays in feature point extraction. Furthermore, the virtual cameras support color correction, resolution matching, and distortion compensation, improving the accuracy of subsequent feature point extraction. Therefore, this embodiment solves the problems of keyframe loss and skeletal-expression misalignment in related technologies, improving image processing accuracy.

[0062] The virtual live streaming system in this embodiment reduces the facial recognition error rate from 8.2% in traditional solutions to 2.1%, reduces energy consumption by 30% through heterogeneous computing (CPU+GPU collaboration), supports multi-platform SDK access such as ROS / Unity3D, eliminates data interference through physical isolation of dual cameras (motion + face), achieves the accuracy of a 100,000-level motion capture system with 2000-level equipment (bone positioning error <2mm), develops a Unity Native plugin to achieve synchronous driving of motion / facial data (latency <15ms), solves the problem of spatial coordinate system consistency between ToF depth data and RGB bone data, and allows the anchor to achieve synchronous driving of full-body movements and micro-expressions through a regular camera.

[0063] In one embodiment, Figure 3 A schematic diagram of the hardware structure of another virtual live streaming system is provided, such as... Figure 3 As shown, the front end includes a main unit and a display screen, with the main unit connected to the display screen. The camera layout is as follows:

[0064] The first camera and the second camera are installed at a preset angle (e.g., 60°). The focal length of the first camera is within a first preset focal length range (e.g., 0.5~1m), and the focal length of the second camera is within a second preset focal length range (e.g., 3~5m). The first preset focal length range is smaller than the second preset focal length range.

[0065] The first camera faces horizontally directly in front of the display screen and is equipped with a close-focus RGB camera at a frame rate no lower than the preset frame rate. The second camera is mounted at a preset downward angle (e.g., 30°), equipped with a wide-angle RGB-D camera, and supports SLAM spatial positioning. This setup facilitates simultaneous high-precision macro capture and coverage of the user's entire range of motion, achieving blind-spot-free collaboration. Ordinary cameras can be used to accurately capture facial and limb movements, requiring no special equipment and resulting in low cost.

[0066] In one embodiment, a face and limb fusion capture method is provided, applied to Figures 1 to 3 The virtual live streaming system of any of the embodiments. Figure 4 This is a flowchart illustrating the face and limb fusion capture method, as shown below. Figure 4 As shown, the process includes the following steps:

[0067] In step S101, the backend uses a first virtual camera to acquire and process the first image captured by the first camera to obtain a first video stream, and uses a second virtual camera to acquire and process the second image captured by the second camera to obtain a second video stream; wherein, the first video stream contains the user's facial image, and the second video stream contains the user's body image.

[0068] The backend can be an electronic device with OBS Studio video software deployed inside. It can create two virtual cameras through OBS Studio video software, capture images from the first and second cameras respectively, process the captured images, adjust the images to the best effect, and then perform feature extraction to improve the accuracy of feature extraction.

[0069] In some embodiments, a first virtual camera crops a first image based on the user's facial region, and performs color correction, resolution matching, and distortion compensation on the cropped first image to obtain a first video stream; a second virtual camera performs color correction, resolution matching, and distortion compensation on a second image to obtain a second video stream; the first video stream and the second video stream are aligned according to timestamps, and then subsequent feature extraction processing is carried out to facilitate the alignment of extracted skeletal nodes and facial joints.

[0070] In step S102, the backend extracts facial key point coordinate data from the first video stream and bone node coordinate data from the second video stream. It integrates the facial key point coordinate data and bone node coordinate data to obtain a single message body and sends the single message body to the frontend.

[0071] The backend also deploys NVIDIA Broadcast software and MediaPipe software to extract image features.

[0072] NVIDIA Broadcast is used to capture facial data and obtain facial key points. Specifically, NVIDIA Broadcast generates 20 facial key points (such as eyes, nose, and mouth) through the Vid2Vid Cameo model. The data structure can include coordinates (x, y) and confidence scores.

[0073] MediaPipe is used to capture motion data and obtain skeletal nodes. MediaPipe coordinates are normalized to [0, 1] by default. Specifically, MediaPipe obtains 33 skeletal nodes (such as shoulder, hip, and knee joints) through the Holistic or Pose module. Each skeletal node contains normalized coordinates (x, y, z) and visibility. For example, the coordinates of the left shoulder can be extracted using the following command:

[0074] landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].

[0075] By integrating facial key point coordinate data and skeletal node coordinate data into a single message body, storage space can be reduced and data transmission efficiency can be improved.

[0076] In step S103, the front end receives and parses the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face.

[0077] Facial key point coordinates are two-dimensional coordinates, while bone node coordinates are three-dimensional coordinates. Through coordinate system unification and spatial calibration, spatial registration of the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the bones can be achieved, thereby unifying the two into a data structure that includes three-dimensional coordinates and confidence scores.

[0078] In step S104, the front end fuses the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live broadcast of the virtual character.

[0079] The three-dimensional absolute pixel coordinates of the face and skeleton are assigned to the target data object; the target data object is then serialized to generate a target sequence, which fuses the three-dimensional absolute pixel coordinates of the face and skeleton. Finally, the target sequence is used to drive the live streaming of the virtual character.

[0080] Specifically, when merging the 3D absolute pixel coordinates of the face and the 3D absolute pixel coordinates of the skeleton, the 33 skeleton nodes and 20 facial key points are filled into the PersonData message according to the protocol order, assigned to the PersonData data object, then serialized into protobuf, and finally transmitted to the application layer for driving. It should be noted that the virtual live streaming system can be functionally divided into a data acquisition layer, a data processing layer, a transmission layer, and an application layer. The data acquisition layer is used to acquire dual-channel images, the data processing layer is used to acquire and process the dual-channel images, and the processing results are sent to the transmission layer, which then transmits them to the application layer for application.

[0081] The following is an example code for assigning values ​​to the PersonData object:

[0082] person_data = PersonData() # Populate skeletal data;

[0083] for landmark in MediaPipe_results.pose_landmarks.landmark:

[0084] person_data.pose_landmarks.add(x=landmark.x,y=landmark.y,

[0085] z=landmark.z,confidence=landmark.visibility)# Fill in the facial data;

[0086] for landmark in nvidia_face_landmarks:

[0087] person_data.face_landmarks.add(x=landmark.x,y=landmark.y,

[0088] z=0.0,confidence=landmark.confidence)

[0089] person_data.timestamp_ms=current_timestamp.

[0090] The following is sample code for serializing to Protobuf:

[0091] serialized_data=person_data.SerializeToString().

[0092] In steps S101 to S104 above, two physical cameras (a first camera and a second camera) are used to capture images of the face and limbs of a person, respectively, resulting in two video streams. Two virtual cameras are then created to acquire and process these two video streams, mitigating conflicts between facial and limb data, reducing keyframe loss, and thus reducing skeletal-expression misalignment. By connecting the virtual cameras to the physical cameras and the backend, a frame buffer is established, further reducing frame loss caused by delays in feature point extraction. Furthermore, the virtual cameras support color correction, resolution matching, and distortion compensation, improving the accuracy of subsequent feature point extraction. Therefore, this embodiment solves the problems of keyframe loss and skeletal-expression misalignment in related technologies, improving image processing accuracy.

[0093] In one embodiment, Figure 5 A flowchart for feature extraction and processing of dual video streams is provided, such as... Figure 5 As shown, the backend extracts facial key point coordinate data from the first video stream and skeletal node coordinate data from the second video stream. It then integrates the facial key point coordinate data and skeletal node coordinate data to obtain a single message body, which is then sent to the frontend. This process includes the following steps:

[0094] Step S201: Extract multiple facial key points including eyes, nose and mouth from the first video stream to obtain facial key point coordinate data including two-dimensional coordinates and confidence scores.

[0095] Specifically, NVIDIA Broadcast generates 20 facial key points (such as eyes, nose, mouth, etc.) using the Vid2Vid Cameo model. The data structure can include coordinates (x, y) and confidence.

[0096] Step S202: Extract multiple bone nodes containing the shoulder, hip, and knee from the second video stream to obtain bone node coordinate data containing three-dimensional coordinates and visibility.

[0097] Specifically, MediaPipe obtains 33 skeletal nodes (such as shoulder, hip, and knee joints) through the Holistic or Pose module. Each skeletal node contains normalized coordinates (x, y, z) and visibility. For example, the coordinates of the left shoulder can be extracted using the following command:

[0098] landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].

[0099] Step S203: Define a single message body containing multiple feature points, wherein each feature point is defined as an independent sub-message containing three-dimensional coordinates and confidence level.

[0100] Based on the Potobuf protocol, 53 feature points (33 skeletal nodes + 20 facial key points) are compressed into a single message body, and each feature point is defined as an independent sub-message containing coordinates (such as x, y, z).

[0101] Step S204: Integrate the facial key point coordinate data and bone node coordinate data into a single message body through the repeated field, and set the high-frequency field as a low-number field.

[0102] The 53 feature points are integrated into a single message body using the `repeated` field, while high-frequency fields (such as coordinate values) are set as low-numbered fields (e.g., numbered 1-15). Considering that low-numbered fields only require 1 byte for encoding (including field number and field type), while high-numbered fields (e.g., field numbers 16-2047) require 2 bytes, Protobuf uses varint encoding. The tag calculation formula is (field_number << 3) | wire_type. For low-numbered fields, the entire tag can be represented by a single byte. Therefore, setting high-frequency fields as low-numbered fields, leveraging Protobuf's single-byte encoding characteristic, reduces data transmission overhead and improves encoding efficiency, achieving more than 5 times the transmission efficiency compared to JSON, and supporting real-time streaming transmission at 8K@60FPS.

[0103] Specifically, the code example for defining the .proto file (containing the composite message for the skeleton and face) is as follows:

[0104] message PersonData {

[0105] message Landmark {

[0106] float x = 1;

[0107] float y=2;

[0108] float z=3; / / If facial data does not have a z-axis, this can be set to the default value;

[0109] float confidence = 4;

[0110] }

[0111] repeated Landmark pose_landmarks=1; / / 33 skeletal nodes;

[0112] repeated Landmark face_landmarks=2; / / 20 facial landmarks;

[0113] int64 timestamp_ms=3; / / Uniform timestamp;

[0114] }

[0115] This embodiment integrates 33 skeletal node data (including 3D coordinates and visibility) from MediaPipe with 20 facial key point data (including 2D coordinates and confidence) from NVIDIA Broadcast, and defines a composite data structure through the Protobuf protocol to achieve unified encapsulation of multi-source spatiotemporal data.

[0116] In one embodiment, Figure 6 A flowchart is provided for the front-end to receive and parse a single message body, such as... Figure 6 As shown, it includes the following steps:

[0117] Step S301: The front end adds an incrementing sequence number to each received single message body and detects whether there is a key frame loss based on the sequence number; wherein, the key frame includes skeletal node data frames cut according to a preset time interval, or facial reference point change frames.

[0118] Step S302: When a keyframe loss is detected, the front end triggers a retransmission request;

[0119] In step S303, the backend will respond to the retransmission request by dynamically reducing the sampling accuracy of the skeletal nodes according to the bandwidth, and adjusting the data transmission interval within a preset time interval to meet the transmission frequency of the key frames.

[0120] In this embodiment, on the one hand, an incremental sequence number is added to each data packet at the application layer, and packet loss is detected based on the sequence number. In this embodiment, the frames corresponding to changes in the reference points of skeletal nodes and facial key points are used as keyframes. When a keyframe loss is detected, a retransmission request is immediately triggered to achieve keyframe retransmission compensation.

[0121] On the other hand, the sampling precision of the skeleton nodes is dynamically reduced based on the bandwidth (e.g., from 30 skeleton nodes to 20 skeleton nodes), and the data transmission interval is adjusted between 20ms and 50ms to prioritize the key frame frequency. When packet loss is detected, the rate is immediately reduced to 50% of the current value to achieve dynamic bitrate adjustment.

[0122] This embodiment reduces the keyframe loss rate and ensures a certain level of data integrity by using keyframe retransmission compensation and dynamic bitrate adjustment. In practical applications, this embodiment can maintain 95% data integrity even with a 20% network packet loss rate.

[0123] In one embodiment, the front end is deployed with a normalized-pixel coordinate bidirectional transformation engine. Figure 7 A flowchart of the coordinate transformation process for the normalized-pixel coordinate bidirectional transformation engine is provided, such as... Figure 7 As shown, the front end receives and parses a single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton. It then performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face. This process includes the following steps:

[0124] Step S401: The normalization-pixel coordinate bidirectional conversion engine converts the facial key point coordinate data and bone node coordinate data according to the front-end image resolution to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the bones, respectively.

[0125] (1) Formula for converting normalized coordinates to absolute pixel coordinates:

[0126] pixel_coord=normalized_coord×image_resolution;

[0127] (2) Formula for converting absolute pixel coordinates to normalized coordinates:

[0128] normalized_coord=pixel_coord÷image_resolution;

[0129] Where pixel_coord represents absolute pixel coordinates, normalized_coord represents normalized coordinates, and image_resolution represents image resolution.

[0130] In some application scenarios, a target sequence needs to synchronously drive virtual characters on multiple live streaming platforms, meaning one backend communicates with multiple frontend devices. These frontend devices may have different image resolutions, leading to incompatibility between devices. The normalized-pixel coordinate bidirectional conversion engine of this embodiment can handle mixed data processing of MediaPipe's [0,1] normalized coordinate system and NVIDIA's relative coordinate system, achieving cross-device consistency through dynamic image resolution adaptation.

[0131] Step S402: Based on the calibration parameters of the first camera and the second camera, project the two-dimensional absolute pixel coordinates of the face into three-dimensional space to obtain the three-dimensional absolute pixel coordinates of the face; or, based on depth estimation, supplement the Z-axis coordinate value of the two-dimensional absolute pixel coordinates of the face to obtain the three-dimensional absolute pixel coordinates of the face.

[0132] When fusing 3D data (such as the z-axis of MediaPipe), the 2D facial coordinates can be projected into 3D space by calibrating camera parameters, or the z-value can be supplemented using MediaPipe's depth estimation. Pseudo-3D coordinates can be generated from NVIDIA's 2D facial data using the MediaPipe depth estimation model, or stereoscopic projection reconstruction can be performed based on dual-camera calibration parameters.

[0133] The Z-axis coordinate value of supplementing the two-dimensional absolute pixel coordinates of the face based on depth estimation can be achieved through the following steps:

[0134] (1) 3DMM model matching: Match the two-dimensional absolute pixel coordinates of the face with the preset 3DMM model, and solve the camera pose parameters by using the PnP algorithm;

[0135] (2) Depth interpolation: Based on the matching results and camera parameters, depth interpolation is performed on the facial region to supplement the corresponding Z-axis depth value for each two-dimensional absolute pixel coordinate of the face.

[0136] The following is sample code for 3DMM model matching and depth interpolation:

[0137] def estimate_face_depth_3dmm(face_2d_points,face_3d_model):

[0138] """

[0139] Facial depth estimation based on 3DMM model;

[0140] """

[0141] / / 1. Solve the camera pose using the PnP algorithm;

[0142] camera_matrix=get_camera_intrinsics()

[0143] success,rotation_vector,translation_vector=cv2.solvePnP(

[0144] face_3d_model.points,face_2d_points,camera_matrix,None);

[0145] / / 2. Project the 3D model to obtain the depth value;

[0146] projected_points=cv2.projectPoints(

[0147] face_3d_model.points,rotation_vector,translation_vector,camera_matrix,None);

[0148] / / 3. Interpolate to calculate the depth of each point on the face;

[0149] depth_map=interpolate_depth(face_2d_points, projected_points)

[0150] return depth_map.

[0151] In some embodiments, the configuration parameters of the virtual live streaming system are as follows:

[0152] Camera configuration:

[0153] First camera (face camera): focal length 0.3-1.2 meters, frame rate ≥60fps, resolution ≥1080p;

[0154] Second camera (limb camera): focal length 2-6 meters, field of view ≥120°, supports depth detection;

[0155] Camera angle: 45°~90°, preferably 60°.

[0156] Data processing parameters:

[0157] Facial key points: 68-468 feature points, detection confidence ≥0.8;

[0158] Skeletal nodes: 25-33 joints, visibility threshold ≥0.5;

[0159] Time synchronization accuracy: ±5 milliseconds;

[0160] Data transmission frequency: 30-120fps, adaptively adjustable.

[0161] Performance metrics:

[0162] End-to-end latency: ≤20 milliseconds;

[0163] Feature point positioning accuracy: ±2 mm;

[0164] Keyframe loss rate: ≤1%.

[0165] In one embodiment, an electronic device is provided, which may be a backend server, and its internal structure diagram may be as follows: Figure 8As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores dual-channel video data. The network interface communicates with external terminals (e.g., live streaming equipment) via a network connection. When executed by the processor, the computer program implements a face and limb fusion capture method.

[0166] In one embodiment, another electronic device is provided, which may be a front-end live streaming device, and its internal structure diagram may be as follows: Figure 9 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a facial and limb fusion capture method. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the device's casing, or an external keyboard, touchpad, or mouse.

[0167] Those skilled in the art will understand that Figure 8 and Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0168] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0169] The backend uses a first virtual camera to capture and process the first image captured by the first camera to obtain a first video stream, and uses a second virtual camera to capture and process the second image captured by the second camera to obtain a second video stream; wherein, the first video stream contains the user's facial image, and the second video stream contains the user's body image;

[0170] The backend extracts facial key point coordinate data from the first video stream and skeletal node coordinate data from the second video stream. It integrates the facial key point coordinate data and skeletal node coordinate data to obtain a single message body, and sends the single message body to the frontend.

[0171] The front end receives and parses a single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton. It then performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face.

[0172] The front end fuses the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live streaming of the virtual character.

[0173] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0174] The first image is cropped based on the user's facial area, and the cropped first image is then subjected to color correction, resolution matching, and distortion compensation to obtain the first video stream.

[0175] The second image is subjected to color correction, resolution matching, and distortion compensation to obtain the second video stream;

[0176] Align the first and second video streams according to their timestamps.

[0177] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0178] Multiple facial key points, including eyes, nose, and mouth, are extracted from the first video stream to obtain facial key point coordinate data containing two-dimensional coordinates and confidence scores.

[0179] Extract multiple bone nodes containing the shoulder, hip, and knee from the second video stream to obtain bone node coordinate data containing three-dimensional coordinates and visibility;

[0180] Define a single message body containing multiple feature points, where each feature point is defined as an independent sub-message containing three-dimensional coordinates and confidence level;

[0181] Facial key point coordinate data and skeletal node coordinate data are integrated into a single message body using the repeated field, and high-frequency fields are set as low-number fields.

[0182] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0183] The front end adds an incrementing sequence number to each received single message body and detects whether there are any missing keyframes based on the sequence number; among them, keyframes include skeletal node data frames cut according to a preset time interval, or facial reference point change frames.

[0184] When a keyframe is detected to be missing, the front end triggers a retransmission request.

[0185] In response to retransmission requests, the backend will dynamically reduce the sampling accuracy of the skeletal nodes based on the bandwidth and adjust the data transmission interval within a preset time interval to meet the transmission frequency of key frames.

[0186] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0187] The normalized-pixel coordinate bidirectional conversion engine converts facial key point coordinate data and bone node coordinate data according to the front-end image resolution, to obtain two-dimensional absolute pixel coordinates of the face and three-dimensional absolute pixel coordinates of the bones, respectively.

[0188] Based on the calibration parameters of the first and second cameras, the two-dimensional absolute pixel coordinates of the face are projected into three-dimensional space to obtain the three-dimensional absolute pixel coordinates of the face; or, based on depth estimation, the Z-axis coordinate value of the two-dimensional absolute pixel coordinates of the face is supplemented to obtain the three-dimensional absolute pixel coordinates of the face.

[0189] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0190] Assign the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to the target data object;

[0191] The target data object is serialized to generate the target sequence.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0193] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0194] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for capturing facial and limb fusion, characterized in that, Applied to a virtual live streaming system, the virtual live streaming system including a backend and a frontend, the method includes: The backend uses a first virtual camera to capture and process a first image captured by the first camera to obtain a first video stream, and uses a second virtual camera to capture and process a second image captured by the second camera to obtain a second video stream; wherein, the first video stream contains the user's facial image, and the second video stream contains the user's body image; The backend extracts facial key point coordinate data from the first video stream and bone node coordinate data from the second video stream. It integrates the facial key point coordinate data and the bone node coordinate data to obtain a single message body and sends the single message body to the frontend. The front end receives and parses the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face. The front end fuses the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live streaming of the virtual character; The process of integrating the facial key point coordinate data and the skeletal node coordinate data to obtain a single message body includes: defining a single message body containing multiple feature points, wherein each feature point is defined as an independent sub-message containing three-dimensional coordinates and confidence level; integrating the facial key point coordinate data and the skeletal node coordinate data into the single message body through the repeated field, and setting the high-frequency field as a low-number field.

2. The face and limb fusion capture method according to claim 1, characterized in that, The backend uses a first virtual camera to acquire and process a first image captured by the first camera to obtain a first video stream, and uses a second virtual camera to acquire and process a second image captured by the second camera to obtain a second video stream, including: The first image is cropped based on the user's facial region, and the cropped first image is then subjected to color correction, resolution matching, and distortion compensation to obtain the first video stream. The second image is subjected to color correction, resolution matching, and distortion compensation to obtain the second video stream; Align the first video stream and the second video stream according to their timestamps.

3. The face and limb fusion capture method according to claim 1, characterized in that, The backend extracts facial key point coordinate data from the first video stream and skeletal node coordinate data from the second video stream, including: Multiple facial key points, including eyes, nose, and mouth, are extracted from the first video stream to obtain the facial key point coordinate data, which includes two-dimensional coordinates and confidence scores. Multiple bone nodes, including those of the shoulder, hip, and knee, are extracted from the second video stream to obtain bone node coordinate data containing three-dimensional coordinates and visibility.

4. The face and limb fusion capture method according to claim 3, characterized in that, The front-end receives and parses the single message body, including: The front end adds an incrementing sequence number to each received single message body and detects whether there is a key frame loss based on the sequence number; wherein, the key frame includes skeletal node data frames cut according to a preset time interval, or facial reference point change frames. When a keyframe loss is detected, the front end triggers a retransmission request; In response to the retransmission request, the backend will dynamically reduce the sampling accuracy of the skeletal nodes based on the bandwidth and adjust the data transmission interval within a preset time interval to meet the transmission frequency of the key frames.

5. The face and limb fusion capture method according to claim 1, characterized in that, The front end is equipped with a normalized-pixel coordinate bidirectional conversion engine; the front end receives and parses the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and performs spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face, including: The normalized-pixel coordinate bidirectional conversion engine converts the facial key point coordinate data and the bone node coordinate data according to the image resolution of the front end, to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the bones, respectively. Based on the calibration parameters of the first camera and the second camera, the two-dimensional absolute pixel coordinates of the face are projected into three-dimensional space to obtain the three-dimensional absolute pixel coordinates of the face; or, the Z-axis coordinate value of the two-dimensional absolute pixel coordinates of the face is supplemented based on depth estimation to obtain the three-dimensional absolute pixel coordinates of the face.

6. The face and limb fusion capture method according to claim 1, characterized in that, The front end fuses the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, including: Assign the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to the target data object; The target data object is serialized to generate a target sequence.

7. A virtual live streaming system, characterized in that, include: The system comprises a data acquisition terminal, a backend, and a frontend; wherein the data acquisition terminal, the backend, and the frontend are sequentially connected in communication. The acquisition device includes a first camera and a second camera; The backend creates a first virtual camera and a second virtual camera. The first virtual camera is used to capture and process a first image captured by the first camera to obtain a first video stream. The second virtual camera is used to capture and process a second image captured by the second camera to obtain a second video stream. The first video stream contains the user's facial image, and the second video stream contains the user's body image. The backend is also used to extract facial key point coordinate data from the first video stream, extract skeletal node coordinate data from the second video stream, integrate the facial key point coordinate data and the skeletal node coordinate data to obtain a single message body, and send the single message body to the frontend; The front end is used to receive and parse the single message body to obtain the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton, and to perform spatial registration on the two-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to obtain the three-dimensional absolute pixel coordinates of the face. The front end is also used to fuse the three-dimensional absolute pixel coordinates of the face and the three-dimensional absolute pixel coordinates of the skeleton to drive the live streaming of the virtual character; The process of integrating the facial key point coordinate data and the skeletal node coordinate data to obtain a single message body includes: defining a single message body containing multiple feature points, wherein each feature point is defined as an independent sub-message containing three-dimensional coordinates and confidence level; integrating the facial key point coordinate data and the skeletal node coordinate data into the single message body through the repeated field, and setting the high-frequency field as a low-number field.

8. The virtual live streaming system according to claim 7, characterized in that, The first camera and the second camera are installed at a preset angle; The focal length of the first camera is within a first preset focal length range, and the focal length of the second camera is within a second preset focal length range, wherein the first preset focal length range is smaller than the second preset focal length range; The first camera is equipped with a near-focus RGB camera with a frame rate of not less than a preset frame rate; The second camera is equipped with a wide-angle RGB-D camera and supports SLAM spatial positioning.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images