Bimodal semantic-driven three-dimensional immersive remote presentation method and system

Abstract

The invention relates to a bimodal semantic-driven three-dimensional immersive remote presentation method and system. The method comprises the following steps: collecting original video data and original audio data; carrying out human voice separation on the original audio data to obtain pure audio semantic features; extracting facial feature points of the original video data through the three-dimensional head model and calculating a head attitude Euler angle to obtain head attitude semantic features; encoding the pure audio semantic feature and the head posture semantic feature, adding a timestamp to obtain a byte stream, and synchronizing the byte stream to a receiving end through network multiplexing; and carrying out image reconstruction by a receiving end according to the byte stream through the synchronous three-dimensional head model, and carrying out frame interpolation processing and super-resolution processing on a generated image based on an online resource scheduling algorithm to obtain a three-dimensional immersive remote presentation video. According to the invention, the video remote presentation delay can be shortened, the equipment capability and the generation quality are balanced, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio - video transmission, and particularly relates to a dual - modal semantic - driven three - dimensional immersive remote presentation method and system. Background Art

[0002] In recent years, three - dimensional immersive remote presentation systems have been applied in fields such as telemedicine, immersive education, and three - dimensional video conferencing. Such systems capture stereoscopic images of users and their surrounding environments using three - dimensional media formats (such as point clouds, meshes, NeRFs, etc.) and transmit them to remote users in real - time through streaming technology, providing users with a brand - new communication experience and greatly enhancing the interactivity and realism of communication.

[0003] However, the current three - dimensional immersive remote presentation systems designed by extracting semantic features face the following problems: Since extracting semantic features requires complex calculations, a trade - off needs to be made among device capabilities, computational costs, and quality; a larger feature - processing window length will result in a longer system initialization startup delay, and noise in semantic feature data (such as background noise in audio) will significantly reduce the quality of three - dimensional image reconstruction; there is a lack of semantic features for interaction. For example, current three - dimensional reconstruction technologies driven by audio semantic features often lack head - pose features, which will lead to a decline in the ability to convey non - verbal cues and reduce the interactivity of the system; to obtain a high - fidelity and immersive three - dimensional experience, high rendering quality is necessary, but the rendering quality of current semantic - driven three - dimensional reconstruction technologies is far lower than the requirements of three - dimensional immersive remote presentation systems. Usually, enhancement technologies such as frame interpolation and super - resolution are used to enhance the generated content. However, when implementing these technologies on client devices with limited resources, their computational complexity is a challenge.

[0004] Therefore, there is an urgent need in this field for a three - dimensional immersive remote presentation method with strong transmission capabilities and high reconstruction quality. Summary of the Invention

[0005] In view of the above - mentioned deficiencies of the prior art, the purpose of the invention is to provide a dual - modal semantic - driven three - dimensional immersive remote presentation method and system.

[0006] The present invention provides a dual - modal semantic - driven three - dimensional immersive remote presentation method, including:

[0007] S1: Collecting original video data and original audio data;

[0008] S2: Separating human voices from the original audio data to obtain pure audio semantic features;

[0009] S3: Extracting facial feature points from the original video data through a three - dimensional head model and calculating the Euler angles of the head pose to obtain head - pose semantic features;

[0010] S4: Encode the pure audio semantic features and the head pose semantic features and add timestamps to obtain a byte stream, and synchronize the byte stream to the receiving end through network multiplexing;

[0011] S5: Through the synchronized three-dimensional head model, the receiving end performs image reconstruction according to the byte stream, and performs frame interpolation processing and super-resolution processing on the generated image based on the online resource scheduling algorithm to obtain a three-dimensional immersive remote presentation video.

[0012] For a dual-modal semantic-driven three-dimensional immersive remote presentation method provided by the present invention, step S2 further includes:

[0013] S21: Store the captured original audio data in a first-in-first-out queue;

[0014] S22: Calculate the mask of the human voice in the original audio data through a pre-trained U-Net network to obtain a human voice mask;

[0015] S23: Extract pure audio semantic features from the original audio data according to the human voice mask.

[0016] For a dual-modal semantic-driven three-dimensional immersive remote presentation method provided by the present invention, step S2 further includes:

[0017] When separating the human voice from the original audio data, dynamically adjust the window length of audio processing, specifically including:

[0018] Set the initial window length;

[0019] When a new audio data packet arrives, increase the window length. The expression for the increase in the window length is:

[0020] l = 2t - 1;

[0021] where l is the window length increased per second and t is the number of seconds;

[0022] When an audio data packet is lost, reduce the window length to 1 / 2 of the window length at the previous moment.

[0023] For a dual-modal semantic-driven three-dimensional immersive remote presentation method provided by the present invention, step S3 further includes:

[0024] S31: According to the original video data, extract a face bounding box through a deep learning-based face detector;

[0025] S32: Through a pre-trained deep learning model, extract feature points from the face image within the bounding box to obtain multiple face feature points;

[0026] S33: Calculate the Euler angles of the head pose through the perspective-n-point algorithm based on multiple facial feature points to obtain the head pose semantic features.

[0027] According to a bimodal semantic-driven three-dimensional immersive telepresence method provided by the present invention, step S3 further includes:

[0028] S34: Adjust the camera pose parameters for collecting the original video data according to the head pose semantic features;

[0029] Step S34 further includes:

[0030] S341: Select the frontal head pose data from the head pose semantic features;

[0031] S342: Load the frontal head pose data into the initial camera pose parameters as the baseline pose data;

[0032] S343: Convert the extracted Euler angles of the head pose to radians and normalize them;

[0033] S344: Multiply the normalized Euler angles by the up-down direction parameters of the baseline pose data to update the rotation matrix of the camera pose, so as to complete the adjustment of the camera pose parameters.

[0034] According to a bimodal semantic-driven three-dimensional immersive telepresence method provided by the present invention, step S4 further includes:

[0035] S41: Allocate independent data streams for the pure audio semantic features and the head pose semantic features respectively; wherein the sliding window size of the audio data stream is 2 times the sliding window size of the head pose data stream;

[0036] S42: Encode the pure audio semantic features and the head pose semantic features respectively to obtain feature streams;

[0037] S43: Add timestamps to the data packet headers of the feature streams to obtain byte streams;

[0038] S44: Synchronize the byte streams to the receiving end through the multiplexing mechanism of QUIC.

[0039] According to a bimodal semantic-driven three-dimensional immersive telepresence method provided by the present invention, in step S5, the online resource scheduling algorithm takes the maximization of the user experience quality as the optimization goal, and based on the maximum number of frames and the super-resolution level, realizes the optimization goal through frame interpolation processing and super-resolution processing respectively.

[0040] A dual-modal semantic-driven three-dimensional immersive telepresence method provided by the present invention, the expression of the optimization objective of the online resource scheduling algorithm is:

[0041]

[0042] where i is the frame number index value, l is the super-resolution level, L is the total number of available super-resolution levels, δ i,l is an indicator variable for selecting the super-resolution level l for frame i, F r is the set of rendered frames, F p is the set of interpolated frames, |·| is the size of the set, μ is a penalty parameter representing the non-smoothness of visual quality, q(·) represents the visual quality related to the resolution level, s.t. is the constraint condition, c r τ r is the rendering resource requirement for a single frame, c p τ p is the interpolation resource requirement for a single frame, c l τ l is the resource requirement for processing a single frame with super-resolution at super-resolution level l, T is the maximum processing time, R is the number of frames in the set of rendered frames, D is the maximum user-perceivable delay tolerable by the client, and θ is the maximum interpolation-to-rendering ratio.

[0043] In a dual-modal semantic-driven three-dimensional immersive telepresence method provided by the present invention, in step S5, the frame interpolation processing is implemented by an IFNet network, and the super-resolution processing is implemented by a residual local feature network.

[0044] The present invention also provides a dual-modal semantic-driven three-dimensional immersive telepresence system, including:

[0045] An acquisition module: used for acquiring original video data and original audio data;

[0046] A semantic feature module: used for separating human voices from the original audio data to obtain pure audio semantic features;

[0047] A head feature module: used for extracting facial feature points of the original video data and calculating the Euler angles of the head pose to obtain head pose semantic features;

[0048] A synchronization module: used for encoding the pure audio semantic features and the head pose semantic features and adding timestamps to obtain a byte stream, and synchronizing the byte stream to the receiving end through network multiplexing;

[0049] Reconstruction module: It is used to perform image reconstruction by the receiving end based on the byte stream through a synchronized three-dimensional head model, and perform frame interpolation processing and super-resolution processing on the generated image based on an online resource scheduling algorithm to obtain a three-dimensional immersive telepresence video.

[0050] A bimodal semantic-driven three-dimensional immersive telepresence method and system provided by the present invention adopt a bimodal semantic-driven mechanism, integrating audio semantic features and head pose semantic features, greatly reducing the dependence of the system on high-end devices and the network bandwidth requirements, making the three-dimensional immersive telepresence technology easier to implement on ordinary consumer devices, and significantly reducing the deployment cost; secondly, the present invention effectively removes background noise by introducing a voice separation module to process the original audio, improves the accuracy of semantic feature extraction, and thus enhances the quality of three-dimensional model rendering and the accuracy of audio-visual synchronization; thirdly, the feature processing module designed by the present invention can dynamically adjust the feature processing window length, with a small initial window length to quickly activate the model and gradually expand to the optimal length, significantly shortening the system initialization startup delay and improving the fluency of the user experience; in addition, by adding head pose semantic features, the system of the present invention can more realistically convey non-verbal cues, greatly enhancing the interactivity and immersion of remote communication.

[0051] The present invention also proposes a multiplexing transmission mechanism based on the QUIC protocol. By adjusting the sliding window size and adding timestamps in the data packet header, it effectively solves the synchronization problem between different semantic feature data streams and ensures the real-time consistency of audio and video; at the same time, the proposed online resource scheduling algorithm can intelligently schedule the number of rendering frames, interpolation frames, and super-resolution processing levels according to the device resource status, achieving the best balance between high-quality content generation and device capabilities under limited resource conditions; by adopting lightweight frame interpolation and super-resolution technologies and combining a carefully designed scheduling strategy, the present invention improves the video frame rate and resolution while maintaining a low computational complexity, enabling the system to run smoothly on resource-constrained terminal devices.

[0052] In summary, by integrating bimodal semantic feature driving, intelligent resource scheduling, and lightweight quality enhancement technologies, the present invention successfully solves the technical problems faced by traditional three-dimensional immersive telepresence systems, such as high device costs, large bandwidth requirements, long startup delays, poor interactivity, and difficulty in balancing rendering quality and resource consumption, providing users with a high-quality, low-latency, and highly interactive three-dimensional immersive telepresence experience. Description of the Drawings

[0053] The accompanying drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference signs denote the same components. Obviously, the drawings in the following description are only some of the embodiments described in the embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings.

[0054] Figure 1 It is a schematic flowchart of a dual-modal semantic-driven three-dimensional immersive remote presentation method provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic structural diagram of a dual-modal semantic-driven three-dimensional immersive remote presentation system provided by an embodiment of the present invention.

[0056] Reference signs:

[0057] 100, acquisition module; 200, semantic feature module; 300, head feature module; 400, synchronization module; 500, reconstruction module. Detailed implementation manners

[0058] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are some, rather than all, of the embodiments of the present invention. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] In addition, in the following description, the descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts disclosed in the present invention.

[0060] In the description of the present invention, it should be noted that unless otherwise clearly specified and defined, the orientation or positional relationship indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. The terms "mounted", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0062] To better understand the present invention, the nouns that appear in the embodiments of the present invention will be explained first below.

[0063] Streaming technology: A technology for real-time transmission of data (such as audio and video), where the data is transmitted and consumed while being generated, without the need for complete downloading.

[0064] NeRF (Neural Radiance Fields): A 3D reconstruction technology based on deep learning that models the light radiance field of a scene through a neural network to generate a realistic 3D model from 2D images.

[0065] QUIC (Quick UDP Internet Connections): A UDP-based transport protocol developed by Google for replacing the TCP+TLS-based HTTP / 3 protocol.

[0066] Multiplexing: A technology for simultaneously transmitting multiple independent signals on a single communication channel.

[0067] U-Net: A convolutional neural network architecture for image segmentation, with a "U" shape structure, including an encoder (downsampling) and a decoder (upsampling).

[0068] OpenCV (Open Source Computer Vision Library): An open-source computer vision library that provides a rich collection of image processing and machine learning algorithms.

[0069] Euler angles: Three angles that describe the rotation of an object in three-dimensional space, including yaw, pitch, and roll.

[0070] Deep learning: A branch of machine learning that automatically learns hierarchical feature representations of data through multi-layer neural networks.

[0071] 3DMM (3D Morphable Model): A 3D deformable model that models the variations of human face shapes and textures through statistical methods.

[0072] IFNet (Intermediate FlowNet): An optical flow estimation algorithm for video frame interpolation that generates smooth transitions by predicting the optical flow of intermediate frames.

[0073] The present invention provides a dual-modal semantic-driven three-dimensional immersive telepresence method, including:

[0074] S1: Collect original video data and original audio data.

[0075] In step S1, first, the video and audio data of the user are collected through a camera and a microphone respectively. Specifically, the video data is collected in real time through the camera, and the audio data is collected in real time through the microphone. The sampling rate is 19200 Hz, using a 16-bit integer format, mono-channel, and each time a data block of 320 bytes is read. The obtained original data will be used as the input for subsequent processing.

[0076] S2: Perform voice separation on the original audio data to obtain pure audio semantic features.

[0077] In order to solve the problem that noise in the semantic feature data (such as background noise in audio) will significantly reduce the quality of three-dimensional image reconstruction, the present invention proposes a step of obtaining pure audio semantic features through voice separation processing. The voice separation processing can filter out background noise, improve the accuracy of semantic feature extraction, and further improve the accuracy of audio-lip synchronization during three-dimensional model rendering.

[0078] Among them, step S2 further includes:

[0079] S21: Store the captured original audio data in a first-in-first-out queue.

[0080] In step S21, the present invention stores the captured original audio data in a First-In-First-Out (FIFO) queue. The FIFO data structure ensures that the audio data is processed in chronological order, with the data collected first being processed first, maintaining the timing of data processing. Additionally, the present invention uses the FIFO queue to support thread-independent design. Through the queue mechanism, the audio processing thread and the inference rendering thread can work independently without resource contention, preventing audio stuttering and audio-video out-of-sync problems.

[0081] S22: Calculate the mask of the human voice in the original audio data through a pre-trained U-Net network to obtain the human voice mask.

[0082] Furthermore, in step S22, the pre-trained U-Net network is used to analyze the original audio data, identify the human voice part and generate the corresponding mask. The mask is a binary or probability map indicating which parts of the audio data belong to the human voice and which parts belong to background noise. The U-Net network is characterized by its ability to effectively retain boundary information and accurately distinguish between the human voice and background noise.

[0083] S23: Extract pure audio semantic features from the original audio data according to the human voice mask.

[0084] After obtaining the human voice mask in step S22, in step S23, the present invention applies the human voice mask to the original audio data, only retaining the parts identified as the human voice and filtering out the background noise. Specifically, the present invention retains the clean speech track as pure audio semantic information for driving the rendering of the model. The processed pure audio semantic features will be used to drive the facial expressions, especially the lip movements, of the three-dimensional head model to achieve a more natural and accurate expression, minimizing the interference of background noise and making the audio and lip synchronization more accurate during the model inference rendering process, thereby producing a higher-quality effect.

[0085] Among them, step S2 also includes:

[0086] When separating the human voice from the original audio data, dynamically adjust the window length of audio processing, specifically including: setting an initial window length; when a new audio data packet arrives, increase the window length. The expression for the increase in the window length is: l = 2t - 1; where l is the increase in the window length per second and t is the number of seconds; when an audio data packet is lost, reduce the window length to 1 / 2 of the window length at the previous moment.

[0087] Furthermore, the present invention also integrates an audio processing unit to dynamically adjust the processing window length and maintains a real-time audio data queue for real-time inference rendering. Initially, the window length is set to 4 (unit: 20 milliseconds) to quickly activate the model. As new audio data arrives, the window length increases by 2t - 1 every t seconds to accommodate new data and discard old data, thereby optimizing resource usage. The maximum window length is extended to 15 to ensure sufficient audio input while minimizing waste. In the case of packet loss, the window is temporarily halved (minimum of 4) and then gradually restored to 15. This adaptive method can accelerate the model inference speed, reduce latency, and ensure continuous audio data flow, thus achieving smooth model rendering.

[0088] S3: Extract facial feature points from the original video data through a three-dimensional head model and calculate the Euler angles of the head pose to obtain the head pose semantic features.

[0089] The extraction of facial feature points from the original video data through the three-dimensional head model and the calculation of the Euler angles of the head pose in step S3 aim to solve the problem of the lack of semantic features for interaction. For example, current three-dimensional reconstruction techniques driven by audio semantic features often lack head pose features, which can lead to a decline in the ability to convey non-verbal cues and reduce the interactivity of the system. By extracting head pose information, the system can increase interactivity and more realistically convey non-verbal cues.

[0090] Among them, step S3 further includes:

[0091] S31: According to the original video data, extract the facial bounding box through a deep learning-based face detector.

[0092] Specifically, the present invention uses OpenCV to call the SCRFD model face detector based on InsightFace, which can extract the bounding box around the detected face. The facial bounding box refers to the rectangular box that encloses the face area, defining the position and size of the face in the image. The extracted bounding box can support accurate face localization under various lighting conditions, angles, and occlusions, providing an accurate region range for subsequent feature point extraction.

[0093] S32: Extract feature points from the facial image within the bounding box through a pre-trained deep learning model to obtain multiple facial feature points.

[0094] After the extraction frame is established in step S31, a pre-trained deep learning model is loaded in step S32. This model inputs each frame of the facial image and outputs 68 facial feature points. These facial feature points refer to the key positions on the human face, such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc. They define the structure and expression of the human face. 68 feature points are the standard configuration in facial feature analysis and can comprehensively describe the geometric structure of the human face.

[0095] S33: According to multiple facial feature points, calculate the Euler angles of the head pose through the Perspective-n-Point (PnP) algorithm to obtain the head pose semantic features.

[0096] Furthermore, the Perspective-n-Point (PnP) algorithm is a method for estimating the camera pose by using known 3D points and their projections in 2D images. In step S33, the correspondence between the pre-defined 3D face model and the extracted 2D facial feature points is used to calculate the pose of the head relative to the camera. The Euler angles are three angles that describe the rotation of an object in three-dimensional space, including the yaw angle, the pitch angle, and the roll angle. Based on the Euler angles, the orientation of the head in space can be defined.

[0097] Among them, step S3 also includes:

[0098] S34: According to the head pose semantic features, adjust the camera pose parameters for collecting the original video data.

[0099] Step S34 further includes:

[0100] S341: Select the frontal head pose data from the head pose semantic features; S342: Load the frontal head pose data into the initial camera pose parameters as the baseline pose data; S343: Convert the extracted head pose Euler angles to radians and normalize them; S344: Multiply the normalized Euler angles by the up and down direction parameters of the baseline pose data to update the rotation matrix of the camera pose, so as to complete the adjustment of the camera pose parameters.

[0101] Specifically, in step S34, first load a set of selected frontal head pose data into the basic pose parameters to eliminate unnatural head jitters. These data will be used as the baseline pose data to initialize the model and effectively start the inference process. The purpose is to establish a stable reference pose, avoid head model jitters caused by detection noise, and provide a smoother and more natural visual experience.

[0102] The Euler angles of the head pose to be extracted later are transmitted to the receiving end as semantic feature information. The receiving end converts the Euler angles to radians and normalizes them, multiplies them by the up and down direction parameters of the baseline pose data, and updates the 3×3 rotation matrix of the camera pose, thereby changing the camera's pose and realizing the interactive control of the head pose. The reason for converting the Euler angles to radians is that most mathematical calculations use radians rather than degrees. Normalization is to limit the values within a standard range for subsequent processing. The rotation matrix is a mathematical tool for describing rotational transformations in three-dimensional space. The 3×3 rotation matrix defines the orientation of the camera. By updating this matrix, the system can precisely control the display direction of the three-dimensional head model on the screen to make it consistent with the user's actual head movements, enhancing the interactivity and immersion.

[0103] S4: Encode the pure audio semantic feature and the head pose semantic feature and add timestamps to obtain a byte stream, and synchronize the byte stream to the receiving end through network multiplexing.

[0104] Step S4 aims to solve the problem that the sending end transmits audio semantic information and head pose semantic information to the receiving end simultaneously, and the differences in data volume and priority between the two may lead to synchronization problems. The present invention ensures that two different types of semantic features can be correctly recombined at the receiving end through encoding and synchronous transmission, realizing a seamless three-dimensional remote presentation experience.

[0105] Among them, step S4 further includes:

[0106] S41: Allocate independent data streams for the pure audio semantic feature and the head pose semantic feature respectively; where the sliding window size of the audio data stream is 2 times the sliding window size of the head pose data stream.

[0107] In step S41 of the present invention, by fine-tuning the sliding window size and allocating half of the window size of the audio feature stream to the head pose data stream, it ensures the priority transmission of audio data, thereby effectively synchronizing these data streams. The sliding window is a mechanism in the network transmission protocol to control data flow, which determines the amount of data that can be sent before receiving an acknowledgment. By allocating different sizes of sliding windows for different types of data streams, the system can control their transmission priorities and rates.

[0108] S42: Encode the pure audio semantic feature and the head pose semantic feature respectively to obtain feature streams.

[0109] Further, the encoding is to convert the extracted semantic features into a data format suitable for network transmission. The encoding process includes data compression, format conversion, etc. The purpose is to reduce the amount of transmitted data and ensure transmission reliability. Specifically, in this embodiment, the pose data is packed by struct.pack, and for the audio semantic features, the processed audio data is serialized.

[0110] S43: Add a timestamp to the data header of the feature stream to obtain a byte stream.

[0111] In step S43, a timestamp also needs to be added to the data header to track the time of the two data streams during transmission. The timestamp is a mark that records the generation or sending time of the data. The purpose of adding the timestamp is to correctly restore the time relationship between different data streams at the receiving end and ensure the synchronization of audio and video.

[0112] S44: Synchronize the byte stream to the receiving end through the multiplexing mechanism of QUIC.

[0113] Further, QUIC (Quick UDP Internet Connections) is a UDP-based transport protocol, and multiplexing is a technology for simultaneously transmitting multiple independent signals on a single communication channel. In step S44 of the present invention, through the multiplexing mechanism of QUIC, multiple data streams are allowed to share the same connection, reducing the overhead of establishing connections. Moreover, each data stream can independently send and receive data, and will not be blocked by other data streams due to the loss of a certain data stream, thereby realizing real-time communication. By using the QUIC protocol, the system can efficiently transmit audio and head pose data simultaneously and maintain the time synchronization relationship between them.

[0114] In addition, the timestamp in step S43 also serves as a synchronization point at the receiving end, guiding the integration of audio and head pose data into the inference engine to generate video frames and synchronize the audio and video playback. At the receiving end, the timestamp is not only used to ensure data synchronization during transmission, but also for the inference and rendering processes at the receiving end to ensure that the final 3D video presented to the user has accurate audio and video synchronization.

[0115] S5: Through the synchronized 3D head model, the receiving end performs image reconstruction based on the byte stream, and performs frame interpolation processing and super-resolution processing on the generated image based on the online resource scheduling algorithm to obtain a 3D immersive remote presentation video.

[0116] In step S5, at the receiving end, the 3D head model is received and stored in the offline phase. In the online phase, first, semantic features are retrieved and then used to render and drive the NeRF-based 3D head model. Subsequently, under the dynamic scheduling of the online resource scheduling algorithm, the generated images selectively enter the quality enhancement module for frame interpolation and super-resolution processing, and finally, high-quality videos are output.

[0117] Among them, in step S5, the frame interpolation process is implemented through the IFNet network, and the super-resolution process is implemented through the residual local feature network.

[0118] To improve efficiency and real-time performance, the present invention selects lightweight and responsive frame interpolation and super-resolution algorithms for quality enhancement implementation. For frame interpolation, the present invention adopts a simplified coarse-to-fine network, IFNet, which can effectively estimate the intermediate optical flow. The super-resolution process uses a residual local feature network that learns from residual local features through three convolutional layers. This network simplifies feature aggregation, thus achieving an optimal balance between model performance and inference time.

[0119] Among them, in step S5, the online resource scheduling algorithm aims to maximize the quality of user experience and, based on the maximum number of frames and the super-resolution level, realizes the optimization goal through frame interpolation processing and super-resolution processing respectively.

[0120] Furthermore, to clearly illustrate the resource allocation problem and its impact on the overall performance of the 3D telepresence system, the present invention presents a comprehensive system model to capture all aspects involved in rendering and enhancing 3D visual output.

[0121] Specifically, a frame can be super-resolved at different levels. For each frame i, an indicator variable δ i,l can be used to indicate whether it selects the super-resolution at level l, and the expression is:

[0122]

[0123] Among them, yes means that frame i selects the super-resolution level at level l. At this time, the indicator variable δ i,l outputs 1, otherwise 0.

[0124] Let L be the total number of available super-resolution levels, and (c l , τ l ) be the resources required for single-frame super-resolution processing at level l. Therefore, the super-resolution level of frame i is The super-resolution resource consumption of frame i is Among them

[0125] Let \(q(\cdot)\) denote the visual quality associated with the resolution level. Thus, the visual quality of frame \(i\) is In addition, the present invention also introduces a parameter \(\mu\) to represent the penalty for non - smooth visual quality, which is modeled as Therefore, the quality of experience of the user can be defined as:

[0126]

[0127] For each frame, it can only be one of the rendered frame or the interpolated frame. So let \(F\) r be the set of rendered frames, and \(F\) p be the set of interpolated frames. The total number of frames is \(|F\) r | + |F\) p |, where \(|\cdot|\) represents the size of the set.

[0128] Since the rendered frames are synchronized with the audio, to maintain the audio - video consistency, the present invention introduces a parameter \(0\lt\theta\lt1\) to represent the maximum interpolation - to - rendering ratio. In addition, the present invention further defines \((c\) r ,\tau\) r ) and \((c\) p ,\tau\) p ) as the rendering and interpolation resource requirements for a single frame. Thus, the resource requirements for rendering and interpolating a single frame are \(|F\) r |c\) r \tau\) r and \(|F\) p |c\) p \tau\) p .

[0129] The present invention uses \(R\) to represent the available 3D immersive conferencing resources at time \(t\), and \(T\) to represent the total number of time slots. Therefore, due to resource constraints, the expression can be obtained:

[0130]

[0131] So, to ensure the smoothness of video playback and maintain the synchronization between video and audio, it is crucial to ensure a constant frame rate.

[0132] Since the rendering of video frames has been postponed by time for maintaining a constant frame rate at the client, based on the delay constraint, the expression of the delay constraint is:

[0133]

[0134] where \(D\) is the maximum user - perceived delay tolerable by the client. Finally, the following client reconstruction quality enhancement problem can be obtained.

[0135] Among them, the optimization objective of the online resource scheduling algorithm, that is, the expression of the client reconstruction quality enhancement problem is:

[0136]

[0137] where \(i\) is the frame number index value, \(l\) is the super-resolution level, \(L\) is the total number of available super-resolution levels, and \(\delta\) i,l is an indicator variable for selecting the super-resolution level \(l\) for frame \(i\), \(F\) r is the set of rendered frames, \(F\) p is the set of interpolated frames, \(|\cdot|\) is the size of the set, \(\mu\) is a penalty parameter representing the non-smoothness of visual quality, \(q(\cdot)\) represents the visual quality associated with the resolution level, s.t. is the constraint condition, and \(c\) r \(\tau\) r is the rendering resource requirement for a single frame, \(c\) p \(\tau\) p is the interpolation resource requirement for a single frame, \(c\) l \(\tau\) l is the resource requirement for a single frame of super-resolution processing at the super-resolution level \(l\), \(T\) is the maximum processing time, \(R\) is the number of frames in the set of rendered frames, \(D\) is the maximum user-perceivable latency tolerable by the client, and \(\theta\) is the maximum interpolation-to-rendering ratio.

[0138] For the problem of enhancing the client reconstruction quality obtained, under limited resources, the user is more sensitive to the frame rate than to the resolution. Since the reduction in resolution has a smaller impact on visual quality than the reduction in frame rate, therefore, when designing the process scheduling strategy, the present invention gives priority to high frame rates and reserves resources for frame rendering and interpolation; in addition, to ensure a consistent viewing experience under resource constraints, the rendering, interpolation, and super-resolution processes must be executed in a periodic manner, which not only optimizes the overall user experience quality but also simplifies the resource allocation strategy.

[0139] Based on this, the present invention provides solutions including determining the maximum number of frames and determining the super-resolution level.

[0140] Specifically, for determining the maximum number of frames, the present invention divides the tolerable interpolation-to-rendering ratio \(\theta\) into \(K\) equal parts and traverses the ratio from 1 to \(K\). For a given ratio \(k\), the number of rendered frames and interpolated frames is subject to the constraint of the processing time \(|F\) r ,k|\tau\) r +|F\) p ,k|\tau\) l \(\leq T\), and the relationship is \(k|F\) r ,k| = |F\) p ,k|.

[0141] Let \(|F\) * r,k | and \(|F\) * p,k | be the maximum number of rendered frames and interpolated frames at the ratio \(k\). Therefore, and Furthermore, the present invention further defines the minimum number of rendered frames and the number of interpolated frames as and where e is the rendering and interpolation resource reservation ratio considering Insight 1.

[0142] For determining the super-resolution level, the present invention focuses on a set of frames consisting of [1 / p] rendered frames and 1 interpolated frame, determines the appropriate super-resolution level for this set, and the super-resolution level determined for this set of frames can be extended to other frames. Further, the present invention divides the ranges of the number of rendered frames and interpolated frames [(|F r ,k ° |,|F p ,k ° |),(|F r ,k * |,|F p ,k * |)] into W equal parts and selects the super-resolution level that optimizes the objective function for each frame.

[0143] As Figure 2 shown, the present invention also provides a dual-modal semantic-driven three-dimensional immersive telepresence system, including:

[0144] Acquisition module 100: for acquiring original video data and original audio data;

[0145] Semantic feature module 200: for separating the human voice from the original audio data to obtain pure audio semantic features;

[0146] Head feature module 300: for extracting facial feature points of the original video data and calculating the Euler angles of the head pose to obtain head pose semantic features;

[0147] Synchronization module 400: for encoding the pure audio semantic features and the head pose semantic features and adding timestamps to obtain a byte stream, and synchronizing the byte stream to the receiving end through network multiplexing;

[0148] Reconstruction module 500: for image reconstruction by the receiving end according to the byte stream through a synchronized three-dimensional head model, and performing frame interpolation processing and super-resolution processing on the generated image based on an online resource scheduling algorithm to obtain a three-dimensional immersive telepresence video.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0150] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0151] Furthermore, a dual-modal semantic-driven three-dimensional immersive remote presentation method and system of the present invention will be described below in combination with specific embodiments.

[0152] At the sending end, first, an audio transmission thread. Specifically, first, loop read audio data blocks audio_data from the device through the Pyaudio library function stream.read, input the audio data blocks into the voice separation module, calculate the audio spectrum and amplitude spectrum, and through a pre-trained U-net model, calculate the mask of the human voice in the audio, separate the spectrum of the human voice in the original audio, obtain the purified human voice audio pure_data, and serialize the data. Then, through client.post, use QUIC to send a POST request. After receiving a successful server connection reply, add a time stamp Time_stamp to the data packet header header, and use the multiplexing feature of QUIC to send data to the server using an independent data stream.

[0153] Next, establish a video processing and pose transmission thread. Specifically, first use cap.read() to read video frames from a monocular camera, call the face detector, use face_detector.detect to extract the face bounding box for face detection. Then, load the pre-trained ONNX type model, and use pose_estimator.solve to extract 68 feature points of the face. After obtaining the feature points, use cv2.decomposeProjectionMatrix to calculate the Euler angles u_angle of the face pose, and then use struct.pack to pack the pose data. After that, use client.post to send a POST request using QUIC. After receiving the server connection success response, add the time stamp Time_stamp to the data packet header. Utilize the multiplexing feature of QUIC to send data to the server using an independent data stream. At the same time, to improve the priority of the audio data stream, adjust the data transmission window size of the head pose data stream to half of the audio data stream window size.

[0154] Specifically, the key functions of the sender are shown in Table 1, and the experimental parameters and conditions of the sender are shown in Table 2.

[0155] Table 1 Key Functions of the Sender

[0156]

[0157]

[0158] Table 2 Experimental Parameters and Conditions of the Sender

[0159]

[0160] On the receiving end, first perform initialization as follows.

[0161] Load configuration: Read the configuration from the command line parameters and initialize the NeRF model, QUIC server, and GUI.

[0162] Initialize the NeRF model: Load the pre-trained model and set the training parameters.

[0163] Initialize the QUIC server: Configure the QUIC protocol and start the HTTP / 3 service.

[0164] Initialize the GUI: Set the camera, background, audio stream, etc., and prepare for rendering.

[0165] Initialize the enhancement model: Load the interpolation model and the super-resolution model

[0166] Next, start the QUIC server as follows.

[0167] Listening Port: The QUIC server starts listening on the specified port, waiting for client connections.

[0168] Request Processing: When a client connects, it processes HTTP / 3 requests and receives audio and head pose data.

[0169] Data Transmission: The received audio data is put into a queue for the ASR module to process; the head pose data is used to update the camera view.

[0170] Next, start the GUI rendering, as follows.

[0171] Main Rendering Loop: Continuously render the output of the NeRF model in the GUI and handle user inputs (such as adjusting the view and playing audio).

[0172] Audio Processing: Retrieve audio data from the queue, extract features through the ASR model, and generate text.

[0173] Video Processing: Update the camera view according to the head pose data and render new frames.

[0174] Enhancement Processing: According to the scheduling instructions, dynamically perform frame interpolation or super-resolution operations on the rendered video frames to improve the video frame quality.

[0175] Finally, perform closing and cleaning, as follows.

[0176] Stop the QUIC Server: When the user exits, close the QUIC server and release resources.

[0177] Close the GUI: Destroy the GUI context and stop the rendering loop.

[0178] Specifically, the key functions of the receiver are shown in Table 3, and the experimental parameters and conditions of the receiver are shown in Table 4.

[0179] Table 3 Key Functions of the Receiver

[0180]

[0181]

[0182] Table 4 Experimental Parameters and Conditions of the Receiver

[0183]

[0184]

[0185] A dual-modal semantic-driven three-dimensional immersive remote presentation method and system provided by the present invention propose a solution for dual-modal semantic-driven three-dimensional immersive remote presentation, solve the problems of high cost, large amount of transmitted data, and high network bandwidth requirement in traditional three-dimensional immersive remote presentation systems, and can reduce equipment costs and bandwidth requirements, and promote the research and development of three-dimensional immersive remote presentation systems. Secondly, the present invention designs a feature processing module to dynamically adjust the window length of feature processing, solves the problem that a relatively large window length of feature processing will lead to a long start-up delay of the system, and can shorten the start-up delay. In addition, a lightweight enhancement technology is used to design an effective scheduling algorithm, which can solve the balance problem between enhancement quality and device capabilities when applying enhancement technology to client devices with limited resources, not only improves the quality of the generated content, but also balances device capabilities and generated quality, and gives users a high-quality content experience.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A dual-modal semantically driven three-dimensional immersive telepresence method, characterized in that: include: S1: collect original video data and original audio data; S2: Separate the original audio data into human voices to obtain pure audio semantic features; S3: extracting facial feature points of the original video data through a three-dimensional head model and calculating the head posture Euler angle to obtain the head posture semantic features; S4: Encode the pure audio semantic feature and the head posture semantic feature and add a timestamp to obtain a byte stream, and synchronize the byte stream to a receiving end through network multiplexing; S5: The receiving end reconstructs an image according to the byte stream through the synchronized three-dimensional head model, and performs frame interpolation processing and super-resolution processing on the generated image based on an online resource scheduling algorithm to obtain a three-dimensional immersive remote presentation video.

2. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 1, characterized in that: Step S2 further comprises: S21: storing the captured original audio data into a first-in-first-out queue; S22: Calculate the voice mask in the original audio data through the pre-trained U-Net network to obtain the voice mask; S23: Extracting pure audio semantic features from the original audio data according to the vocal mask.

3. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 2, characterized in that: Step S2 also includes: When performing voice separation on the original audio data, dynamically adjusting the window length of the audio processing specifically includes: Set the initial window length; When a new audio data packet arrives, the window length is increased. The expression for the increase in window length is: l＝2t-1； Where l is the window length that increases per second, and t is the number of seconds; When an audio data packet is lost, the window length is reduced to 1 / 2 of the window length at the previous moment.

4. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 1, characterized in that: Step S3 further comprises: S31: Extracting facial bounding boxes through a deep learning-based face detector based on the original video data; S32: extracting feature points from the facial image within the bounding box through a pre-trained deep learning model to obtain a plurality of facial feature points; S33: According to the multiple facial feature points, the Euler angles of the head posture are calculated by using the mutual perspective n-point algorithm to obtain the semantic features of the head posture.

5. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 1, characterized in that: Step S3 also includes: S34: adjusting the camera posture parameters used to collect the original video data according to the head posture semantic features; Step S34 further comprises: S341: Selecting frontal head posture data based on the head posture semantic features; S342: Loading the front head posture data into the initial camera posture parameters as baseline posture data; S343: converting the extracted head posture Euler angle into radians and normalizing it; S344: updating the rotation matrix of the camera attitude according to the multiplication of the normalized Euler angle and the up-down direction parameter of the baseline attitude data, so as to complete the adjustment of the camera attitude parameters.

6. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 1, characterized in that: Step S4 further comprises: S41: Allocate independent data streams to the pure audio semantic features and the head posture semantic features respectively; wherein the sliding window size of the audio data stream is twice the sliding window size of the head posture data stream; S42: Encode the pure audio semantic feature and the head posture semantic feature respectively to obtain a feature stream; S43: adding a timestamp to the data packet header of the feature stream to obtain a byte stream; S44: Synchronize the byte stream to the receiving end through the multiplexing mechanism of QUIC.

7. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 1, characterized in that: In step S5, the online resource scheduling algorithm takes maximizing the user experience quality as the optimization goal, and based on the maximum number of frames and the super-resolution level, achieves the optimization goal through frame interpolation processing and super-resolution processing respectively.

8. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 7, characterized in that: The expression of the optimization objective of the online resource scheduling algorithm is: Among them, i is the frame index value, l is the super-resolution level, L is the total number of available super-resolution levels, δ i,l is the indicator variable for selecting super-resolution level l for frame i, F r is the set of rendered frames, F p is the interpolated frame set, |·| is the size of the set, μ is the penalty parameter indicating the non-smooth visual quality, q(·) represents the visual quality related to the resolution level, st is the constraint condition, c r τ r is the rendering resource requirement for a single frame, c p τ p is the interpolation resource requirement for a single frame, c l τ l is the resource requirement for super-resolution processing of a single frame at super-resolution level l, T is the maximum processing time, R is the number of frames in the rendered frame set, D is the maximum user-perceived delay that the client can tolerate, and θ is the maximum interpolation-to-rendering ratio.

9. A dual-modal semantically driven three-dimensional immersive telepresence method according to claim 7, characterized in that: In step S5, the frame interpolation processing is implemented through the IFNet network, and the super-resolution processing is implemented through the residual local feature network.

10. A dual-modal semantically driven three-dimensional immersive telepresence system, characterized in that: include: Acquisition module: used to collect original video data and original audio data; Semantic feature module: used to separate the human voice from the original audio data and obtain the pure audio semantic features; Head feature module: used to extract facial feature points of the original video data and calculate the Euler angles of the head posture to obtain semantic features of the head posture; Synchronization module: used for encoding the pure audio semantic features and the head posture semantic features and adding timestamps to obtain a byte stream, and synchronizing the byte stream to a receiving end through network multiplexing; Reconstruction module: used to reconstruct the image according to the byte stream by the receiving end through the synchronized three-dimensional head model, and perform frame interpolation and super-resolution processing on the generated image based on the online resource scheduling algorithm to obtain a three-dimensional immersive remote presentation video.

Images