Three-dimensional digital human streaming media quality evaluation method and system suitable for 6g communication

Abstract

The application discloses a three-dimensional digital human streaming media quality evaluation method and system suitable for 6G communication, and the method comprises the following steps: frame-by-frame rendering is performed on the three-dimensional digital human streaming media transmitted through a 6G simulation channel; video synthesis is performed on all the rendered frames, and an audio track is added, a digital human video with audio is synthesized at a user end, and the digital human video with audio is used to represent the three-dimensional digital human streaming media; quality of service features of the three-dimensional digital human streaming media are extracted; content quality features of the digital human video are extracted; regression is performed on the quality of service features and the content quality features, and finally, a quality score of the digital human video, i.e., a three-dimensional digital human streaming media quality score, is obtained. The application can be effectively used for three-dimensional digital human streaming media quality evaluation under 6G communication conditions.

Description

Technical Field

[0001] This invention relates to the field of streaming video processing technology, and more specifically, to a method, system, medium, and terminal for evaluating the quality of three-dimensional digital human streaming media suitable for 6G communication. Background Technology

[0002] Currently, the demand for immersive media such as virtual reality and augmented reality is growing, and 3D media is gradually gaining attention. 3D digital human streaming media, with its features of human appearance, movement, and personality, has become a major component of 3D media. However, compared to traditional video streaming media that only needs to transmit a complete video, 3D digital human streaming media requires the transmission of a large number of 3D digital human model sequences, corresponding texture sequences, and audio media. This leads to a dramatic increase in the amount of data transmitted, placing extremely high demands on communication systems. If the communication system cannot support the transmission of massive amounts of data, the 3D digital human streaming media received by the user will suffer from distortion, blurring, buffering, and stuttering, reducing the user's audiovisual experience.

[0003] Fortunately, with the increasingly widespread application of 5G communication, the next-generation 6G communication technology has also been put on the research and development agenda. 6G communication technology is expected to achieve further breakthroughs in transmission speed, capacity, and latency, providing stronger support for future media data transmission. The development of 6G will alleviate or even solve the problem of traditional network bandwidth and speed being insufficient to meet the requirements of large-scale 3D data transmission, making 3D digital human streaming media possible. Therefore, it is necessary to conduct an effectiveness evaluation of 3D digital human streaming media in the context of 6G communication, which is of great significance for the design of 3D digital human streaming media communication systems and 3D streaming media audiences. Summary of the Invention

[0004] To address the gaps in existing technologies, the purpose of this invention is to provide a method, system, medium, and terminal for evaluating the quality of 3D digital human streaming media suitable for 6G communication. This method can easily and effectively evaluate the quality of 3D digital human streaming media transmitted under 6G communication conditions, which is of great significance to the development of fields such as digital human design and production, digital human communication system design, and 3D display technology.

[0005] According to one aspect of the present invention, a method for evaluating the quality of three-dimensional digital human streaming media suitable for 6G communication is provided, comprising:

[0006] Frame-by-frame rendering of 3D digital human streaming media transmitted via a 6G simulated channel;

[0007] Video compositing and audio tracks are added to all rendered frames to synthesize digital human videos with audio on the user end, thus representing 3D digital human streaming media.

[0008] Extract the quality of service characteristics of the three-dimensional digital human streaming media;

[0009] Extract the content quality features of the digital human video;

[0010] Regression is performed on the service quality features and the content quality features to obtain the quality score of the digital human video, i.e., the quality score of the 3D digital human streaming media.

[0011] Preferably, the 3D digital human streaming media includes a 3D digital human model sequence, a 3D digital human texture sequence, and an audio file.

[0012] Preferably, the step of rendering the 3D digital human streaming media transmitted via a 6G simulated channel frame by frame includes:

[0013] f t =Open3D(M t ,I t )

[0014] t = 1, 2, 3, ..., n

[0015] Wherein, Open3D(·) indicates that Open3D is used to render the t-th 3D digital human in the 3D digital human streaming media as frame f. t The process is as follows: n represents the total number of digital human models contained in the 3D digital human streaming media; each rendering operation requires two items, namely the 3D digital human model M and the corresponding 3D digital human texture I.

[0016] Preferably, the step of performing video compositing and adding audio tracks to all rendered frames, synthesizing a digital human video with audio on the user end, and using this to represent the 3D digital human streaming media, includes:

[0017] V = f1 + f2 + f3 + ... + f n

[0018]

[0019] Here, + indicates the process of stitching together n rendered frames using FFmpeg to create a silent video V. This indicates the process of using FFmpeg for audio and video mixing. V s A and A represent the digital human video with audio and the original audio file of the digital human, respectively;

[0020] V s It retains the main information of the 3D digital human streaming media, avoiding computation on the massive 3D data stream, and uses V... s Using the 3D digital human streaming media itself as the evaluation object reduces computational overhead.

[0021] Preferably, the extraction of the quality of service features of the three-dimensional digital human streaming media includes:

[0022] The quality of service (QoS) features of the 3D digital human streaming media are extracted by combining the encoding parameters, communication parameters, and decoding parameters, namely:

[0023] EP = {qp, qt, sr, jc}

[0024] TP = {D, C}

[0025] DP={T re ,T st}

[0026] Wherein, EP represents encoding parameters, including the number of bits for quantization of the 3D digital human's geometric position (qp), the number of bits for quantization of the geometric texture (qt), the 3D digital human texture downsampling rate (sr), and the JPEG compression level (jc); TP represents communication parameters, including the transmission distance (D) and the corresponding channel capacity (C); DP represents decoding parameters, including the loading buffer duration (T). re and video stuttering duration T st .

[0027] Preferably, the extraction of content quality features from the digital human video includes:

[0028] Semantic feature extraction from digital human videos using the CLIP model:

[0029] SF = CLIP(V s )

[0030] Where CLIP(·) indicates that the V model is trained using a pre-trained CLIP model. s The process of semantic feature extraction, SF represents V s It possesses semantic features;

[0031] Technical feature extraction from digital human videos using the NIQE algorithm:

[0032] TF = NIQE(V s )

[0033] Wherein, NIQE(·) represents the NIQE algorithm for V s The process of extracting technical features, where TF represents V s It possesses semantic features.

[0034] Preferably, the regression analysis of the service quality features and the content quality features to obtain the final quality score of the digital human video, i.e., the three-dimensional digital human streaming media quality score, includes:

[0035] Score = SVR(EP,TP,DP,SF,TF)

[0036] Where EP, TP, and DP are service quality features, SF and TF are content quality features, SVR(·) represents the process of performing support vector regression on all acquired and extracted parameters and features, and Score represents the digital human video V. s The quality of the 3D digital human streaming media.

[0037] According to a second aspect of the present invention, a three-dimensional digital human streaming media quality evaluation system suitable for 6G communication is provided, comprising:

[0038] Rendering module: Renders the 3D digital human streaming media transmitted through a 6G simulated channel frame by frame;

[0039] Equivalent module: Performs video synthesis on all rendered frames and adds audio tracks to synthesize digital human videos with audio on the user end, and uses this to represent the 3D digital human streaming media.

[0040] Service Quality Module: Extracts the service quality features of the 3D digital human streaming media;

[0041] Content quality module: Extracts content quality features from the digital human video;

[0042] Evaluation module: Regresses the service quality features and content quality features to obtain the quality score of the digital human video, i.e., the quality score of the 3D digital human streaming media.

[0043] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform the method described therein, or to run the system described therein.

[0044] According to a fourth aspect of the present invention, a terminal is provided, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described thereon, or to run the system described thereon.

[0045] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0046] The method and system for evaluating the quality of 3D digital human streaming media suitable for 6G communication in this invention embodiment closely resemble real human perception and can effectively evaluate 3D digital human streaming media transmitted under 6G communication conditions. Attached Figure Description

[0047] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a flowchart of a three-dimensional digital human streaming media quality evaluation method applicable to 6G communication according to one embodiment of the present invention;

[0049] Figure 2 This is a framework diagram of a three-dimensional digital human streaming media quality evaluation method applicable to 6G communication in one embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the process of obtaining a quality score in a preferred embodiment of the present invention;

[0051] Figure 4 This is a structural diagram of a three-dimensional digital human streaming media quality evaluation system suitable for 6G communication according to an embodiment of the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0053] like Figure 1 and Figure 2 The diagram shows a flowchart and framework of a three-dimensional digital human streaming media quality evaluation method applicable to 6G communication, according to an embodiment of the present invention. This evaluation method is a quality assessment method for three-dimensional digital human streaming media under 6G communication conditions that combines service quality characteristics and content quality characteristics. The method can be performed according to the following steps:

[0054] The first step is to render the 3D digital human streaming media transmitted through a 6G simulated channel frame by frame.

[0055] The second step is to perform video synthesis on all rendered frames and add audio tracks to synthesize digital human videos with audio on the user end, and use this to represent the 3D digital human streaming media.

[0056] The third step is to extract the service quality features of the three-dimensional digital human streaming media obtained in the second step.

[0057] The fourth step is to extract the content quality features of the original digital human video from the first step.

[0058] The fifth step involves regressing the service quality features from the third step and the content quality features from the fourth step to obtain the quality score of the digital human video, i.e., the quality score of the three-dimensional digital human streaming media.

[0059] The evaluation method described in the above embodiments closely aligns with real human perception and can effectively evaluate 3D digital human streaming media transmitted under 6G communication conditions. Generally, 3D digital human streaming media includes a 3D digital human model sequence Ms, a 3D digital human texture sequence Is, and a voice file Ss.

[0060] A sequence of 3D digital human models is a set of 3D digital human models. Typically, a single 3D digital human model is presented as an .obj file. A set of .obj files that are sequential in time constitutes a model sequence. Each .obj model originally only contains geometric information, lacking color information, which is supplemented by textures. Therefore, each .obj file corresponds to a texture, usually in JPEG or PNG format. Similarly, a set of textures that are sequential in time constitutes a texture sequence. Audio files are streaming audio, primarily referring to audio without background noise, containing only the digital human's voice. This audio is sequential in time, so the name does not need to use "sequence".

[0061] Given a 3D digital human streaming media with a duration of T seconds and a frame rate of F frames per second, it can be calculated that in this 3D digital human streaming media, Ms includes T·F 3D digital human models, and correspondingly, Is includes T·F 3D digital human textures. In addition, the audio of the entire streaming media can be regarded as Ss.

[0062] In a preferred embodiment of the present invention, a preferred first-step process is provided. In this embodiment, Open3D is used to perform frame-by-frame rendering of the 3D digital human streaming media transmitted through a 6G simulated channel. The specific process is as follows:

[0063] f t =Open3D(M t ,I t )

[0064] t = 1, 2, 3, ..., n

[0065] Wherein, Open3D(·) indicates that Open3D is used to render the t-th 3D digital human in the 3D digital human streaming media as frame f. t The process involves n, where n represents the total number of digital human models contained in the 3D digital human streaming media. Each rendering operation requires two items: the 3D digital human model M and its corresponding 3D digital human texture I.

[0066] In this embodiment, the open-source library Open3D is used for rendering, which provides good support for video capture. In other embodiments, similarly, open-source 3D visualization libraries such as OpenGL can be used to achieve video sequence capture.

[0067] In a preferred embodiment of the present invention, a second preferred step is provided. In this embodiment, a preferred step is selected.

[0068] FFmpeg performs video compositing on all rendered frames and adds audio tracks, creating a digital human video with audio on the user's end. The specific process is as follows:

[0069] V = f1 + f2 + f3 + ... + f n

[0070]

[0071] Here, + indicates the process of stitching together the rendering results of n frames using FFmpeg to synthesize a silent video V. This indicates the process of using FFmpeg for audio and video mixing. V s A and V represent the digital human video with audio and the original audio file of the digital human, respectively. Considering V... s The main information of the 3D digital human streaming media was preserved, and to avoid computation on the massive 3D data stream, V was further used. s The 3D digital human streaming media itself is used as the evaluation object to reduce computational overhead.

[0072] In this embodiment, FFmpeg is an open-source computer program that can be used to record, convert, and stream digital audio and video. It provides a complete solution for recording, converting, and streaming audio and video. In other embodiments, related audio and video processing functions can be implemented using Python, such as moviepy and pydub; or professional audio and video editing software such as Adobe Premiere can be used to achieve the same functionality.

[0073] In a preferred embodiment of the present invention, a third preferred step is provided. Specifically, as shown below... Figure 3 As shown, the service quality characteristics of the 3D digital human streaming media are examined by combining encoding parameters, communication parameters, and decoding parameters:

[0074] EP = {qp, qt, sr, jc}

[0075] TP = {D, C}

[0076] DP={T re ,T st}

[0077] Where EP represents the encoding parameters, including the quantization bits for the 3D digital human's geometric position (qp), the quantization bits for the geometric texture (qt), the 3D digital human texture downsampling rate (sr), and the JPEG compression level (jc); TP represents the communication parameters, including the transmission distance (D) and the corresponding channel capacity (C); DP represents the decoding parameters, including the loading buffer duration (T). re and video stuttering duration T st .

[0078] In a preferred embodiment of the present invention, a fourth step of the optimization process is provided. In this embodiment, the quality of service characteristics include semantic characteristics and technical characteristics. The process of obtaining the above two characteristics is as follows: Figure 3 As shown, specifically:

[0079] Semantic feature extraction from digital human videos is performed using the CLIP model, which can simultaneously understand and reason about images and text.

[0080] SF = CLIP(V s )

[0081] Where CLIP(·) indicates that the V model is trained using a pre-trained CLIP model. s The process of semantic feature extraction. SF represents V. s It possesses semantic features.

[0082] Technical feature extraction from digital human videos using the NIQE algorithm:

[0083] TF = NIQE(V s )

[0084] Wherein, NIQE(·) represents the NIQE algorithm for V s The process of extracting technical features. TF represents V. s It possesses semantic features. The NIQE algorithm itself has prior knowledge about the natural scene, which enables it to effectively extract video features. Furthermore, it has low computational complexity, short processing time, and is simple and easy to use.

[0085] Of course, in other embodiments, models with similar modalities to the CLIP model can be used to extract semantic features, such as BLIP, BLIP2, etc.; other technical feature extraction models can also be used, such as BRISQUE, VIIDEO, TLVQM, V-BLIINDS, etc., as long as they can objectively evaluate the quality of the video, there is no single limitation.

[0086] In a preferred embodiment of the present invention, a fifth step of the optimization process is provided. Specifically, the extracted features are regressed using the support vector regression method to finally obtain the quality score of the digital human video, i.e., the quality score of the 3D digital human streaming media:

[0087] Score = SVR(EP,TP,DP,SF,TF)

[0088] Where SVR(·) represents the process of performing support vector regression on all acquired and extracted parameters and features, and Score represents the digital human video V. s The quality of the 3D digital human streaming media.

[0089] Based on the same inventive concept, other embodiments of the present invention provide a three-dimensional digital human streaming media quality evaluation system suitable for 6G communication, such as... Figure 4 As shown, the system includes a rendering module, an equivalence module, a service quality module, a content quality module, and an evaluation module. The rendering module renders the 3D digital human streaming media frame-by-frame through a 6G simulated channel. The equivalence module performs video synthesis and adds audio tracks to all rendered frames, synthesizing a digital human video with audio at the user end, which represents the 3D digital human streaming media. The service quality module extracts the service quality features of the 3D digital human streaming media. The content quality module extracts the content quality features of the digital human video. The evaluation module performs regression analysis on the service quality features and content quality features to ultimately obtain the quality score of the digital human video, i.e., the 3D digital human streaming media quality score.

[0090] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the aforementioned three-dimensional digital human streaming media quality evaluation method for 6G communication, or to run the aforementioned three-dimensional digital human streaming media quality evaluation system for 6G communication.

[0091] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the aforementioned three-dimensional digital human streaming media quality evaluation method for 6G communication, or to run the aforementioned three-dimensional digital human streaming media quality evaluation system for 6G communication.

[0092] Implementation results:

[0093] In a specific embodiment of this invention, the 3D digital human streaming media quality evaluation method applicable to 6G communication described in the above embodiments was validated on the 6G-DHQA dataset and compared with mainstream video quality evaluation methods such as VSFA, SimpVQA, and FAST-VQA, and mainstream streaming media quality evaluation methods such as FTW, VsQM, and PXNR. The results show that the 3D digital human streaming media quality evaluation method best reflects real human perception among all the compared methods, further demonstrating that the method in the above embodiments can effectively evaluate 3D digital human streaming media transmitted under 6G communication conditions.

[0094] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for evaluating the streaming quality of three-dimensional digital human media suitable for 6G communication, characterized in that, include: Frame-by-frame rendering of 3D digital human streaming media transmitted via a 6G simulated channel; Video compositing and audio tracks are added to all rendered frames to synthesize digital human videos with audio on the user end, thus representing 3D digital human streaming media. Extract the quality of service characteristics of the three-dimensional digital human streaming media; Extract the content quality features of the digital human video; Regression is performed on the service quality features and the content quality features to obtain the quality score of the digital human video, i.e., the quality score of the three-dimensional digital human streaming media. The 3D digital human streaming media includes a 3D digital human model sequence, a 3D digital human texture sequence, and audio files; The extraction of the service quality features of the three-dimensional digital human streaming media includes: The quality of service (QoS) features of the 3D digital human streaming media are extracted by combining the encoding parameters, communication parameters, and decoding parameters, namely: ； in, This represents the encoding parameters, including the number of bits used for quantization of the geometric position of the 3D digital human. Geometric texture quantization bit depth 3D digital human texture downsampling rate and JPEG compression level ; Indicates communication parameters, including transmission distance. and corresponding channel capacity ; This indicates decoding parameters, including the load buffer duration. and video buffering duration ; The extraction of content quality features from the digital human video includes: Semantic feature extraction from digital human videos using the CLIP model: ； in, This indicates the use of a pre-trained CLIP model. The process of semantic feature extraction express It possesses semantic features; Digital human videos with audio; Technical feature extraction from digital human videos using the NIQE algorithm: ； in, Indicates the NIQE algorithm for The process of extracting technical features express It possesses technical characteristics.

2. The method for evaluating the quality of three-dimensional digital human streaming media suitable for 6G communication according to claim 1, characterized in that, The frame-by-frame rendering of the 3D digital human streaming media transmitted via a 6G simulated channel includes: ； in, This indicates the use of Open3D to analyze the third 3D digital human streaming media. A 3D digital human is rendered into frames. The process This indicates the total number of digital human models contained in the 3D digital human streaming media; each rendering operation requires two items: the 3D digital human models. and the corresponding 3D digital human texture. .

3. The method for evaluating the quality of three-dimensional digital human streaming media suitable for 6G communication according to claim 2, characterized in that, The process of compositing all rendered frames into a video and adding an audio track to synthesize a digital human video with audio on the user's end, and using this to represent the 3D digital human streaming media, includes: ； in, This means using FFmpeg to... Frame rendering results are stitched together to create a silent video. The process This describes the process of using FFmpeg for audio and video mixing. and These represent the digital human video with audio and the original audio file of the digital human, respectively. It retains the main information of the 3D digital human streaming media, avoiding computation on the massive 3D data stream, and uses... Using the 3D digital human streaming media itself as the evaluation object reduces computational overhead.

4. The method for evaluating the quality of three-dimensional digital human streaming media suitable for 6G communication according to claim 1, characterized in that, The regression analysis of the service quality features and content quality features ultimately yields the quality score of the digital human video, i.e., the three-dimensional digital human streaming media quality score, including: ； in, EP,TP,DP For service quality characteristics, SF,TF As a feature of content quality, This represents the process of performing support vector regression on all acquired and extracted parameters and features. Digital human video The quality of the 3D digital human streaming media.

5. A three-dimensional digital human streaming media quality evaluation system suitable for 6G communication, used to implement the method of claim 1, characterized in that, include: Rendering module: Renders the 3D digital human streaming media transmitted through a 6G simulated channel frame by frame; Equivalent module: Performs video synthesis on all rendered frames and adds audio tracks to synthesize digital human videos with audio on the user end, and uses this to represent the 3D digital human streaming media. Service Quality Module: Extracts the service quality features of the 3D digital human streaming media; Content quality module: Extracts content quality features from the digital human video; Evaluation module: Regresses the service quality features and content quality features to obtain the quality score of the digital human video, i.e., the quality score of the 3D digital human streaming media.

6. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-4.

Images