Digital twin interactive media content streaming method in weak network environment

Abstract

The invention particularly relates to a digital twin interactive media content streaming method in a weak network environment. According to the digital twin interactive media content streaming method in the weak network environment, a client sends a dynamic address request and receives fed back dynamic address information; the media server receives the SDP proposal and the ICE candidate sent by the client, and forwards the SDP proposal and the ICE candidate to the digital twin program; the client and the digital twin program establish a media content transmission channel with the media server according to the dynamic address information and the fixed address information; and the digital twinning program transmits the media content data to a client for display through a media content transmission channel stream by adopting a quality of service (QoS) processing mode. According to the digital twin interactive media content streaming method in the weak network environment, the problems of low quality and high time delay caused by bandwidth limitation and a severe network environment are solved, the quality of the media content received by the client is improved, the time delay in the transmission process of the media content data is reduced, and the transmission efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to a method for streaming interactive media content of digital twins in a weak network environment. Background Art

[0002] As an online digital simulation technology, digital twin technology plays an important role in industrial applications. In existing digital twin projects based on B / S architecture, the following technical solutions are usually adopted for media content streaming:

[0003] (1) Directly transmit media content encoded data based on the Reliable Transport Protocol (RTP);

[0004] (2) Based on the Real Time Streaming Protocol (RTSP), the transmission and control of real-time media content data are realized.

[0005] Among them, the first solution does not address issues such as data loss and disorder, and the second solution does not provide measures to ensure the reliability of media content streaming. Therefore, in summary, the existing technical solutions generally have the following defects:

[0006] (1) The ability to resist weak networks is poor. When the network bandwidth is small or the delay is large, problems such as distortion and discontinuity in the streaming of digital twin media content may occur.

[0007] (2) When the network bandwidth fluctuates greatly, the digital twin media content streaming will experience freezes, packet loss, and other phenomena.

[0008] (3) The digital twin media content is not filtered, denoised, or processed, resulting in poor quality of the media content received by the client.

[0009] Existing digital twin media content streaming implementation technologies fail to provide good QoS (Quality of Service) guarantee measures, and therefore cannot meet the high standards required by users in terms of real-time performance and reliability.

[0010] Based on the limitations of the above-mentioned prior art, the present invention proposes a method for streaming digital twin interactive media content in a weak network environment. Summary of the Invention

[0011] In order to overcome the shortcomings of the prior art, the present invention provides a simple and efficient method for streaming digital twin interactive media content in a weak network environment.

[0012] The present invention is achieved through the following technical solutions:

[0013] A method for streaming interactive media content of a digital twin in a weak network environment includes the following steps:

[0014] Step S1: The client sends a dynamic address request to the network address translation traversal server and receives the dynamic address information fed back by the network address translation traversal server;

[0015] Step S2: The client sends the SDP proposal and ICE candidate to the media server, which forwards it to the digital twin program. The media server establishes a WebSocket link with the client and the digital twin program to forward the message.

[0016] Step S3: The client and the digital twin program create a remote description based on the returned response information, and establish a media content transmission channel with the media server based on the dynamic address information and the fixed address information;

[0017] Step S4: The digital twin program collects media content data and, based on the quality of service (QoS) processing method in WebRTC, a web real-time communication technology, streams the media content data to the client through a media content transmission channel for display;

[0018] When the digital twin program streams media content, it uses at least one of the following technologies: acoustic echo cancellation (AEC), automatic gain control (AGC), and active noise reduction (ANC) for data quality processing.

[0019] For data transmission processing, at least one of negative acknowledgment NACK, forward error correction EFC, network equalizer NetEQ, Google receiver estimated maximum bit rate Goog-REMB, Google thickness control congestion algorithm Goog-TCC and traffic shaper Pacer technology is adopted.

[0020] In step S2, the client sends the SDP proposal to the media server based on the temporary call address generated by the media server, and the media server forwards it to the digital twin program and receives the corresponding SDP response.

[0021] In step S2, the client accesses the digital twin product page through a browser and sends a request to the backend server, which then determines the client's permissions.

[0022] When the background server verifies the permission, it sends a request to the media server, which calls up the digital twin program and sends the permission verification pass information and the temporary call address generated by the media server to the client.

[0023] In step S3, the client and the digital twin program first create a remote description based on the response information and add the address based on the fixed address information;

[0024] Then, a media content transmission channel is established with the media server according to the dynamic address information and the fixed address information.

[0025] A digital twin interactive media content streaming system in a weak network environment, including a client, a media server, a background server, a network address translation penetration server and a digital twin program;

[0026] The client is responsible for sending a dynamic address request to the network address translation traversal server and receiving the dynamic address information fed back by the network address translation traversal server; then, it sends the SDP proposal and ICE candidate to the media server;

[0027] The media server establishes a WebSocket link with the client and the digital twin program, and is responsible for forwarding the SDP proposal and ICE candidate to the digital twin program;

[0028] The client and the digital twin program are responsible for creating a remote description based on the returned response information, and establishing a media content transmission channel with the media server based on the dynamic address information and the fixed address information;

[0029] The digital twin program is responsible for collecting media content data and streaming the media content data to the client through the media content transmission channel based on the quality of service (QoS) processing method in WebRTC, a web real-time communication technology;

[0030] The client is responsible for displaying the received media content data.

[0031] The media server is responsible for generating a temporary call address, and the client is responsible for sending the SDP proposal to the media server according to the temporary call address;

[0032] The media server is responsible for forwarding the SDP proposal to the digital twin program after receiving it and receiving the corresponding SDP response.

[0033] The client is responsible for accessing the digital twin product page through the browser and sending a request to the backend server;

[0034] The backend server is responsible for determining the client's permissions and sending a request to the media server after the client passes the permission verification;

[0035] The media server is responsible for invoking the digital twin program and sending permission verification pass information and a temporary call address generated by the media server to the client.

[0036] The client and the digital twin program first create a remote description based on the response information and add the address based on the fixed address information;

[0037] Then, a media content transmission channel is established with the media server according to the dynamic address information and the fixed address information.

[0038] A digital twin interactive media content streaming device in a weak network environment is characterized by comprising a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0039] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0040] The beneficial effects of the present invention are: the digital twin interactive media content streaming method in a weak network environment utilizes the powerful service quality assurance capability of WebRTC, a real-time communication technology for web pages, to provide real-time and reliability guarantees for digital twin products based on media content streaming in a weak network environment, solves the low quality and high latency problems caused by bandwidth limitations and harsh network environments, improves the quality of media content received by the client, reduces the latency during media content data transmission, and improves transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Attachment Figure 1 Schematic diagram of the digital twin interactive media content streaming method in a weak network environment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0044] The concept of simulation has evolved from physical simulation (simulation) that completely uses physical models to computer simulation that is completely based on digital models, and further evolved into modeling simulation that integrates digital models and physical models.

[0045] A digital twin product is a digital model of a physical object that can evolve in real time by receiving data from the physical object, thus remaining consistent with the physical object throughout its life cycle.

[0046] Currently, there are numerous digital twin products in various application scenarios across various industries. However, digital twin products that use media content streaming as a form of delivery have the following main shortcomings:

[0047] (1) When the network bandwidth is small, the media content data stream is distorted and discontinuous.

[0048] (2) When the network bandwidth fluctuates greatly, media content streaming may experience freezes, packet loss, and other phenomena.

[0049] (3) The media content data is not filtered or processed, resulting in poor quality of the media content received by the client.

[0050] The method for streaming interactive media content of digital twins in a weak network environment includes the following steps:

[0051] Step S1: The client sends a dynamic address request to the network address translation traversal server and receives the dynamic address information fed back by the network address translation traversal server;

[0052] Step S2: The client sends the SDP proposal (Session Description Protocol proposal) and ICE candidate (Interactive Connection Establishment Candidate) to the media server, which forwards it to the digital twin program. The media server establishes a WebSocket link with the client and the digital twin program to forward the message.

[0053] Step S3: The client and the digital twin program create a remote description based on the returned response information, and establish a media content transmission channel with the media server based on the dynamic address information and the fixed address information;

[0054] Step S4: The digital twin program collects media content data and, based on the quality of service (QoS) processing method in WebRTC, a web real-time communication technology, streams the media content data to the client through a media content transmission channel for display;

[0055] When the digital twin program streams media content, it uses at least one of the following technologies: acoustic echo cancellation (AEC), automatic gain control (AGC), and active noise reduction (ANC) for data quality processing.

[0056] For data transmission processing, at least one of negative acknowledgment NACK, forward error correction EFC, network equalizer NetEQ, Google receiver estimated maximum bit rate Goog-REMB, Google thickness control congestion algorithm Goog-TCC and traffic shaper Pacer technology is adopted.

[0057] In step S2, the client sends the SDP proposal to the media server based on the temporary call address generated by the media server, and the media server forwards it to the digital twin program and receives the corresponding SDP response.

[0058] In step S2, the client accesses the digital twin product page through a browser and sends a request to the backend server, which then determines the client's permissions.

[0059] When the background server verifies the permission, it sends a request to the media server, which calls up the digital twin program and sends the permission verification pass information and the temporary call address generated by the media server to the client.

[0060] In step S3, the client and the digital twin program first create a remote description based on the response information and add the address based on the fixed address information;

[0061] Then, a media content transmission channel is established with the media server according to the dynamic address information and the fixed address information.

[0062] The digital twin interactive media content streaming system in the weak network environment includes a client, a media server, a background server, a network address translation penetration server and a digital twin program;

[0063] The client is responsible for sending a dynamic address request to the network address translation traversal server and receiving the dynamic address information fed back by the network address translation traversal server; then, it sends the SDP proposal and ICE candidate to the media server;

[0064] The media server establishes a WebSocket link with the client and the digital twin program, and is responsible for forwarding the SDP proposal and ICE candidate to the digital twin program;

[0065] The client and the digital twin program are responsible for creating a remote description based on the returned response information, and establishing a media content transmission channel with the media server based on the dynamic address information and the fixed address information;

[0066] The digital twin program is responsible for collecting media content data and streaming the media content data to the client through the media content transmission channel based on the quality of service (QoS) processing method in WebRTC, a web real-time communication technology;

[0067] The client is responsible for displaying the received media content data.

[0068] The media server is responsible for generating a temporary call address, and the client is responsible for sending the SDP proposal to the media server according to the temporary call address;

[0069] The media server is responsible for forwarding the SDP proposal to the digital twin program after receiving it and receiving the corresponding SDP response.

[0070] The client is responsible for accessing the digital twin product page through the browser and sending a request to the backend server;

[0071] The backend server is responsible for determining the client's permissions and sending a request to the media server after the client passes the permission verification;

[0072] The media server is responsible for invoking the digital twin program and sending permission verification information and a temporary call address generated by the media server to the client.

[0073] The client and the digital twin program first create a remote description based on the response information and add the address based on the fixed address information;

[0074] Then, a media content transmission channel is established with the media server according to the dynamic address information and the fixed address information.

[0075] The digital twin interactive media content streaming device in a weak network environment is characterized by comprising a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0076] The readable storage medium is characterized in that a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0077] Compared with existing technologies, this method for streaming interactive media content using digital twins in a weak network environment has the following characteristics:

[0078] First, during the establishment of the digital twin program and the client, since the client is located on a private network and is equipped with a firewall or network address translation service, before establishing the media content transmission channel, the media server will first receive the network address translation penetration service address request sent by the client to request the address of the corresponding network address translation penetration server (STUN / TURN server); the media server forwards the feedback information of the network address translation penetration server to the client, so that the client obtains the corresponding dynamic conversion address, and then interacts with the media server via SDP and ICE, thereby establishing a media content transmission channel with the digital twin program, ensuring that subsequent clients can receive the media content data sent by the digital twin sequence, so as to improve the quality and transmission efficiency of the digital twin media content streaming.

[0079] Second, the digital twin program uses the WebRTC preset service quality QoS processing method to stream media content to the client, which solves the problems of data packet loss, jitter, disorder, noise and other problems in the media content streaming of existing digital twin products, improves the quality of media content received by the client, reduces the delay in the media content data transmission process, and improves transmission efficiency.

[0080] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for streaming interactive media content of digital twins in a weak network environment, characterized by: The following steps are involved: Step S1: The client sends a dynamic address request to the network address translation traversal server and receives the dynamic address information fed back by the network address translation traversal server; Step S2: The client sends the SDP proposal and ICE candidate to the media server, which forwards it to the digital twin program. The media server establishes a WebSocket link with the client and the digital twin program to forward the message. Step S3: The client and the digital twin program create a remote description based on the returned response information, and establish a media content transmission channel with the media server based on the dynamic address information and the fixed address information; Step S4: The digital twin program collects media content data and, based on the quality of service (QoS) processing method in WebRTC, a web real-time communication technology, streams the media content data to the client through a media content transmission channel for display; When the digital twin program streams media content, it uses at least one of the following technologies: acoustic echo cancellation (AEC), automatic gain control (AGC), and active noise reduction (ANC) for data quality processing. For data transmission processing, at least one of negative acknowledgment NACK, forward error correction EFC, network equalizer NetEQ, Google receiver estimated maximum bit rate Goog-REMB, Google thickness control congestion algorithm Goog-TCC and traffic shaper Pacer technology is adopted.

2. The method for streaming digital twin interactive media content in a weak network environment according to claim 1 is characterized by: In step S2, the client sends the SDP proposal to the media server based on the temporary call address generated by the media server, and the media server forwards it to the digital twin program and receives the corresponding SDP response.

3. The method for streaming interactive media content of digital twins in a weak network environment according to claim 2 is characterized by: In step S2, the client accesses the digital twin product page through a browser and sends a request to the backend server, which then determines the client's permissions. When the background server verifies the permission, it sends a request to the media server, which calls up the digital twin program and sends the permission verification pass information and the temporary call address generated by the media server to the client.

4. The method for streaming interactive media content of digital twins in a weak network environment according to claim 1, characterized in that: In step S3, the client and the digital twin program first create a remote description based on the response information and add the address based on the fixed address information; Then, a media content transmission channel is established with the media server according to the dynamic address information and the fixed address information.

5. A digital twin interactive media content streaming system in a weak network environment, characterized by: Including client, media server, background server, network address translation penetration server and digital twin program; The client is responsible for sending a dynamic address request to the network address translation traversal server and receiving the dynamic address information fed back by the network address translation traversal server; then, it sends the SDP proposal and ICE candidate to the media server; The media server establishes a WebSocket link with the client and the digital twin program, and is responsible for forwarding the SDP proposal and ICE candidate to the digital twin program; The client and the digital twin program are responsible for creating a remote description based on the returned response information, and establishing a media content transmission channel with the media server based on the dynamic address information and the fixed address information; The digital twin program is responsible for collecting media content data and streaming the media content data to the client through the media content transmission channel based on the quality of service (QoS) processing method in WebRTC, a web real-time communication technology; The client is responsible for displaying the received media content data.

6. The digital twin interactive media content streaming system in a weak network environment according to claim 5 is characterized by: The media server is responsible for generating a temporary call address, and the client is responsible for sending the SDP proposal to the media server according to the temporary call address; The media server is responsible for forwarding the SDP proposal to the digital twin program after receiving it and receiving the corresponding SDP response.

7. The digital twin interactive media content streaming system in a weak network environment according to claim 6, characterized in that: The client is responsible for accessing the digital twin product page through the browser and sending a request to the backend server; The backend server is responsible for determining the client's permissions and sending a request to the media server after the client passes the permission verification; The media server is responsible for invoking the digital twin program and sending permission verification information and a temporary call address generated by the media server to the client.

8. The method for streaming digital twin interactive media content in a weak network environment according to claim 5, characterized in that: The client and the digital twin program first create a remote description based on the response information and add the address based on the fixed address information; Then, a media content transmission channel is established with the media server according to the dynamic address information and the fixed address information.

9. A digital twin interactive media content streaming device in a weak network environment, characterized by: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method steps according to any one of claims 1 to 4 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the method steps according to any one of claims 1 to 4.

Images