Data transmission method, apparatus, device, and medium

Abstract

The application discloses a data transmission method and device, equipment and medium, comprising: acquiring user posture data sent by a VR device; generating a picture frame to be displayed on the VR device based on the user posture data; and transmitting the picture frame to the VR device based on an SRT protocol. In this way, the picture frame to be displayed on the VR device is generated based on the user posture data acquired from the VR device, and then the picture frame is transmitted to the VR device based on the SRT protocol, so that the delay of data transmission between the computer device and the VR device can be reduced under the premise of ensuring reliable transmission.

Description

Data transmission methods, devices, equipment and media Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to data transmission methods, apparatus, devices and media. Background Technology

[0002] In recent years, VR (Virtual Reality) devices have become increasingly popular. In the user experience of VR devices, the transmission of images between VR devices and PC (Personal Computer) devices and the timely motion feedback are important factors affecting the user experience.

[0003] Currently, data streaming between VR devices and PCs is typically done via direct transmission over Wi-Fi hotspots. While this method offers low latency, it can lead to data loss during high-volume transmissions. Streaming media requires real-time audio, and mobile-ffmpeg often uses HTTP (Hypertext Transfer Protocol) or RTMP (Real-Time Messaging Protocol) for encoding and compression. HTTP lacks real-time performance. RTMP is a widely used streaming protocol, particularly in live streaming, and is well-suited for large-scale distribution to multiple users. However, end-to-end video transmission using RTMP requires an RTMP server, resulting in significant latency. Even with a user-hosted RTMP server, the latency is around 1 second. RTMP has cumulative latency because it relies on TCP (Transmission Control Protocol), preventing packet loss. RTSP server implementation is complex and lacks firewall penetration capabilities.

[0004] In summary, reducing the latency of data transmission between computer devices and VR devices while ensuring reliable transmission is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a data transmission method, apparatus, device, and medium that can reduce the latency of data transmission between computer equipment and VR devices while ensuring reliable transmission. The specific solution is as follows:

[0006] In a first aspect, this application discloses a data transmission method applied to a computer device, comprising:

[0007] Acquire user gesture data sent by the VR device;

[0008] The user's posture data is used to generate the frame to be displayed on the VR device;

[0009] The image frames are transmitted to the VR device based on the SRT protocol.

[0010] Optionally, transmitting the image frames to the VR device based on the SRT protocol includes:

[0011] The image frames are compressed using a preset compression processing logic to obtain compressed frames;

[0012] The compressed frame is transmitted to the VR device based on the SRT protocol.

[0013] Optionally, the step of compressing the image frame using preset compression processing logic to obtain a compressed frame includes:

[0014] Group the video frames;

[0015] Determine the base frame and non-base frames in any group, predict the difference information of the non-base frames based on the base frames, compress the base frames and the difference information to obtain the compressed frames corresponding to the base frames and the compressed frames corresponding to the non-base frames.

[0016] Optionally, it also includes: establishing a wireless hotspot connection with the VR device; correspondingly,

[0017] The acquisition of user gesture data sent by the VR device includes: acquiring user gesture data sent by the VR device based on a wireless hotspot connection.

[0018] Optionally, generating the screen frames to be displayed on the VR device based on the user posture data includes:

[0019] The game engine is used to render the frames to be displayed on the VR device based on the user's posture data.

[0020] Optionally, transmitting the image frames to the VR device based on the SRT protocol includes:

[0021] Send a handshake request to the VR device and receive a handshake response from the VR device;

[0022] Send parameter notifications to the VR device and receive parameter responses returned by the VR device;

[0023] The image frame and its timestamp are transmitted to the VR device.

[0024] Secondly, this application discloses a data transmission method applied to a VR device, comprising:

[0025] Send user gesture data to computer devices;

[0026] The computer device receives screen frames sent by the computer device based on the SRT protocol. The screen frames are screen frames to be displayed on the VR device generated by the computer device based on the user posture data.

[0027] Thirdly, this application discloses a data transmission device applied to a computer device, comprising:

[0028] The user data acquisition module is used to acquire user gesture data sent by the VR device.

[0029] A frame generation module is used to generate frame images to be displayed on the VR device based on the user posture data;

[0030] The image frame transmission module is used to transmit the image frames to the VR device based on the SRT protocol.

[0031] Fourthly, this application discloses an electronic device, including a memory and a processor, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program to implement the aforementioned data transmission method.

[0034] Fifthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data transmission method.

[0035] As can be seen, this application first acquires user posture data sent by the VR device, then generates a frame to be displayed on the VR device based on the user posture data, and transmits the frame to the VR device based on the SRT protocol. That is, this application first generates a frame to be displayed on the VR device based on the user posture data acquired from the VR device, and then transmits the frame to the VR device based on the SRT protocol. By using the SRT protocol, which is based on a bidirectional UDP point-to-point connection and has a good packet loss retransmission mechanism and congestion control mechanism, the latency of data transmission between the computer device and the VR device can be reduced while ensuring reliable transmission. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 is a flowchart of a data transmission method disclosed in this application;

[0038] Figure 2 is a schematic diagram of a specific SRT protocol transmission disclosed in this application;

[0039] Figure 3 is a schematic diagram of data transmission between a specific VR device and a PC disclosed in this application;

[0040] Figure 4 is a flowchart of another data transmission method disclosed in this application;

[0041] Figure 5 is a schematic diagram of a data transmission device disclosed in this application;

[0042] Figure 6 is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Currently, data streaming between VR devices and PCs is typically done via direct transmission over Wi-Fi hotspots. While this method offers low latency, it can lead to data loss during high-volume transmissions. Streaming media requires real-time audio, and mobile-ffmpeg primarily uses HTTP or RTMP protocols for encoding and compression. HTTP offers limited real-time performance. RTMP, a widely used streaming protocol, is ideal for live broadcasting and large-scale distribution to multiple users. However, end-to-end video transmission requires an RTMP server, resulting in significant latency. Even with a user-hosted RTMP server, latency is around 1 second. RTMP suffers from cumulative latency because it relies on TCP and avoids packet loss. RTSP server implementation is complex and lacks router / firewall penetration capabilities. Therefore, this application provides a data transmission solution that reduces latency between computer and VR devices while ensuring reliable transmission.

[0045] Referring to Figure 1, this application discloses a data transmission method applied to a computer device, including:

[0046] Step S11: Obtain user posture data sent by the VR device.

[0047] In one implementation, the user posture data can be the user's head posture data; of course, in other embodiments, it can also be the posture data of other parts of the user's body.

[0048] Furthermore, this embodiment of the application can establish a wireless hotspot connection with the VR device; correspondingly, it acquires user posture data sent by the VR device based on the wireless hotspot connection. It should be noted that, since the posture data is relatively small, the VR device can directly send the user posture data to a computer device via the wireless hotspot connection. The computer device can be a personal computing device.

[0049] Step S12: Generate the screen frames to be displayed on the VR device based on the user posture data.

[0050] In a specific implementation, a game engine can be used to render the image frames to be displayed on the VR device based on the user's posture data.

[0051] Step S13: Transmit the image frame to the VR device based on the SRT protocol.

[0052] In a specific implementation, a handshake request can be sent to the VR device, and a handshake response can be received from the VR device; a parameter notification can be sent to the VR device, and a parameter response can be received from the VR device; the image frame and its timestamp can be transmitted to the VR device. It is understood that this embodiment can utilize a wireless hotspot connection and transmit the image frame to the VR device based on the SRT protocol.

[0053] For example, as shown in Figure 2, which is a specific SRT protocol transmission diagram disclosed in an embodiment of this application, the PC acts as the caller, encoding the video feed and sending it to the VR terminal, which acts as the listener. During transmission, all streaming media is decomposed into video frames and timestamps using the SRT protocol. Simultaneously, smaller sensor data, such as user gesture data from the VR terminal, can be transmitted back to the PC terminal via a wireless hotspot for response. The SRT protocol includes two types of data packets: Data Packets and Control Packets, distinguished by the highest bit (flag bit) of the SRT header: 0 represents a Data Packet, and 1 represents a Control Packet. Control Packets include various types such as Handshake, Acknowledgment (ACK), Negative Acknowledgment (NAK), Acknowledgment of Acknowledgment (ACKACK), Keepalive, and Shutdown.

[0054] Furthermore, in a specific implementation, the image frames can be compressed using a preset compression processing logic to obtain compressed frames; the compressed frames are then transmitted to the VR device based on the SRT protocol.

[0055] In one implementation, the frame can be grouped; a base frame and non-base frames in any group can be determined, and the difference information of the non-base frames can be predicted based on the base frames. The base frames and the difference information can be compressed to obtain the compressed frames corresponding to the base frames and the compressed frames corresponding to the non-base frames.

[0056] Tests showed that the SRT protocol is approximately 2.5 to 3.2 times faster than the RTMP protocol. It's worth noting that the SRT protocol is an improvement on the UDT protocol. The SRT protocol retains most of the core concepts and mechanisms of the UDT protocol while introducing several improvements and enhancements, primarily including flow control for real-time audio and video, enhanced congestion control, modification of control data, and improved encryption mechanisms. The UDP protocol is an unreliable datagram-based transport protocol designed primarily to maximize data transmission rates, providing only unreliable data transmission services with a best-effort delivery. Although UDP is an unreliable transport protocol and unstable in the jitter and packet loss environment of the internet, the SRT protocol's strong data recovery capabilities, employing Forward Correction (FEC) technology, minimize the possibility of network packet loss, ensuring SRT transmission stability. Furthermore, the SRT protocol can perform AES (Advanced Encryption Standard) encryption, ensuring data security during transmission. In other words, the SRT protocol is based on bidirectional UDP point-to-point connection, which is secure, reliable and low-latency. It allows direct connection between signal source and target, and is suitable for point-to-point high-quality, low-latency and reliable transmission. This application adopts the SRT protocol, which can reduce latency, eliminate central bottlenecks and reduce network costs.

[0057] The following section uses a 6DoF head-mounted display (HMD) as an example to illustrate the data transmission scheme provided in this application. Referring to Figure 3, which is a schematic diagram of data transmission between a VR device and a PC according to an embodiment of this application, the 6DoF head-mounted display tracks and captures head posture data. Since posture data is much smaller than video data, it can be directly transmitted to the PC using a protocol hotspot (hostap). The game engine on the PC generates rendering frames (i.e., screen frames) based on the head posture data and sends them back to the VR device, where they are decoded, sent, and displayed. Specifically, the 6DoF engine determines the head posture data based on the eye buffer (D3D11 texture). The PC-side game engine transmits the rendering frames and head posture data to the encoding module. The encoder in the encoding module encodes the head posture data into H.264 format for other references and encodes the rendering frames into H.264 format, sending them back to the VR end based on the SRT protocol. The VR end receives the SRT encoded bitstream of the compressed rendering frames, synchronously decompresses the compressed frames, performs color correction and reprojection operations on the rendering frames, and then sends them to the HMD's display for projection. The wireless module is responsible for encoding and decoding 6DoF head pose data and compressing the SRT data stream of the rendered frames. It primarily uses SRT wireless transmission technology for low-latency transmission, resulting in a better user experience. The VR player transmits HMD pose position information to the VR renderer at a specified rate. The player receives the encoded bitstream of the rendered frames wirelessly and decodes / synchronizes the frames before sending them to the VR stack to perform necessary lens distortion and chromatic aberration correction. The VR renderer receives the HMD pose position information from the player and uses it to set the orientation and position of the VR camera. The VR device transmits the latest head pose data to the PC. This embodiment uses H.264 compression. By compressing the image with H.264 before transmission to the PC and using SRT transmission to achieve data streaming between the VR device and the PC, the transmission speed is increased while maintaining low latency during image synchronization. Of course, in other embodiments, H.265 compression can also be used. In this way, based on end-to-end Wi-Fi connections, the SRT protocol is used for streaming media transmission. This allows for the streaming of high-volume content (such as high-definition video resources) while maintaining low latency, and provides a robust packet loss retransmission mechanism and rich congestion control statistics, thereby improving the user's picture quality. Furthermore, the UDT protocol offers low latency and low encryption security requirements for short-range end-to-end transmission.

[0058] H.264 compression defines three types of frames: I-frame (fully encoded), P-frame (frame containing only the differences from the preceding I-frame), and B-frame (frame encoded by referencing the preceding and following frames). The core algorithms used in H.264 are intra-frame compression and inter-frame compression: intra-frame compression generates I-frames, while inter-frame compression generates B-frames and P-frames. Compression methods include: Grouping: Several frames are grouped into a single group (GOP, a sequence). To prevent motion distortion, the number of frames should not be excessive. Frame definition: Each frame within a group is defined as one of three types: I-frame, B-frame, or P-frame. Frame prediction: Using the I-frame as the base frame, P-frames are predicted from the I-frame, and B-frames are predicted from the I-frame and P-frame. Data transmission: Finally, the difference between the I-frame data and the predicted data is stored and transmitted.

[0059] As can be seen, this application embodiment first acquires user posture data sent by the VR device, then generates a screen frame to be displayed on the VR device based on the user posture data, and transmits the screen frame to the VR device based on the SRT protocol. That is, this application first generates a screen frame to be displayed on the VR device based on the user posture data acquired from the VR device, and then transmits the screen frame to the VR device based on the SRT protocol. By using the SRT protocol, which is based on a bidirectional UDP point-to-point connection and has a good packet loss retransmission mechanism and congestion control mechanism, the latency of data transmission between the computer device and the VR device can be reduced while ensuring reliable transmission.

[0060] Furthermore, the image frames are compressed using a preset compression processing logic to obtain compressed frames; the compressed frames are then transmitted to the VR device based on the SRT protocol, thereby increasing the transmission speed while maintaining low latency.

[0061] Referring to Figure 4, this application discloses a data transmission method applied to a VR device, including:

[0062] Step S21: Send user gesture data to the computer device.

[0063] In a specific implementation, a wireless hotspot connection is established with the computer device; correspondingly, user gesture data is transmitted based on the wireless hotspot connection.

[0064] Step S22: Receive screen frames sent by the computer device based on the SRT protocol. The screen frames are screen frames to be displayed on the VR device generated by the computer device based on the user posture data.

[0065] In a specific implementation, a handshake response can be returned upon receiving a handshake request from a computer device; a parameter response can be returned upon receiving a parameter notification from a computer device; and then a screen frame and the timestamp of the screen frame can be received from the computer device.

[0066] The image frame is a frame rendered by a computer device using a game engine based on the user's gesture data. Furthermore, in one embodiment, the received frame is a compressed image frame, obtained by the computer device compressing the image frame using preset compression processing logic.

[0067] As can be seen, this embodiment sends user gesture data to a computer device; and receives screen frames sent by the computer device based on the SRT protocol. These screen frames are generated by the computer device based on the user gesture data and are to be displayed on the VR device. Thus, by employing the SRT protocol, which is based on a bidirectional UDP point-to-point connection and has a robust packet loss retransmission mechanism and congestion control mechanism, the latency of data transmission between the computer device and the VR device can be reduced while ensuring reliable transmission.

[0068] Referring to Figure 5, this application discloses a data transmission device applied to a computer device, comprising:

[0069] User data acquisition module 11 is used to acquire user posture data sent by VR device;

[0070] The frame generation module 12 is used to generate a frame to be displayed on the VR device based on the user posture data;

[0071] The image frame transmission module 13 is used to transmit the image frame to the VR device based on the SRT protocol.

[0072] As can be seen, this application embodiment first acquires user posture data sent by the VR device, then generates a screen frame to be displayed on the VR device based on the user posture data, and transmits the screen frame to the VR device based on the SRT protocol. That is, this application first generates a screen frame to be displayed on the VR device based on the user posture data acquired from the VR device, and then transmits the screen frame to the VR device based on the SRT protocol. By using the SRT protocol, which is based on a bidirectional UDP point-to-point connection and has a good packet loss retransmission mechanism and congestion control mechanism, the latency of data transmission between the computer device and the VR device can be reduced while ensuring reliable transmission.

[0073] The image frame transmission module 13 includes:

[0074] The compression submodule is used to compress the image frames using preset compression processing logic to obtain compressed frames;

[0075] The frame transmission submodule is used to transmit the compressed frame to the VR device based on the SRT protocol.

[0076] In one implementation, the compression submodule is specifically used to group the frame images; determine the base frame and non-base frames in any group, predict the difference information of the non-base frames based on the base frames, compress the base frames and the difference information, and obtain the compressed frames corresponding to the base frames and the compressed frames corresponding to the non-base frames.

[0077] Furthermore, the device also includes a wireless connection establishment module for establishing a wireless hotspot connection with the VR device; correspondingly, the user data acquisition module 11 is specifically used to acquire user posture data sent by the VR device based on the wireless hotspot connection.

[0078] In one embodiment, the frame generation module 12 is used to render the frame to be displayed on the VR device using a game engine and based on the user posture data.

[0079] Furthermore, the frame transmission module 13 is specifically used to send a handshake request to the VR device and receive a handshake response returned by the VR device; send a parameter notification to the VR device and receive a parameter response returned by the VR device; and transmit the frame and the timestamp of the frame to the VR device.

[0080] Referring to Figure 6, this application discloses an electronic device, including a processor 21 and a memory 22; wherein, the memory 22 is used to store a computer program; and the processor 21 is used to execute the computer program, the data transmission method disclosed in the foregoing embodiments.

[0081] For details regarding the specific process of the above data transmission method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0082] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data transmission method disclosed in the foregoing embodiments.

[0083] For details regarding the specific process of the above data transmission method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0085] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0086] The data transmission method, apparatus, device, and medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data transmission method, characterized in that, This application, used in computer equipment, includes: establishing a wireless hotspot connection with a VR device; acquiring user posture data sent by the VR device based on the wireless hotspot connection; wherein the user posture data is directly sent from the VR device to the computer device via the wireless hotspot connection; the user posture data is the user's head posture data; rendering based on the user posture data to obtain a frame to be displayed on the VR device; and transmitting the frame to the VR device based on the SRT protocol; wherein transmitting the frame to the VR device based on the SRT protocol includes: sending a handshake request to the VR device and receiving a handshake response returned by the VR device; sending a parameter notification to the VR device and receiving a parameter response returned by the VR device; and converting the frame into a display frame. The timestamps of the image frames are transmitted to the VR device, and the image frames are compressed using preset compression processing logic to obtain compressed frames. The compressed frames are then transmitted to the VR device based on the SRT protocol, so that the VR device can receive the SRT encoded bitstream of the compressed frames, synchronize and decompress the compressed frames, and then perform color correction and reprojection operations on the image frames before sending them to the display for projection. The step of compressing the image frames using preset compression processing logic to obtain compressed frames includes: grouping the image frames; determining the base frame and non-base frames in any group, predicting the difference information of the non-base frames based on the base frames, compressing the base frames and the difference information to obtain compressed frames corresponding to the base frames and compressed frames corresponding to the non-base frames.

2. The data transmission method according to claim 1, characterized in that, The step of generating the screen frame to be displayed on the VR device based on the user posture data includes: using a game engine and rendering based on the user posture data to obtain the screen frame to be displayed on the VR device.

3. A data transmission method, characterized in that, This technology is applied to VR devices and includes: establishing a wireless hotspot connection with a computer device; directly sending user posture data to the computer device based on the wireless hotspot connection; the user posture data being the user's head posture data; receiving a frame from the computer device based on the SRT protocol, wherein the frame is a frame to be displayed on the VR device, rendered by the computer device based on the user posture data, and the transmission process of the frame is as follows: the computer device sends a handshake request to the VR device and receives a handshake response from the VR device; the computer device sends a parameter notification to the VR device and receives a parameter response from the VR device; and the frame and its time are then displayed. The data is transmitted to the VR device, and the image frames are compressed using a preset compression processing logic to obtain compressed frames. The compressed frames are transmitted based on the SRT protocol. The VR device receives the SRT encoded bitstream of the compressed frames, synchronizes and decompresses the compressed frames, then performs color correction and reprojection operations on the image frames, and sends them to the display for projection. The step of compressing the image frames using the preset compression processing logic to obtain compressed frames includes: grouping the image frames; determining the base frame and non-base frames in any group, predicting the difference information of the non-base frames based on the base frames, compressing the base frames and the difference information to obtain compressed frames corresponding to the base frames and compressed frames corresponding to the non-base frames.

4. A data transmission device, characterized in that, This device, applied to computer equipment, includes: a connection establishment module for establishing a wireless hotspot connection with a VR device; a user data acquisition module for acquiring user posture data sent by the VR device based on the wireless hotspot connection; wherein the user posture data is directly sent from the VR device to the computer device based on the wireless hotspot connection; the user posture data is the user's head posture data; a frame generation module for rendering based on the user posture data to obtain the frame to be displayed on the VR device; and a frame transmission module for transmitting the frame to the VR device based on the SRT protocol; wherein the frame transmission module includes: a compression submodule for grouping the frame; determining the base frame and non-base frames in any group, and based on the base frame... The system predicts the difference information of non-base frames, compresses the base frame and the difference information to obtain compressed frames corresponding to the base frame and compressed frames corresponding to the non-base frames; the screen frame transmission submodule is used to transmit the compressed frames to the VR device based on the SRT protocol; the screen frame transmission module is also used to: send a handshake request to the VR device and receive a handshake response returned by the VR device; send a parameter notification to the VR device and receive a parameter response returned by the VR device; transmit the screen frame and the timestamp of the screen frame to the VR device so that the VR device can receive the SRT encoded bit stream of the compressed frame, synchronize and decompress the compressed frame, and then perform color correction and reprojection operations on the screen frame before sending it to the display for projection.

5. An electronic device, characterized in that, The system includes a memory and a processor, wherein: the memory is used to store a computer program; and the processor is used to execute the computer program to implement the data transmission method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the data transmission method as described in any one of claims 1 to 3.

Images