A data processing method, storage medium and system for network users to share immersive experiences

Abstract

The application provides a data processing method for network users to share immersive experience, so that network users can share immersive experience by transmitting immersive reality content data. The application also provides a sending device for immersive reality content data, a receiving device for immersive reality content data, a system for network users to share immersive experience, and a computer readable storage medium.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer Internet, more particularly, to the technical field of Virtual Reality (VR); in particular, a data processing method for network users to share immersive experience, a storage medium and a system are provided. BACKGROUND

[0002] In the current digital era, virtual reality technology has changed the traditional network user experience more and more deeply. Virtual reality involves a wide range of computer technologies, including computer graphics, human-computer interaction, artificial intelligence, sensor technology, computer network technology, and even wireless communication technology as virtual reality technology elements with the popularization of 5G applications. In simple terms, virtual reality is to present a virtual environment to the user, and the user experiences, perceives and reacts to the virtual environment in the virtual environment. In this process, the system tracks changes and provides visual, auditory, and tactile sensory experiences to make the user feel as if they were there.

[0003] Taking e-sports live broadcast as an example, many live broadcast platforms pay attention to global e-sports competitions every year, such as the League of Legends World Championship. Broadcasting e-sports competitions has become a competitive stage for major live broadcast platforms. Therefore, various live broadcast platforms in the industry try every means to provide users with more levels and more ways of watching experience, which has derived many technical solutions in this technical field, including: AI intelligent anti-blocking bullet screen from network user interaction; 360-degree high-definition picture and 4K+HDR ultra-high definition to improve visual effects, making colors more vibrant and game scenes more delicate; and dotting playback of live broadcast highlights, users can quickly review the highlights according to various annotations on the progress bar…… These technologies are the use experience brought to users by immersive technology. In short, immersive technology refers to a system composed of audio-visual devices, sensors, motors, and interactive software, which blurs the boundaries between the physical world and the real world, providing users with immersive and interactive experiences.

[0004] However, in the existing live broadcast process, the above technical means are mostly concentrated in the audio-visual technology field, whether it is bullet screen content or video content is concentrated in the visual and auditory perception range of users. The limitations of the perception method make users who watch e-sports live broadcast or review unable to obtain the same experience as e-sports players in the game. In short, users urgently hope to obtain more game experience through watching live broadcast or playback, as if they personally experienced the game, and better obtain immersive experience.

[0005] Therefore, how to transmit the feelings of players in the e-sports match to the users watching the live broadcast or playback as much as possible without difference and make the users obtain better immersive experience has become an urgent demand of network users. SUMMARY

[0006] The present application aims to provide a data processing method for network users to share immersive experience, and network users can share immersive experience. The present application also provides a sending device of immersive reality content data, a receiving device of immersive reality content data, a system for network users to share immersive experience and a computer readable storage medium.

[0007] To achieve the above-mentioned purpose, the embodiment of the present application provides a data processing method for network users to share immersive experience, comprising:

[0008] The first user equipment captures immersive reality content data through a data generation module;

[0009] The first user equipment encodes the immersive reality content data to generate a protocol data packet containing time information;

[0010] The first user equipment sends the protocol data packet to the second user equipment;

[0011] The second user equipment receives the protocol data packet;

[0012] The second user equipment parses the protocol data packet to obtain the time information and the immersive reality content data;

[0013] The second user equipment processes the immersive reality content data through a data processing module.

[0014] The immersive reality content data contains audio data, video data or gamepad vibration data.

[0015] The protocol data packet complies with H.264 protocol, H.265 protocol or H.266 protocol.

[0016] The gamepad vibration data is generated by a gamepad vibration motor or a gamepad vibration module.

[0017] To achieve the above-mentioned purpose, the embodiment of the present application also provides a sending device of immersive reality content data, comprising:

[0018] A processor,

[0019] A memory configured to provide instructions and operation data to the processor;

[0020] data generating apparatus configured to capture immersive reality content data and store the immersive reality content data in the memory; and

[0021] data transmitting apparatus configured to transmit a protocol data packet; wherein

[0022] the protocol data comprises a data format encoding the immersive reality content data to contain time information.

[0023] To achieve the above object, the embodiment of the present application further provides a receiving apparatus of immersive reality content data, comprising:

[0024] a processor;

[0025] a memory configured to provide instructions and operation data to the processor; and

[0026] data receiving apparatus configured to receive a protocol data packet;

[0027] a data processing module configured to process immersive reality content data;

[0028] wherein

[0029] the processor parses the protocol data packet to obtain the time information and the immersive reality content data.

[0030] To achieve the above object, the embodiment of the present application further provides a system for sharing immersive experience by network users, comprising:

[0031] at least one processor;

[0032] at least one memory coupled to the at least one processor;

[0033] at least one data generating apparatus configured to capture immersive reality content data and store the immersive reality content data in the at least one memory;

[0034] at least one data transmitting apparatus configured to transmit a protocol data packet;

[0035] at least one data receiving apparatus configured to receive a protocol data packet; and

[0036] at least one data processing module configured to process immersive reality content data;

[0037] wherein

[0038] the protocol data comprises a data format encoding the immersive reality content data to contain time information.

[0039] The at least one processor parses the protocol data packet to obtain the time information and the immersive reality content data.

[0040] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium for storing the immersive reality content data.

[0041] Obviously, implementing any product or method of the present application does not necessarily need to achieve all the advantages mentioned above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0042] To more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 The data processing method for network user sharing immersive experience provided by the embodiment of the present application is shown in the figure.

[0044] Figure 2 The data processing module provided by the embodiment of the present application processes the immersive reality content data, as shown in the figure. DETAILED DESCRIPTION

[0045] The essence of the inventive idea of the present application will be further introduced below in combination with the drawings in the embodiments of the present application. It should be noted that the technical solutions contained in the following embodiments are intended to clearly describe the essential idea of the present application. Moreover, the specific embodiments listed are only one embodiment of the present application, not a comprehensive list. Those skilled in the art can obtain all other embodiments from the technical solutions described in the drawings and specific embodiments without creative labor, which still fall within the scope of protection of the present application. In particular, various configurations and combinations of the embodiments shown in the drawings naturally belong to the scope of protection of the present application.

[0046] In addition, the terms "contain", "include", "have" in the present application are intended to indicate a non-exclusive containing relationship; so that the technical solutions disclosed in the present application can contain other prior art content in addition to the processes, steps, methods, components, modules already listed. The terms "first", "second" are used to distinguish the description, and cannot be understood as indicating or implying relative importance. In the description of the embodiments, the concepts of first user equipment, second user equipment, etc. are used.

[0047] To achieve the above object, the embodiments of the present application provide a data processing method, a storage medium and a system for network users to share immersive experience. The data processing method, the storage medium and the system can be applied to various access network electronic devices, including but not limited to the client, the server and various hardware carrier forms of network node devices mentioned in the subsequent content of the present application. Generally, the network node device refers to an electronic device connected to the network with an independent address and data transmission or reception function.

[0048] As shown in Figure 1 One aspect 100 of the specific embodiments of the present application discloses a data processing method for network users to share immersive experience, which transmits immersive reality content data to users watching live or playback as much as possible without distinction, so as to obtain better immersive experience. Specifically, relying on streaming media technology, a first user device obtains immersive reality content data through a data generation module; the first user device encodes the immersive reality content data to generate protocol data packets containing time information; the first user device sends the protocol data packets to a second user device; the second user device receives the protocol data packets; the second user device parses the protocol data packets to obtain the time information and the immersive reality content data; and the second user device processes the immersive reality content data through a data processing module. Wherein, the immersive reality content data is to digitize the information of the real world, such as video data, audio data, and sensory data such as tactile data by means of sensors and mechanical devices.

[0049] Step S101: The first user device captures immersive reality content data through a data generation module.

[0050] The first user device in step S101 can be a network electronic device connected through the Internet, or a device in the communication network. In summary, the first user device has an independent address and identifiable identity information in the network. Typically, it is a game device for live streaming in network games. In the game process, the immersive reality content data brings the game user 111 an immersive feeling, which has become the main feature that distinguishes it from traditional digital games. The immersive feeling comes from multiple aspects, with multi-angle and multi-level perception characteristics 103, including visual, auditory, tactile, motion perception, etc.

[0051] In an embodiment of the present application, the immersive reality content data is video image data or sound data in a game scene, or vibration data of a gamepad. The following takes the vibration data of the gamepad 106 as an example to illustrate the generation manner of the immersive reality content data. In the game, all operations of the player are game triggering events (Events), the player presses the right move key "→", and the game character controlled by the player responds to the right move event. When the player's operation event needs to be fed back, the motor of the gamepad will vibrate. The player obtains immersive experience by vibration touch, and feels the game scene information. Generally, the gamepad has at least two motors, and different game events trigger specific motors. The two motors can correspond to different areas of the screen respectively, or correspond to different events in the game scene. Assuming that the two motors are identified as motor A and motor B respectively, there are three matching situations of responding to events, i.e. motor A, motor B, and motor A+motor B. Meanwhile, the rotation speed of the motor can also be set to different rpm values (Revolutions Per Minute rotation speed values) to produce different vibration intensities. In this way, the game player judges the character state in the game scene according to the touch of the gamepad vibration, and is immersed in the game scene. Therefore, the gamepad has a dual role, which not only serves as an input device, but also provides the player with a sense of substitution of the game scene as an immersive device.

[0052] The first user equipment captures the state and parameters of the motor A and the motor B in the gamepad as part of the immersive reality content data through the data generation module. Common capturing manners include: ① API Hook 104 can cross the access limit between different processes to capture function parameter values; and ② game programming library functions such as XInputSetState and XInputGetState are provided for third-party manufacturers to use. With the development of technology, motors composed of linear vibration modules have replaced traditional electric motors.

[0053] Step S102: The first user equipment encodes the immersive reality content data to generate protocol data packets containing time information.

[0054] The immersive reality content data captured by the first user equipment is packaged into data protocol packets conforming to the Stream transmission manner (stream media technology). In an embodiment of the present application, the immersive reality content data is sent together with audio and video data in the stream transmission manner, so that the code stream output by the audio and video encoder is sent according to a specific transmission protocol after encapsulation.

[0055] Take H.264 protocol as an example, a supplemental enhancement information field SEI (Supplemental Enhancement Information) is provided. The SEI field is not indispensable when the 108 video encoder outputs the video code stream, and the SEI can not be processed in the transmission, analysis and decoding process; however, in the actual application of stream media, it is an effective method for transmitting messages in the audio and video data, and does not cause the technical problem of abnormal audio and video data synchronization. According to the standard of H.264 protocol, the stream transmission is divided into two layers: video coding layer (Video Coding Layer, VCL for short) and network abstraction layer (Network Abstraction Layer, NAL for short). SEI is specified by H.264 protocol as NAL; further, H.264 protocol has specified the byte number range of SEI, so the byte number representing SEI is floating. The SEI field is specified in H.264, H.265, H.266 and various stream transmission technology protocols, and there is a unit with the same function.

[0056] In an embodiment of the present application, the immersive reality content data is transmitted in the SEI field, and also needs to contain time information. However, the SEI content itself does not have a timestamp, and the audio and video bare stream also does not contain a timestamp. However, the timestamp of each frame of bare stream will be attached in the common video packaging format, such as flv, ts, mp4, mov, mkv, webm……, and the timestamp is generally incremented with the system clock. The timestamp is divided into PTS and DTS. DTS (Decoding Time Stamp) is called decoding timestamp; the meaning of DTS timestamp is to inform the decoder of the time of decoding the data of this frame. PTS (Presentation Time Stamp) is called display timestamp, and the PTS timestamp is used to inform the decoder of the time of displaying the data of this frame. PTS is the play timestamp, and the frame corresponds to the timestamp at the play time. DTS is the decoding timestamp, which should be monotonically increasing. When the video bare stream has B frame (intermediate frame in bidirectional prediction), the PTS will not be monotonically increasing, and at this time, there is a CTS Offset (PTS and DTS time offset). The video bare stream (H264, H265, H266) data does not contain a timestamp, only the upper layer packaging container format (such as FLV, MP4, TS, WEBM, MKV, MOV) contains a timestamp. Because the vibration data is bound with the picture frame data, when it is packaged into the packaging container, the vibration data and the picture frame obtain and share the same timestamp.

[0057] In an embodiment of the present application, the first user equipment adds the immersive reality content data to the SEI field and adds the time offset of the CTS Offset without affecting the original audio and video stream. Thus, the data packet satisfying the requirement is obtained.

[0058] Step S103: The first user equipment sends the protocol data packet to the second user equipment.

[0059] The first user equipment sends the prepared protocol data packet to the second user equipment in the form of Stream.

[0060] In an embodiment of the present application, the sending is in the form of RTSP (Real Time Streaming Protocol). The first user equipment sends the data protocol packet to the RTMP (Real Time Message Protocol) server first, and then the RTMP server 110 sends the data protocol packet to the second user equipment. In the field of live broadcast, the game host sends the protocol data packet of the game information to the server, and then the server sends the data packet to the live broadcast audience.

[0061] Step S104: The second user equipment receives the protocol data packet.

[0062] The second user equipment receives the data protocol packet from the first user equipment. In an embodiment of the present application, the second user equipment receives the data protocol packet from the first user equipment through the RTMP server. After receiving the data packet, the second user equipment stores it in the receiving buffer according to the receiving order.

[0063] Step S105: The second user equipment parses the protocol data packet to obtain the time information and the immersive reality content data.

[0064] In an embodiment of the present application, the received data packet is unpacked according to the order of the RTP sequence number by the second user equipment. Taking the H.264 protocol as an example, the SEI field is output by the video encoder in the video code stream, and does not affect the playing of the audio and video data. The second user equipment decodes to obtain the immersive reality content data and the time information such as the CTS Offset in the SEI field.

[0065] Step S106: The second user equipment processes the immersive reality content data through the data processing module.

[0066] As Figure 2As shown, in an embodiment of the present application, the second user equipment processes the immersive reality content data through the data processing module to bring immersive experience to the user. For example, the second user equipment parses the handle vibration data in the immersive reality content data and calibrates according to the PTS. Thus, the live or recorded audience cannot feel the delay of the picture and vibration, and obtains the same experience as the host.

[0067] Corresponding to the method embodiment, the present application also provides a sending device of immersive reality content data. The sending device of immersive reality content data comprises a processor, a memory configured to provide instructions and operation data to the processor, a data generation device for capturing immersive reality content data and storing in the memory, and a data sending device for sending protocol data packet, wherein the protocol data contains data format with time information generated by encoding the immersive reality content data.

[0068] In an embodiment of the present application, the sending device of immersive reality content data accesses the Internet as a device in the Internet, has an independent address and identifiable identity information. The sending device captures the game player through the data generation device, and generates the immersive reality content data in the game process, including vision, hearing, touch, motion perception, etc.

[0069] Taking the game player obtaining immersive experience through game handle vibration in the game as an example, the motor of the game handle will vibrate; different vibration touch feeling is generated according to the information of different game scenes. Generally, the game handle contains at least two vibration motors. With the development of the present technology, the game handle also uses the vibration module technical means to replace the motor vibration. The data generation device of the sending device can capture the motor state and various parameters of the vibration module providing vibration feeling in the game handle, all of which are part of the immersive reality content data. Common capture methods: ① API Hook can cross the access limit between different processes to capture function parameter value; ② provide game programming library functions for third-party manufacturers, such as XInputSetState, XInputGetState, etc.

[0070] The processor of the sending device edits the immersive reality content data into a data format conforming to the requirements of the supplemental enhancement information (SEI) field according to the provisions of the H.264 protocol. The SEI field is specified in H.264, H.265, H.266, and various stream transmission technology protocols, or units with the same function exist. At the same time, the SEI content itself does not have a timestamp, and the audio and video raw stream also does not contain a timestamp. However, the raw stream of each frame does not contain a timestamp in common video packaging formats such as flv, ts, mp4, mov, mkv, webm, and the like. The H264, H265, and H266 video raw stream data do not contain a timestamp, and only the upper packaging container format (such as FLV, MP4, TS, WEBM, MKV, and MOV) contains a timestamp. Because the vibration data is bound to the picture frame data, when the vibration data is packaged into the packaging container, the vibration data and the picture frame obtain and share the same timestamp. The sending device adds the immersive reality content data to the SEI field and adds a time offset of the CTS Offset. Thus, a data packet (Data Packet) that meets the requirements is obtained.

[0071] After completing the data packet packaging operation conforming to the H.264 protocol, the data sending device of the sending device sends the data packet to the server in the RTSP manner.

[0072] Corresponding to the method embodiments described above, the present application also provides a receiving device for immersive reality content data. The receiving device for immersive reality content data comprises a processor, a memory configured to provide instructions and operation data to the processor, a data receiving device for receiving a protocol data packet, and a data processing module for processing immersive reality content data. The processor parses the protocol data packet to obtain the time information and the immersive reality content data.

[0073] In an embodiment of the present application, the receiving device receives the data protocol packet through an RTMP server and stores it in a receiving buffer in the order of reception. Then, the receiving device unpacks in the order of the RTP sequence number. Taking the H.264 protocol as an example, the SEI field is output by the video encoder in the video code stream and does not affect the playback of the audio and video data. Moreover, the immersive reality content data in the SEI field and the time information, such as the CTS Offset, can be obtained.

[0074] The receiving device processes the immersive reality content data through the data processing module to bring an immersive experience to the user. For example, the second user equipment parses the handle vibration data in the immersive reality content data and calibrates according to the PTS. Thus, the live or recorded audience does not feel the delay between the picture and the vibration, and obtains the same experience as the host.

[0075] Corresponding to the method embodiments described above, the present application also provides a system for network users to share immersive experience. The system for network users to share immersive experience comprises at least one processor, at least one memory coupled to the at least one processor, at least one data generating device for capturing immersive reality content data and storing the data in the at least one memory, at least one data sending device for sending protocol data packets, at least one data receiving device for receiving protocol data packets, and at least one data processing module for processing immersive reality content data, wherein the protocol data contains data format with time information generated by encoding the immersive reality content data, and the at least one processor parses the protocol data packets to obtain the time information and the immersive reality content data.

[0076] In an embodiment of the present application, the system accesses the Internet as a device in the Internet, has an independent address and identifiable identity information. The sending device captures the game player through the data generating device, and the immersive reality content data generated during the game includes visual, auditory, tactile, motion perception, etc.

[0077] The at least one data generating device is used to capture immersive reality content data and store the data in the at least one memory. Taking the game player obtaining immersive experience through gamepad vibration in the game as an example, the motor of the gamepad will vibrate; according to the information of different game scenes, different vibration touch feelings are generated. Generally, the gamepad contains at least two vibration motors. With the development of current technology, some gamepads also use vibration module technology to replace the motor vibration. The data generating device of the sending device can capture the motor state and various parameters of the vibration module providing vibration feeling in the gamepad as part of the immersive reality content data. Common capturing methods are: ① API Hook can be used to cross the access limit between different processes to capture function parameter values; ② game programming library functions can be provided for third-party manufacturers, such as XInputSetState, XInputGetState, etc.

[0078] At least one data sending device is used to send a protocol data packet. According to the agreement of H.264 protocol, the data format is edited to meet the requirements of the Supplemental Enhancement Information (SEI) field. The SEI field is specified in H.264, H.265, H.266 and various Stream transmission technology protocols, or there is a unit with the same function. At the same time, the SEI content itself does not have a timestamp, and the audio and video raw stream also does not contain a timestamp. However, the raw stream of each frame does not contain a timestamp in the common video packaging format, such as flv, ts, mp4, mov, mkv, webm… The raw stream (H264, H265, H266) data does not contain a timestamp, and only the upper packaging container format (such as FLV, MP4, TS, WEBM, MKV, MOV) contains a timestamp. Because the vibration data is bound to the picture frame data, when it is packaged into a packaging container, the vibration data and the picture frame obtain and share the same timestamp. The sending device adds immersive reality content data to the SEI field and adds a time offset of CTS Offset. Thus, the required data packet (Data Packet) is obtained. After completing the data packet packaging operation in accordance with the H.264 protocol, the data sending device of the sending device sends the data packet to the server in the RTSP manner.

[0079] At least one data receiving device is used to receive a protocol data packet. The data protocol packet is received through an RTMP server; and stored in a receiving buffer in the order of reception. Then, the receiving device unpacks in the order of the RTP sequence number. Taking the H.264 protocol as an example, the SEI field is output by the video encoder in the video code stream, which does not affect the playback of audio and video data; and the immersive reality content data in the SEI field and the time information such as CTS Offset can be obtained.

[0080] The immersive reality content data is processed by a data processing module to bring an immersive experience to a user. For example, the second user equipment analyzes the handle vibration data in the immersive reality content data and calibrates according to the PTS. Thus, the live or recorded audience cannot feel the delay of the picture and the vibration, and obtains the same experience as the host.

[0081] The application also provides a computer readable storage medium for storing program code instructions and operation data corresponding to the above method embodiments.

[0082] In an embodiment of the application, the computer readable storage medium includes various computer readable or electronic device readable storage devices, such as DVD optical disc, Blu-ray optical disc, CD optical disc, floppy disk, magnetic disk, magnetic disc, random access memory (RAM), read-only memory (ROM) and erasable programmable read-only memory (E2PROM), storage device smart card, flash memory, magnetic tape, etc.

[0083] For example, the program code on the storage medium can be written in any combination of one or more programming languages used to implement the present application, and the computer program design languages involved include but are not limited to C, C++, C#, Java, Delphi, etc. The program design language can run on a computer, or run on multiple computers in a distributed computing manner, or run on a server. When a remote computer is involved, the remote computer can be connected through any kind of network, including a local area network and a wide area network.

[0084] The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0085] It should be noted that the terms "first", "second" and the like used in the present application are only used for distinguishing purpose, and are used to distinguish one entity, operation, step, component or functional module from another entity, operation, step, component or functional module, and do not imply any relative or absolute relationship, order or importance between the defined entities, operations, steps, components or functional modules. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0086] Various embodiments are described in related manner in the specification, and the same or similar parts among various embodiments can refer to each other, and each embodiment focuses on the difference from other embodiments. Especially, for the device embodiment, the equipment embodiment, the computer readable storage medium embodiment, and the computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the related parts can refer to the part of the method embodiment.

[0087] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method for sharing immersive experiences among network users, characterized in that, include: The first user device captures immersive reality content data through a data generation module, the immersive reality content data including game controller vibration data; The first user equipment encodes the immersive reality content data into a protocol data packet containing time information. The immersive reality content data is transmitted in the supplemental enhancement information field of the transmission protocol. The immersive reality content data is transmitted together with the audio and video data. The time information is obtained by encapsulating the immersive reality content data and the audio and video data into the same encapsulation container. The immersive reality content data and the screen frame share the same timestamp. The first user equipment sends the protocol data packet to the second user equipment; The second user equipment receives the protocol data packet; The second user equipment parses the protocol data packets to obtain the time information and the immersive reality content data; The second user equipment processes the immersive reality content data through a data processing module.

2. The data processing method for sharing immersive experiences among network users according to claim 1, characterized in that, The immersive reality content data also includes audio or video data.

3. The data processing method for sharing immersive experiences among network users according to claim 2, characterized in that, The protocol data packets conform to the H.264, H.265, or H.266 protocols.

4. The data processing method for sharing immersive experiences among network users according to claim 3, characterized in that, The game controller vibration data is generated by the game controller vibration motor or game controller vibration module.

5. A device for transmitting immersive reality content data, characterized in that, include: processor, A memory configured to provide instructions and operational data to the processor; A data generation device for capturing immersive reality content data and storing it in the memory, the immersive reality content data including game controller vibration data; and, Data transmission device, used to transmit protocol data packets; in, The protocol data packet contains a data format that encodes the immersive reality content data into time information. The immersive reality content data is transmitted in the supplemental enhancement information field of the transmission protocol. The time information is obtained by encapsulating the immersive reality content data and audio / video data into the same encapsulation container. The immersive reality content data and the screen frames share the same timestamp.

6. The immersive reality content data transmission device according to claim 5, characterized in that, The immersive reality content data also includes audio or video data.

7. The immersive reality content data transmission device according to claim 6, characterized in that, The protocol data packets conform to the H.264, H.265, or H.266 protocols.

8. The immersive reality content data transmission device according to claim 7, characterized in that, The game controller vibration data is generated by the game controller vibration motor or game controller vibration module.

9. A receiving device for immersive reality content data, characterized in that, include: processor; A memory configured to provide instructions and operational data to the processor; as well as, A data receiving device for receiving protocol data packets; A data processing module is used to process immersive reality content data, including game controller vibration data. in, The processor parses the protocol data packet to obtain time information and the immersive reality content data. The immersive reality content data is transmitted in the supplemental enhancement information field of the transmission protocol. The time information is obtained by encapsulating the immersive reality content data and audio / video data into the same encapsulation container. The immersive reality content data and the screen frame share the same timestamp.

10. The receiving device for immersive reality content data according to claim 9, characterized in that, The immersive reality content data also includes audio or video data.

11. The receiving device for immersive reality content data according to claim 10, characterized in that, The protocol data packets conform to the H.264, H.265, or H.266 protocols.

12. The receiving device for immersive reality content data according to claim 11, characterized in that, The game controller vibration data is generated by the game controller vibration motor or game controller vibration module.

13. A system for sharing an immersive experience among network users, characterized in that, include: At least one processor; At least one memory, said at least one memory being coupled to said at least one processor; At least one data generation device is used to capture immersive reality content data and store it in the at least one memory, the immersive reality content data including game controller vibration data; At least one data transmitting device is used to transmit protocol data packets; At least one data receiving device is used to receive protocol data packets; as well as, At least one data processing module for processing immersive reality content data; in, The protocol data packet contains a data format that encodes the immersive reality content data into a time information format. At least one of the processors parses the protocol data packet to obtain the time information and the immersive reality content data. The immersive reality content data is transmitted in the supplemental enhancement information field of the transmission protocol. The immersive reality content data is transmitted together with the audio and video data. The time information is obtained by encapsulating the immersive reality content data and the audio and video data into the same encapsulation container. The immersive reality content data and the frame share the same timestamp.

14. The system for sharing an immersive experience among network users according to claim 13, characterized in that, The immersive reality content data also includes audio or video data.

15. The system for sharing an immersive experience among network users according to claim 14, characterized in that, The protocol data packets conform to the H.264, H.265, or H.266 protocols.

16. The system for sharing an immersive experience among network users according to claim 15, characterized in that, The game controller vibration data is generated by the game controller vibration motor or game controller vibration module.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code instructions, which, when executed by a computer, implement the method as described in any one of claims 1 to 4.

18. A computer program product comprising program code instructions, wherein when the program product is executed by a computer, the program code instructions cause the computer to perform the method of any one of claims 1 to 4.

Images