Data processing method, system and device

Abstract

The embodiment of the invention provides a data processing method, system and device, and relates to the technical field of media, and the method comprises the steps: terminal equipment can obtain the format information of perception and interaction data supported by display equipment, determines the format of the perception and interaction data to be transmitted based on the format information, and transmits the perception and interaction data to the terminal equipment; therefore, the transmission of the sensing and interaction data in the format with the display device is realized. Therefore, no matter how the format of the perception and interaction data supported by the display equipment is, the terminal equipment can be compatible with the display equipment to transmit the perception and interaction data, so that the acquisition of the image data is realized, and the compatibility of the terminal equipment with various display equipment is realized.

Description

[0001] This application claims priority to the Chinese patent application No. 202410476542.X, filed on April 17, 2024, entitled "Data processing method, system and device", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of media technology, and in particular to a data processing method, system and device. BACKGROUND

[0003] With the continuous development of AR (Augmented Reality) and VR (Virtual Reality) technologies, VR / AR movies, VR / AR games and VR / AR product demonstrations have gradually entered the user's field of vision.

[0004] The current box terminal is used in conjunction with AR and VR devices, which makes it difficult for the current box terminal to be compatible with various display devices. SUMMARY

[0005] To solve the above technical problems, the present application provides a data processing method and device. In the method, the terminal device can obtain the format information of the perception and interaction data supported by the display device, and determine the format of the perception and interaction data to be transmitted based on the format information, so as to realize the transmission of the perception and interaction data of the format between the terminal device and the display device. In this way, no matter what format of the perception and interaction data supported by the display device is, the terminal device can be compatible with the display device to transmit the perception and interaction data, and further realize the acquisition of image data, thereby realizing the compatibility with various display devices.

[0006] In one possible implementation, the present application provides a data processing method. A terminal device is in communication connection with a display device, and the method comprises: the terminal device receives first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; the terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; the terminal device receives the perception and interaction data of the first format from the display device; and the terminal device obtains multimedia data based on the perception and interaction data.

[0007] The format information provided by the first capability information can be information carried in the first capability information, or information obtained based on the first capability information. For example, the first capability information carries information about multiple resolutions of the perception and interaction data supported by the display device, and the format information provided by the first capability information is not limited to the multiple resolutions of the perception and interaction data, but can further include information about a range of resolutions defined by the multiple resolutions of the perception and interaction data, so as to determine at least one of a maximum resolution and a minimum resolution of the perception and interaction data supported by the display device.

[0008] The terminal device of the present application can obtain the first capability information from the display device, the first capability information can provide format information of the perception and interaction data supported by the display device (which can be perception and interaction data collected or results obtained by processing collected perception and interaction data), and determine the format of the perception and interaction data to be transmitted by the terminal device within the capability range of the display device based on the format information. Then, the perception and interaction data received by the terminal device from the display device is also in the determined format. Therefore, regardless of the format of the perception and interaction data supported by the display device, the terminal device of the present application can compatibly receive the perception and interaction data from the display device, so that the terminal device of the present application can be compatible with display devices of various manufacturers and models to transmit perception and interaction data.

[0009] In this way, the terminal device of the present application can be compatible with wearable display devices (such as AR glasses, VR glasses, AR helmets, VR helmets, etc.) of various data transmission protocols, to realize immersive virtual-real fusion and spatial interaction, and meet the experience of VR, AR, etc. In addition, the terminal device 100 can also be compatible with ordinary display devices (such as displays, televisions, vehicle display screens, projectors, etc.) of various data transmission protocols, so that the terminal device can be used as a portable micro host.

[0010] For example, a user can use one terminal device to cooperate with display devices (such as display screens, projectors, vehicle display screens, AR, VR, etc.) of different manufacturers and different specifications. It can also be used with different perception units (3DOF / 6DOF / grid / illumination / materials), different interaction units (gestures / eye movements / handheld devices / keyboards, etc.).

[0011] In one possible implementation, the terminal device obtains multimedia data based on the perception and interaction data, including: the terminal device obtains multimedia data by performing inference on the perception and interaction data through an artificial intelligence (AI) module.

[0012] The terminal device can include an AI module, and the terminal device can use the AI module to infer multimedia data based on the perception and interaction data (or a calculation result of the perception and interaction data).

[0013] In some scenarios, the terminal device can also interact with a cloud server. For example, the terminal device can send the perception and interaction data to the cloud to request the cloud to perform AI inference, so as to obtain multimedia data returned by the cloud.

[0014] That is, the terminal device can use an AI module of the terminal device itself or an AI module of the cloud to perform inference to obtain the multimedia data. The use of a single-side AI module or a double-side AI module can be flexibly selected according to the computing power and complexity of the inference. In this way, the display effect of the obtained multimedia data can be improved.

[0015] In a possible implementation, the terminal device obtains multimedia data based on the perception and interaction data, including: the terminal device performs inference on the perception and interaction data by using an AI module to obtain a first multimedia element; the terminal device renders a second multimedia element based on the perception and interaction data; and the terminal device obtains multimedia data based on the first multimedia element and the second multimedia element.

[0016] Similarly, the AI module can be an AI module of the terminal device or an AI module of a server, which is not limited herein.

[0017] The first multimedia element and the second multimedia element can be any multimedia element such as an image, a text, a video, a model, and an audio.

[0018] The terminal device can process (for example, synthesize, which is not limited) the first multimedia element and the second multimedia element to obtain the multimedia data.

[0019] In this way, the embodiments of the present application can obtain multimedia data by using rendering and AI inference. The text and the image can be used as elements displayed in the multimedia data, the audio in the audio and the video can be used as audio in the multimedia data, and the model in the first multimedia element and / or the second multimedia element can be used as a two-dimensional model or a three-dimensional model in the multimedia data.

[0020] In a possible implementation, the multimedia data includes at least one of the following: an image, a video, and a model.

[0021] In a possible implementation, before the terminal device receives the perception and interaction data in the first format from the display device, the method further includes: the terminal device receiving second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; the terminal device determining second link information based on the second capability information and the first format; and the terminal device configuring the first transmission link based on the second link information.

[0022] The first link information can include, but is not limited to, the number (for example, the maximum number) of open transmission lanes, the transmission rate of the lanes, and the like.

[0023] The second capability information can provide first link information supported by the display device for a first transmission link for transmitting the perception and interaction data, which can include, but is not limited to, the number (for example, the maximum number) of open transmission lanes, the transmission rate of the lanes, the maximum transmission rate (for example, the transmission rate corresponding to x8), and the like.

[0024] The terminal device 100 can determine second link information according to the format (the first format) of the perception and interaction data within the maximum capability of the display device for the first transmission link, and configure the first transmission link based on the second link information.

[0025] The second link information can include information indicating whether to start decompression and decoding, and optionally, at least one of the following: the number of lanes (for example, x4) allocated to the first transmission link, the reception frequency of the data, the format of the data transmitted by the first transmission link, and the like.

[0026] For example, the display device supports 8 transmission lanes for the perception and interaction data, and the terminal device supports 4 reception lanes for the perception and interaction data, then according to the first format of the perception and interaction data to be transmitted, a corresponding number of transmission lanes (up to 4) can be selected from 1 to 4 transmission lanes, to transmit the perception and interaction data through the first transmission link, to realize the compatibility of the transmission link of the perception and interaction data of the terminal device to various display devices.

[0027] In a possible implementation, the second link information includes information indicating to start decompression, and the terminal device receiving the perception and interaction data in the first format from the display device includes: the terminal device receiving, through the configured first transmission link, a first code stream of the perception and interaction data in the first format compressed and encoded by the display device; and the method further includes: the terminal device decoding the first code stream based on the second link information to obtain the perception and interaction data in the first format.

[0028] The terminal device can determine a data transmission rate of the perception and interaction data to be transmitted based on the first format, and determine a maximum transmission rate of the transmission link of the perception and interaction data of the display device based on the second capability information. When the maximum transmission rate is less than the data transmission rate, it indicates that the current bandwidth cannot meet the transmission requirement of a large amount of perception and interaction data, and compression encoding is required.

[0029] For example, when the terminal device 100 determines that the maximum data amount max4 (determined based on the first format) is greater than the maximum data amount max3 (determined based on the first link information) that the transmission link of the perception and interaction data of the display device 200 supports, it indicates that the maximum bandwidth of the transmission link of the display device 200 cannot support the transmission of the perception and interaction data with the data amount max4 to the terminal device 100, which exceeds the transmission capability of the transmission link of the perception and interaction data of the display device 200. Then, the terminal device 100 can configure to start decompression and decoding of the transmitted perception and interaction data when configuring the transmission link of the local perception and interaction data, wherein the data amount of the compressed and encoded perception and interaction data is less than or equal to the maximum data amount max3.

[0030] In this way, the display device 200 can compress and encode the perception and interaction data to be transmitted, and then transmit the compressed and encoded data to the terminal device 100 through the transmission link of the perception and interaction data, so that the transmission of the perception and interaction data can be realized under low latency and low bandwidth even when the data amount of the perception and interaction data to be transmitted by the display device 200 is large. By encoding and decoding the perception and interaction data, higher compression efficiency can be obtained, and the code rate and the required bandwidth for transmission can be reduced, and the latency can be reduced.

[0031] In a possible implementation, the method further includes: receiving, by the terminal device, third capability information, wherein the third capability information provides format information of display data supported by the display device; determining, by the terminal device based on the third capability information, a second format of display data to be transmitted, wherein the second format includes an image format, and the image format is an image format after synthesis of a virtual image and a real scene image, or an image format of a virtual image; and obtaining, by the terminal device based on the perception and interaction data, multimedia data, including: obtaining, by the terminal device based on the perception and interaction data, a first image sequence in the second format.

[0032] The multimedia data can include a first image sequence. The first image sequence can be one frame of image, or multiple frames of image.

[0033] For example, the first image sequence can be a monocular image (an example of one frame of image), or a binocular or multi-view image (an example of multiple frames of image).

[0034] In addition, the first image sequence obtained by the terminal device is also described as the to-be-displayed data. The first image sequence can be generated by the terminal device itself based on the perception and interaction data in the first format, or the first image sequence can be generated by the cloud based on the perception and interaction data in the first format, so that the terminal device can obtain the first image sequence from the cloud.

[0035] Unlike the terminal device in the prior art, the terminal device in the present application can receive third capability information of the display device. The third capability information can provide format information of display data supported by the display device. The format information can include not only image formats but also resolutions, frame rates, and the like of to-be-displayed data. Then, the terminal device can determine a second format of the to-be-transmitted display data based on the third capability information. The second format is also within the format information provided by the third capability information. The second format can be a virtual-real fused image format (for example, an RGB format) or a virtual image format (for example, an RGBAZ format or an RGBA format). In this way, the terminal device can obtain display data (for example, first image data) in the second format based on the format of the display data supported by the display device. Thus, the terminal device in the present application can be compatible with sending a virtual-real fused image to the display device for display on the display device side, and can also be compatible with sending a virtual image to the display device for the display device to realize fusion and display of virtual-real images.

[0036] In a possible implementation, the method further includes: the terminal device receiving fourth capability information, wherein the fourth capability information provides third link information supported by the display device for a second transmission link; the terminal device determining fourth link information based on the fourth capability information and the second format; and the terminal device configuring the second transmission link based on the fourth link information.

[0037] The third link information can be the number (for example, the maximum number) of open transmission lanes, the transmission rate of the lanes, and the like.

[0038] The fourth capability information can provide third link information supported by the display device for a second transmission link for transmitting display data. The third link information can include but is not limited to the number (for example, the maximum number) of open transmission lanes, the transmission rate of the lanes, and the like, and the maximum transmission rate (for example, the transmission rate corresponding to x8).

[0039] Then, the terminal device 100 can determine fourth link information according to the format (the second format described above) of the display data within the maximum capability of the display device for the second transmission link, and configure the second transmission link based on the fourth link information.

[0040] The fourth link information can include information indicating whether to start compression decoding, and optionally, at least one of the following: a number of channels allocated to the second transmission link (e.g., x4), a receiving frequency of data, a transceiving data format of the second transmission link, etc.

[0041] For example, the display device supports 8 transmission channels for display data, and the terminal device supports 4 receiving channels for display data, the terminal device can select a corresponding number of transmission channels (up to 4) in 1-4 transmission channels according to the second format of the display data to be transmitted, and transmit the display data through the second transmission link, so as to realize the compatibility of the terminal device to the transmission link of the display data of various display devices.

[0042] In a possible implementation, the fourth link information includes information indicating to start compression, and the method further includes: the terminal device compressively encodes the first image sequence based on the fourth link information to obtain a second code stream of the first image sequence; and the terminal device sends the second code stream to the display device through the configured second transmission link.

[0043] The terminal device can determine a data transmission rate of the display data to be transmitted based on the second format, and determine a maximum transmission rate of the display link of the display device based on the fourth capability information. When the maximum transmission rate is less than the data transmission rate, it indicates that the current bandwidth cannot meet the transmission requirement of a large amount of display data, and compression encoding is required.

[0044] For example, the terminal device 100 can determine the fourth link information according to the format of the display data to be displayed within the maximum capability of the display device to the transmission link, and configure the transmission link for transmitting the display data based on the fourth link information.

[0045] The fourth link information can include information indicating whether to start compression decoding, and optionally, at least one of the following: a number of channels allocated to the second transmission link (e.g., x4), a receiving frequency of data, a transceiving data format of the second transmission link, etc.

[0046] When the terminal device 100 determines that the maximum data amount max2 is greater than the maximum data amount maxl that can be transmitted by the display data transmission link of the display device 200, it indicates that the maximum bandwidth of the display device 200 to the transmission link cannot support the transmission of the data amount max2 of the display data from the terminal device 100. Then, the terminal device 100 can configure to start compression encoding of the display data to be transmitted, so that the data amount of the compressed display data to be transmitted is less than or equal to the maximum data amount maxl.

[0047] In this way, when the maximum capability of the transmission link of the display peripheral 200 for transmitting the display data to be displayed is less than the maximum data amount max2 of the display data to be displayed sent by the terminal device 100 through the local display interface, the terminal device 100 can compress and encode the display data to be transmitted, and send the compressed and encoded data to the display peripheral 200 through the transmission link of the display data, so that the transmission of the display data can be realized in the case of low latency and low bandwidth when the data amount of the display data to be displayed is large. Higher compression efficiency can be obtained by encoding and decoding the display data (the first image sequence), and the code rate and the required bandwidth for transmission are reduced, and the latency is reduced.

[0048] In a possible implementation, the method further includes: the terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain a code stream of the first image sequence.

[0049] In the embodiments of the present application, the display data can be compressed in combination with the perception and interaction data, so as to improve the compression efficiency, reduce the code rate and the required bandwidth for transmission, and reduce the latency.

[0050] In a possible implementation, the first image sequence includes at least two-view images, the at least two-view images include a first-view image and a second-view image, and the perception and interaction data include first-view data.

[0051] The first-view image and the first-view data correspond to the same view, for example, both are left eyes.

[0052] In a possible implementation, the perception and interaction data further include second-view data, and the terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain a second code stream of the first image sequence, including:

[0053] The terminal device compresses and encodes the first-view image based on the first-view data, and compresses and encodes the second-view image based on the second-view data to obtain the second code stream of the first image sequence.

[0054] The first-view image and the first-view data correspond to the same view, for example, both are left eyes.

[0055] The second-view image and the second-view data correspond to the same view, for example, both are right eyes.

[0056] In the embodiments, in a binocular or multi-view (more than binocular) scenario, the corresponding display data for different views can be compressed and encoded in combination with the perception and interaction data, so as to reduce the code rate and the latency.

[0057] In a possible implementation, the terminal device encodes the first image sequence based on the perception and interaction data to obtain a second code stream of the first image sequence, including: the terminal device encodes the first image to obtain a code stream of the first image and a reconstructed image of the first image; the terminal device pre-processes the reconstructed image based on the first target data to obtain a reference image matched with the second image; and the terminal device encodes the second image based on the reference image to obtain a code stream of the second image.

[0058] In the embodiments of the present application, the reconstructed image of the first image can be pre-processed (for example, morphing conversion) by means of the perception and interaction data, so that the reference image obtained by pre-processing is more close to the second image in pixel content. Then, the second image is encoded based on the reference image, which can improve the compression rate of the second image. In this way, the second image can be compressed by using the redundant information of the dual-purpose, so as to improve the compression efficiency of the first image sequence.

[0059] In a possible implementation, the first format includes at least one of the following: a data format (for example, a type of supported sensor), a resolution (for example, a resolution of image data generated by the sensor), a frame rate (for example, a frame rate of image data generated by the sensor), and a number of eyes (for example, a number of eyes corresponding to the sensor).

[0060] In a possible implementation, the second format includes at least one of the following: a data format (for example, an RGB format, an RGBA format, an RGBAZ format, etc.), a resolution (for example, a resolution of display data), a frame rate (for example, a frame rate of display data), and a number of eyes (for example, a number of eyes corresponding to the display data).

[0061] In a possible implementation, the terminal device determines the first format of the perception and interaction data to be transmitted based on the first capability information, including: the terminal device determines the first format of the perception and interaction data to be transmitted based on the first capability information and target information, where the target information is at least one of a computing capability of the perception and interaction data and an application scenario.

[0062] The terminal device 100 can set the format of the perception and interaction data to be transmitted within the format range of the perception and interaction data supported by the display device 200 in combination with at least one of an application scenario (for example, a rendering requirement of an application) and a computing capability (referred to as computing power) of the terminal device 100 for the perception and interaction data. In this way, the display device can cooperate with the terminal device in different computing power and application scenarios to realize transmission of the perception and interaction data in different formats.

[0063] In a possible implementation, the terminal device determines the second format of the display data to be transmitted based on the third capability information, including: the terminal device determines the second format of the display data to be transmitted based on the third capability information and target information, wherein the target information is at least one of the computing capability and application scenario of the display data.

[0064] The terminal device 100 can set the second format of the display data to be transmitted within the range of display data formats supported by the display peripheral device 200, based on the application scenario and at least one of the terminal device 100's computing power for displaying the data (referred to as computing power). In this way, the same display device, such as mixed reality (MR) glasses, can be driven by terminal devices with different computing powers (or in different application scenarios) to achieve different display effects (such as full load rendering / center rendering / foveated rendering) of the first image sequence.

[0065] In a possible implementation, the first capability information provides format information of perception and interaction data collected and supported by the display device, or format information of perception and interaction data calculated and supported by the display device.

[0066] Among them, when the display device has strong computing power and supports calculation of the collected perception and interaction data, the format information of the calculated perception and interaction data (also called the calculation results of the perception and interaction data) supported by it can be provided to the terminal device through the first capability information.

[0067] In a possible implementation, the method may further include: the terminal device collecting perception and interaction data; the terminal device obtaining a first image sequence based on the collected perception and interaction data and the received perception and interaction data in the first format.

[0068] like Figure 3d As shown, not only can the XR helmet 201 shoot the surrounding environment (taking the user as an example here) to obtain perception and interaction data and transmit it to the terminal device 100, but also because the terminal device 100 has N camera modules, the terminal device 100 can also shoot the surrounding environment (taking the user as an example). In this way, the perception and interaction data based on which the terminal device 100 generates data to be displayed comes not only from the XR helmet 201, but also from the image data shot by the terminal device 100 itself; finally, the terminal device 100 can send the obtained data to be displayed (generated by the terminal device 100 or generated by the cloud and obtained from the cloud) to the XR helmet 201 for output.

[0069] In this way, the terminal device 100 can cooperate with the XR helmet to cooperatively collect perception and interaction data, so as to meet the collection requirements of perception and interaction data in AR, VR, MR and other scenarios.

[0070] In a possible implementation, the present application provides a data processing method. The method comprises: a display device sending first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; the display device obtaining perception and interaction data in a first format, wherein the format information comprises the first format; the display device sending the perception and interaction data in the first format; and the display device receiving multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

[0071] For example, the display device can send the first capability information to the terminal device, or the display device can send the first capability information to the terminal device through the cloud, which is not limited here.

[0072] Similarly, the display device can receive the first image sequence from the terminal device, or the display device can receive the first image sequence from the cloud, which is not limited here.

[0073] The display device of the present application can be compatible with terminal devices of different manufacturers and different models, so as to realize compatibility between the display device and the terminal device.

[0074] In a possible implementation, before the display device obtains the perception and interaction data in the first format, the method further comprises: the display device receiving the first format; and the display device initializing and setting a sensor for collecting perception and interaction data according to the first format.

[0075] The display device can receive the first format from the terminal device, or the display device can receive the first format from the cloud, which is not limited here.

[0076] In this way, the display device can initialize and set the sensor according to the format of the perception and interaction data set by the terminal device, so that the initialized sensor can collect the perception and interaction data in the first format.

[0077] In a possible implementation, the method further comprises: the display device sending second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; the display device receiving second link information, wherein the second link information is determined based on the second capability information and the first format; and the display device configuring the first transmission link based on the second link information.

[0078] The display device can send the second capability information to the terminal device, send the second capability information to the terminal device through a cloud, or send the second capability information to the cloud, which is not limited here.

[0079] The method can realize compatibility of the transmission link of the perception and interaction data between the display device and the terminal device.

[0080] In a possible implementation, the second link information includes information indicating starting compression or decompression, and the method further includes: the display device performs compression encoding on the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; and the display device sends the first code stream through the first transmission link configured.

[0081] In this way, the display peripheral 200 can perform compression encoding on the perception and interaction data to be sent, and then send the compressed and encoded data through the transmission link of the perception and interaction data, for example, to the terminal device 100, so that the transmission of the perception and interaction data can be realized in a low latency and low bandwidth case when the amount of the perception and interaction data to be sent by the display peripheral 200 is large. The compression efficiency of the perception and interaction data is higher through encoding and decoding, the code rate and the required bandwidth for transmission are reduced, and the latency is reduced.

[0082] In a possible implementation, the perception and interaction data in the first format includes a real scene image and image data other than the real scene image, and the display device performs compression encoding on the obtained perception and interaction data in the first format to obtain a first code stream, including: the display device performs compression encoding on the real scene image based on the image data to obtain the first code stream.

[0083] For example, the real scene image can be VST data (VST data collected by a VST sensor), and the image data other than the real scene image can include, but is not limited to, image data collected by other perception and interaction sensors. The other perception and interaction sensors are sensors (for example, a black-and-white camera, a depth camera, etc.) other than the sensor used to collect the real scene image.

[0084] The image data other than the real scene image can be at least one of an eye movement image and a depth image. The display device can calculate eye movement information based on the eye movement image, calculate depth information based on the depth image, and use at least one of the eye movement information and the depth information to perform compression encoding on the real scene image, so as to improve the compression rate of the real scene image and reduce the code rate.

[0085] In a possible implementation, the present application provides a data processing method, a terminal device is in communication connection with a display device, and the method comprises the following steps: the display device sends first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; the terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; the display device sends the perception and interaction data in the first format to the terminal device; the terminal device obtains multimedia data based on the perception and interaction data in the first format; the terminal device sends the multimedia data to the display device; and the display device displays an image based on the multimedia data.

[0086] In a possible implementation, before the display device sends the perception and interaction data in the first format to the terminal device, the method further comprises the following steps: the terminal device sends the first format to the display device; the display device sets the sensor for collecting perception and interaction data according to the first format; and the display device obtains the perception and interaction data in the first format based on the sensor after the setting.

[0087] In a possible implementation, the method further comprises the following steps: the display device sends second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for a first transmission link; the terminal device determines second link information based on the second capability information and the first format; the terminal device sends the second link information to the display device; and the display device configures the first transmission link based on the second link information.

[0088] In a possible implementation, the second link information comprises information indicating starting decompression, and the method further comprises the following steps: the display device performs compression encoding on the perception and interaction data in the first format obtained based on the second link information, to obtain a first code stream; the display device sends the first code stream to the terminal device through the configured first transmission link; and the terminal device decodes the first code stream based on the second link information, to obtain the perception and interaction data in the first format.

[0089] The effects of the method of each implementation of the terminal device and the display device are similar to those of the data processing method executed by the terminal device and the data processing method executed by the display device in the above implementations, and thus will not be described here.

[0090] In a possible implementation, the present application provides a data processing system. The system comprises a terminal device and a display device connected in communication; the display device is configured to send first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; the terminal device is configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; the display device is further configured to send the perception and interaction data in the first format to the terminal device; the terminal device is further configured to obtain multimedia data based on the perception and interaction data in the first format; the terminal device is further configured to send the multimedia data to the display device; and the display device is further configured to display an image based on the multimedia data.

[0091] In a possible implementation, the terminal device is further configured to send the first format to the display device; the display device is further configured to initialize a sensor for collecting perception and interaction data according to the first format; and the display device is further configured to obtain the perception and interaction data in the first format based on the initialized sensor, and send the perception and interaction data in the first format to the terminal device.

[0092] In a possible implementation, the display device is further configured to send second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for a first transmission link; the terminal device is further configured to determine second link information based on the second capability information and the first format; the terminal device is further configured to configure the first transmission link based on the second link information; the terminal device is further configured to send the second link information to the display device; and the display device is further configured to configure the first transmission link based on the second link information.

[0093] In a possible implementation, the second link information comprises information indicating starting decompression; the display device is specifically configured to compress and encode the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; the display device is specifically configured to send the first code stream to the terminal device through the configured first transmission link; and the terminal device is specifically configured to decode the first code stream based on the second link information to obtain the perception and interaction data in the first format.

[0094] In a possible implementation, the display device is further configured to send third capability information to the terminal device, where the third capability information provides format information of display data supported by the display device; the terminal device is further configured to determine a second format of display data to be transmitted based on the third capability information, where the second format includes an image format, and the image format is an image format after a virtual image and a real scene image are synthesized, or an image format of the virtual image; and the terminal device is specifically configured to obtain a first image sequence in the second format based on the perception and interaction data.

[0095] In a possible implementation, the terminal device is further configured to send the second format to the display device; the display device is further configured to perform display setting initialization according to the second format; and the display device is specifically configured to display an image based on the initialized display setting and the first image sequence.

[0096] In a possible implementation, the display device is further configured to send fourth capability information to the terminal device, where the fourth capability information provides third link information supported by the display device for a second transmission link; the terminal device is further configured to determine fourth link information based on the fourth capability information and the second format; the terminal device is further configured to configure the second transmission link based on the fourth link information; the terminal device is further configured to send the fourth link information to the display device; and the display device is further configured to configure the second transmission link based on the fourth link information.

[0097] In a possible implementation, the fourth link information includes information indicating starting compression; the terminal device is specifically configured to perform compression encoding on the first image sequence based on the fourth link information, to obtain a second code stream of the first image sequence; the terminal device is specifically configured to send the second code stream to the display device through the configured second transmission link; and the display device is specifically configured to decode the second code stream based on the fourth link information, to obtain the first image sequence in the second format.

[0098] Effects of the system in each of the above implementations are similar to effects of the data processing method performed by the terminal device and the display device in each of the above implementations, which will not be repeated here.

[0099] In a possible implementation, the present application provides a data processing apparatus. The data processing apparatus is communicatively connected with a display device, and the data processing apparatus comprises: a first receiving module configured to receive first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; a first determining module configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; a second receiving module configured to receive the perception and interaction data in the first format from the display device; and an obtaining module configured to obtain a first image sequence based on the perception and interaction data.

[0100] The data processing apparatus can be a standalone device (for example, a terminal device) or a part of a larger device. For example, the data processing apparatus can be implemented in the following forms:

[0101] (1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally comprise a storage component for storing data and instructions; (3) a module that can be embedded in other devices; (4) a vehicle-mounted device; and (5) other forms.

[0102] In a possible implementation, the apparatus further comprises: the first receiving module is further configured to receive second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; a second determining module configured to determine second link information based on the second capability information and the first format; and a first configuring module configured to configure the first transmission link based on the second link information.

[0103] In a possible implementation, the second link information comprises information indicating to start decompression, the second receiving module is configured to receive, through the configured first transmission link, a first code stream of the perception and interaction data in the first format that is compressed and encoded from the display device; and the apparatus further comprises: a decoding module configured to decode the first code stream based on the second link information to obtain the perception and interaction data in the first format.

[0104] In a possible implementation, the apparatus further includes: the first receiving module is further configured to receive third capability information, where the third capability information provides format information of display data supported by the display device; a third determining module is configured to determine a second format of display data to be transmitted based on the third capability information, where the second format includes an image format, the image format is an image format after a virtual image and a real scene image are synthesized, or an image format of the virtual image; and the obtaining module is specifically configured to obtain a first image sequence in the second format based on the perception and interaction data.

[0105] In a possible implementation, the apparatus further includes: the first receiving module is further configured to receive fourth capability information, where the fourth capability information provides third link information supported by the display device for a second transmission link; a fourth determining module is configured to determine fourth link information based on the fourth capability information and the second format; and a second configuring module is configured to configure the second transmission link based on the fourth link information.

[0106] In a possible implementation, the fourth link information includes information indicating starting compression, and the apparatus further includes: an encoding module is configured to compress and encode the first image sequence based on the fourth link information to obtain a second code stream of the first image sequence; and a sending module is configured to send the second code stream to the display device through the configured second transmission link.

[0107] In a possible implementation, the encoding module is further configured to compress and encode the first image sequence based on the perception and interaction data to obtain a code stream of the first image sequence.

[0108] In a possible implementation, the first image sequence includes at least two-view images, the at least two-view images include a first-view image and a second-view image, and the perception and interaction data includes first-view data.

[0109] In a possible implementation, the perception and interaction data further includes second-view data, and the encoding module is specifically configured to compress and encode the first-view image based on the first-view data, and compress and encode the second-view image based on the second-view data to obtain a second code stream of the first image sequence.

[0110] In a possible implementation, the encoding module is specifically configured to: compress and encode the first-eye image to obtain a bitstream of the first-eye image and a reconstructed image of the first-eye image; pre-process the reconstructed image based on the first-eye data to obtain a reference image matched with the second-eye image; and compress and encode the second-eye image based on the reference image to obtain a bitstream of the second-eye image.

[0111] In a possible implementation, the first format includes at least one of the following: a data format, a resolution, a frame rate, and a number of eyes.

[0112] In a possible implementation, the second format includes at least one of the following: a data format, a resolution, a frame rate, and a number of eyes.

[0113] In a possible implementation, the first determining module is specifically configured to determine the first format of the perception and interaction data to be transmitted based on the first capability information and target information, where the target information is at least one of a computing capability and an application scenario of the perception and interaction data.

[0114] In a possible implementation, the third determining module is specifically configured to determine the second format of the display data to be transmitted based on the third capability information and target information, where the target information is at least one of a computing capability and an application scenario of the display data.

[0115] In a possible implementation, the first capability information provides format information of the perception and interaction data supported by the display device for collection, or format information of the perception and interaction data supported by the display device for calculation.

[0116] The effects of the data processing apparatus in the above implementations are similar to those of the data processing method performed by the terminal device in the above implementations, and are not described here again.

[0117] In a possible implementation, the present application provides a data processing apparatus. The apparatus includes: a first sending module configured to send first capability information, where the first capability information provides format information of perception and interaction data supported by the display device; an obtaining module configured to obtain perception and interaction data in a first format, where the format information includes the first format; a second sending module configured to send the perception and interaction data in the first format; and a first receiving module configured to receive multimedia data, where the multimedia data is obtained based on the perception and interaction data in the first format.

[0118] The data processing apparatus can be a standalone device (for example, a display device) or a part of a larger device. For example, the data processing apparatus can be implemented in the following forms:

[0119] (1) an independent integrated circuit (IC), or chip, or chip system or subsystem; (2) a set of one or more ICs, optionally including memory elements that store data, instructions, or both; (3) a module that is to be embedded within another device; (4) a vehicle, etc.; (5) other, etc.

[0120] In a possible implementation, the apparatus includes a second receiving module configured to receive the first format; and a first configuring module configured to initialize a sensor for collecting perception and interaction data according to the first format.

[0121] In a possible implementation, the apparatus further includes the first sending module is further configured to send second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; a third receiving module configured to receive second link information, wherein the second link information is determined based on the second capability information and the first format; and a second configuring module configured to configure the first transmission link based on the second link information.

[0122] In a possible implementation, the second link information includes information indicating to start compression or decompression, and the apparatus further includes an encoding module configured to encode the perception and interaction data of the first format obtained based on the second link information to obtain a first code stream; and a third sending module configured to send the first code stream through the first transmission link configured.

[0123] In a possible implementation, the perception and interaction data of the first format includes a live image and image data other than the live image, and the encoding module is specifically configured to encode the live image based on the image data to obtain a first code stream.

[0124] The effects of the data processing apparatuses in the above embodiments are similar to those of the data processing methods performed by the display devices in the above embodiments, which will not be repeated here.

[0125] In a possible implementation, the present application provides a data processing apparatus. The data processing apparatus includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the processor can implement the method performed by the terminal device or the method performed by the display device in any of the above embodiments.

[0126] The effects of the data processing device of this embodiment are similar to the effects of the data processing methods of the above embodiments, and will not be described in detail here.

[0127] In one possible implementation, the present application provides a computer-readable storage medium storing a computer program that, when executed on a computer or processor, causes the computer or processor to execute the method executed by a terminal device or a display device in any of the above implementations.

[0128] The effects of the computer-readable storage medium of this embodiment are similar to the effects of the data processing methods of the above embodiments, and are not described in detail here.

[0129] In one possible implementation, the present application provides a computer program product. The computer program product includes a software program, which, when executed by a computer or a processor, causes the method executed by the terminal device or the method executed by the display device in any of the above implementations to be executed.

[0130] The effects of the computer program product of this embodiment are similar to the effects of the data processing methods of the above embodiments, and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Figure 1 This is a schematic diagram showing the structure of a VR split machine;

[0132] Figure 2a is a schematic diagram showing an exemplary device interaction;

[0133] Figure 2b is a schematic diagram showing an exemplary internal structure of a device;

[0134] Figure 2c is a schematic diagram showing an exemplary internal structure of a device;

[0135] Figure 3a is a schematic diagram illustrating an exemplary application scenario;

[0136] Figure 3b is a schematic diagram illustrating an exemplary application scenario;

[0137] Figure 3c is a schematic diagram illustrating an exemplary application scenario;

[0138] Figure 3d is a schematic diagram illustrating an exemplary application scenario;

[0139] Figure 3e is a schematic diagram illustrating an exemplary application scenario;

[0140] Figure 4 A schematic diagram of a data processing procedure shown for illustration;

[0141] Figure 5a A schematic diagram of an encoding procedure shown for illustration;

[0142] Figure 5b A schematic diagram of an encoding procedure shown for illustration;

[0143] Figure 6 A schematic diagram of a data processing procedure shown for illustration;

[0144] Figure 7 A schematic diagram of a data processing procedure shown for illustration;

[0145] Figure 8 A schematic diagram of a data processing procedure shown for illustration;

[0146] Figure 9 A schematic diagram of a data processing procedure shown for illustration;

[0147] Figure 10 A schematic diagram of a data processing procedure shown for illustration;

[0148] Figure 11 A schematic diagram of a device provided by an embodiment of the present application;

[0149] Figure 12 A schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0150] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0151] The term “and / or” in the present application is merely used to describe an association relationship of associated objects, and indicates that there can be three relationships, for example, A and / or B can represent three cases of existence of A alone, existence of A and B simultaneously, and existence of B alone.

[0152] The terms “first” and “second” and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe a specific order of the target objects.

[0153] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be construed as preferred or advantageous over other embodiments or designs. In fact, the purpose of using the words "exemplary" or "for example" is to present concepts in a concrete manner.

[0154] In the description of the embodiments of the present application, "a plurality of" means two or more, unless otherwise specified. For example, a plurality of processing units means two or more processing units; a plurality of systems means two or more systems.

[0155] Before introducing the technical solutions of the present application, the technical terms involved in the present application are first explained and described:

[0156] XR, representing any one of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0157] XR split machine, representing display units, sensing units, interaction units, etc. The units are integrated in different terminal devices respectively with the computing unit. The two terminal devices are connected through wired or wireless connection to interact data to realize XR function.

[0158] Simultaneous Localization and Mapping (SLAM) camera, which completes the sensing work of the environment through the camera, and generates an image which can be called pose graph.

[0159] Video See-Through (VST) through the camera, which is an AR technology that superimposes virtual content in the real world through real-time video stream, so that users can see the augmented reality scene through the camera.

[0160] Mono camera, a black and white camera, is a camera with only one light-sensitive element, which is different from traditional RGB color cameras. Its principle is to use the light-sensitive element to perceive the light in the environment and convert it into an electrical signal, which forms a monochrome image or video after processing.

[0161] Eye tracking camera, which can be a Mono camera, refers to a sensor or camera system specifically designed to track the movement of the human eye. This technology monitors and records the movement of the eye, providing information about the user's gaze point, gaze duration, blink frequency, etc.

[0162] Downward Camera refers to a camera or lens installed on a device or vehicle, facing downward. Such a camera is usually used to capture the scene or information below the device.

[0163] Depth Camera is a camera that can capture the depth information of a scene. Traditional cameras mainly capture the surface information of an image, while depth cameras can capture the distance of each pixel point in the image from the camera, thereby obtaining the depth value or distance value of each pixel point.

[0164] Inertial Measurement Unit (IMU) can be used to obtain the motion information of a device, such as linear acceleration and angular velocity.

[0165] Figure 1 A structure diagram of a VR split machine in a related technology is shown as an example.

[0166] As shown in Figure 1 , the VR split machine can be composed of a physically separated computing unit 1000 and a unit 2000 that integrates perception, interaction, and display, wherein the computing unit 1000 and the unit 2000 are connected by a cable.

[0167] As shown in Figure 1 , the computing unit 1000 can include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Neural-Network Processing Units (NPU), etc. In addition, the computing unit 1000 also includes a Universal Serial Bus (USB) interface and a DisplayPort (DP) interface.

[0168] As shown in Figure 1 , the unit 2000 can also include a USB interface and a DP interface, in addition to a sensor unit and a camera unit. The sensor unit can include an Inertial Measurement Unit (IMU), a handle, a keyboard, a microphone, etc. for obtaining interaction data. The camera unit can be used to obtain perception data, and can include a VST camera, a black-and-white camera, etc.

[0169] As shown in Figure 1 , the unit 2000 can also include a display module, which can include a left-eye display screen and a right-eye display screen for displaying VR images.

[0170] As shown in Figure 1 , the sensor unit and the camera unit are connected with the USB interface, and the display module is connected with the DP interface. In this way, the unit 2000 can send the above-mentioned interaction data and the above-mentioned perception data collected by the sensor unit and the camera unit respectively to the computing unit 1000 through the USB interface. And the computing unit 1000 can send the to-be-displayed data (such as images) to the display module of the unit 2000 through the DP interface for display, so as to realize the display of the VR video.

[0171] The above-mentioned unit 2000 can be deployed on a wearable display device (such as a VR glasses or a VR helmet), and the above-mentioned computing unit 1000 is deployed on another terminal device (such as a computer).

[0172] Among them, the transmission protocol of the USB interface of the unit 2000 is customized by the manufacturer of the wearable display device, and the perception data and the interaction data are transmitted by the customized transmission protocol.

[0173] Therefore, Figure 1 , the terminal device in which the same computing unit 1000 is located cannot be compatible with wearable display devices of various manufacturers and various models.

[0174] In addition, in the scene of the XR split machine as shown in Figure 1 , the computing unit 1000 and other perception, interaction, display and the like units 2000 are in a separated state. In the XR scene, the amount of data to be transmitted on both sides is large, which will bring great pressure to the bandwidth and transmission delay, and it is difficult to realize the transmission of XR data under the condition of meeting low delay and low bandwidth.

[0175] Therefore, as shown in Figure 2a , the present application provides a terminal device 100, a display device (such as a display peripheral 200) and a system. The system can include a terminal device 100 and a display peripheral 200 in communication connection (such as wired or wireless connection) with the terminal device 100. The wireless connection can be Wi-Fi, Bluetooth, etc., which is not limited here.

[0176] The terminal device 100 can obtain the capability information of the sensing and interaction data from the display device 200, and optionally obtain the capability information of the display data. The capability information of the display data can provide the format information of the display data (e.g. format, resolution, frame rate, etc.) that the display device 200 can support. The capability information of the sensing and interaction data can provide the format information of the sensing and interaction data (e.g. sensor type, format of the data generated by the sensor (e.g. RGB, etc.), resolution, frame rate, etc.) that the display device 200 can support. Then, the terminal device 100 can determine the format (also referred to as the first format) of the sensing and interaction data to be transmitted based on the capability information of the sensing and interaction data, and optionally determine the format (also referred to as the second format) of the display data to be transmitted based on the capability information of the display data. The format determined by the terminal device 100 is within the format information provided by the corresponding capability information. The terminal device 100 can receive the sensing and interaction data in the first format from the display device 200, and obtain the display data (also referred to as the first image sequence) based on the sensing and interaction data in the first format.

[0177] In some embodiments, the terminal device 100 can generate the display data based on the sensing and interaction data in the first format.

[0178] In other embodiments, the terminal device 100 can send the sensing and interaction data in the first format to a server (e.g. cloud), and the server can generate the display data based on the sensing and interaction data in the first format. Then, the terminal device 100 can receive the display data from the server to obtain the display data.

[0179] That is, the present application can generate the display data based on the sensing and interaction data by the terminal device 100 itself to obtain the display data, or can generate the display data based on the sensing and interaction data by the cloud and obtain the display data from the cloud by the terminal device 100. The terminal device 100 can select one of the two ways to obtain the display data according to the application scenario, which is not limited here.

[0180] Optionally, the terminal device 100 can send the obtained display data to the display device 200. In this way, the format of the display data sent by the terminal device 100 to the display device 200 belongs to the format of the display data supported by the display device 200.

[0181] In this way, the terminal device 100 can receive the perception and interaction data in a format (e.g., the first format) supported by the display peripheral 200, and the terminal device 100 can be compatible with the display peripheral of any transmission protocol.

[0182] In addition, the to-be-displayed data, i.e., the first image sequence, can be one frame of image or multiple frames of image.

[0183] In one transmission moment, the first image sequence can be one frame of image or multiple frames of image corresponding to multiple eyes (two eyes or more than two eyes).

[0184] For example, when the terminal device 100 is connected to a common display device (e.g., a vehicle display screen), in one transmission moment, the first image sequence can be one frame of image, so that the vehicle display screen can receive one frame of image in the same transmission moment for on-screen display, so that the user can browse the one frame of image through the vehicle display screen.

[0185] When the terminal device 100 is connected to an XR device (a device related to display of two or more than two eyes, e.g., a VR glasses), in one transmission moment, the first image sequence can be two frames of image or more than two frames of image. For example, when the XR device is a VR glasses, the VR glasses can receive two frames of image in the same transmission moment, which are an image corresponding to a left eye display screen and an image corresponding to a right eye display screen, respectively. In this way, the VR glasses can perform on-screen display of the two frames of image corresponding to two eyes on the left eye display screen and the right eye display screen in the same moment. The same applies to more than two eyes, which will not be described here.

[0186] The terminal device 100 can be a desktop device or a portable device. The portable form of the terminal device 100 can be a box type (e.g., the terminal device 100 shown in FIG. 1A), a neck-hanging type (e.g., the terminal device 100 shown in FIG. 1B), a waist-hanging type, etc., which will not be limited here. Figures 3a to 3d The terminal device 100 can be a desktop device or a portable device. The portable form of the terminal device 100 can be a box type (e.g., the terminal device 100 shown in FIG. 1A), a neck-hanging type (e.g., the terminal device 100 shown in FIG. 1B), a waist-hanging type, etc., which will not be limited here. Figure 3e The terminal device 100 can be a desktop device or a portable device. The portable form of the terminal device 100 can be a box type (e.g., the terminal device 100 shown in FIG. 1A), a neck-hanging type (e.g., the terminal device 100 shown in FIG. 1B), a waist-hanging type, etc., which will not be limited here. The terminal device 100 can be a desktop device or a portable device. The portable form of the terminal device 100 can be a box type (e.g., the terminal device 100 shown in FIG. 1A), a neck-hanging type (e.g., the terminal device 100 shown in FIG. 1B), a waist-hanging type, etc., which will not be limited here.

[0187] The display peripheral 200 can be any device with a display screen, which will not be limited here. The display peripheral 200 can be any device with a display screen, which will not be limited here.

[0188] For example, the display peripheral 200 can include but is not limited to any of the following: a display, a personal computer (PC), a mobile phone, a television, a projector, a vehicle display screen, a wearable display device. The wearable display device can be an XR glasses or an XR helmet, etc. The display peripheral 200 can be any device with a display screen, which will not be limited here.

[0189] In this way, the terminal device 100 of the present application can be compatible with wearable display devices (such as AR glasses, VR glasses, AR helmets, VR helmets, etc.) of various data transmission protocols to achieve immersive virtual-real fusion and spatial interaction to meet VR, AR, etc. experience; in addition, the terminal device 100 can also be compatible with ordinary display devices (such as displays, televisions, vehicle display screens, projectors, etc.) of various data transmission protocols, etc., so that the terminal device can be used as a portable micro host.

[0190] combined with Figure 2a , Figure 2b The internal structure and interaction schematic diagram of the display peripheral 200 and the terminal device 100 of the present application are shown.

[0191] As shown in Figure 2b , the display peripheral 200 can include but is not limited to: a display module, an audio module.

[0192] The display module may, for example, be a screen, and the audio module may, for example, be a loudspeaker, which can be used to display or play multimedia data. In the present application, the data output by the display module of the display peripheral 200 can be defined as display data. Optionally, the display data can also include data output by the audio module.

[0193] As shown in Figure 2b , the display peripheral 200 can further include: a perception module, an interaction module, a microphone module.

[0194] As shown in Figure 2b , the microphone module, the perception module (such as a depth camera for collecting depth information), and the interaction module (such as an eye tracking camera for collecting eye movement images) in the display peripheral 200 can be sensors for collecting input information. Among them, the microphone module can be used to collect audio (such as environmental sound, voice, etc.); the perception module can be used for spatial positioning and visual information for sensing information around the positioning location; the interaction module can be used to obtain visual information (such as images) for interaction, and the interaction module is commonly used in glasses-type display peripherals. In the present application, the data received by the perception module, the interaction module, and optionally the microphone module in the display peripheral 200 can be defined as perception and interaction data.

[0195] As shown in Figure 2b , the display peripheral 200 can further include: a VST module.

[0196] Among them, the VST module is a visual sensor, also referred to as a real scene camera, which is used to collect real scene images, commonly used in MR glasses and some VR glasses, and can be used to collect visual information in real time. The vision restores the real world, and in the present application, the VST data collected by the VST module is also defined as perception and interaction data.

[0197] As Figure 2b shown, the display peripheral device 200 and the terminal device 100 further comprise respective sensing and interaction interfaces, through which the display peripheral device 200 and the terminal device 100 can send the sensing and interaction data collected by the display peripheral device 200 to the terminal device 100.

[0198] As Figure 2b shown, the display peripheral device 200 and the terminal device 100 further comprise respective display interfaces, through which the terminal device 100 can generate display data to be displayed and send the display data to the display interface of the display peripheral device 200.

[0199] The display interface of the display peripheral device 200 can be used to receive the display data.

[0200] Optionally, the display peripheral device 200 can further comprise a processing module and a corresponding storage module.

[0201] In some embodiments, the processing module can be used to calculate the sensing and interaction data to obtain the sensing and interaction data to be transmitted to the terminal device 100.

[0202] In some embodiments, the processing module can be used to calculate the display data to be displayed.

[0203] The processing module of the display peripheral device 200 can include but is not limited to CPU, GPU, NPU, etc.

[0204] The storage module can include internal memory and external storage, and can be used to store data such as the device type of the display peripheral device 200, the capability information of the sensing and interaction data, the capability information of the display data, algorithm data, user local and temporary data, etc.

[0205] The device type, the capability information of the sensing and interaction data, and the capability information of the display data will be described and explained in detail in the embodiment shown in FIG. 3.

[0206] Based on the different specifications of the display peripheral device 200, the computing power of the processing module can be different. The processing module of the display peripheral device 200 can generally realize the basic functions of driving its own display module and audio module.

[0207] Some display peripheral devices, mainly wearable devices (such as XR glasses and XR headsets), can also have a microphone module, a sensing module, an interaction module, and a VST module as Figure 2b shown.

[0208] In some embodiments, the processing module of the display peripheral 200 can be used to manage the sensors that collect input data, and image signal processing (ISP) and the like.

[0209] In some embodiments, the processing module of the display peripheral 200 has strong computing power, and the processing module can also calculate the VST data collected by the VST module to obtain displayable real scene image data, realizing the closed loop between the VST module and the display module. In this way, the display peripheral 200 does not need to transmit the VST data collected by the VST module to the terminal device 100.

[0210] In some embodiments, the processing module of the display peripheral 200 has strong computing power, and the processing module can also calculate the perception and interaction data collected by the perception module and the interaction module, so as to output the calculated perception and interaction data (also referred to as the calculation result of the perception and interaction data) to the terminal device 100 through the perception module and the interaction module. In this way, the perception and interaction data received by the terminal device 100 can be the result of spatial positioning (such as pose), local map positioning, spatial geometric grid, light information, gesture point, gaze point, limb key point, facial expression base, and the like.

[0211] The structure of the terminal device 100 as shown in FIG. 1 is described in detail below. Figure 2b

[0212] The terminal device 100 can include a processing module, an analysis and decoding module interacting with a perception and interaction interface, an analysis and encoding module interacting with a display interface, an AI module, a storage module, a battery, and a power interface.

[0213] The analysis and decoding module can be used to decompress and decode the perception and interaction data received by the perception and interaction interface, and can also be used to analyze the information from the display peripheral 200 to obtain the capability information of the perception and interaction data.

[0214] The analysis and encoding module can be used to compress and encode the display data to be sent to the display interface, and can also be used to analyze the information from the display peripheral 200 to obtain the capability information of the display data.

[0215] The processing module (such as CPU, GPU, NPU, etc.) of the terminal device 100 can be used to implement the data processing method of the terminal device 100, and render the perception and interaction data (or the calculation result of the perception and interaction data) to generate display data, and the like.

[0216] ​The Artificial Intelligence (AI) module of the terminal device 100 can be used to infer the received perception and interaction data (or the calculation result of the perception and interaction data) by an AI model, a language large model, etc., to generate the to-be-displayed data.

[0217] In some embodiments, the processing module and the AI module of the terminal device 100 can respectively generate part of the elements of the to-be-displayed data, so as to generate all the elements of the complete to-be-displayed data by the cooperation of the processing module and the AI module.

[0218] The elements of the to-be-displayed data can include but are not limited to text, audio (such as voice), image, video, model, etc., which are not limited here.

[0219] In some embodiments, the processing module and the AI module of the terminal device 100 can both generate all the elements of the to-be-displayed data, and the terminal device 100 can select one of the processing module and the AI module to independently generate the to-be-displayed data according to the application scenario and the computing power requirement.

[0220] For example, the AI module can generate text, audio, image, video, model, etc. based on the received perception and interaction data (or the calculation result of the perception and interaction data) as part of the elements of the to-be-displayed data.

[0221] In some embodiments, the AI module can also complete the generation of the to-be-displayed data by cooperating with the server (such as the cloud) through communication.

[0222] For example, in a scenario where the computing power required for the generation of the to-be-displayed data is high, the AI module can send a request for AI processing to the cloud based on the above-mentioned perception and interaction data (or the calculation structure of the perception and interaction data) to obtain the to-be-displayed data generated by the cloud.

[0223] The storage module of the terminal device 100 can be used to store data information when implementing the data processing method, and the storage module can be an internal memory or an external storage.

[0224] Optionally, the terminal device 100 can further include a communication module, which can support various wireless communication protocols to communicate with the display peripheral 200.

[0225] The communication module of the terminal device 100 can also be used to communicate with the server (such as the cloud), so that the AI module can interact with the cloud through the communication module, so that the terminal device 100 can realize the inference of the AI model and the calculation of the language large model by means of the processing capability of the cloud.

[0226] For example, the communication module can be a Wi-Fi module, a Bluetooth module, etc., which is not limited here.

[0227] The battery of the terminal device 100 can be interconnected with the power interface of the display peripheral 200 through the power interface to supply power to the display peripheral 200.

[0228] Combined with Figure 2b , Figures 3a to 3d The schematic diagrams of various application scenarios of the terminal device 100 of the present application are shown respectively.

[0229] 1. Entertainment scenario:

[0230] As shown in Figure 3a , the terminal device 100 can be connected with the XR helmet 201 in a wired or wireless manner, wherein the VR helmet is an example of the display peripheral 200 of the present application.

[0231] The XR helmet 201 can collect perception and interaction data through the VST module, perception module, and interaction module as shown in Figure 2b . The perception and interaction data can include but are not limited to IMU data, VST data, black and white images (such as eye movement images, lip images, limb images, hand gesture images), depth images, etc. The above-mentioned perception and interaction data can be obtained through real-time images collected by sensors and cameras (also described as cameras) of the XR helmet 201 (which can include real scene images around the user wearing the XR helmet 201 (an example of VST data)).

[0232] The XR helmet 201 can transmit the perception and interaction data to the terminal device 100, and the terminal device 100 can superimpose virtual images generated by the terminal device 100 on the above-mentioned real-time images based on the perception and interaction data, generate virtual-real fusion display data, and send the display data to the XR helmet 201. The display data can include virtual-real fusion image data and audio data, and the XR helmet 201 can output the audio data through a loudspeaker and output the virtual-real fusion image data through two display screens of the XR helmet 201.

[0233] In this way, the terminal device 100 of the present application can be compatible with the XR helmet to realize the transmission of perception and interaction data and display data, so as to realize the virtual-real fusion spatial interaction experience.

[0234] 2. Office scenario:

[0235] As shown in Figure 3b , the terminal device 100 can be connected with the display 202 and the keyboard 203 in a wired (or wireless) manner, wherein the display 202 and the keyboard 203 together serve as an example of the display peripheral 200 of the present application.

[0236] The display screen 202 may be a television, a projector, a vehicle-mounted display screen, a desktop display screen, etc., which is not limited here.

[0237] The terminal device 100 can receive keyboard input data (a type of interactive data) from the keyboard 203. The terminal device 100 can generate data to be displayed (such as an image) based on the keyboard input data, and send the data to be displayed to the display 202 for on-screen display.

[0238] In this way, the terminal device 100 of the present application is compatible with a display screen and a keyboard to achieve the effect of expanding the display screen of the terminal device 100 .

[0239] 3. Independent shooting scene:

[0240] Back to Figure 2b The terminal device 100 optionally includes N camera modules, microphone modules, audio modules, and display modules.

[0241] Among them, the camera module can be used to capture images, the microphone module can be used to capture audio, the audio module (such as a speaker) can be used to output audio, and the display module (such as a display screen) can be used to display multimedia data (such as images or videos).

[0242] Combined with Figure 2b ,like Figure 3c As shown, the terminal device 100 uses a camera module to perform multi-purpose three-dimensional (3D) shooting of the user, and uses a microphone module to record audio in the three-dimensional space on the spot, and finally outputs the 3D-shot image or video through a display module and an audio module. This makes the audio in the output video have a sense of presence as if it were recorded on the spot.

[0243] In addition, combined with Figure 2b ,like Figure 3c As shown, the N camera modules may include a camera module with a micro single-level shooting capability, and the terminal device 100 can use the camera module with the micro single-level shooting capability to shoot images or videos, so that the terminal device 100 of the present application can be equipped with a micro single-level shooting capability and the image shooting quality is high.

[0244] 4. Collaborative XR glasses shooting scene:

[0245] Different from the first entertainment scenario mentioned above (e.g. Figure 3a ), in this scenario (e.g. Figure 3d ), such as Figure 3dAs shown, not only can the XR helmet 201 capture the surrounding environment (here, the user as an example) to obtain perception and interaction data and transmit to the terminal device 100, but also because the terminal device 100 has N camera modules, the terminal device 100 can also capture the surrounding environment (the user as an example), so that the perception and interaction data on which the terminal device 100 is based when generating the to-be-displayed data not only comes from the XR helmet 201, but also can come from the image data captured by the terminal device 100 itself; finally, the terminal device 100 can send the to-be-displayed data obtained (generated by the terminal device 100 or obtained from the cloud by being generated by the cloud) to the XR helmet 201 for output.

[0246] In this way, the terminal device 100 of the present application can cooperate with the XR helmet to perform cooperative shooting to meet the shooting requirements of the real environment in the XR scene.

[0247] 5, Application scenario of the neck-hung terminal device

[0248] The terminal device in the above Figures 3a to 3d is a box-shaped terminal device, Figure 3e The terminal device 100 shown in the figure is a neck-hung terminal device 100.

[0249] As Figure 3e shown, the terminal device 100 can interact with the XR helmet 201, and the specific interaction process is the same as that described in Figure 3a , which will not be described here.

[0250] As Figure 3e shown, the neck-hung terminal device 100 can be worn on the neck of the user 10, so that the user 10 can use the worn terminal device 100 and the XR helmet 201 to immerse in the game, video, etc. of XR.

[0251] It should be understood that, of course, the neck-hung terminal device 100 can also be applied to the scenario shown in the above Figures 3b to 3d , and the specific implementation principle can refer to the description in Figures 3b to 3d , which will not be described here.

[0252] The terminal device of the present application provides more space for camera modules and audio modules compared with mobile phones, can layout micro-single level shooting capability, multi-purpose 3D shooting, spatial audio recording, and the terminal device layout positioning perception module capability can realize cooperative shooting of the terminal device and glasses, and solve the core scene demand of daily work, entertainment and life.

[0253] In order to make the like Figure 2a and Figure 2b and Figures 3a to 3dThe terminal device 100 shown can implement the functions mentioned in the present application, and the present application further provides a data processing method to ensure the implementation of the corresponding functions of the terminal device 100.

[0254] Figure 4 The process schematic diagram of the data processing method of the present application is exemplarily shown.

[0255] Based on Figure 2b , Figure 2c The internal structures and data interactions of the display peripheral 200 and the terminal device 100 are further described.

[0256] The functions of the modules and structures shown will be described below in combination with Figure 4 the process shown. Figure 2c

[0257] Among them, Figure 4 The steps indicated by the dashed boxes and dashed arrows in the method are optional steps in the implementation process of the method.

[0258] As Figure 4 shown, the process can include the following steps:

[0259] S100, the terminal device 100 is in communication connection with the display peripheral 200.

[0260] The connection can be wired or wireless connection, which is not limited here.

[0261] Optionally, S101, the terminal device 100 determines the device type of the connected display peripheral 200.

[0262] Among them, as described above, the device type of the display peripheral 200 can include but is not limited to any one of the following: display, personal computer (PC), mobile phone, television, projector, vehicle-mounted display screen, wearable display device. Among them, the wearable display device can be XR glasses or XR helmet, etc.

[0263] Among them, the terminal device 100 can read the information indicating the device type from the display peripheral 200.

[0264] S102, the terminal device 100 obtains the capability information based on the device type.

[0265] As Figure 2c shown, the storage module of the display peripheral 200 can store the capability information of perception and interaction data and the capability information of display data.

[0266] As Figure 4 ​As shown, the capability information of the perception and interaction data can include data capability information (also referred to as first capability information) of the perception and interaction data, and capability information (also referred to as second capability information) of a transmission link of the perception and interaction data as shown by the dashed arrow. The transmission link is a link used by the display peripheral 200 to transmit the perception and interaction data.

[0267] The capability information of the display data can include data capability information (also referred to as third capability information) of the display data, and capability information (also referred to as fourth capability information) of a transmission link of the display data as shown by the dashed arrow. The transmission link is a link used by the display peripheral 200 to transmit the display data.

[0268] The data capability information of the perception and interaction data can provide format information of the perception and interaction data supported by the display device 200. The perception and interaction data transmitted can be perception and interaction data collected by the display device 200, or a result obtained by performing calculation on the collected perception and interaction data.

[0269] The data capability information of the display data can provide format information of the display data supported by the display device 200.

[0270] In some embodiments, when it is determined that the device type of the display peripheral 200 is a general display, the general display can include but is not limited to a display, a personal computer (PC), a mobile phone, a television, a projector, and a vehicle-mounted display screen. The perception and interaction data of the display peripheral 200 is generally microphone data of voice input, remote control type, and input data of a keyboard and a mouse, etc. The display peripheral 200 can process these perception and interaction data, and thus the general display does not need to transmit the perception and interaction data to the terminal device 100 for processing by the terminal device 100. Then, as shown, Figure 4 When the terminal device 100 determines that the display peripheral 200 is a general display, it is not necessary to obtain the capability information of the perception and interaction data, but only the capability information of the display data needs to be obtained from the display peripheral 200. As shown, Figure 4 The capability information of the display data can include data capability information of the display data, and optionally capability information of a transmission link of the display data as shown by the dashed arrow.

[0271] In some embodiments, when it is determined that the device type of the display peripheral 200 is a wearable display device (such as an XR glasses, an XR helmet, etc.), the terminal device 100 can obtain the capability information of the perception and interaction data from the display peripheral 200, and optionally the capability information of the display data. As shown, Figure 4As shown, the terminal device 100 can obtain the data capability information of the perception and interaction data from the display peripheral 200, and optionally obtain the capability information of the transmission link of the perception and interaction data shown by the dashed arrow. Optionally, as shown in FIG. 2B, the terminal device 100 can obtain the data capability information of the display data from the display peripheral 200, and optionally obtain the capability information of the transmission link of the display data. Figure 4 As shown, the terminal device 100 can obtain the data capability information of the display data from the display peripheral 200, and optionally obtain the capability information of the transmission link of the display data.

[0272] The data capability information of the display data can be used to provide the format information of the display data supported by the display device 200 for transmission.

[0273] The format information of the display data can include, but is not limited to, the data format of the display data, the resolution of the display data, the frame rate of the display data, the number of eyes (for example, binocular, or multi-eyes above binocular) of the display data, etc.

[0274] For example, the data format of the display data supported by a general display is generally a Red Green Blue (RGB) format, and the resolution and frame rate (unit: hertz (hz)) of the display data supported by the general display are generally 1920*1080*60hz, 3840*2160*120hz, 1280*720*60hz, etc., which are not limited herein. Resolution*frame rate is taken as an example herein.

[0275] For another example, the data format of the display data supported by the XR glasses, the XR helmet, etc. can include the image format after the synthesis of virtual images and real images (also described as the data format that can be directly displayed), such as the RGB format, etc.; the data format of the display data can also include the image format of the virtual image (also described as the data format to be displayed after the synthesis of virtual and real images), such as the RGBA format or the RGBAZ format, etc.

[0276] The data format to be displayed after the synthesis of virtual and real images is RGBA or RGBAZ. Compared with the RGB format, the RGBAZ format has an additional transparency channel A and a depth channel Z. In the synthesis of virtual and real images, the transparency channel A is used as a mask, and the depth channel Z provides correct occlusion relationship in the synthesis to realize virtual and real occlusion. Compared with the RGB format, the RGBA format has an additional transparency channel A, which is used as a mask in the synthesis of virtual and real images.

[0277] The AR glasses support a display data of 2, and the resolution and frame rate of the display data can include 2*1920*1080*60hz, 2*1536*1760*120hz; the VR glasses support a display data of 2, and the data format of the display data can include 2*2000*2000*90hz, 2*3000*3000*90Hz; the MR glasses support a display data of 2, and the data format of the display data can be 2*3840*3600*90Hz. Since the XR glasses have two display screens, the above resolution is multiplied by 2. Taking 2*1920*1080*60hz as an example, the resolution and frame rate of each display data are 1920*1080*60hz.

[0278] The AR glasses and the VR glasses described above can support two resolutions of each display data. For example, the VR glasses support a resolution of each display data of full high definition, and the resolution of full high definition is 3000*3000*90hz, and the VR glasses also support a resolution of each display data of a gaze point rendering, and the resolution of the gaze point rendering is 2000*2000*90hz.

[0279] The resolution and frame rate of the display data supported by the AR glasses, the VR glasses and the XR glasses described above are only examples, and the resolution and frame rate of the display data supported by the XR glasses are not limited by the present application, and depend on the XR glasses.

[0280] The data capability information of the perception and interaction data can provide format information of the perception and interaction data supported by the display device 200 for transmission. The format information can include but is not limited to the sensor type supported by the display device 200, the data format of the perception and interaction data supported by the sensor, the resolution of the perception and interaction data, the frame rate (also described as the refresh rate) of the perception and interaction data, etc.

[0281] For example, the XR glasses are near-eye display, and support the realization of the three characteristics of positioning, perception and interaction in use. When the XR glasses realize the positioning characteristic, the IMU, the binocular or four-mirror black-and-white camera Y image stream can be supported. When the XR glasses realize the perception characteristic, the depth stream using the depth camera can be supported. When the XR glasses realize the interaction characteristic, the camera used for realizing the positioning characteristic can be multiplexed, and two additional black-and-white cameras are added. The interaction characteristic can include the eye movement detection characteristic. When the XR glasses support the realization of the eye movement detection characteristic, 2 to 4 eye tracking cameras (which are also black-and-white cameras) can be supported. If the MR glasses, two VST cameras can also be supported for collecting color images, and the format of the color images can be YUV or RGB format, etc.

[0282] So in some scenarios, the data capability information of the perception and interaction data acquired by the terminal device 100 from the XR glasses can include the following information: the supported sensor types include: IMU, 6-10 black-and-white cameras, 1 depth camera, and 2 VST color cameras. Among them, the black-and-white camera supports the generation of a luminance (Luminance, Y) image stream, and the resolution and frame rate (unit: FPS) of the Y image stream supported by the black-and-white camera include 1280*1280*60FPS and 640*480*60FPS. The depth camera supports a depth image resolution and frame rate of 640*480*30FPS. The above-mentioned 2 VST color cameras support the generation of color image streams in YUV ("Y" represents luminance (Luminance, Luma), and "U" and "V" are chrominance, concentration (Chrominance, Chroma)) or RGB format, and the color image stream supported by the color camera has a resolution and frame rate of 4000*3000*60FPS.

[0283] In some scenarios, part of the wearable display device (such as MR glasses) is also equipped with a powerful processing module, which can directly complete the whole link closed loop from VST data acquisition to color image display, and can calculate perception and interaction data other than VST data, such as spatial positioning perception and interaction calculation, thereby directly outputting the calculation results of the perception and interaction data. Such a display peripheral 200 does not need to transmit VST data to the terminal device 100, so the data capability information of the perception and interaction data of the display peripheral 200 can include the format information of the calculated perception and interaction data supported by the display peripheral 200, such as the format information can include but not limited to: spatial positioning results, local positioning maps, spatial geometric grids, lighting information, gesture points, gaze points, limb key points, facial expression bases, etc.

[0284] Optionally, when implementing the above S102, the terminal device 100 can also acquire the capability information of the display data transmission link from the display peripheral 200. The transmission link is the transmission link between the display interfaces of the two ends as shown in Figure 2b and Figure 2c .

[0285] The capability information of the display data transmission link (also referred to as the fourth capability information) can provide the third link information supported by the display peripheral 200 on the second transmission link. Among them, the second transmission link is a link for transmitting display data.

[0286] The third link information can include, but is not limited to, the number of lanes (e.g., maximum number) supported by the display peripheral 200 to be opened for the second transmission link, the transmission rate of the lanes, and the like. In this way, the terminal device 100 can determine the link rate (e.g., maximum rate) supported by the second transmission link based on the fourth capability information.

[0287] For example, the number of lanes of the display data transmission link supported by the display peripheral 200 to be opened includes x1, x2, x4, and x8. Wherein x1, x2, x4, and x8 respectively represent that the display data transmission link of the display peripheral 200 is opened with 1 lane, 2 lanes, 4 lanes, and 8 lanes. Wherein the transmission link supports up to 8 lanes to be opened, that is, the maximum link rate is the product of x8 and the lane transmission rate. The number of lanes of the display data transmission link opened by the display peripheral 200 can affect the link rate and bandwidth of the transmission link, and the more the number of lanes opened, the faster the link rate and the wider the bandwidth.

[0288] Similarly, optionally, when implementing S102, the terminal device 100 can obtain the capability information of the transmission link of the sensing and interaction data from the display peripheral 200. The transmission link is the transmission link between the sensing and interaction interfaces of the two ends as shown in Figure 2b and Figure 2c .

[0289] The capability information (also described as second capability information) of the transmission link of the sensing and interaction data can provide the first link information supported by the display peripheral 200 to the first transmission link. Wherein the first transmission link is a link for transmitting sensing and interaction data.

[0290] The first link information can include, but is not limited to, the number of lanes (e.g., maximum number) supported by the display peripheral 200 to be opened for the first transmission link, the transmission rate of the lanes, and the like. In this way, the terminal device 100 can determine the link rate (e.g., maximum rate) supported by the first transmission link based on the second capability information.

[0291] For example, the number of lanes of the sensing and interaction data transmission link supported by the display peripheral 200 to be opened includes x1 and x2.

[0292] Combined with Figure 2c, the terminal device 100 can send a display module capability request to the display peripheral 200, and the display peripheral 200 can read the display data capability information from the storage module in response to the capability request, and send the display data capability information to the display interface of the terminal device 100 through the display interface of the display peripheral 200; the display data capability information parsing module of the terminal device 100 can read and parse the display data capability information from the display interface, to obtain the data format of the display data supported by the display peripheral 200, the resolution of the display data supported by the display peripheral 200, the frame rate of the display data supported by the display peripheral 200, and the like.

[0293] Wherein, the terminal device 100 can send two requests to obtain the data capability information of the display data of the display peripheral 200 and the capability information of the transmission link in sequence, or can obtain the two capabilities by sending one request, which is not limited here.

[0294] Similarly, in combination with Figure 2c , the terminal device 100 can send a sensor capability request to the display peripheral 200 when obtaining the capability information of the sensing and interaction data of the display peripheral 200, and the display peripheral 200 can read the capability information of the sensing and interaction data from the storage module in response to the capability request, and send the sensing and interaction capability information to the sensing and interaction interface of the terminal device 100 through the sensing and interaction interface of the display peripheral 200; the sensing and interaction data capability information parsing module of the terminal device 100 can read and parse the sensing and interaction data capability information from the sensing and interaction interface, to obtain the sensor type supported by the display peripheral 200, the format, resolution, and frame rate (also described as refresh rate) of the data supported by the sensor, and the like.

[0295] Wherein, the terminal device 100 can send two requests to obtain the data capability information of the sensing and interaction data of the display peripheral 200 and the capability information of the transmission link in sequence, or can obtain the two capabilities by sending one request, which is not limited here.

[0296] Continuing to Figure 4 After the terminal device 100 obtains the data capability information of the display data of the display peripheral 200 through S102, S103a and S104a can be executed, and the corresponding display peripheral 200 can execute S201a after S104a. Based on the data capability information of the display data of the display peripheral 200, the data format of the to-be-displayed data to be transmitted between the terminal device 100 and the display peripheral 200 this time is uniformly configured.

[0297] S103a, the terminal device 100 determines the format of the to-be-displayed data to be transmitted based on the data capability information of the display data.

[0298] Optionally, the terminal device 100 can set the format of the to-be-displayed data to be transmitted within the data capability range of the display data supported by the display peripheral 200 in combination with at least one of the application scenario and the computing capability (referred to as computing power) of the terminal device 100 on the to-be-displayed data.

[0299] For example, the display peripheral 200 is a general display, and the data capability information (also referred to as third capability information) of the display data of the display peripheral 200 includes that the data format of the display data supported by the display peripheral 200 is an RGB format, and the resolution and frame rate of the display data supported by the display peripheral 200 are three kinds: 1920*1080*60hz, 3840*2160*120hz, and 1280*720*60hz.

[0300] Then, the format information of the display data supported by the display peripheral 200 provided by the third capability information of the display peripheral 200 can include that the data format is an RGB format, and the resolution and frame rate of the display data are 1920*1080*60hz, 3840*2160*120hz, and 1280*720*60hz. In addition, the format information of the display data also implicitly expresses that the minimum resolution of the display data supported by the display peripheral 200 is 1280*720 among the above three resolutions, the maximum resolution of the display data supported by the display peripheral 200 is 3840*2160, the minimum frame rate of the display data supported by the display peripheral 200 is 60hz, and the maximum frame rate of the display data supported by the display peripheral 200 is 120hz.

[0301] Then, the terminal device 100 can set the data format, resolution, and frame rate of the to-be-displayed data to be transmitted within the above data capability range (for example, the format information of the display data supported by the display peripheral provided by the third capability information) of the display peripheral 200. For example, the set data format is an RGB format, and the set resolution and frame rate can be selected from one of the above three kinds. Alternatively, the set resolution is an arbitrary resolution a*b within the resolution range of the maximum resolution (for example, 3840*2160) and the minimum resolution (for example, 1280*720), wherein a / b=1280 / 720=3840 / 2160=1920 / 1080, and the set frame rate is an arbitrary frame rate c within the frame rate range of the maximum frame rate (for example, 120hz) and the minimum frame rate (60hz), wherein 60hz≤c≤120hz, for example, c=90hz.

[0302] That is, the terminal device 100 can set the data format of the to-be-displayed data to be transmitted between the terminal device 100 and the display device 200 this time in the format information of the display data supported by the display device 200, and the set data format cannot exceed the maximum data capacity (for example, 3840*2160*120hz here) of the display data supported by the display device 200.

[0303] For example, the general display device only supports display data in RGB format, and the terminal device 100 determines that the data format of the to-be-displayed data to be transmitted this time can only be RGB and cannot be other formats.

[0304] When the display device 200 supports multiple formats of the same parameter (for example, resolution) of the display data, for example, the supported resolutions are 1920*1080, 3840*2160, and 1280*720 respectively. Then, the terminal device 100 can determine a resolution from the above three resolutions as the resolution of the to-be-displayed data to be transmitted in combination with at least one of the computing capacity of the terminal device 100 for the to-be-displayed data and the application scenario.

[0305] For example, the computing capacity of the terminal device 100 for the to-be-displayed data is low, and only supports gaze point rendering, so a lower resolution, for example, 1280*720, can be selected.

[0306] For another example, the rendering requirement of the application currently running in the terminal device 100 for the image is full-load rendering, so a higher resolution, for example, 3840*2160, can be selected.

[0307] Similarly, for example, the display device 200 is an XR eyewear, and the data capacity information of the display data of the display device 200 is that the data format is RGBA or RGBAZ format, and the supported two resolutions and frame rates are 2000*2000*90hz (for example, corresponding to the resolution of gaze point rendering) and 3000*3000*90hz (for example, corresponding to the resolution of full high definition). Then, the terminal device 100 can set the data format, resolution, and frame rate of the to-be-displayed data to be transmitted this time within the above data capacity range of the display device 200. For example, the set data format is RGBAZ format or RGBA format; the set resolution and frame rate can be selected from one of the two examples above. Or, the set resolution is any resolution e*f within the resolution range of the maximum resolution (for example, 3000*3000) and the minimum resolution (for example, 2000*2000), where e / f=3000 / 3000=2000 / 2000=1 / 1, and the set frame rate g, where g≤90hz, g is a positive integer, for example, g=60hz.

[0308] Then the terminal device 100 can set the data format of the to-be-displayed data to be transmitted between the two ends this time in combination with the application requirement and the format information of the to-be-displayed data supported by the display peripheral 200, and the set data format cannot exceed the maximum data capacity (for example, 3000*3000*90hz here) of the to-be-displayed data supported by the display peripheral 200.

[0309] That is, the terminal device 100 of the present application can reasonably determine the format of the to-be-displayed data to be transmitted through the display interface in the display capability range of the display data supported by the display peripheral 200, in combination with the application requirement and the computing power of the terminal device 100 itself, so that the determined format of the to-be-displayed data is not only within the range of the data format supported by the display peripheral 200, but also can meet the display requirement of the image of the currently running application, and can match the computing power of the terminal device 100 on the to-be-displayed data. In this way, the terminal device 100 of the present application can be compatible with the formats of the display data supported by various display peripherals, so as to be compatible with the data display functions of various display peripherals.

[0310] Optionally, in S104a, the terminal device 100 sends the format of the to-be-displayed data to the display peripheral 200.

[0311] Among them, it can be determined through S103a that the terminal device 100 can transmit what format of to-be-displayed data through the display interface. Then, as shown in Figure 2c , the terminal device 100 can send the format of the to-be-displayed data determined through S103a to the display peripheral 200 through the display interface.

[0312] In S201a, the display peripheral 200 enables and initializes the display module based on the received format of the to-be-displayed data.

[0313] For example, as shown in Figure 2b , the display peripheral 200 can initialize the display rendering driver of the display module and the processing module of the display peripheral 200 according to the format of the to-be-displayed data (such as the display format, resolution, frame rate, etc.) received from the terminal device 100, so that the display peripheral 200 can drive the display module (such as the left eye display screen and the right eye display screen) of the display peripheral 200 to display the display data according to the format of the display data (including the data format, resolution and frame rate) determined by the terminal device 100.

[0314] Continuing to Figure 4Similar to the principles of S103a, S104a, and S201a, after the terminal device 100 obtains the sensing and interaction data capability information of the display peripheral 200 through S102, S103b and S104b can be executed, and the corresponding display peripheral 200 can execute S201b after S104b. Based on the display capability information of the sensing and interaction data of the display peripheral 200, the format of the sensing and interaction data to be transmitted between the terminal device 100 and the display peripheral 200 is uniformly configured.

[0315] The application does not limit the execution order of S103a and S103b, which can be executed in series or in parallel.

[0316] S103b, the terminal device 100 determines the format of the sensing and interaction data to be transmitted based on the sensing and interaction data capability information.

[0317] Optionally, the terminal device 100 can set the format of the sensing and interaction data to be transmitted within the sensing and interaction data capability range supported by the display peripheral 200 in combination with at least one of the application scenario and the computing power of the terminal device 100 for the sensing and interaction data.

[0318] As described above, when the device type of the display peripheral 200 is a general display, the terminal device 100 does not need to obtain the sensing and interaction energy information, and does not need to execute S103b, S104b, S201b, and the following S105b, S106, and S202b.

[0319] When the display peripheral 200 is a wearable display device such as an XR glasses or an XR helmet, the execution of S103b is triggered.

[0320] The execution principle of S103b is the same as that of S103a described above. The terminal device 100 can set the first format of the sensing and interaction data to be transmitted within the format information provided by the sensing and interaction data capability (also referred to as first capability information) supported by the display peripheral 200. The format information is the format information of the sensing and interaction data supported by the display peripheral 200.

[0321] In one possible implementation, the display peripheral 200 is an XR glasses, and the sensing and interaction data capability information (also referred to as first capability information) of the display peripheral 200 can include but is not limited to the following information:

[0322] The supported sensor types include: IMU, 6-10 black and white cameras, 1 depth camera, and 2 VST cameras.

[0323] The format of the data generated by the black and white camera is a Y image stream, and the resolution and frame rate of the Y image stream supported by the black and white camera include 1280*1280*60FPS and 640*480*60FPS.

[0324] The depth camera supports the resolution and frame rate of the generated depth image to be 640*480*30FPS.

[0325] The two VST cameras support generating color image streams (eg, real-scene image streams) in one of YUV and RGB formats. Each color camera supports generating a color image stream with a resolution and frame rate of 4000*3000*60FPS.

[0326] Then the format information of the perception and interaction data supported by the display peripheral 200 that can be provided by the first capability information may include but is not limited to: the number of supported cameras (for example, binocular), support for IMU, support for starting 6 to 10 black and white cameras, support for starting up to one depth camera, support for starting up to two VST cameras. And the data format supported by the black and white camera is Y image stream, the resolution of the Y image stream may include 1280*1280*60FPS and 640*480*60FPS. In addition, the resolution of the Y image stream can be h*h, where h≤1280, h is a positive integer, and the resolution of the Y image stream can also be j*k, where j / k=640 / 480, and the frame rate of the Y image stream is m≤60FPS, m is a positive integer. The principle of the value range of the resolution and frame rate of the above-mentioned depth image and color image stream provided by the first capability information is similar to the principle of the value range of the resolution and frame rate of the above-mentioned Y image stream, and will not be repeated here.

[0327] Then, the terminal device 100 may, within the data capability range of the above-mentioned perception and interaction data supported by the display peripheral device 200, optionally set the format of the perception and interaction data to be transmitted this time in combination with the application scenario and the computing capability of the perception and interaction data. The format may specifically include: the type of sensor to be activated, the data format of the perception and interaction data obtained by the activated sensor (collected by the sensor or after calculation of the data collected by the sensor), the resolution and frame rate of the perception and interaction data, etc.

[0328] For example, the format of the perception and interaction data to be transmitted by the terminal device 100 includes: starting the IMU, starting eight black-and-white cameras (or starting fewer than eight black-and-white cameras), starting a depth camera, and starting two VST cameras. The type and number of sensors started are all within the above data capability range. For example, it is not possible to drive 15 black-and-white cameras because the display peripheral 200 has a maximum of 10 black-and-white cameras that can be used to collect perception and interaction data.

[0329] The format of the perception and interaction data to be transmitted set by the terminal device 100 also includes: the data format of the black and white camera is a Y image stream, and the resolution and frame rate of the Y image stream are 1280*1280*60FPS. The resolution and frame rate of the image captured by the depth camera are 640*480*30FPS. The VST data (color image) captured by the VST camera is in YUV format, with a resolution and frame rate of 4000*3000*60FPS.

[0330] That is, the terminal device 100 of the present application can, within the capabilities of the perception and interaction data supported by the display peripheral 200, optionally combine application requirements and the computing power of the terminal device 100 itself for the perception and interaction data to reasonably determine the format of the perception and interaction data to be transmitted through the perception and interaction interface, so that the determined format of the perception and interaction data is not only within the data format range of the perception and interaction data supported by the display peripheral 200, but also can meet the requirements of the currently running application for perception and interaction data, and can also match the computing power of the terminal device 100 for perception and interaction data. In this way, the terminal device 100 of the present application can be compatible with the formats of the perception and interaction data supported by various display peripherals, thereby being compatible with the perception and interaction data capabilities of various display peripherals.

[0331] In another possible implementation, the display peripheral device 200 is an XR glasses, such as Figure 2b As shown, the processing module of the display peripheral device 200 has a processing unit with strong computing capabilities, allowing the display peripheral device 200 to calculate the perception and interaction data collected by the perception module, interaction module, etc. to obtain the calculation results of the perception and interaction data. The data capabilities of the perception and interaction data obtained by the terminal device 100 from the display peripheral device 200 are: the corresponding sensor type of the calculation result of the perception and interaction data, the data type and data format of the calculation result, etc.

[0332] For example, the data capabilities of the perception and interaction data include: spatial geometric grids, lighting information, gesture points, gaze points, limb key points and other data types. In addition, each data type can also have its own data format.

[0333] Then, the terminal device 100 can select one or more data types and corresponding data formats from the above data types within the scope of the data capability, combined with application requirements and its own computing power.

[0334] S104b, the terminal device 100 sends the format of the perception and interaction data to be transmitted to the display peripheral device 200.

[0335] In S103a, the terminal device 100 determines the format of the perception and interaction data to be transmitted through the perception and interaction interface. Then, as shown in Figure 2c S103b, the terminal device 100 transmits the perception and interaction data to be transmitted to the display device 200 through the perception and interaction interface.

[0336] S201b, the display device 200 enables and initializes the sensors based on the format of the perception and interaction data to be transmitted received from the terminal device 100.

[0337] For example, as shown in Figure 2b S104b, the display device 200 configures the sensors based on the format of the perception and interaction data to be transmitted received from the terminal device 100.

[0338] For example, although the display device 200 supports sensor types including IMU, 6-10 black and white cameras, 1 depth camera, and 2 VST cameras, according to the format of the perception and interaction data to be transmitted determined by the terminal device 100, the display device 200 only initializes and starts the IMU and 6 black and white cameras, 1 depth camera, and 2 VST cameras. In addition, the display device 200 can configure the resolution and frame rate of the black and white cameras to be smaller, such as 640*480*60FPS, when initializing the configuration of the above-mentioned sensors according to the format of the perception and interaction data to be transmitted received in S104b.

[0339] Continuing to Figure 4 After S103a, the terminal device 100 can also perform S105a and S106a, and the corresponding display device 200 can perform S202a after S106a. To achieve the configuration of the display data transmission link of the terminal device 100 and the initialization of the display data transmission link of the display device 200.

[0340] Optionally, S105a, the terminal device 100 configures the display data transmission link (also referred to as the second transmission link) based on the capability information of the display data transmission link (also referred to as the fourth capability information) and the format of the display data to be displayed determined by the above-mentioned S103a.

[0341] The capability information of the transmission link of the display data can provide a maximum transmission rate supported by the transmission link of the display device 200 for the display data, for example, a transmission rate corresponding to x4. Then the terminal device 100 can determine a maximum data amount maxl of the display data that can be transmitted by the transmission link based on the maximum transmission rate.

[0342] Similarly, after determining the format of the to-be-displayed data, the terminal device 100 can determine a maximum data amount max2 of the to-be-displayed data sent by the terminal device 100 to the outside through the display interface based on the format of the to-be-displayed data, for example, a product of resolution and frame rate.

[0343] Then the terminal device 100 can determine fourth link information according to the format of the to-be-displayed data within the range of the maximum capability of the transmission link of the display device, and configure the transmission link for transmitting the display data based on the fourth link information.

[0344] The fourth link information can include an indication of whether to start compression encoding, and optionally, at least one of the following: a number of channels (for example, x4) allocated to the second transmission link, a data sending frequency, a data receiving and sending format of the second transmission link, and the like.

[0345] Wherein, when the terminal device 100 determines that the maximum data amount max2 is greater than the maximum data amount maxl that can be transmitted by the transmission link of the display device 200 for the display data, it means that the maximum bandwidth of the transmission link of the display device 200 is difficult to support the transmission of the data amount max2 of the to-be-displayed data from the terminal device 100. Then the terminal device 100 can configure to start compression encoding for the to-be-displayed data transmitted, so that the data amount of the compressed to-be-displayed data is less than or equal to the maximum data amount maxl when configuring the transmission link of the display data.

[0346] In this way, when the maximum capability of the transmission link of the display device 200 for transmitting the to-be-displayed data is less than the maximum data amount max2 of the to-be-displayed data sent by the terminal device 100 through the display interface of the terminal device, the to-be-displayed data transmitted by the terminal device 100 can be compressed and encoded, and the compressed and encoded data can be sent to the display device 200 through the transmission link of the display data, so that the transmission of the XR display data can be realized in the case of low latency and low bandwidth when the data amount of the to-be-displayed data is large.

[0347] On the contrary, when the terminal device 100 determines that the maximum data amount max2 is less than or equal to the maximum data amount maxl that the transmission link of the display device 200 can support, it indicates that the maximum bandwidth of the display device 200 can support the transmission of the data amount of the display data from the terminal device 100. Then, the terminal device 100 can configure the transmission link of the display data to be closed to the compression encoding of the display data to be transmitted.

[0348] In S106a, the terminal device 100 sends the configuration information (for example, the fourth link information) of the transmission link of the display data to the display device 200.

[0349] After S105a, the terminal device 100 can send the configuration information after the configuration of the transmission link of the display data in S105a to the display device 200, which can include but is not limited to: indicating whether to start compression encoding, and optionally, at least one of the following: the number of channels (for example, x4) allocated to the second transmission link, the data transmission frequency, the data transmission format of the second transmission link, and the like. So that the display device 200 can configure the transmission link (also referred to as the second transmission link) of the display data on the display device 200 side based on the configuration information, and the display device 200 can decompress and decode the compressed display data received.

[0350] In one possible implementation, in S202a, the display device 200 configures and initializes the format of the transmission link of the display data on the display device 200 based on the received configuration information of the transmission link of the display data.

[0351] In one possible implementation, in S202a, the display device 200 configures and initializes the format of the transmission link of the display data on the display device 200 based on the received configuration information of the transmission link of the display data.

[0352] In this way, the display device 200 can configure the transmission link of the display data on the display device 200 according to the configuration information of the transmission link of the display data of the terminal device 100, and start the corresponding transmission link according to the corresponding configuration, for example, start the channel number x4.

[0353] Optionally, after S202a, the terminal device 100 can further perform link state confirmation of the display data transmission link with the display device 200, complete the handshake of the data transceiver end, and the implementation can refer to the link state confirmation process of the transmission link in the prior art, which is not limited or described herein.

[0354] In a possible implementation, continuing to return to Figure 4 After S103b, the terminal device 100 can further perform S105b and S106b, and the corresponding display device 200 can perform S202b after S106b. To realize the configuration of the perception and interaction data transmission link of the terminal device 100 and the initialization of the perception and interaction data transmission link of the display device 200. The implementation principle of the process is the same as that of the setting process of the display data transmission link described above S105a, S106a, S202a, which is not described herein.

[0355] Optionally, S105b, the terminal device 100 configures the perception and interaction data transmission link (also referred to as the first transmission link) based on the capability information (also referred to as the second capability information) of the perception and interaction data transmission link, and the format (also referred to as the first format) of the perception and interaction data to be transmitted determined by S103b.

[0356] The capability information of the perception and interaction data transmission link can provide the maximum transmission rate supported by the perception and interaction data transmission link of the display device 200, for example, the transmission rate corresponding to x8. Then the terminal device 100 can determine the maximum data amount max3 of the perception and interaction data that can be transmitted by the transmission link based on the maximum transmission rate.

[0357] Similarly, after determining the format of the perception and interaction data, the terminal device 100 can determine the maximum data amount max4 of the perception and interaction data received by the terminal device 100 through the perception and interaction interface based on the format of the perception and interaction data to be transmitted, for example, the product of the resolution and the frame rate corresponding to each started sensor, and the product is accumulated based on the number of started sensors to obtain the maximum data amount max4.

[0358] Then the terminal device 100 can determine the second link information according to the format (the first format described above) of the perception and interaction data within the maximum capability range of the display device for the transmission link, and configure the transmission link for transmitting the perception and interaction data based on the second link information.

[0359] The second link information can include an indication of whether to start decompression decoding, and can optionally include at least one of the following: a number of channels allocated to the first transmission link (for example, x4), a receiving frequency of data, a transceiving data format of the first transmission link, and the like.

[0360] When the terminal device 100 determines that the maximum data amount max4 is greater than the maximum data amount max3 that the transmission link of the display peripheral 200 supporting the perception and interaction data can transmit, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 cannot support the transmission of the perception and interaction data with the data amount max4 to the terminal device 100, which exceeds the transmission capability of the transmission link of the perception and interaction data of the display peripheral 200. Then, the terminal device 100 can configure to start decompression decoding of the transmitted perception and interaction data when configuring the transmission link of the local perception and interaction data, wherein the data amount of the compressed and encoded perception and interaction data is less than or equal to the maximum data amount max3.

[0361] In this way, when the maximum capability of the transmission link of the display peripheral 200 for transmitting the perception and interaction data is less than the maximum data amount max4 of the perception and interaction data sent by the display peripheral 200 to the terminal device 100, the display peripheral 200 can compress and encode the perception and interaction data to be sent, and then send the compressed and encoded data to the terminal device 100 through the bilateral transmission link of the perception and interaction data, so that the transmission of the perception and interaction data can be realized in the case of low latency and low bandwidth when the data amount of the perception and interaction data to be sent by the display peripheral 200 is large, and the transmission of the display data of XR is realized.

[0362] On the contrary, when the terminal device 100 determines that the maximum data amount max4 is less than or equal to the maximum data amount max3 that the transmission link of the perception and interaction data of the display peripheral 200 can support, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 can support the transmission of the perception and interaction data with the data amount max4 to the terminal device 100. Then, the terminal device 100 can configure to close the decompression decoding of the received perception and interaction when configuring the transmission link of the local perception and interaction data.

[0363] S106b, the terminal device 100 sends configuration information of the transmission link of the perception and interaction data to the display peripheral 200.

[0364] After S105b, the terminal device 100 can send the configuration information after configuring the transmission link of the local sensing and interaction data in S105b to the display device 200, for example, whether to start decompression decoding, and optionally, the channel (for example, x8) of the transmission link can be allocated, the receiving frequency of the configuration data, the transceiving data format of the transmission link is configured, etc., so that the display device 200 can configure the transmission link of the sensing and interaction data on the display device 200 side based on the configuration information, and the display device 200 can compress and encode the sensing and interaction data to be sent.

[0365] In a possible implementation, S202b, the display device 200 confirms and initializes the format of the transmission link of the sensing and interaction data of the display device 200 based on the received configuration information of the transmission link of the sensing and interaction data.

[0366] In a possible implementation, S202b, the display device 200 confirms and initializes the format of the transmission link of the sensing and interaction data of the display device 200 based on the received configuration information of the transmission link of the sensing and interaction data.

[0367] In this way, the display device 200 can perform corresponding matching configuration on the transmission link of the sensing and interaction data on the display device 200 side according to the configuration information of the terminal device 100 on the transmission link of the sensing and interaction data of the display device 200, and start the corresponding transmission link according to the corresponding configuration, for example, the number of channels is x8.

[0368] Optionally, after S202b, the terminal device 100 can also perform link state confirmation on the transmission link of the sensing and interaction data with the display device 200, complete the handshake of the data transceiving end, and the specific implementation can refer to the link state confirmation process of the transmission link in the prior art, which is not limited and elaborated here.

[0369] Continuing to Figure 2c , the terminal device 100 can generate display data based on Figure 4 the configuration of the transmission link of the display data in S105a, the terminal device 100 can determine whether to compress and encode the display data.

[0370] In a possible implementation, the transmission link of the display data of the terminal device 100 in S105a is configured to start compression and encoding. Then, as shown in Figure 4 , the process can also include S107a, S108a, S203a.

[0371] S107a: The terminal device 100 compresses and encodes the data to be displayed.

[0372] As described above, the amount of data to be displayed generated by the terminal device 100 is max2, which exceeds the transmission capacity of the data to be displayed supported by the display peripheral device 200 (for example, the maximum data amount is max1). Figure 2c As shown, the terminal device 100 can use the encoding module for displaying data to compress and encode the generated data to be displayed to obtain a code stream of the data to be displayed.

[0373] S108a: The terminal device 100 sends a code stream of data to be displayed to the display peripheral device 200.

[0374] like Figure 2c As shown, the terminal device 100 can send the compressed code stream to the display peripheral device 200 through the display interface of the local end and the transmission link of the display data configured at the local end.

[0375] S203a, the display peripheral device 200 decodes the received data to be displayed.

[0376] like Figure 2c As shown, after the display data transmission link of the display interface of the display peripheral 200 receives the code stream of the data to be displayed, the code stream can be decompressed and decoded by the display data decoding module to obtain the data to be displayed for display on the screen.

[0377] In a possible implementation, when the amount of data to be displayed generated by the terminal device 100 is max2 and does not exceed the transmission capacity of the data to be displayed supported by the display peripheral 200 (for example, the maximum amount of data is max1), the compression encoding is not started in the above S105a. Then, when the terminal device 100 sends the data to be displayed to the display peripheral 200, it is not necessary to go through the encoding operation of the encoding module of the display data of the terminal device 100. After receiving the code stream of the data to be displayed, the display peripheral 200 does not need to go through the encoding operation of the encoding module of the display data of the terminal device 100. Figure 2c The decoding module of the display data shown performs decoding operation and can directly transmit the data to be displayed according to the transmission link configured by both parties.

[0378] Continue back Figure 2c , the display peripheral device 200 can generate perception and interaction data based on Figure 4 In S105b , the transmission link for the perception and interaction data is configured, and the display peripheral device 200 can determine whether to compress and encode the perception and interaction data.

[0379] In a possible implementation, the transmission link configuration of the perception and interaction data of the terminal device 100 in S105b starts decompression and decoding.Figure 4 As shown, the process can further include S203b, S204b, S107b.

[0380] S203b, the display peripheral 200 compressively encodes the generated perception and interaction data.

[0381] As described above, the data volume of the perception and interaction data generated by the display peripheral 200 is max4, which exceeds the transmission capability of the perception and interaction data supported by the display peripheral 200 for transmission (for example, the maximum data volume is max3). As shown, the display peripheral 200 can utilize the encoding module of the perception and interaction data to compressively encode the generated perception and interaction data to obtain the code stream of the perception and interaction data. Figure 2c

[0382] S204b, the display peripheral 200 transmits the code stream of the perception and interaction data to the terminal device 100.

[0383] As shown, the display peripheral 200 can transmit the compressed code stream to the terminal device 100 through the perception and interaction interface of the local terminal and the transmission link of the perception and interaction data configured by the local terminal. Figure 2c

[0384] S107b, the terminal device 100 decompressively decodes the received code stream of the perception and interaction data.

[0385] As shown, after the transmission link of the perception and interaction interface of the terminal device 100 receives the code stream of the perception and interaction data, the terminal device 100 can utilize the decoding module of the perception and interaction data to decompressively decode the code stream to obtain the perception and interaction data. Figure 2c

[0386] In a possible implementation, when the data volume of the perception and interaction data generated by the display peripheral 200 is max4, which does not exceed the transmission capability of the perception and interaction data supported by the display peripheral 200 for transmission (for example, the maximum data volume is max2), the decompressive decoding in S105b is not started, and then the display peripheral 200 does not need to perform the encoding operation of the encoding module of the perception and interaction data when transmitting the perception and interaction data to the terminal device 100. After the terminal device 100 receives the code stream of the perception and interaction data, the terminal device 100 does not need to perform the decoding operation through the decoding module of the perception and interaction data as shown. The transmission of the perception and interaction data can be directly performed according to the transmission link configured by both parties. Figure 2c

[0387] In addition, the order between the two steps of transmitting the to-be-displayed data by the terminal device 100 to the display peripheral 200 and transmitting the perception and interaction data by the display peripheral 200 to the terminal device 100 is not limited.​​​​

[0388] For example, in Figure 4 , S204b can be performed prior to S108a, in other words, the display peripheral 200 can first send the collected perception and interaction data to the terminal device 100, and then the terminal device 100 sends the to-be-displayed data to the display peripheral 200.

[0389] In one possible implementation, the terminal device 100 can obtain the to-be-displayed data by using the perception and interaction data (the to-be-displayed data can be generated by the terminal device 100, or obtained from the cloud and generated by the cloud).

[0390] In one possible implementation, the terminal device 100 can also compress and encode the obtained to-be-displayed data by using the perception and interaction data.

[0391] The process of compressing and encoding the to-be-displayed data by using the perception and interaction data by the terminal device 100 will be described below in conjunction with Figure 5a .

[0392] The to-be-displayed data can be image or video data. As explained above for the first image sequence, the to-be-displayed data can be monocular image (i.e., one frame of image), binocular image (i.e., two frames of image), or multiocular image (i.e., more than two frames of image).

[0393] In Figure 5a , taking the to-be-displayed data as binocular to-be-displayed data as an example, the to-be-displayed data can include a first-view image and a second-view image. Taking the display peripheral 200 as having a left-eye display screen and a right-eye display screen as an example, the first-view image can be an image to be displayed on the left-eye display screen, and the second-view image can be an image to be displayed on the right-eye display screen.

[0394] Similarly, the perception and interaction data can also include monocular perception and interaction data, or binocular perception and interaction data, or multiocular perception and interaction data. For example, the perception and interaction data can include first-view data, and optionally, second-view data, for example, the first-view data is perception and interaction data about the left eye, and the second-view data is perception and interaction data about the right eye.

[0395] Although Figure 5a the process of compressing and encoding the to-be-displayed data is described by taking binocular as an example, the method of the present application is not limited to the binocular scenario, and when the to-be-displayed data includes multiocular (the number of views is greater than 2) to-be-displayed data, the method is the same, and thus will not be described here.

[0396] In some embodiments, the terminal device 100 can encode and compress the first-view image based on the first-view data, and encode and compress the second-view image based on the second-view data, to obtain a bitstream of the to-be-displayed data.

[0397] In the embodiment, in a binocular or multi-view scene with two or more views, the to-be-displayed data of the respective views can be encoded in combination with the perception and interaction data, to reduce the code rate and the latency. The principle of the process of encoding the to-be-displayed data of the respective views by using the perception and interaction data can refer to the principle of the implementation process of S3011 in the embodiment of Figure 5a , which will not be repeated here.

[0398] In the Figure 5a embodiment, the image data of the to-be-displayed data of the two views can be compressed in sequence, for example, the first-view image is compressed first, and then the second-view image is compressed. Since the distance between the binocular cameras (for example, the left-eye camera and the right-eye camera) on the display peripheral 200 is relatively small, at the same time, the contents of the two images (the first-view image and the second-view image) captured by the binocular cameras are relatively similar. Therefore, the method of the present application can use the first-compressed image as the reference image of the second-compressed image, so as to remove the redundant information and reduce the code rate.

[0399] As shown in Figure 5a , the process can include the following steps:

[0400] S301, encoding and compressing the first-view image in the to-be-displayed data to obtain a bitstream of the first-view image and a reconstructed image of the first-view image.

[0401] The reconstructed image is generated by encoding and compressing the first-view image to obtain a bitstream of the first-view image, and then decoding and decompressing the bitstream to obtain the reconstructed image of the first-view image.

[0402] S302, preprocessing the reconstructed image of the first-view image to obtain a similar image.

[0403] The similar image is an image obtained by preprocessing the first-view image and matching the second-view image.

[0404] The preprocessing can be morphing conversion, projection transformation between two images (for example, projection transformation based on depth information, etc.), and the like, which will not be limited here.

[0405] Through S302, the reconstructed image can be processed into an image (hereinafter referred to as a similar image) with more similar (or matching) pixel content to the second-view image.

[0406] S303: Based on the similar image, compress and encode the second image in the data to be displayed to obtain a code stream of the second image.

[0407] The residual between the second image and the similar image can be calculated, and the residual can be compressed and encoded to obtain a code stream of the second image.

[0408] In an embodiment of the present application, when compressing the second image, the reconstructed image of the first image can be first processed (for example, deformation transformation) to process the reconstructed image into a similar image that better matches the pixel content of the second image; then, based on the second image, the similar image is referenced to obtain a residual, and the residual is used to compress and encode the second image, so that the redundant information of the two images can be used to compress the second image, thereby improving the compression efficiency of the data to be displayed.

[0409] In a specific implementation, the relevant parameter data involved in the compression encoding process of the dual-purpose display data can be added to the code stream of the first-purpose image or the code stream of the second-purpose image, so that the display peripheral 200 can accurately decode the code stream of the display data.

[0410] Then, in the above S301 to S303, any step can refer to the perception and interaction data to implement the corresponding step.

[0411] like Figure 5a As shown, in a possible implementation manner, when executing the above S301, it can be implemented through S3011.

[0412] S3011: compress and encode a first image in the data to be displayed based on the first data in the perception and interaction data to obtain a code stream of the first image and a reconstructed image of the first image.

[0413] When compressing and encoding the first-purpose image, the first-purpose data in the perception and interaction data may be referred to to assist in image compression.

[0414] For example, the first eye data may include a depth image captured by a depth camera for the left eye.

[0415] Then the terminal device 100 can use the depth image to obtain the first destination depth information.

[0416] When using the first-order data for auxiliary compression, the depth information can be used, for example, to adjust the compression quality of an image region in the compressed image (here, the first-order image). The compression quality can be adjusted by adjusting encoding parameters of the encoder, such as adjusting the encoder's quantization coefficient or quantization step size.

[0417] For example, image areas corresponding to larger depths can be compressed with lower quality (for example, when encoding, the quantization coefficient is set larger and the quantization step size is set smaller), and image areas corresponding to smaller depths can be compressed with higher quality (for example, when encoding, the quantization coefficient is set smaller and the quantization step size is set larger).

[0418] like Figure 5a As shown, in a possible implementation manner, when executing the above S302, it can be implemented through S3021.

[0419] S3021: Preprocess the reconstructed image of the first image based on the first data in the perception and interaction data to obtain a similar image.

[0420] In one example, taking deformation conversion as an example of the preprocessing, the first-object data may include a depth image captured by a depth camera for the left eye. The terminal device 100 may use this depth image to obtain first-object depth information for constructing a deformation function. The deformation function is then used to perform deformation processing on the reconstructed image of the first-object image to obtain a similar image.

[0421] Compared with the similar image obtained through S302, the deformation function constructed based on the first target depth information is more accurate. Therefore, the similar image obtained after deforming the reconstructed image using the more accurate deformation function can be closer to the second target image.

[0422] Thus, in this embodiment, during the process of preprocessing the reconstructed image of the first image, the reconstructed image can be preprocessed with reference to the perception and interaction data. Since the perception and interaction data provides depth information about the first image, the similar image obtained based on the perception and interaction data can be closer to the second image, thereby reducing the bit rate.

[0423] like Figure 5a As shown, in a possible implementation manner, when executing the above S303, it can be implemented through S3031.

[0424] S3031: compress and encode the second image in the display data based on the second data in the perception and interaction data and the similar image to obtain a code stream of the second image.

[0425] The principle of using the perception interaction data in S3031 to compress the second image is similar to the principle of the above-mentioned S3011. Please refer to the introduction of S3011 for details and will not be repeated here.

[0426] so, Figure 5a An example description Figure 4 S107a is the process of compression encoding of display data.

[0427] In binocular or multi-eye scenarios, this application can use the depth information of a single eye to assist in compressing multi-eye images to increase the bit rate and reduce bandwidth requirements.

[0428] In another scenario, not limited to binocular or multi-eye scenarios, the data to be displayed can be compressed and encoded based on the eye movement information calculated from the eye movement images in the perception and interaction data.

[0429] Specifically, eye movement information can be extracted from the eye movement images captured by the eye tracking camera in the perception and interaction data to determine the area where the line of sight is focused (i.e., the gaze point) in the data to be displayed. Then, a higher compression quality is used for the area at the gaze point in the data to be displayed, and a lower compression quality can be used for the area outside the gaze point in the data to be displayed. The compression quality can be adjusted by adjusting the encoding parameters of the encoder, such as adjusting the quantization coefficient of the encoder, or adjusting the quantization step size, etc. In this way, the image quality of the image area on which the line of sight can be focused after compression is higher than the image quality of the image area outside the image area after compression. Of course, this scheme of using eye movement information to assist in compressing the data to be displayed can also be applied to binocular or multi-purpose scenarios.

[0430] exist Figure 5a The perception and interaction data used in S3011, S3021, and S3031 shown may be eye movement information, and the first eye data and the second eye data are both the same eye movement information.

[0431] In addition, Figure 5a In S3011, S3021, and S3031, the perception and interaction data used can be depth information alone, for example, the first-eye data is the first-eye depth information and the second-eye data is the second-eye depth information. It can also be eye movement information alone, or it can include depth information and eye movement information. There is no limitation here.

[0432] Accordingly, refer to Figure 4 When the display peripheral device 200 performs S203a to decode the code stream of the data to be displayed to obtain the data to be displayed, Figure 5a The decoding process is implemented by the inverse process of the process shown in FIG. 1 , and the principle of the decoding process corresponds to the principle of the encoding process.

[0433] For example, the display of the peripheral device 200 is as follows Figure 2c The decoding module for displaying the data shown in FIG. 1 may first decode the encoded first image to obtain a reconstructed image of the first image; then, the reconstructed image of the first image is processed by the encoding end (eg Figure 2cThe same preprocessing operation (e.g., deformation transformation) corresponding to the encoding module of the display data shown in FIG is performed to obtain a reconstructed image after preprocessing (also referred to as a similar image). In addition, as Figure 2c The display data decoding module shown can also decode the code stream of the second image and, based on the decoded result of the second image and the aforementioned similar image, obtain a reconstructed image of the second image. Specifically, the code stream of the second image can be decoded to obtain a residual of the second image; this residual is then processed (e.g., superimposed) with the aforementioned similar image to obtain a reconstructed image of the second image. In this way, the reconstructed image of the first image and the reconstructed image of the second image are decoded. Parameter data for the processing process of the reconstructed image of the first image can be obtained by parsing the code stream.

[0434] In the above decoding process, similar to the above S3011, S3021, and S3031, the display peripheral 200 can refer to the first-order data in the perception and interaction data when preprocessing the reconstructed image of the first-order image. In addition, when decoding the code stream of the first-order image, it can refer to the first-order data in the perception and interaction data. In addition, when decoding the code stream of the second-order image, it can refer to the first-order data in the perception and interaction data.

[0435] In an embodiment of the present application, the terminal device 100 can use the perception and interaction data to compress and encode the data to be displayed, and the display peripheral 200 can also use the perception and interaction data to decompress and decode the encoded code stream of the data to be displayed, so that effective perception and interaction information can be reasonably used to assist in the compression processing of the data to be displayed, which can make the encoding prediction more accurate and improve the compression efficiency. Moreover, the perception and interaction data can be used to adjust the encoding parameters of the encoder of the terminal device 100 (such as the encoding module of the display data) to make the encoded image quality more suitable for the human eye senses. Therefore, this method can obtain higher compression efficiency than the prior art, reduce the bit rate and the bandwidth required to transmit the code stream of the data to be displayed, and reduce latency. Moreover, the image quality of the video generated by decoding is also better, which can enhance the user's viewing experience.

[0436] based on Figure 4 In S203b, the process of encoding the perception and interaction data by the display peripheral device 200 is described below.

[0437] For example, the perception and interaction data to be encoded may be binocular data, and the perception and interaction data may include data collected by various sensors of the display peripheral device 200. Figure 2bAs shown, the perception and interaction data can include, but is not limited to, at least one of the following: black and white images, depth images, IMU data, and the like obtained from the perception module and the interaction module; VST data (e.g., real scene images, and the like) obtained from the VST module, and the like. The perception and interaction data can be a single image or a video obtained based on a period of time. The perception and interaction data can include first-eye data (e.g., perception and interaction data about the left eye) and second-eye data (e.g., perception and interaction data about the right eye). Alternatively, more than two eyes can be included, which is not limited herein.

[0438] The perception and interaction data can be divided into two categories: one category is VST data, and the other category is other perception and interaction data than the VST data.

[0439] First, the encoding process of the VST data is introduced.

[0440] For example, as shown in the encoding module of the perception and interaction data of the display peripheral 200, the VST data can be compressed and encoded based on other perception and interaction data than the VST data. Figure 2c

[0441] The process of the display peripheral 200 compressing and encoding the VST data based on other perception and interaction data than the VST data is described below in combination with Figure 5b

[0442] As described above, taking the VST data as visual image data for example, the visual image data can include first-eye images and second-eye images.

[0443] In this embodiment, the image data of each eye of the two-eye VST data can be compressed in sequence, for example, the first-eye images are compressed first, and then the second-eye images are compressed. Since the distance between the two-eye cameras (e.g., the left-eye camera and the right-eye camera) on the display peripheral 200 is relatively close, at the same time, the contents of the two images (the first-eye images and the second-eye images) captured by the two-eye cameras are relatively similar. Therefore, the method of the present application can use the first compressed image as a reference image of the second compressed image, so as to remove the redundant information and reduce the code rate.

[0444] As shown in the process, the process can include the following steps: Figure 5b

[0445] S401, the first-eye images in the VST data are compressed and encoded to obtain the code stream of the first-eye images and the reconstructed images of the first-eye images.

[0446] S402, the reconstructed images of the first-eye images are preprocessed to obtain similar images.

[0447] ​​​The similar image is an image obtained after preprocessing the first image.

[0448] S403: Based on the similar image, compress and encode the second image in the VST data to obtain a code stream of the second image.

[0449] In an embodiment of the present application, when compressing the second image in the VST data, the reconstructed image of the first image can be processed once (for example, deformation transformation) to process the reconstructed image into a similar image that better matches the pixel content of the second image; then, based on the second image, the similar image is referenced to obtain a residual, and the residual is used to perform compression encoding of the second image, so that the second image can be compressed using the redundant information of both images to improve the compression efficiency of the VST data.

[0450] In specific implementation, the relevant parameter data involved in the compression encoding process of the dual-purpose VST data can be added to the code stream of the first-purpose image or the code stream of the second-purpose image, so that the terminal device 100 can accurately decode the code stream of the VST data.

[0451] Then, in the above S401 to S403, any step can refer to other perception and interaction data other than VST data to implement the corresponding step.

[0452] like Figure 5b As shown, in a possible implementation manner, when executing the above S401, it can be implemented through S4011.

[0453] S4011: compress and encode the first image in the VST data based on the first data in the other perception and interaction data other than the VST data to obtain a code stream of the first image and a reconstructed image of the first image.

[0454] When compressing and encoding the first-purpose image, reference may be made to the first-purpose data in other perception and interaction data other than the VST data to assist in image compression.

[0455] Among them, the other perception and interaction data other than the VST data used in S4011, S4021, and S4031 are the other perception and interaction data other than the original VST data that has not been compressed.

[0456] In addition to the VST data, other perception and interaction data can be acquired. The other perception and interaction data can be decompressed to obtain the original other perception and interaction data, or the other perception and interaction data can be used to assist in compressing the VST data before the other perception and interaction data is compressed. The application does not limit the implementation.

[0457] As shown in Figure 5b , in a possible implementation, when S402 is performed, S4021 can be used to implement S402.

[0458] S4021, based on the first-view data in the other perception and interaction data than the VST data, pre-processes the reconstructed image of the first-view image to obtain a similar image.

[0459] As shown in Figure 5b , in a possible implementation, when S403 is performed, S4031 can be used to implement S403.

[0460] S4031, based on the second-view data in the other perception and interaction data than the VST data and the similar image, compresses and encodes the second-view image in the VST data to obtain a code stream of the second-view image.

[0461] In this way, Figure 5b , the process of S203b in Figure 4 compressing and encoding the VST data in the perception and interaction data is described.

[0462] Figure 5b The implementation principle of the process shown in Figure 5a is the same as the implementation principle of the process shown in Figure 5a , and the only difference is that the objects to be compressed are different (the to-be-displayed data and the VST data, respectively), and the difference is that the perception and interaction data referred to when compressing the object can be different. In Figure 5b , the perception and interaction data can be the VST data or the other perception and interaction data than the VST data, and in Figure 5b , the perception and interaction data is the other perception and interaction data than the VST data. The principles of other processes are the same, and therefore, the specific implementation details of Figure 5a are not described in detail here, and can be referred to the description of .

[0463] Correspondingly, referring to Figure 4When the terminal device 100 decodes the code stream of the VST data in the perception and interaction data to obtain the VST data by performing S107b, the terminal device 100 can Figure 5b The decoding process is implemented in the reverse process of the encoding process shown in FIG. 7, and the principle of the decoding process is corresponding to the principle of the encoding process. Details are not described herein again. For details, refer to the principle of decoding the code stream of the to-be-displayed data by the display peripheral 200 to obtain the to-be-displayed data, which is not described herein again.

[0464] In the embodiments of the present application, the display peripheral 200 can compress and encode the VST data by using other perception and interaction data in addition to the VST data, and the terminal device 100 can also decompress and decode the code stream of the encoded VST data by using other perception and interaction data in addition to the VST data, so that the effective perception and interaction information can be reasonably used to assist the compression of the VST data, the encoding prediction can be made more accurate, and the compression efficiency can be improved. Moreover, the encoding parameters of the encoder (for example, the encoding module of the perception and interaction data) of the display peripheral 200 can be adjusted by using the other perception and interaction data, so that the image quality of the encoded image is more suitable for human visual sense. Therefore, the method can obtain higher compression efficiency than the prior art, reduce the code rate and the bandwidth required for transmitting the code stream of the VST data, and reduce the time delay. Moreover, the quality of the video generated by decoding is also better, and the user's viewing experience can be improved.

[0465] As described above, the perception and interaction data can include not only the VST data but also other perception and interaction data in addition to the VST data, such as at least one of black-and-white images, depth images, IMU data corresponding to spatial data, gesture data, eye movement data, face data, body data, and depth data.

[0466] In some embodiments, the data amount of the other perception and interaction data is relatively small compared with the VST data, so the display peripheral 200 can not compress and encode the other perception and interaction data, but can directly transmit the other perception and interaction data to the terminal device 100.

[0467] In some embodiments, the display peripheral 200 can also encode the other perception and interaction data based on Figure 5b The other perception and interaction data is encoded by using the encoding process for encoding the VST data shown in FIG. 7, and the principle of encoding the other perception and interaction data is implemented by using the principles of S401, S402, and S403 shown in FIG. 4 (specifically, the compressed data object is replaced from the VST data to the other perception and interaction data), without the aid of the perception and interaction data for auxiliary compression (that is, S4011, S4021, and S4031 are not performed). Figure 5b The other perception and interaction data is encoded by using the encoding process for encoding the VST data shown in FIG. 7, and the principle of encoding the other perception and interaction data is implemented by using the principles of S401, S402, and S403 shown in FIG. 4 (specifically, the compressed data object is replaced from the VST data to the other perception and interaction data), without the aid of the perception and interaction data for auxiliary compression (that is, S4011, S4021, and S4031 are not performed).

[0468] Similarly, the terminal device 100 can also decompress the code stream of the compressed other perception and interaction data from the display peripheral 200 according to the principle of the inverse process of the process of S401, S402, S403 in the method, to obtain the other perception and interaction data. Figure 5b

[0469] The implementation process of the processing method of the present application will be described below in combination with specific examples.

[0470] Example 1

[0471] Figure 6 The process schematic diagram of the processing method of the present application is exemplarily shown.

[0472] In the Figure 6 , the display peripheral 200 connected with the terminal device 100 is specifically an MR device 301.

[0473] The MR device 301 can be an MR glasses or an MR helmet, etc.

[0474] Figure 6 The two devices shown are Figure 2b A specific schematic diagram of the two devices shown.

[0475] In the present example 1, the processing chip of the MR device 301 has strong computing capability, which can include a perception and interaction data computing module, which can be used to calculate other perception and interaction data in addition to the real scene image, and send the calculated perception and interaction data (i.e. the calculation result of the perception and interaction data) to the terminal device 100.

[0476] For example, as shown in Figure 6 , the calculated perception and interaction data can include but are not limited to: the result of spatial positioning, spatial geometric grid, lighting information, gesture point, gaze point, face geometry, limb key point, positioning local map, expression base, etc.

[0477] In addition, since the MR device 301 is equipped with a strong processing chip, it can directly complete the link closed loop from the VST camera (one or more sensor modules) to the display of the VST image, therefore, the MR device 301 does not need to send VST data to the terminal device 100.

[0478] In addition, the MR device 301 has a display data processing module, which can be used to perform virtual-real synthesis and other calculation processing on the image to be displayed, therefore the format of the display data supported by the MR 301 is a virtual frame in RGBAZ format or RGBA format.

[0479] ​Specifically, the terminal device 100 can acquire the data capability information of the display data of the MR device 301 and the capability information of the transmission link thereof, the data capability information of the perception and interaction data and the capability information of the transmission link thereof, and determine the format of the to-be-transmitted display data, the format of the to-be-transmitted perception and interaction data, and the link configuration of the two corresponding transmission links based on the various capability information, and instruct the display peripheral 200 to perform corresponding initialization according to the determined various information.

[0480] For example, in the scenario of Figure 6 , the terminal device 100 can determine that the format of the to-be-transmitted display data is RGBAZ or RGBA format, and determine that the resolution of each-eye image in the display data supported by the MR device 301 is full high definition resolution (or also can be the resolution of gaze point rendering display).

[0481] In addition, the terminal device 100 can determine that in the case of rendering each-eye image at full high definition resolution, the maximum transmission rate of the transmission link of the display data can reach 100 Gbps (G represents gigabyte, bps represents bit per second), and compression encoding needs to be started.

[0482] For example, in the scenario of Figure 6 , the terminal device 100 can determine that the format of the to-be-transmitted perception and interaction data includes the format of the calculation result of the perception and interaction data, such as the format of the above-mentioned spatial positioning result, spatial geometric grid, illumination information, gesture point, gaze point, Blender shape, limb key point, subMap, positioning local map, expression base, etc.

[0483] In addition, the data amount of the above-mentioned calculation result is small, and compression encoding can not be started.

[0484] Then, as shown in Figure 6 , the MR device 301 can collect perception and interaction data through a sensor module. The sensor module can include but is not limited to 2 VST cameras, 4 Mono cameras, 2 eye tracking cameras, 2 downward cameras, a depth camera, an IMU, etc., and the application does not limit the type and number of sensors.

[0485] Among them, the VST camera can collect VST data (also referred to as VST image, or real scene image), the Mono camera can collect Mono image (also referred to as Y image in the above), the eye tracking camera can collect eye movement image (since the eye tracking camera is also a Mono camera, it is also referred to as Y image or Mono image, or black and white image), the downward camera can collect downward image (since the downward camera is also a Mono camera, it is also referred to as Y image or Mono image, or black and white image), the depth camera can collect depth image, and the IMU can collect motion information.

[0486] Then, the ISP of the MR device 301 can perform calculation on the collected perception and interaction data, wherein the real scene image from the VST camera can be calculated to obtain a signal of the real scene image, and the signal of the real scene image is sent to the processing module of the display data for fusion with the virtual frame.

[0487] In addition, the ISP can further calculate the calculated perception and interaction data by the calculation module of the perception and interaction data to obtain the calculation result of the perception and interaction data. The calculation can include but is not limited to the following aspects: spatial geometric grid, illumination, gesture, eye movement, face, body, etc. The perception and interaction data can include but is not limited to at least one of the following: data from the ISP, motion information from the IMU, depth information obtained by performing depth calculation on the depth image collected by the depth camera, etc.

[0488] Here, the depth calculation on the depth image to obtain the depth information can be used for compression encoding and decompression decoding of the display data (which will be described in detail in the embodiments of Figure 5a .

[0489] Continuing to refer to Figure 6 , the calculation module of the perception and interaction data can send the calculation result of the calculated perception and interaction data to the terminal device 100 through the perception and interaction interface.

[0490] The terminal device 100 can generate a virtual frame in RGBAZ format or RGBA format based on the calculation result of the perception and interaction data, and optionally compress the virtual frame using the received calculated perception and interaction data (such as depth information, eye movement information, etc.), and send the compressed code stream to the MR device 301 through the display interface.

[0491] The MR device 301 can perform decompression decoding processing on the received code stream of the virtual frame to obtain the virtual frame.

[0492] The processing module of the display data of the MR device 301 can use the calculation result of the calculated perception and interaction data calculated by the calculation module of the perception and interaction data, the signal of the real scene image calculated by the ISP, and the decoded virtual frame to perform processing and calculation of the display data, which can include fusion of the virtual frame and the real scene image, and optionally can also include anti-distortion processing of the virtual frame and the real scene image respectively, and finally, the display driving is performed to display on the screen. Here, the screen is the left eye display screen and the right eye display screen, and the two screens display the image of the binocular vision.

[0493] Optionally, as Figure 6As shown, the terminal device 100 may include two main cameras for image capture. In the XR scenario, the two main cameras can also be used to realize the collection of perception and interaction data on the terminal device 100 side. Accordingly, the processing chip of the terminal device 100 can also realize the algorithm calculation of the main camera's shooting and the algorithm calculation of the perception and interaction data collected by itself. In addition, the terminal device 100 can also cooperate with the display peripheral 100 for shooting.

[0494] like Figure 6 As shown, the processing chip can provide a two-dimensional user interface (UI), such as an entrance to a two-dimensional button or other control, or a three-dimensional UI (such as an entrance to a three-dimensional button or other control). After the user clicks the corresponding control through the display interface (such as the game interface), the processing chip can provide a two-dimensional window or a three-dimensional model, and the rendering engine module ( Figure 2b An example of a processing module shown, such as a CPU, a GPU) is used to render the two-dimensional window or the three-dimensional model to generate data to be displayed, such as a virtual frame.

[0495] In some embodiments, the processing chip may also include an AI module (e.g., an AI processing chip). The AI ​​module may generate some or all elements in the virtual frame based on the calculation results of the received perception and interaction data through AI model reasoning, calculation of a large language model, and other methods.

[0496] The terminal device 100 can flexibly choose to generate a virtual frame through one or more modules of the rendering engine module and the AI ​​module according to the application scenario and computing power.

[0497] The positioning module is used to position the terminal device 100 itself.

[0498] The reconstruction module can be used to reconstruct the environment where the terminal device 100 is located into a three-dimensional model.

[0499] like Figure 6 As shown, the terminal device 100 may have an audio module for collecting audio; a microphone module for collecting audio; and a display screen (eg, 720P) for displaying media data such as images and videos.

[0500] like Figure 6 As shown, the terminal device 100 may have a Wi-Fi module to communicate with various display peripheral devices 100 via Wi-Fi.

[0501] In the present example 1, the terminal device 100 can be compatible with the display peripheral equipped with the MR glasses with a stronger processing module. This type of MR glasses can complete the link closed loop from the VST module of the real scene to the display of the entire real scene at the glasses end, and complete the display of the virtual-real synthesis at the glasses end. The MR glasses can also realize the calculation of spatial positioning perception and interaction at the glasses end, and directly output the calculation results of the perception and interaction data, which can include the calculation results of positioning, perception, and interaction.

[0502] In the present example 1, when generating the elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation results of the perception and interaction data through the rendering engine module, and obtain the elements in the to-be-displayed data through the AI module in the form of model inference. The terminal device 100 finally obtains the to-be-displayed data (for example, the virtual frame) synthesized by multiple elements, and sends the to-be-displayed data to the MR device 301. The MR device 301 performs virtual-real synthesis based on the received virtual frame and the real scene image obtained at the MR device 301 side, and displays the image after the virtual-real synthesis on the left eye display screen and the right eye display screen.

[0503] Different from the prior art, the display interface of the terminal device 100 of the present application can support the transmission of the to-be-virtual-real-synthesized display data, and the perception and interaction interface of the terminal device 100 can support the transmission of the calculated perception and interaction data. The to-be-displayed data supported by the terminal device 100 for transmission can be the data format RGBAZ (or RGBA format) to be virtually-real-synthesized: one transparency channel A and one depth channel Z are added. The transparency channel is used as a mask in virtual-real synthesis, and the depth channel provides the correct occlusion relationship in synthesis to realize virtual-real occlusion. The perception and interaction data supported by the terminal device 100 for transmission is the result calculated by the algorithm: for example, the result of spatial positioning, the local positioning map, the spatial geometric grid, the light information, the gesture point, the gaze point, the limb key point, and the facial expression base. The transmission can reduce the bandwidth occupation and the time delay.

[0504] In addition, the present example 1 can use the information obtained by the perception and interaction data to perform reference auxiliary processing on the multi-view video to be processed when encoding and decoding the to-be-displayed data, so as to obtain higher compression efficiency than the prior art, reduce the code rate and the required bandwidth for transmission, reduce the time delay, and obtain better subjective video quality, thereby improving the subjective experience of the user.

[0505] Example 2

[0506] Figure 7 The present application is exemplarily shown in the process schematic diagram of the processing method.

[0507] In Figure 7 the display peripheral 200 connected to the terminal device 100 is specifically the MR device 302.

[0508] The MR device 302 may be MR glasses or a MR helmet.

[0509] Figure 7 The two devices shown are Figure 2b A detailed schematic diagram of the two devices shown.

[0510] Compared with Example 1 Figure 6 , please refer to Figure 7 This example 2 is mostly the same as example 1, with the following differences:

[0511] Difference 1: If Figure 7 As shown, the terminal device 100 may have, for example Figure 6 The computing module for sensing and interaction data deployed in the MR device 301 is used to compute sensing and interaction data from the MR device 302. In other words, the terminal device 100 of the present application may have the computing capability for sensing and interaction data of devices such as XR glasses.

[0512] Difference 2: If Figure 7 As shown, the MR device 302 can compress and encode the perception and interaction data to be sent to the terminal device 100, for example, compress and encode the real scene image (such as the VST image), and optionally compress and encode the depth image, black and white image, motion information, etc., and then send it to the terminal device 100.

[0513] Difference 3: If Figure 7 As shown, the terminal device 100 may have, for example Figure 6 The display data processing module deployed in the MR device 301 is used to perform computational processing on the display data, for example, to generate a fusion of virtual and real images. In other words, the terminal device 100 of the present application may have the display data processing capability of devices such as XR glasses.

[0514] Difference 4: If Figure 7 As shown, the terminal device 100 generates data to be displayed using a display data processing module. The data to be displayed is a virtual fused image in RGB format. Optionally, the terminal device 100 can compress and encode the data to be displayed and then send it to the MR device 302.

[0515] In this way, the embodiment of the present application can integrate the computing power of glasses or helmet-type devices such as AR glasses, VR glasses, and MR glasses on the terminal device 100 side, and the terminal device 100 calculates the perceived interaction data and performs virtual and real synthesis on the display data to reduce the computing power requirements of the display peripherals and reduce the device weight and power consumption of the display peripherals.

[0516] The following combinationFigure 7 , mainly describe the difference with Figure 6 the process, the same will not be described, please refer to the description of example 1.

[0517] In Figure 7 Embodiments, terminal device 100 can determine the display data supported by MR glasses 302 is RGB format, the resolution includes full high definition resolution and gaze point rendering display resolution, by obtaining the data capability information of the display data of MR glasses 302 and the capability information of the corresponding transmission link, terminal device 100 can determine the upper limit of the transmission rate of the to-be-displayed data to be transmitted is 60Gbps.

[0518] In addition, terminal device 100 can also determine that the activated sensors supported by MR glasses 302 can include 2 VST cameras, 8 black and white cameras (of which, eye tracking camera and downward camera are also black and white cameras), 1 depth camera, and IMU, by obtaining the data capability information of the perception and interaction data of MR glasses 302 and the capability information of the corresponding transmission link, terminal device 100 can determine that the upper limit of the data amount of the to-be-transmitted perception and interaction data is close to 30Gbps.

[0519] Then, terminal device 100 can determine the format of the to-be-displayed data to be transmitted, the format of the to-be-transmitted perception and interaction data, and the link configuration of the two corresponding transmission links based on various capability information, and instruct display peripheral 200 to perform corresponding initialization according to the determined various information.

[0520] For example, in the scenario of Figure 7 , terminal device 100 can determine that the format of the to-be-displayed data to be transmitted is RGB, and determine that the resolution of each eye image in the to-be-displayed data supported by the MR device 302 is full high definition resolution (or also can be gaze point rendering display resolution). The transmission rate of the data stream of the full high definition to-be-displayed image can reach 60Gbps, which exceeds the transmission capability of the display data transmission link, so the display data transmission link can be configured to open compression encoding to ensure that the transmission rate of the display data transmission link can meet the upper limit requirement of 60Gbps.

[0521] For example, in the scenario of Figure 7In this scenario, the terminal device 100 can determine the format of the perception and interaction data to be transmitted, including: activating two VST cameras, activating eight black and white cameras, activating one depth camera, activating one IMU, and the resolution and frame rate of the images captured by each sensor. Based on the format of the perception and interaction data, the terminal device 100 can determine that the transmission rate of the data stream of the VST images captured by the two VST cameras reaches 30Gbps, and can enable compression encoding for the transmission link configuration of the perception and interaction data on the terminal device 100 side.

[0522] For other perception and interaction data other than VST images, such as depth images, black and white images, IMU data, etc., since their data volume is small, they can be compressed and encoded (for example, Figure 7 motion information shown).

[0523] Afterwards, if Figure 7 As shown, the MR device 302 can collect perception and interaction data through the sensor module, including real-scene images (such as VST images), depth images, black-and-white images, etc. The MR device 302 can compress and encode the real-scene images to obtain a real-scene image code stream, and send it to the terminal device 100 through the perception and interaction interface. The MR glasses 302 can directly send other perception and interaction data other than the real-scene images to the terminal device 100 through the above-mentioned transmission link of the perception and interaction interface without compressing them.

[0524] Alternatively, as Figure 7 As shown by the dotted arrows, in order to compress and encode the VST image and decompress and decode it with the help of the perception and interaction data, and to compress and encode the display data and decompress and decode it with the help of the perception and interaction data. Figure 7 As shown, the MR device 302 can calculate eye movement information based on the eye movement image captured by the sensor module (e.g., the eye movement image captured by the eye tracking camera, which is also a black and white image). Optionally, the MR device 302 can calculate depth information based on the depth image captured by the depth camera. This eye movement information and depth information are also sent to the terminal device 100 via the perception and interaction interface.

[0525] The MR glasses 302 may use at least one of the eye movement information and the depth information to compress and encode the VST image to obtain a bit stream of VST data.

[0526] The data volume of perception and interaction data such as depth images, black and white images, motion information, etc. is relatively small and may be compressed or not compressed, and there is no restriction here. These perception and interaction data are other perception and interaction data except VST data mentioned above.

[0527] The terminal device 100 can decompress the received perception and interaction data to obtain the perception and interaction data.

[0528] For the process of compressing and decompressing the perception and interaction data, please refer to the introduction of Figure 5b and related solutions, which will not be repeated here.

[0529] Continuing to refer to Figure 7 , the terminal device 100 can utilize the calculation module of the perception and interaction data in the processing chip to calculate the decoded perception and interaction data to obtain the calculated perception and interaction data (for example, the results of spatial positioning, spatial geometric grid, lighting information, gesture point, etc. as shown in Figure 6 , the specific process and principles are the same as those in Example 1, which will not be repeated here.

[0530] In addition, as shown in Figure 7 , the terminal device 100 can generate a virtual frame (RGBA format or RGBA format) based on the calculated perception and interaction data through the rendering engine module; in addition, the terminal device 100 can also generate part or all of the elements in the virtual frame through the AI module. For example, the AI module can generate part or all of the elements in the virtual frame based on the calculated perception and interaction data; and through the display data processing module in the MR real scene rendering synthesis module, the virtual frame and the decoded real scene image are fused to obtain a virtual-real fused image (for example, an image in RGB format).

[0531] Then, the terminal device 100 can compress and encode the virtual-real fused image in RGB format (for example, using the above-mentioned depth information and eye movement information), and send the code stream of the image to the MR glasses 302 through the display interface.

[0532] The MR glasses 302 can decode the code stream of the image, and display the decoded virtual-real fused image on the screen through display driving to display the binocular image on the left eye display screen and the right eye display screen.

[0533] In the embodiments of the present application, the display interface of the terminal device 100 transmits the display data in RGB format which can be directly displayed, and the terminal device 100 of the present application can be compatible with the display peripherals such as ordinary MR glasses to realize the calculation, virtual-real synthesis and other processing of the display data. In addition, the perception and interaction interface of the terminal device 100 of the present application can transmit the image data collected by the sensor, and can decompress the compressed perception and interaction data.

[0534] In addition, the display peripheral side perception and interaction interface transmitted data in this example 2 includes real scene image data, and other perception and interaction data (such as depth image, IMU data, etc.) in addition to the real scene image data. This method can use the information obtained from other perception and interaction data for the perception and interaction data, especially the real scene image data, and can perform reference auxiliary processing on the multi-view video to be processed when encoding the real scene image data, thereby obtaining higher compression efficiency than the prior art, reducing the code rate and the required bandwidth for transmission, reducing the latency, and obtaining a better subjective quality of the video, which can improve the subjective experience of the user.

[0535] In addition, the processing chip of the terminal device can include an AI module. When generating the elements in the virtual frame, the terminal device can render the elements in the to-be-displayed data based on the calculation result of the perception and interaction data through the rendering engine module, and can also obtain the elements in the to-be-displayed data through the AI module in a model inference manner. The terminal device finally obtains the to-be-displayed elements, which can be displayed in the form of AR on the AR device.

[0536] In this example 1, when generating the elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation result of the perception and interaction data through the rendering engine module, and can also obtain the elements in the to-be-displayed data through the AI module in a model inference manner. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI module. The terminal device 100 can obtain the image after fusion of virtual and real by fusing the finally obtained virtual frame and the real scene image through the MR real scene rendering synthesis module. And send the image after fusion of virtual and real to the MR device 302 for display.

[0537] Example 3

[0538] Figure 8 An example of the process of the processing method of the present application is shown.

[0539] In Figure 8 , the display peripheral 200 connected with the terminal device 100 is specifically an MR device 303.

[0540] The MR device 303 can be an MR glasses or an MR helmet, etc.

[0541] Figure 8 The two devices shown are Figure 2b A specific diagram of the two devices shown.

[0542] In comparison with the Figure 7 in example 2, please refer to Figure 8The example 3 is same as the example 2 in most part, the difference is mainly that: when the terminal device 100 accesses the MR device 303, the VST image captured by the VST camera of the MR device 303 can be directly processed and sent to the display in the MR device 303, therefore, the MR device 303 does not need to transmit the high-definition real scene image (for example, the VST image) to the terminal device 100 through the perception and interaction interface for processing. In addition, the MR device 303 can send other perception and interaction data (for example, data captured by various cameras and sensors, such as depth image, black and white image, motion information) to the terminal device 100, so that the terminal device 100 can perform calculation and processing on the other perception and interaction data.

[0543] In addition, the difference between example 3 and example 2 also includes that the calculation and processing of the display data (for example, the fusion of the virtual frame and the real scene image) is completed in the MR device 303, which is the same as example 1.

[0544] The process of the method of the present application implemented by the system shown will be briefly described below. Figure 8 The process of the method of the present application implemented by the system shown will be briefly described below.

[0545] Firstly, the terminal device 100 can obtain the data capability information of the display data of the MR glasses 303 and the capability information of the corresponding transmission link, and the data capability information of the perception and interaction data and the capability information of the corresponding transmission link.

[0546] The terminal device parses the capability information to determine that the format of the display data supported by the MR glasses 303 is virtual frame (for example, RGBAZ format or RGBA format), the resolution and frame rate are 2*3840*3600*90Hz. Based on this, it can be determined that the upper limit of the transmission rate of the to-be-displayed data is 100Gbps.

[0547] The terminal device 100 parses the capability information to determine the format of the perception and interaction data to be transmitted by the MR glasses 303: it does not include the real scene image of the two VST cameras, the format of the perception and interaction data supported by the MR glasses 303 includes the down-sampled image (for example, the resolution is 640*480) of the two VST cameras, the image data of the eight black and white cameras, the image data of the one depth camera, and the IMU data, so the transmission rate of the data to be transmitted by the perception and interaction interface is only 1Gbps-2Gbps.

[0548] Then, the terminal device 100 can determine the format of the to-be-displayed data to be transmitted, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links based on various capability information, and instruct the display peripheral 200 to perform corresponding initialization according to the determined various information.

[0549] For example, inFigure 8 In the scenario, the terminal device 100 can determine that the format of the data to be transmitted and displayed is RGBAZ or RGBA format, and determine that the resolution of each image in the data to be displayed supported by the MR device 303 is full HD resolution (or the resolution of foveated rendering display). Then the transmission rate of the data stream of the full HD image to be displayed can reach 100Gbps. This transmission rate exceeds the transmission capacity of the transmission link for the display data. In this case, compression encoding can be enabled for the transmission link for the display data to ensure that the transmission rate of the transmission link for the display data can meet the upper limit requirement of 100Gbps.

[0550] For example, in Figure 8 In the scenario, the terminal device 100 can determine that the format of the perception and interaction data to be transmitted includes: starting 2 VST cameras, the obtained VST image has a downsampled image resolution of, for example, 640*480, starting 8 black and white cameras, starting 1 depth camera, starting an IMU, and the resolution and frame rate of the images collected by each sensor. Based on the format of the perception and interaction data, the terminal device 100 can determine that the amount of perception and interaction data to be transmitted is small, and the transmission link of the perception and interaction data can meet the requirements. Among them, since there is no need to transmit VST data, only the compressed image of the VST image with a small amount of data and some sensor data need to be transmitted. Therefore, the compression encoding can be started without setting the transmission link.

[0551] Afterwards, if Figure 8 As shown, the MR device 303 can collect perception and interaction data through the sensor module, including VST images, depth images, black and white images, etc. The MR device 302 can process the VST images from calculation to display without transmitting the VST images.

[0552] The MR device 303 can transmit other perception and interaction data besides the real scene image, such as depth image, 8 black and white images, motion information and other perception and interaction data, to the perception and interaction interface of the terminal device 100 through the perception and interaction interface.

[0553] The terminal device 100 can calculate the perception and interaction data through the perception and interaction data calculation module. The specific calculation content is similar to the above examples 1 and 2 and will not be repeated here.

[0554] In addition, the terminal device 100 may also generate some or all elements in the virtual frame through the AI ​​module. For example, the AI ​​module may generate some or all elements in the virtual frame based on the calculated perception and interaction data.

[0555] In addition, the terminal device 100 can generate the to-be-displayed data (for example, a virtual frame in the RGBAZ format or the RGBA format) based on the calculated perception and interaction data.

[0556] When generating the elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation result of the perception and interaction data through the rendering engine module, and can also obtain the elements in the to-be-displayed data through the model inference manner of the AI module. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI module. The terminal device 100 can send the finally obtained virtual frame to the MR device 303, so that the MR device 303 displays the virtual-real synthesized result on the MR device 303.

[0557] In some embodiments, the terminal device 100 can compress the to-be-displayed data, for example, using the depth information and eye movement information calculated by the terminal device 100 based on the perception and interaction data to compress the to-be-displayed data, and transmitting the compressed virtual frame to the MR glasses 303 through the display interface.

[0558] Then, the MR glasses 303 can decompress and decode the compressed virtual frame to obtain a virtual frame in the RGBAZ format or the RGBA format.

[0559] Next, the display data processing module of the MR glasses 303 can process (for example, virtual-real synthesis) the real scene image and the virtual frame based on the information processed by the ISP, and perform on-screen display through the display driver.

[0560] For example, the dual-purpose virtual-real synthesized image can be displayed on the left eye display screen and the right eye display screen, respectively.

[0561] In this example 3, the terminal device 100 of the present application can be compatible with a display peripheral of an MR glasses type, which is a device capable of supporting VST data processing and directly sending to display. The VST data of the MR glasses 303 does not need to be transmitted to the terminal device 100, and can be processed, synthesized and displayed in the MR glasses 303.

[0562] Different from the above example 2, the display interface of the terminal device 100 of the present application transmits data in the RGBAZ format or the RGBA format, which is the same as example 1. Different from the above example 2, in this example 3, the data transmitted by the perception and interaction interface does not include VST data, but image data down-sampled from the VST. In this way, the data transmitted by the perception and interaction interface can include image data of various sensors, the down-sampled image data and IMU data, which has a small amount of data and does not need to be compressed, and can be directly transmitted. Compared with example 1, this example 3 can reduce the time delay of the VST channel.

[0563] In this example 3, the rendering of the image, and the calculation of the perception and interaction data are all deployed on the terminal device 100 side, which also takes into account reducing the power consumption and weight of the glasses-type peripheral device.

[0564] Example 4

[0565] Figure 9 An exemplary process diagram of the processing method of the present application is shown.

[0566] In Figure 9 , the display peripheral device 200 connected with the terminal device 100 is specifically an AR device 304.

[0567] The AR device 304 can be AR glasses or an AR helmet, etc.

[0568] Figure 9 The two devices shown are Figure 2b A specific diagram of the two devices shown.

[0569] In comparison with the Figure 8 in example 3, please refer to Figure 9 , most of the content of example 4 is the same as example 3, the main difference is that the sensor module of the AR glasses 304 does not include a sensor for collecting real scene images, such as a VST camera, so the perception and interaction data transmitted by the AR glasses 304 to the terminal device 100 does not include real scene images; and the ISP of the AR device 304 also does not need to send VST images to the display data processing module. Other content of example 4 is the same as example 3, which will not be repeated here.

[0570] In addition, the terminal device 100 can also generate part or all of the elements in the virtual frame through the AI module. For example, the AI module can generate part or all of the elements in the virtual frame based on the calculated perception and interaction data;

[0571] In addition, the processing chip of the terminal device 100 can include an AI module, and when the terminal device 100 generates elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation results of the perception and interaction data through the rendering engine module, and also can obtain the elements in the to-be-displayed data through the model inference of the AI module. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI module. The terminal device 100 can send the final to-be-displayed data (such as the virtual frame) to the AR device 304, and display it on the AR device 304 in the form of AR. In this way, the terminal device 100 of the present application can be compatible with the AR device.

[0572] Example 5

[0573] Figure 10 An exemplary process diagram of the processing method of the present application is shown.

[0574] In Figure 10 The display peripheral device 200 connected with the terminal device 100 is specifically a VR device 305.

[0575] The VR device 305 can be a VR glasses or a VR helmet, etc., which is not limited here.

[0576] Figure 10 The two devices shown are Figure 2b A specific schematic diagram of the two devices shown.

[0577] Most of the contents of the present example 5 are the same as those of example 4, and specific reference can be made to the introduction of example 4, which will not be repeated here. The main difference is that the display peripheral device connected with the terminal device 100 is a VR device, not an AR device, but the process of interaction between the terminal device 100 and the VR device 305 is the same as that of the interaction between the terminal device 100 and the AR device 304 in example 4, and the virtual frame sent to the VR device 305 is displayed on the VR device 304 in the form of complete immersive VR. In this way, the terminal device 100 of the present application can be compatible with the VR device.

[0578] In any of the above embodiments, when encoding and decoding the perception and interaction data, or the data to be displayed, a software encoder / decoder or a hardware encoder / decoder can be used, and in addition, a standard encoder / decoder or a non-standard encoder / decoder can also be used. The present application does not limit the implementation form of the encoder / decoder.

[0579] Next, a device provided by an embodiment of the present application is introduced. As Figure 11 shown:

[0580] Figure 11 A structural schematic diagram of a data processing device provided by an embodiment of the present application is shown. As Figure 11 shown, the device 500 can include a processor 501, optionally a transceiver 505, and optionally further include a memory 502.

[0581] The transceiver 505 can be referred to as a transceiving unit, a transceiver, or a transceiving circuit, etc., for realizing the transceiving function. The transceiver 505 can include a receiver and a transmitter, the receiver can be referred to as a receiver or a receiving circuit, etc., for realizing the receiving function; the transmitter can be referred to as a transmitter or a transmitting circuit, etc., for realizing the transmitting function.

[0582] The transceiver 505 can also be a communication interface.

[0583] The memory 502 may store a computer program or software code or instruction 504, which may also be referred to as firmware. The processor 501 may implement the data processing methods provided in various embodiments of the present application by running the computer program or software code or instruction 503 therein, or by calling the computer program or software code or instruction 504 stored in the memory 502. The processor 501 may be a central processing unit (CPU), and the memory 502 may be, for example, a read-only memory (ROM) or a random access memory (RAM).

[0584] The processor 501 and transceiver 505 described in this application can be implemented on an integrated circuit (IC), an analog IC, a radio frequency integrated circuit RFIC, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc.

[0585] The above-mentioned device 500 may further include an antenna 506. The modules included in the device 500 are only for illustration and are not limited in this application.

[0586] For example, the structure of the data processing device may not be affected by Figure 11 The data processing device may be an independent device or may be part of a larger device. For example, the data processing device may be implemented as follows:

[0587] (1) An independent integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally including a storage component for storing data and instructions; (3) A module that can be embedded in other devices; (4) In-vehicle equipment, etc.; (5) Others, etc.

[0588] For the case where the data processing device is implemented as a chip or a chip system, see Figure 12 Schematic diagram of the chip structure shown. Figure 12 The chip shown includes a processor 601 and an interface 602. There may be one or more processors 601, and there may be multiple interfaces 602. Optionally, the chip or chip system may include a memory 603. The processor 601 is configured to call and execute instructions from the interface 602. When the processor 601 executes the instructions, the steps of the above-described method embodiment may be performed.

[0589] All the related content of each step involved in the above method embodiments can be cited to the function description of the corresponding function module, and will not be repeated here.

[0590] Based on the same technical concept, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program includes at least one code, which can be executed by a computer to control the computer to implement the above method embodiments.

[0591] Based on the same technical concept, the embodiments of the present application also provide a computer program, which, when executed, implements the above method embodiments.

[0592] The program can be stored in a storage medium packaged with the processor, or partially or entirely stored in a storage medium not packaged with the processor.

[0593] Based on the same technical concept, the embodiments of the present application also provide a chip including a processor. The processor can implement the above method embodiments.

[0594] The steps of the method or algorithm described in connection with the embodiments disclosed herein can be implemented in hardware, or be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a Random Access Memory (RAM), a flash memory, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disk (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0595] Those skilled in the art should be aware that, in the above one or more examples, the functions described in the embodiments of the present application can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or code on the computer readable medium. The computer readable medium includes computer storage medium and communication medium, and the communication medium includes any medium that facilitates transfer of computer program from one place to another. The storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

[0596] Finally, it should be noted that the above examples are merely used to illustrate the technical solutions of the present application, rather than limit the same; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features thereof can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized by, A terminal device is in communication connection with a display device, and the method comprises: The terminal device receives first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; The terminal device receives the perception and interaction data in the first format from the display device; The terminal device obtains multimedia data based on the perception and interaction data.

2. The method of claim 1, wherein, The terminal device obtains multimedia data based on the perception and interaction data, comprising: The terminal device obtains multimedia data by reasoning the perception and interaction data through an artificial intelligence (AI) module.

3. The method of claim 1, wherein, The terminal device obtains multimedia data based on the perception and interaction data, comprising: The terminal device obtains a first multimedia element by reasoning the perception and interaction data through an AI module; The terminal device renders a second multimedia element based on the perception and interaction data; The terminal device obtains multimedia data based on the first multimedia element and the second multimedia element.

4. The method according to any one of claims 1 to 3, characterized in that, The multimedia data comprises at least one of the following: an image, a video, and a model.

5. The method according to any one of claims 1 to 4, characterized in that, Before the terminal device receives the perception and interaction data in the first format from the display device, the method further comprises: The terminal device receives second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; The terminal device determines second link information based on the second capability information and the first format; The terminal device configures the first transmission link based on the second link information.

6. The method of claim 5, wherein, The second link information comprises information indicating to start decompression, and the terminal device receives the perception and interaction data in the first format from the display device, comprising: The terminal device receives a first code stream of the perception and interaction data in the first format compressed and encoded from the display device through the configured first transmission link; The method further comprises: The terminal device decodes the first code stream based on the second link information to obtain the perception and interaction data in the first format.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: The terminal device receives third capability information, wherein the third capability information provides format information of display data supported by the display device; The terminal device determines a second format of display data to be transmitted based on the third capability information, wherein the second format comprises an image format, and the image format is an image format synthesized from a virtual image and a real scene image, or an image format of a virtual image; The terminal device obtains multimedia data based on the perception and interaction data, comprising: The terminal device obtains a first image sequence in the second format based on the perception and interaction data.

8. The method of claim 7, wherein, The method further comprises: The terminal device receives fourth capability information, wherein the fourth capability information provides third link information supported by the display device for a second transmission link; The terminal device determines fourth link information based on the fourth capability information and the second format; The terminal device configures the second transmission link based on the fourth link information.

9. The method of claim 8, wherein, The fourth link information includes information indicating starting compression, and the method further includes: The terminal device encodes the first image sequence based on the fourth link information to obtain a second code stream of the first image sequence; The terminal device sends the second code stream to the display device through the configured second transmission link.

10. The method of claim 9, wherein, The method further includes: The terminal device encodes the first image sequence based on the perception and interaction data to obtain a code stream of the first image sequence.

11. The method of claim 10, wherein, The first image sequence includes at least two-eye images, the at least two-eye images include a first-eye image and a second-eye image, and the perception and interaction data includes first-eye data.

12. The method of claim 11, wherein, The perception and interaction data further includes second-eye data, and the terminal device encodes the first image sequence based on the perception and interaction data to obtain a second code stream of the first image sequence, including: The terminal device encodes the first-eye image based on the first-eye data and encodes the second-eye image based on the second-eye data to obtain the second code stream of the first image sequence.

13. The method according to claim 11 or 12, characterized in that, The terminal device encodes the first-eye image to obtain a code stream of the first-eye image and a reconstructed image of the first-eye image; The terminal device pre-processes the reconstructed image based on the first-eye data to obtain a reference image matching the second-eye image; The terminal device encodes the second-eye image based on the reference image to obtain a code stream of the second-eye image. The first format includes at least one of the following:

14. The method according to any one of claims 1 to 13, characterized in that, Data format, resolution, frame rate, and number of eyes. The second format includes at least one of the following:

15. The method according to any one of claims 7 to 13, characterized in that, Data format, resolution, frame rate, and number of eyes. The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, including:

16. The method according to any one of claims 1 to 14, characterized in that, The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information and target information, wherein the target information is at least one of a computing capability of the perception and interaction data and an application scenario. The terminal device determines a second format of display data to be transmitted based on the third capability information, including:

17. The method according to any one of claims 7 to 13 or 15, characterized in that, The terminal device determines a second format of display data to be transmitted based on the third capability information and target information, wherein the target information is at least one of a computing capability of the display data and an application scenario. The first capability information provides format information of perception and interaction data supported by the display device for collection, or format information of perception and interaction data supported by the display device for calculation.

18. The method according to any one of claims 1 to 17, characterized in that, The method includes:

19. A data processing method, characterized by, ​ The display device sends first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; The display device obtains perception and interaction data in a first format, and the format information comprises the first format; The display device sends the perception and interaction data in the first format; The display device receives multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

20. The method of claim 19, wherein, Before the display device obtains the perception and interaction data in the first format, the method further comprises: The display device receives the first format; The display device initializes and sets a sensor for collecting perception and interaction data according to the first format.

21. The method of claim 19 or 20, wherein, The method further comprises: The display device sends second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; The display device receives second link information, wherein the second link information is determined based on the second capability information and the first format; The display device configures the first transmission link based on the second link information.

22. The method of claim 21, wherein, The second link information comprises information indicating starting compression or decompression, and the method further comprises: The display device compressively encodes the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; The display device sends the first code stream through the configured first transmission link.

23. The method of claim 22, wherein, The perception and interaction data in the first format comprises a live image and image data other than the live image, and the display device compressively encodes the obtained perception and interaction data in the first format to obtain a first code stream, comprising: The display device compressively encodes the live image based on the image data to obtain a first code stream.

24. A data processing method, characterized by, The terminal device is in communication connection with the display device, and the method comprises: The display device sends first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; The display device sends the perception and interaction data in the first format to the terminal device; The terminal device obtains multimedia data based on the perception and interaction data in the first format; The terminal device sends the multimedia data to the display device; The display device displays an image based on the multimedia data.

25. The method of claim 24, wherein, Before the display device sends the perception and interaction data in the first format to the terminal device, the method further comprises: The terminal device sends the first format to the display device; The display device initializes and sets a sensor for collecting perception and interaction data according to the first format; The display device obtains perception and interaction data in a first format based on the initialized sensor.

26. The method of claim 24 or 25, wherein, The method further comprises: The display device sends second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for the first transmission link; The terminal device determines second link information based on the second capability information and the first format; The terminal device sends the second link information to the display device; The display device configures the first transmission link based on the second link information.

27. The method of claim 26, wherein, The second link information includes information indicating starting decompression, and the method further includes: The display device encodes the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; The display device sends the first code stream to the terminal device through the configured first transmission link; The terminal device decodes the first code stream based on the second link information to obtain the perception and interaction data in the first format.

28. A data processing system comprising: The system includes a terminal device and a display device in communication connection; The display device is configured to send first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device is configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; The display device is further configured to send the perception and interaction data in the first format to the terminal device; The terminal device is further configured to obtain multimedia data based on the perception and interaction data in the first format; The terminal device is further configured to send the multimedia data to the display device; The display device is further configured to display an image based on the multimedia data.

29. A data processing apparatus, characterized by The data processing apparatus is in communication connection with a display device, and the data processing apparatus includes: A first receiving module configured to receive first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; A first determining module configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; A second receiving module configured to receive the perception and interaction data in the first format from the display device; An obtaining module configured to obtain multimedia data based on the perception and interaction data.

30. A data processing apparatus, characterized in that, The apparatus includes: A first sending module configured to send first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; An obtaining module configured to obtain perception and interaction data in a first format, wherein the format information includes the first format; A second sending module configured to send the perception and interaction data in the first format; A first receiving module configured to receive multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

31. A computer readable storage medium, characterized in that, A computer program comprising computer instructions which, when executed on a computer or processor, cause the computer or processor to perform the method of any one of claims 1 to 18, or the method of any one of claims 19 to 27.

32. A data processing apparatus, characterized in that, An apparatus comprising one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory and transmit the signals to the processors, the signals comprising computer instructions stored in the memory; when the processors execute the computer instructions, the processors are configured to perform the method of any one of claims 1 to 18, or the method of any one of claims 19 to 27.

33. A computer program product, characterised in that, The computer program product comprises a software program which, when executed by a computer or processor, causes the steps of the method of any one of claims 1 to 18, or the method of any one of claims 19 to 27 to be performed.

Images