Data processing method, system and apparatus

Abstract

The embodiments of the present application relate to the technical field of media. Provided are a data processing method, system and apparatus. The method comprises: a terminal device being capable of acquiring format information of sensing and interaction data supported by a display device, and on the basis of the format information, determining the format of sensing and interaction data to be transmitted, so that transmission of the sensing and interaction data in said format is realized between the terminal device and the display device. Thus, regardless of the format of the sensing and interaction data supported by the display device, the terminal device can be compatible with the display device to perform the transmission of the sensing and interaction data, and thus realize the acquisition of image data, thereby achieving compatibility of the terminal device with various display devices.

Description

Data processing method, system and device

[0001] The present application claims priority to a Chinese patent application No. 202410476542.X, filed on April 17, 2024, and entitled "Data processing method, system and device", the contents of which are incorporated herein by reference in its entirety. The present application also claims priority to a Chinese patent application No. 202410808963.8, filed on June 20, 2024, and entitled "Data processing method, system and device", the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of media technology, and in particular to a data processing method, system and device. BACKGROUND

[0003] With the continuous development of AR (Augmented Reality) and VR (Virtual Reality) technologies, VR / AR movies, VR / AR games and VR / AR product demonstrations have gradually entered the field of vision of users.

[0004] Currently, the box terminal is used in conjunction with AR and VR devices, which makes it difficult for the current box terminal to be compatible with various display devices. SUMMARY

[0005] To solve the above technical problems, the present application provides a data processing method and device. In the method, the terminal device can obtain the format information of the sensing and interaction data supported by the display device, and determine the format of the sensing and interaction data to be transmitted based on the format information, so as to realize the transmission of the sensing and interaction data of the format between the terminal device and the display device. In this way, no matter what format of the sensing and interaction data supported by the display device is, the terminal device can be compatible with the display device to transmit the sensing and interaction data, and further obtain the image data, thereby realizing the compatibility with various display devices.

[0006] In one possible implementation, the present application provides a data processing method. A terminal device is in communication connection with a display device, and the method comprises: the terminal device receiving first capability information, wherein the first capability information provides format information of sensing and interaction data supported by the display device; the terminal device determining a first format of sensing and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; the terminal device receiving the sensing and interaction data of the first format from the display device; and the terminal device obtaining multimedia data based on the sensing and interaction data.

[0007] The format information provided by the first capability information can be information carried in the first capability information, or information derived based on the first capability information. For example, the first capability information carries information about multiple resolutions of the perception and interaction data supported by the display device, and the format information provided by the first capability information is not limited to the multiple resolutions, but can also include information about the range of resolutions defined by the multiple resolutions, so as to determine at least one of the maximum resolution and the minimum resolution of the perception and interaction data supported by the display device.

[0008] The terminal device of the present application can obtain the first capability information from the display device, which can provide format information of the perception and interaction data supported by the display device (which can be perception and interaction data collected or results calculated based on the collected perception and interaction data), and determine the format of the perception and interaction data to be transmitted by the terminal device within the capability range of the display device based on the format information. Then, the perception and interaction data received by the terminal device from the display device is also in the determined format. Therefore, regardless of the format of the perception and interaction data supported by the display device, the terminal device of the present application can compatibly receive the perception and interaction data from the display device, so that the terminal device of the present application can be compatible with various manufacturers and models of display devices for transmission of perception and interaction data.

[0009] In this way, the terminal device of the present application can be compatible with wearable display devices (such as AR glasses, VR glasses, AR helmets, VR helmets, etc.) of various data transmission protocols, to realize immersive virtual-real fusion and spatial interaction, and to meet the experience of VR, AR, etc. In addition, the terminal device 100 can also be compatible with ordinary display devices (such as displays, televisions, vehicle display screens, projectors, etc.) of various data transmission protocols, so that the terminal device can be used as a portable micro host.

[0010] For example, a user can use one terminal device to cooperate with display devices (such as display screens, projectors, vehicle display screens, AR, VR, etc.) of different manufacturers and different specifications. It can also be used with different perception units (3DOF / 6DOF / grid / illumination / materials), different interaction units (gestures / eye movements / handheld devices / keyboards, etc.).

[0011] In a possible implementation, the terminal device obtains multimedia data based on the perception and interaction data, including: the terminal device obtains multimedia data by performing inference on the perception and interaction data through an artificial intelligence (AI) module.

[0012] The terminal device can include an AI module, and the terminal device can use the AI module to infer multimedia data based on the perception and interaction data (or a calculation result of the perception and interaction data).

[0013] In some scenarios, the terminal device can also interact with a cloud server. For example, the terminal device can send the perception and interaction data to the cloud to request the cloud to perform AI inference, so as to obtain multimedia data returned by the cloud.

[0014] That is, the terminal device can use an AI module of the terminal device itself or an AI module of the cloud to perform inference to obtain the multimedia data. The use of a single-side AI module or a double-side AI module can be flexibly selected according to the computing power and complexity of the inference. In this way, the display effect of the obtained multimedia data can be improved.

[0015] In a possible implementation, the terminal device obtains multimedia data based on the perception and interaction data, including: the terminal device uses an AI module to infer the perception and interaction data to obtain a first multimedia element; the terminal device renders a second multimedia element based on the perception and interaction data; and the terminal device obtains multimedia data based on the first multimedia element and the second multimedia element.

[0016] Similarly, the AI module can be an AI module of the terminal device or an AI module of a server, which is not limited here.

[0017] The first multimedia element and the second multimedia element can be any multimedia element such as an image, a text, a video, a model, and an audio.

[0018] The terminal device can process (for example, synthesize, which is not limited) the first multimedia element and the second multimedia element to obtain the multimedia data.

[0019] In this way, the embodiments of the present application can obtain multimedia data by rendering and AI inference. The text, the image, and the like can be used as an element displayed in the multimedia data, the audio in the audio and the video can be used as an audio in the multimedia data, and the model in the first multimedia element and / or the second multimedia element can be used as a two-dimensional model or a three-dimensional model in the multimedia data.

[0020] In a possible implementation, the multimedia data includes at least one of the following: an image, a video, and a model.

[0021] In a possible implementation, before the terminal device receives the perception and interaction data in the first format from the display device, the method further includes: the terminal device receiving second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; the terminal device determining second link information based on the second capability information and the first format; and the terminal device configuring the first transmission link based on the second link information.

[0022] The first link information can include, but is not limited to, the number (for example, the maximum number) of open transmission lanes, the transmission rate of the lanes, and the like.

[0023] The second capability information can provide first link information supported by the display device for the first transmission link for transmitting the perception and interaction data, which can include, but is not limited to, the number (for example, the maximum number) of open transmission lanes, the transmission rate of the lanes, the maximum transmission rate (for example, the transmission rate corresponding to x8), and the like.

[0024] The terminal device 100 can determine second link information according to the format (the first format) of the perception and interaction data within the maximum capability of the display device for the first transmission link, and configure the first transmission link based on the second link information.

[0025] The second link information can include information indicating whether to start decompression and decoding, and optionally, at least one of the following: the number of lanes (for example, x4) allocated to the first transmission link, the reception frequency of the data, the format of the data transmitted by the first transmission link, and the like.

[0026] For example, the display device supports 8 transmission lanes for the perception and interaction data, and the terminal device supports 4 reception lanes for the perception and interaction data, then according to the first format of the perception and interaction data to be transmitted, a corresponding number of transmission lanes (up to 4) can be selected from 1 to 4 transmission lanes, to transmit the perception and interaction data through the first transmission link, to realize the compatibility of the transmission link of the perception and interaction data of the terminal device to various display devices.

[0027] In a possible implementation, the second link information includes information indicating to start decompression, and the terminal device receiving the perception and interaction data in the first format from the display device includes: the terminal device receiving, through the configured first transmission link, a first code stream of the perception and interaction data in the first format compressed and encoded by the display device; and the method further includes: the terminal device decoding the first code stream based on the second link information to obtain the perception and interaction data in the first format.

[0028] The terminal device can determine a data transmission rate of the perception and interaction data to be transmitted based on the first format, and determine a maximum transmission rate of the transmission link of the perception and interaction data of the display device based on the second capability information. When the maximum transmission rate is less than the data transmission rate, it indicates that the current bandwidth cannot meet the transmission requirement of a large amount of perception and interaction data, and compression encoding is required.

[0029] For example, when the terminal device 100 determines that the maximum data amount max4 (determined based on the first format) is greater than the maximum data amount max3 (determined based on the first link information) that the transmission link of the perception and interaction data of the display device 200 supports, it indicates that the maximum bandwidth of the transmission link of the display device 200 cannot support the transmission of the perception and interaction data with the data amount max4 to the terminal device 100, which exceeds the transmission capability of the transmission link of the perception and interaction data of the display device 200. Then, the terminal device 100 can configure to start decompression and decoding of the transmitted perception and interaction data when configuring the transmission link of the local perception and interaction data, wherein the data amount of the compressed and encoded perception and interaction data is less than or equal to the maximum data amount max3.

[0030] In this way, the display device 200 can compress and encode the perception and interaction data to be transmitted, and then transmit the compressed and encoded data to the terminal device 100 through the transmission link of the perception and interaction data, so that the transmission of the perception and interaction data can be realized under low latency and low bandwidth even when the data amount of the perception and interaction data to be transmitted by the display device 200 is large. By encoding and decoding the perception and interaction data, higher compression efficiency can be obtained, and the code rate and the required bandwidth for transmission can be reduced, and the latency can be reduced.

[0031] In a possible implementation, the method further includes: receiving, by the terminal device, third capability information, wherein the third capability information provides format information of display data supported by the display device; determining, by the terminal device based on the third capability information, a second format of display data to be transmitted, wherein the second format includes an image format, and the image format is an image format after synthesis of a virtual image and a real scene image, or an image format of a virtual image; and obtaining, by the terminal device based on the perception and interaction data, multimedia data, including: obtaining, by the terminal device based on the perception and interaction data, a first image sequence in the second format.

[0032] The multimedia data can include a first image sequence. The first image sequence can be one frame of image, or multiple frames of image.

[0033] For example, the first image sequence can be a monocular image (an example of one frame of image), or a binocular or multi-view image (an example of multiple frames of image).

[0034] In addition, the first image sequence obtained by the terminal device is also described as the to-be-displayed data, which can be generated by the terminal device itself based on the perception and interaction data in the first format, or can be generated by the cloud based on the perception and interaction data in the first format, so that the terminal device can obtain the first image sequence from the cloud.

[0035] Unlike the terminal device in the prior art, the terminal device in the present application can receive third capability information of the display device, which can provide format information of display data supported by the display device, which can include not only image formats but also resolutions, frame rates, etc. of to-be-displayed data. Then the terminal device can determine a second format of the to-be-transmitted display data based on the third capability information, which is also within the format information provided by the third capability information. The second format can be a virtual-real fused image format (such as an RGB format), or a virtual image format (such as an RGBAZ format or an RGBA format). In this way, the terminal device can obtain display data (such as first image data) in the second format based on the format of the display data supported by the display device. So that the terminal device in the present application can be compatible with sending virtual-real fused images to the display device for display on the display device side, and also compatible with sending virtual images to the display device for the display device to realize the fusion and display of virtual-real images.

[0036] In a possible implementation, the method further includes: the terminal device receiving fourth capability information, wherein the fourth capability information provides third link information supported by the display device for a second transmission link; the terminal device determining fourth link information based on the fourth capability information and the second format; and the terminal device configuring the second transmission link based on the fourth link information.

[0037] The third link information can be the number (such as the maximum number) of open transmission lanes, the transmission rate of the lanes, etc.

[0038] The fourth capability information can provide third link information supported by the display device for a second transmission link for transmitting display data, which can include but is not limited to the number (such as the maximum number) of open transmission lanes, the transmission rate of the lanes, etc., and the maximum transmission rate (such as the transmission rate corresponding to x8).

[0039] Then the terminal device 100 can determine the fourth link information according to the format (the second format described above) of the display data within the maximum capability of the display device for the second transmission link, and configure the second transmission link based on the fourth link information.

[0040] The fourth link information can include information indicating whether to start compression decoding, and optionally, at least one of the following: a number of channels allocated to the second transmission link (e.g., x4), a receiving frequency of data, a transceiving data format of the second transmission link, etc.

[0041] For example, the display device supports 8 transmission channels for display data, and the terminal device supports 4 receiving channels for display data, the terminal device can select a corresponding number of transmission channels (up to 4) in 1-4 transmission channels according to the second format of the display data to be transmitted, and transmit the display data through the second transmission link, so as to realize the compatibility of the terminal device to the transmission link of the display data of various display devices.

[0042] In a possible implementation, the fourth link information includes information indicating to start compression, and the method further includes: the terminal device compressively encodes the first image sequence based on the fourth link information to obtain a second code stream of the first image sequence; and the terminal device sends the second code stream to the display device through the configured second transmission link.

[0043] The terminal device can determine a data transmission rate of the display data to be transmitted based on the second format, and determine a maximum transmission rate of the transmission link of the display data of the display device based on the fourth capability information. When the maximum transmission rate is less than the data transmission rate, it indicates that the current bandwidth cannot meet the transmission requirement of a large amount of display data, and compression encoding is required.

[0044] For example, the terminal device 100 can determine the fourth link information according to the format of the display data to be displayed within the maximum capability of the display device to the transmission link, and configure the transmission link for transmitting the display data based on the fourth link information.

[0045] The fourth link information can include information indicating whether to start compression decoding, and optionally, at least one of the following: a number of channels allocated to the second transmission link (e.g., x4), a receiving frequency of data, a transceiving data format of the second transmission link, etc.

[0046] When the terminal device 100 determines that the maximum data amount max2 is greater than the maximum data amount max1 that can be transmitted by the display data transmission link of the display device 200, it indicates that the maximum bandwidth of the display device 200 to the transmission link is difficult to support the transmission of the data amount max2 of the display data from the terminal device 100. Then, the terminal device 100 can configure to start compression encoding of the display data to be transmitted when configuring the transmission link of the display data, so that the data amount of the compressed and encoded display data to be transmitted is less than or equal to the maximum data amount max1.

[0047] In this way, when the maximum capability of the transmission link of the display peripheral 200 for transmitting the display data to be displayed is less than the maximum data amount max2 of the display data to be displayed sent by the terminal device 100 through the local display interface, the terminal device 100 can compress and encode the display data to be transmitted, and send the compressed and encoded data to the display peripheral 200 through the transmission link of the display data, so that the transmission of the display data can be realized in the case of low latency and low bandwidth when the data amount of the display data to be displayed is large. Higher compression efficiency can be obtained by encoding and decoding the display data (the first image sequence), and the code rate and the required bandwidth for transmission are reduced, and the latency is reduced.

[0048] In a possible implementation, the method further includes: the terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain a code stream of the first image sequence.

[0049] In the embodiments of the present application, the display data can be compressed in combination with the perception and interaction data, so as to improve the compression efficiency, reduce the code rate and the required bandwidth for transmission, and reduce the latency.

[0050] In a possible implementation, the first image sequence includes at least two-view images, the at least two-view images include a first-view image and a second-view image, and the perception and interaction data include first-view data.

[0051] The first-view image and the first-view data correspond to the same view, for example, both are left eyes.

[0052] In a possible implementation, the perception and interaction data further include second-view data, and the terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain a second code stream of the first image sequence, including:

[0053] The terminal device compresses and encodes the first-view image based on the first-view data, and compresses and encodes the second-view image based on the second-view data to obtain the second code stream of the first image sequence.

[0054] The first-view image and the first-view data correspond to the same view, for example, both are left eyes.

[0055] The second-view image and the second-view data correspond to the same view, for example, both are right eyes.

[0056] In the embodiments, in a binocular or multi-view (more than binocular) scenario, the corresponding display data for different views can be compressed and encoded in combination with the perception and interaction data, so as to reduce the code rate and the latency.

[0057] In a possible implementation, the terminal device encodes the first image sequence based on the perception and interaction data to obtain a second code stream of the first image sequence, including: the terminal device encodes the first image to obtain a code stream of the first image and a reconstructed image of the first image; the terminal device pre-processes the reconstructed image based on the first target data to obtain a reference image matched with the second image; and the terminal device encodes the second image based on the reference image to obtain a code stream of the second image.

[0058] In the embodiments of the present application, the reconstructed image of the first image can be pre-processed (for example, morphing conversion) by means of the perception and interaction data, so that the reference image obtained by pre-processing is more close to the second image in pixel content. Then, the second image is encoded based on the reference image, which can improve the compression rate of the second image. In this way, the second image can be compressed by using the redundant information of the dual-purpose, so as to improve the compression efficiency of the first image sequence.

[0059] In a possible implementation, the first format includes at least one of the following: a data format (for example, a type of supported sensor), a resolution (for example, a resolution of image data generated by the sensor), a frame rate (for example, a frame rate of image data generated by the sensor), and a number of eyes (for example, a number of eyes corresponding to the sensor).

[0060] In a possible implementation, the second format includes at least one of the following: a data format (for example, an RGB format, an RGBA format, an RGBAZ format, etc.), a resolution (for example, a resolution of display data), a frame rate (for example, a frame rate of display data), and a number of eyes (for example, a number of eyes corresponding to the display data).

[0061] In a possible implementation, the terminal device determines the first format of the perception and interaction data to be transmitted based on the first capability information, including: the terminal device determines the first format of the perception and interaction data to be transmitted based on the first capability information and target information, where the target information is at least one of a computing capability of the perception and interaction data and an application scenario.

[0062] The terminal device 100 can set the format of the perception and interaction data to be transmitted within the format range of the perception and interaction data supported by the display device 200 in combination with at least one of an application scenario (for example, a rendering requirement of an application) and a computing capability (referred to as computing power) of the terminal device 100 for the perception and interaction data. In this way, the display device can cooperate with the terminal device in different computing power and application scenarios to realize transmission of the perception and interaction data in different formats.

[0063] In a possible implementation, the terminal device determines the second format of the display data to be transmitted based on the third capability information, including: the terminal device determines the second format of the display data to be transmitted based on the third capability information and target information, where the target information is at least one of a computing capability of the display data and an application scenario.

[0064] The terminal device 100 can set the second format of the display data to be transmitted within the range of formats of the display data supported by the display device 200 in combination with at least one of an application scenario and a computing capability (referred to as computing power) of the terminal device 100 on the display data to be displayed. In this way, the same display device, for example, a mixed reality (MR) glasses, can be driven to achieve different display effects (full-load rendering / center rendering / focal point rendering, etc.) of the first image sequence using terminal devices with different computing power (or in different application scenarios).

[0065] In a possible implementation, the first capability information provides format information of the display device supporting the collected perception and interaction data, or format information of the display device supporting the calculated perception and interaction data.

[0066] In this case, when the display device has strong computing power and supports computing on the collected perception and interaction data, the format information of the calculated perception and interaction data (also referred to as the computing result of the perception and interaction data) supported by the display device can be provided to the terminal device through the first capability information.

[0067] In a possible implementation, the method can further include: the terminal device collects perception and interaction data; and the terminal device obtains the first image sequence based on the collected perception and interaction data and the received first format of the perception and interaction data.

[0068] As shown in FIG. 3d, not only the XR headgear 201 can capture the surrounding environment (here, the user is taken as an example) to obtain the perception and interaction data and transmit the data to the terminal device 100, but also the terminal device 100 can capture the surrounding environment (the user is taken as an example) because the terminal device 100 has N camera modules, so that the perception and interaction data based on which the terminal device 100 generates the display data not only comes from the XR headgear 201, but also comes from the image data captured by the terminal device 100 itself; finally, the terminal device 100 can send the obtained (generated by the terminal device 100 or obtained from the cloud) display data to the XR headgear 201 for output.

[0069] In this way, the terminal device 100 can cooperate with the XR helmet to cooperatively collect perception and interaction data, so as to meet the collection requirements of perception and interaction data in AR, VR, MR and other scenarios.

[0070] In a possible implementation, the present application provides a data processing method. The method comprises: a display device sending first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; the display device obtaining perception and interaction data in a first format, wherein the format information comprises the first format; the display device sending the perception and interaction data in the first format; and the display device receiving multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

[0071] For example, the display device can send the first capability information to the terminal device, or the display device can send the first capability information to the terminal device through the cloud, which is not limited here.

[0072] Similarly, the display device can receive the first image sequence from the terminal device, or the display device can receive the first image sequence from the cloud, which is not limited here.

[0073] The display device of the present application can be compatible with terminal devices of different manufacturers and different models, so as to realize compatibility between the display device and the terminal device.

[0074] In a possible implementation, before the display device obtains the perception and interaction data in the first format, the method further comprises: the display device receiving the first format; and the display device initializing and setting a sensor for collecting perception and interaction data according to the first format.

[0075] The display device can receive the first format from the terminal device, or the display device can receive the first format from the cloud, which is not limited here.

[0076] In this way, the display device can initialize and set the sensor according to the format of the perception and interaction data set by the terminal device, so that the initialized sensor can collect the perception and interaction data in the first format.

[0077] In a possible implementation, the method further comprises: the display device sending second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; the display device receiving second link information, wherein the second link information is determined based on the second capability information and the first format; and the display device configuring the first transmission link based on the second link information.

[0078] The display device can send the second capability information to the terminal device, send the second capability information to the terminal device through a cloud, or send the second capability information to the cloud, which is not limited here.

[0079] The method can realize compatibility of the transmission link of the perception and interaction data between the display device and the terminal device.

[0080] In a possible implementation, the second link information includes information indicating starting compression or decompression, and the method further includes: the display device performs compression encoding on the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; and the display device sends the first code stream through the first transmission link configured.

[0081] In this way, the display peripheral 200 can perform compression encoding on the perception and interaction data to be sent, and then send the compressed and encoded data through the transmission link of the perception and interaction data, for example, to the terminal device 100, so that the transmission of the perception and interaction data can be realized in a low latency and low bandwidth case when the amount of the perception and interaction data to be sent by the display peripheral 200 is large. The compression efficiency of the perception and interaction data is higher through encoding and decoding, the code rate and the required bandwidth for transmission are reduced, and the latency is reduced.

[0082] In a possible implementation, the perception and interaction data in the first format includes a real scene image and image data other than the real scene image, and the display device performs compression encoding on the obtained perception and interaction data in the first format to obtain a first code stream, including: the display device performs compression encoding on the real scene image based on the image data to obtain the first code stream.

[0083] For example, the real scene image can be VST data (VST data collected by a VST sensor), and the image data other than the real scene image can include, but is not limited to, image data collected by other perception and interaction sensors. The other perception and interaction sensors are sensors (for example, a black-and-white camera, a depth camera, etc.) other than the sensor used to collect the real scene image.

[0084] The image data other than the real scene image can be at least one of an eye movement image and a depth image. The display device can calculate eye movement information based on the eye movement image, calculate depth information based on the depth image, and use at least one of the eye movement information and the depth information to perform compression encoding on the real scene image, so as to improve the compression rate of the real scene image and reduce the code rate.

[0085] In a possible implementation, the application provides a data processing method, a terminal device is communicatively connected with a display device, and the method comprises the following steps: the display device sends first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; the terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; the display device sends the perception and interaction data in the first format to the terminal device; the terminal device obtains multimedia data based on the perception and interaction data in the first format; the terminal device sends the multimedia data to the display device; and the display device displays an image based on the multimedia data.

[0086] In a possible implementation, before the display device sends the perception and interaction data in the first format to the terminal device, the method further comprises the following steps: the terminal device sends the first format to the display device; the display device sets the sensor for collecting perception and interaction data according to the first format; and the display device obtains the perception and interaction data in the first format based on the sensor after the setting.

[0087] In a possible implementation, the method further comprises the following steps: the display device sends second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for a first transmission link; the terminal device determines second link information based on the second capability information and the first format; the terminal device sends the second link information to the display device; and the display device configures the first transmission link based on the second link information.

[0088] In a possible implementation, the second link information comprises information indicating starting decompression, and the method further comprises the following steps: the display device performs compression encoding on the perception and interaction data in the first format obtained based on the second link information, to obtain a first code stream; the display device sends the first code stream to the terminal device through the configured first transmission link; and the terminal device decodes the first code stream based on the second link information, to obtain the perception and interaction data in the first format.

[0089] The effects of the method of each implementation of the terminal device and the display device are similar to those of the data processing method performed by the terminal device and the data processing method performed by the display device in the above implementations, and thus will not be described here.

[0090] In a possible implementation, the present application provides a data processing system. The system comprises a terminal device and a display device connected in communication; the display device is configured to send first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; the terminal device is configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; the display device is further configured to send the perception and interaction data in the first format to the terminal device; the terminal device is further configured to obtain multimedia data based on the perception and interaction data in the first format; the terminal device is further configured to send the multimedia data to the display device; and the display device is further configured to display an image based on the multimedia data.

[0091] In a possible implementation, the terminal device is further configured to send the first format to the display device; the display device is further configured to initialize a sensor for collecting perception and interaction data according to the first format; and the display device is further configured to obtain the perception and interaction data in the first format based on the initialized sensor, and send the perception and interaction data in the first format to the terminal device.

[0092] In a possible implementation, the display device is further configured to send second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for a first transmission link; the terminal device is further configured to determine second link information based on the second capability information and the first format; the terminal device is further configured to configure the first transmission link based on the second link information; the terminal device is further configured to send the second link information to the display device; and the display device is further configured to configure the first transmission link based on the second link information.

[0093] In a possible implementation, the second link information comprises information indicating starting decompression; the display device is specifically configured to compress and encode the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; the display device is specifically configured to send the first code stream to the terminal device through the configured first transmission link; and the terminal device is specifically configured to decode the first code stream based on the second link information to obtain the perception and interaction data in the first format.

[0094] In a possible implementation, the display device is further configured to send third capability information to the terminal device, where the third capability information provides format information of display data supported by the display device; the terminal device is further configured to determine a second format of display data to be transmitted based on the third capability information, where the second format includes an image format, and the image format is an image format after a virtual image and a real scene image are synthesized, or an image format of the virtual image; and the terminal device is specifically configured to obtain a first image sequence in the second format based on the perception and interaction data.

[0095] In a possible implementation, the terminal device is further configured to send the second format to the display device; the display device is further configured to perform display setting initialization according to the second format; and the display device is specifically configured to display an image based on the initialized display setting and the first image sequence.

[0096] In a possible implementation, the display device is further configured to send fourth capability information to the terminal device, where the fourth capability information provides third link information supported by the display device for a second transmission link; the terminal device is further configured to determine fourth link information based on the fourth capability information and the second format; the terminal device is further configured to configure the second transmission link based on the fourth link information; the terminal device is further configured to send the fourth link information to the display device; and the display device is further configured to configure the second transmission link based on the fourth link information.

[0097] In a possible implementation, the fourth link information includes information indicating starting compression; the terminal device is specifically configured to perform compression encoding on the first image sequence based on the fourth link information, to obtain a second code stream of the first image sequence; the terminal device is specifically configured to send the second code stream to the display device through the configured second transmission link; and the display device is specifically configured to decode the second code stream based on the fourth link information, to obtain the first image sequence in the second format.

[0098] Effects of the system in each of the above implementations are similar to effects of the data processing method performed by the terminal device and the display device in each of the above implementations, which will not be repeated here.

[0099] In a possible implementation, the present application provides a data processing apparatus. The data processing apparatus is communicatively connected with a display device, and the data processing apparatus comprises: a first receiving module configured to receive first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; a first determining module configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; a second receiving module configured to receive the perception and interaction data in the first format from the display device; and an obtaining module configured to obtain a first image sequence based on the perception and interaction data.

[0100] The data processing apparatus can be a standalone device (for example, a terminal device) or a part of a larger device. For example, the data processing apparatus can be implemented in the following forms:

[0101] (1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally comprise a storage component for storing data and instructions; (3) a module that can be embedded in other devices; (4) a vehicle-mounted device; and (5) other forms.

[0102] In a possible implementation, the apparatus further comprises: the first receiving module is further configured to receive second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; a second determining module configured to determine second link information based on the second capability information and the first format; and a first configuring module configured to configure the first transmission link based on the second link information.

[0103] In a possible implementation, the second link information comprises information indicating to start decompression, the second receiving module is configured to receive, through the configured first transmission link, a first code stream of the perception and interaction data in the first format that is compressed and encoded from the display device; and the apparatus further comprises: a decoding module configured to decode the first code stream based on the second link information to obtain the perception and interaction data in the first format.

[0104] In a possible implementation, the apparatus further includes: the first receiving module is further configured to receive third capability information, where the third capability information provides format information of display data supported by the display device; a third determining module is configured to determine a second format of display data to be transmitted based on the third capability information, where the second format includes an image format, the image format is an image format after a virtual image and a real scene image are synthesized, or an image format of the virtual image; and the obtaining module is specifically configured to obtain a first image sequence in the second format based on the perception and interaction data.

[0105] In a possible implementation, the apparatus further includes: the first receiving module is further configured to receive fourth capability information, where the fourth capability information provides third link information supported by the display device for a second transmission link; a fourth determining module is configured to determine fourth link information based on the fourth capability information and the second format; and a second configuring module is configured to configure the second transmission link based on the fourth link information.

[0106] In a possible implementation, the fourth link information includes information indicating starting compression, and the apparatus further includes: an encoding module is configured to compress and encode the first image sequence based on the fourth link information to obtain a second code stream of the first image sequence; and a sending module is configured to send the second code stream to the display device through the configured second transmission link.

[0107] In a possible implementation, the encoding module is further configured to compress and encode the first image sequence based on the perception and interaction data to obtain a code stream of the first image sequence.

[0108] In a possible implementation, the first image sequence includes at least two-view images, the at least two-view images include a first-view image and a second-view image, and the perception and interaction data includes first-view data.

[0109] In a possible implementation, the perception and interaction data further includes second-view data, and the encoding module is specifically configured to compress and encode the first-view image based on the first-view data, and compress and encode the second-view image based on the second-view data to obtain a second code stream of the first image sequence.

[0110] In a possible implementation, the encoding module is specifically configured to: compress and encode the first-eye image to obtain a bitstream of the first-eye image and a reconstructed image of the first-eye image; pre-process the reconstructed image based on the first-eye data to obtain a reference image matched with the second-eye image; and compress and encode the second-eye image based on the reference image to obtain a bitstream of the second-eye image.

[0111] In a possible implementation, the first format includes at least one of the following: a data format, a resolution, a frame rate, and a number of eyes.

[0112] In a possible implementation, the second format includes at least one of the following: a data format, a resolution, a frame rate, and a number of eyes.

[0113] In a possible implementation, the first determining module is specifically configured to determine the first format of the perception and interaction data to be transmitted based on the first capability information and target information, where the target information is at least one of a computing capability and an application scenario of the perception and interaction data.

[0114] In a possible implementation, the third determining module is specifically configured to determine the second format of the display data to be transmitted based on the third capability information and target information, where the target information is at least one of a computing capability and an application scenario of the display data.

[0115] In a possible implementation, the first capability information provides format information of the perception and interaction data supported by the display device for collection, or format information of the perception and interaction data supported by the display device for calculation.

[0116] The effects of the data processing apparatus in the above implementations are similar to those of the data processing method performed by the terminal device in the above implementations, and are not described here again.

[0117] In a possible implementation, the present application provides a data processing apparatus. The apparatus includes: a first sending module configured to send first capability information, where the first capability information provides format information of perception and interaction data supported by the display device; an obtaining module configured to obtain perception and interaction data in a first format, where the format information includes the first format; a second sending module configured to send the perception and interaction data in the first format; and a first receiving module configured to receive multimedia data, where the multimedia data is obtained based on the perception and interaction data in the first format.

[0118] The data processing apparatus can be a standalone device (for example, a display device) or a part of a larger device. For example, the data processing apparatus can be implemented in the following forms:

[0119] (1) an independent integrated circuit (IC), or chip, or chip system or subsystem; (2) a set of one or more ICs, optionally including memory elements that store data, instructions, or both; (3) a module that is to be embedded within another device; (4) a vehicle, etc.; (5) other, etc.

[0120] In a possible implementation, the apparatus includes a second receiving module configured to receive the first format; and a first configuring module configured to initialize a sensor for collecting perception and interaction data according to the first format.

[0121] In a possible implementation, the apparatus further includes the first sending module is further configured to send second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; a third receiving module configured to receive second link information, wherein the second link information is determined based on the second capability information and the first format; and a second configuring module configured to configure the first transmission link based on the second link information.

[0122] In a possible implementation, the second link information includes information indicating to start compression or decompression, and the apparatus further includes an encoding module configured to encode the perception and interaction data of the first format obtained based on the second link information to obtain a first code stream; and a third sending module configured to send the first code stream through the first transmission link configured.

[0123] In a possible implementation, the perception and interaction data of the first format includes a live image and image data other than the live image, and the encoding module is specifically configured to encode the live image based on the image data to obtain a first code stream.

[0124] The effects of the data processing apparatuses in the above embodiments are similar to those of the data processing methods performed by the display devices in the above embodiments, which will not be repeated here.

[0125] In a possible implementation, the present application provides a data processing apparatus. The data processing apparatus includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the processor can implement the method performed by the terminal device or the method performed by the display device in any of the above embodiments.

[0126] The data processing apparatus of the embodiment has similar effects to those of the data processing method of each of the above embodiments, which will not be repeated here.

[0127] In a possible implementation, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, which, when executed on a computer or a processor, causes the computer or the processor to perform the method performed by the terminal device or the method performed by the display device in any one of the above embodiments.

[0128] The computer readable storage medium of the embodiment has similar effects to those of the data processing method of each of the above embodiments, which will not be repeated here.

[0129] In a possible implementation, the present application provides a computer program product. The computer program product contains a software program, which, when executed by a computer or a processor, causes the method performed by the terminal device or the method performed by the display device in any one of the above embodiments to be performed.

[0130] The computer program product of the embodiment has similar effects to those of the data processing method of each of the above embodiments, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0131] FIG. 1 is a structural schematic diagram of a VR split machine;

[0132] FIG. 2a is a schematic diagram of device interaction;

[0133] FIG. 2b is a schematic diagram of the internal structure of a device;

[0134] FIG. 2c is a schematic diagram of the internal structure of a device;

[0135] FIG. 3a is a schematic diagram of an application scenario;

[0136] FIG. 3b is a schematic diagram of an application scenario;

[0137] FIG. 3c is a schematic diagram of an application scenario;

[0138] FIG. 3d is a schematic diagram of an application scenario;

[0139] FIG. 3e is a schematic diagram of an application scenario;

[0140] FIG. 4 is a schematic diagram of a data processing process;

[0141] FIG. 5a is a schematic diagram of an encoding process;

[0142] FIG. 5b is a schematic diagram of an encoding process, according to an example;

[0143] FIG. 6 is a schematic diagram of a scenario of a data processing process, according to an example;

[0144] FIG. 7 is a schematic diagram of a scenario of a data processing process, according to an example;

[0145] FIG. 8 is a schematic diagram of a scenario of a data processing process, according to an example;

[0146] FIG. 9 is a schematic diagram of a scenario of a data processing process, according to an example;

[0147] FIG. 10 is a schematic diagram of a scenario of a data processing process, according to an example;

[0148] FIG. 11 is a schematic diagram of a structure of an apparatus, according to an example;

[0149] FIG. 12 is a schematic diagram of a structure of a chip, according to an example. DETAILED DESCRIPTION

[0150] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0151] The term "and / or" in the present document is only used to describe an association relationship of associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone.

[0152] The terms "first" and "second" and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe a specific order of the target objects.

[0153] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of "exemplary" or "for example" is merely intended to present concepts in a concrete manner. Therefore, it is to be understood that embodiments or designs shown as "exemplary" or "for example" are not necessarily to be preferred in themselves or preferred over other embodiments or designs.

[0154] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0155] Before introducing the technical solutions of the present application, first explain and describe the technical terms involved in the present application:

[0156] XR, representing any one of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0157] XR split machine, representing display unit, perception unit, interaction unit, etc. The units are integrated in different terminal devices with the computing unit, and the two terminal devices are connected through wired or wireless connection to realize data interaction to realize XR function.

[0158] Simultaneous Localization and Mapping (SLAM) camera, which completes the perception of the environment through the camera, and generates an image called pose graph.

[0159] Video See-Through (VST) through the camera, which is an AR technology that superimposes virtual content in the real world through real-time video stream, so that users can see the augmented reality scene through the camera.

[0160] Mono camera, a black and white camera, is a camera with only one light-sensitive element, which is different from traditional RGB color cameras. Its principle is to use the light-sensitive element to perceive the light in the environment, and convert the light into an electrical signal, which is processed to form a monochrome image or video.

[0161] Eye tracking camera, which can be a Mono camera, refers to a sensor or camera system specifically designed to track the movement of the human eye. This technology monitors and records eye movements to provide information about the user's gaze point, gaze duration, blink frequency, etc.

[0162] Downward camera, which refers to a camera or camera installed on a device or carrier, facing downward. This camera is usually used to capture the scene or information below the device.

[0163] A depth camera is a camera device capable of capturing the depth information of a scene. A traditional camera mainly captures the surface information of an image, while a depth camera can capture the distance from each pixel point in the image to the camera at the same time, thereby obtaining the depth value or distance value of each pixel point.

[0164] An inertial sensor (IMU) can be used to obtain the motion information of a device, such as linear acceleration and angular velocity.

[0165] FIG. 1 is a structural schematic diagram of a VR split machine in the related art.

[0166] As shown in FIG. 1, the VR split machine can be composed of a physically separated computing unit 1000 and a unit 2000 integrating perception, interaction and display, wherein the computing unit 1000 and the unit 2000 are connected by a cable.

[0167] As shown in FIG. 1, the computing unit 1000 can include a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), etc. The computing unit 1000 also includes a universal serial bus (USB) interface and a display interface (DisplayPort, DP).

[0168] As shown in FIG. 1, the unit 2000 can also include a USB interface and a DP interface, in addition to a sensor unit and a camera unit. The sensor unit can include an inertial sensor (IMU), a handle, a keyboard, a microphone, etc. for obtaining interaction data. The camera unit can be used to obtain perception data, and can include a VST camera, a black-and-white camera, etc.

[0169] As shown in FIG. 1, the unit 2000 can also include a display module, which can include a left-eye display screen and a right-eye display screen for displaying VR images.

[0170] As shown in FIG. 1, the sensor unit and the camera unit are connected with the USB interface, and the display module is connected with the DP interface. In this way, the unit 2000 can send the above-mentioned interaction data and the above-mentioned perception data collected by the sensor unit and the camera unit respectively to the computing unit 1000 through the USB interface. And the computing unit 1000 can send the to-be-displayed data (such as images) to the display module of the unit 2000 through the DP interface for display, so as to realize the display of the VR video.

[0171] The above-mentioned unit 2000 can be deployed on a wearable display device (such as a VR glasses or a VR helmet), and the above-mentioned computing unit 1000 is deployed on another terminal device (such as a computer).

[0172] Among them, the transmission protocol of the USB interface of the unit 2000 is customized by the manufacturer of the wearable display device, and the perception data and the interaction data are transmitted by the customized transmission protocol.

[0173] Therefore, the terminal device in which the same computing unit 1000 in FIG. 1 is located cannot be compatible with wearable display devices of various manufacturers and various models.

[0174] In addition, in the scene of the XR split machine shown in FIG. 1, the computing unit 1000 and other perception, interaction, display and the like units 2000 are in a separated state. In the XR scene, the amount of data to be transmitted on both sides is large, which will bring great pressure to the bandwidth and transmission delay, and it is difficult to realize the transmission of XR data under the condition of meeting low delay and low bandwidth.

[0175] Therefore, as shown in FIG. 2a, the present application provides a terminal device 100, a display device (such as a display peripheral 200) and a system, which can include the terminal device 100 and the display peripheral 200 in communication connection (such as wired or wireless connection) with the terminal device 100. The wireless connection can be Wi-Fi, Bluetooth, etc., which is not limited here.

[0176] The terminal device 100 can obtain the capability information of the sensing and interaction data from the display device 200, and optionally obtain the capability information of the display data. The capability information of the display data can provide the format information of the display data (e.g. format, resolution, frame rate, etc.) that the display device 200 can support. The capability information of the sensing and interaction data can provide the format information of the sensing and interaction data (e.g. sensor type, format of the data generated by the sensor (e.g. RGB, etc.), resolution, frame rate, etc.) that the display device 200 can support. Then, the terminal device 100 can determine the format (also referred to as the first format) of the sensing and interaction data to be transmitted based on the capability information of the sensing and interaction data, and optionally determine the format (also referred to as the second format) of the display data to be transmitted based on the capability information of the display data. The format determined by the terminal device 100 is within the format information provided by the corresponding capability information. The terminal device 100 can receive the sensing and interaction data in the first format from the display device 200, and obtain the display data (also referred to as the first image sequence) based on the sensing and interaction data in the first format.

[0177] In some embodiments, the terminal device 100 can generate the display data based on the sensing and interaction data in the first format.

[0178] In other embodiments, the terminal device 100 can send the sensing and interaction data in the first format to a server (e.g. cloud), and the server can generate the display data based on the sensing and interaction data in the first format. Then, the terminal device 100 can receive the display data from the server to obtain the display data.

[0179] That is, the present application can generate the display data based on the sensing and interaction data by the terminal device 100 itself to obtain the display data, or can generate the display data based on the sensing and interaction data by the cloud and obtain the display data from the cloud by the terminal device 100. The terminal device 100 can select one of the two ways to obtain the display data according to the application scenario, which is not limited here.

[0180] Optionally, the terminal device 100 can send the obtained display data to the display device 200. In this way, the format of the display data sent by the terminal device 100 to the display device 200 belongs to the format of the display data supported by the display device 200.

[0181] In this way, the terminal device 100 can receive the perception and interaction data in a format (e.g., the first format) supported by the display peripheral 200, and the terminal device 100 can be compatible with the display peripheral of any transmission protocol.

[0182] In addition, the to-be-displayed data, i.e., the first image sequence, can be one frame of image or multiple frames of image.

[0183] In one transmission moment, the first image sequence can be one frame of image or multiple frames of image corresponding to multiple eyes (two eyes or more than two eyes).

[0184] For example, when the terminal device 100 is connected to a common display device (e.g., a vehicle display screen), in one transmission moment, the first image sequence can be one frame of image, so that the vehicle display screen can receive one frame of image in the same transmission moment for on-screen display, so that the user can browse the one frame of image through the vehicle display screen.

[0185] When the terminal device 100 is connected to an XR device (a device related to display of two or more than two eyes, e.g., a VR glasses), in one transmission moment, the first image sequence can be two frames of image or more than two frames of image. For example, when the XR device is a VR glasses, the VR glasses can receive two frames of image in the same transmission moment, which are an image corresponding to a left eye display screen and an image corresponding to a right eye display screen, respectively. In this way, the VR glasses can perform on-screen display of the two frames of image corresponding to two eyes on the left eye display screen and the right eye display screen in the same moment. The same applies to more than two eyes, which will not be described here.

[0186] The terminal device 100 can be a desktop device or a portable device. The portable form of the terminal device 100 can be a box type (e.g., the terminal device 100 shown in FIGS. 3a-3d), a neck-hanging type (e.g., the terminal device 100 shown in FIG. 3e), a waist-hanging type, etc., which will not be limited here.

[0187] The display peripheral 200 can be any device with a display screen, which will not be limited here.

[0188] For example, the display peripheral 200 can include but is not limited to any of the following: a display, a personal computer (PC), a mobile phone, a television, a projector, a vehicle display screen, a wearable display device. The wearable display device can be an XR glasses or an XR helmet, etc.

[0189] In this way, the terminal device 100 of the present application can be compatible with wearable display devices (such as AR glasses, VR glasses, AR helmets, VR helmets, etc.) of various data transmission protocols to achieve immersive virtual-real fusion and spatial interaction to meet VR, AR, and other experiences; in addition, the terminal device 100 can also be compatible with ordinary display devices (such as displays, televisions, vehicle display screens, projectors, etc.) of various data transmission protocols, etc., so that the terminal device can be used as a portable micro host.

[0190] In conjunction with FIG. 2a, FIG. 2b shows the internal structure and interaction schematic diagram of the display peripheral 200 and the terminal device 100 of the present application.

[0191] As shown in FIG. 2b, the display peripheral 200 can include but is not limited to a display module and an audio module.

[0192] The display module may, for example, be a screen, and the audio module may, for example, be a loudspeaker. The display module and the audio module can be used to display or play multimedia data. In the embodiments of the present application, the data output by the display module of the display peripheral 200 can be defined as display data. Optionally, the display data can also include the data output by the audio module.

[0193] As shown in FIG. 2b, the display peripheral 200 can further include a perception module, an interaction module, and a microphone module.

[0194] As shown in FIG. 2b, the microphone module, the perception module (such as a depth camera for collecting depth information), and the interaction module (such as an eye tracking camera for collecting eye movement images) in the display peripheral 200 can be sensors for collecting input information. Among them, the microphone module can be used to collect audio (such as environmental sound, voice, etc.); the perception module can be used for spatial positioning and perception of visual information around the positioning position; and the interaction module can be used to obtain visual information (such as images) for interaction, which is commonly seen in display peripherals such as glasses. In the embodiments of the present application, the data received by the perception module, the interaction module, and optionally the microphone module in the display peripheral 200 can be defined as perception and interaction data.

[0195] As shown in FIG. 2b, the display peripheral 200 can further include a VST module.

[0196] The VST module is a visual sensor, also referred to as a real scene camera, which is used to collect real scene images and is commonly seen in MR glasses and some VR glasses. It can be used to collect visual information in real time, and vision restores the real world. In the embodiments of the present application, the VST data collected by the VST module is also defined as perception and interaction data.

[0197] As shown in FIG. 2b, the display peripheral device 200 and the terminal device 100 also include respective sensing and interaction interfaces. The display peripheral device 200 and the terminal device 100 can transmit the sensing and interaction data collected by the display peripheral device 200 to the terminal device 100 through the respective sensing and interaction interfaces.

[0198] As shown in FIG. 2b, the display peripheral device 200 and the terminal device 100 also include respective display interfaces. The terminal device 100 can generate the display data to be displayed through the processing module of the terminal device 100, and transmit the display data to be displayed to the display interface of the display peripheral device 200 through the display interface of the terminal device 100.

[0199] The display interface of the display peripheral device 200 can be configured to receive the display data to be displayed.

[0200] Optionally, the display peripheral device 200 can further include a processing module and a corresponding storage module.

[0201] In some embodiments, the processing module can be configured to calculate the sensing and interaction data to obtain the sensing and interaction data to be transmitted to the terminal device 100.

[0202] In some embodiments, the processing module can be configured to calculate the display data to be displayed to obtain the display data.

[0203] The processing module of the display peripheral device 200 can include, but is not limited to, a CPU, a GPU, an NPU, etc.

[0204] The storage module can include an internal memory and an external storage, and can be configured to store data, such as the device type of the display peripheral device 200, the capability information of the sensing and interaction data, the capability information of the display data, algorithm data, user local and temporary data, etc.

[0205] The device type, the capability information of the sensing and interaction data, and the capability information of the display data will be described and explained in detail in the embodiment shown in FIG. 3.

[0206] The computing capability of the processing module can vary based on the specifications of the display peripheral device 200. The processing module of the display peripheral device 200 can generally implement the basic functions of driving the display module and the audio module of the display peripheral device 200.

[0207] Some display peripheral devices, mainly wearable devices (such as XR glasses and XR headsets), can also have a microphone module, a sensing module, an interaction module, and a VST module as shown in FIG. 2b.

[0208] In some embodiments, the processing module of the display peripheral 200 can be used to manage the sensors that collect input data, and image signal processing (ISP) and the like.

[0209] In some embodiments, the processing module of the display peripheral 200 has strong computing power, and the processing module can also calculate the VST data collected by the VST module to obtain displayable real scene image data, realizing the closed loop between the VST module and the display module. In this way, the display peripheral 200 does not need to transmit the VST data collected by the VST module to the terminal device 100.

[0210] In some embodiments, the processing module of the display peripheral 200 has strong computing power, and the processing module can also calculate the perception and interaction data collected by the perception module and the interaction module, so as to output the calculated perception and interaction data (also referred to as the calculation result of the perception and interaction data) to the terminal device 100 through the perception and interaction module. In this way, the perception and interaction data received by the terminal device 100 can be the result of spatial positioning (such as pose), local map positioning, spatial geometric grid, light information, gesture point, gaze point, limb key point, facial expression base, and the like.

[0211] The structure of the terminal device 100 as shown in FIG. 2b is mainly introduced as follows.

[0212] The terminal device 100 can include a processing module, an analysis and decoding module interacting with the perception and interaction interface, an analysis and encoding module interacting with the display interface, an AI module, a storage module, a battery, and a power interface.

[0213] The analysis and decoding module can be used to decompress and decode the perception and interaction data received by the perception and interaction interface, and can also be used to analyze the information from the display peripheral 200 to obtain the capability information of the perception and interaction data.

[0214] The analysis and encoding module can be used to compress and encode the to-be-displayed data to be sent to the display interface, and can also be used to analyze the information from the display peripheral 200 to obtain the capability information of the display data.

[0215] The processing module (such as CPU, GPU, NPU, etc.) of the terminal device 100 can be used to implement the data processing method of the terminal device 100, and render the perception and interaction data (or the calculation result of the perception and interaction data) to generate to-be-displayed data, and the like.

[0216] The Artificial Intelligence (AI) module of the terminal device 100 can be used to infer and generate the to-be-displayed data from the received perception and interaction data (or the calculation result of the perception and interaction data) by using an AI model, a language large model, or the like.

[0217] In some embodiments, the processing module and the AI module of the terminal device 100 can respectively generate part of the elements of the to-be-displayed data, so as to generate all the elements of the to-be-displayed data by the cooperation of the processing module and the AI module.

[0218] The elements of the to-be-displayed data can include, but are not limited to, text, audio (such as voice), images, videos, models, and the like, which are not limited herein.

[0219] In some embodiments, the processing module and the AI module of the terminal device 100 can both generate all the elements of the to-be-displayed data, and the terminal device 100 can select one of the processing module and the AI module to independently generate the to-be-displayed data according to the application scenario and the computing power requirement.

[0220] For example, the AI module can generate text, audio, images, videos, models, and the like based on the received perception and interaction data (or the calculation result of the perception and interaction data) as part of the elements of the to-be-displayed data.

[0221] In some embodiments, the AI module can also communicate with a server (such as a cloud) to complete the generation of the to-be-displayed data by the cooperation of the AI module and the cloud.

[0222] For example, in a scenario where the computing power required for the generation of the to-be-displayed data is high, the AI module can send a request for AI processing to the cloud based on the above-mentioned perception and interaction data (or the calculation result of the perception and interaction data), so as to obtain the to-be-displayed data generated by the cloud.

[0223] The storage module of the terminal device 100 can be used to store data information when implementing the data processing method, and the storage module can be an internal memory or an external storage.

[0224] Optionally, the terminal device 100 can further include a communication module, which can support various wireless communication protocols to communicate with the display device 200.

[0225] The communication module of the terminal device 100 can also be used to communicate with a server (such as a cloud), and the AI module can interact with the cloud through the communication module, so that the terminal device 100 can use the processing capability of the cloud to implement the inference of the AI model and the calculation of the language large model.

[0226] For example, the communication module can be a Wi-Fi module, a Bluetooth module, etc., which is not limited here.

[0227] The battery of the terminal device 100 can be interconnected with the power interface of the display peripheral 200 through the power interface to supply power to the display peripheral 200.

[0228] In combination with FIG. 2b, FIGS. 3a-3d respectively show schematic diagrams of various application scenarios of the terminal device 100 of the present application.

[0229] 1. Entertainment scenario:

[0230] As shown in FIG. 3a, the terminal device 100 can be connected with the XR helmet 201 in a wired or wireless manner, wherein the VR helmet is an example of the display peripheral 200 of the present application.

[0231] The XR helmet 201 can collect perception and interaction data through the VST module, the perception module, and the interaction module as shown in FIG. 2b. The perception and interaction data can include but are not limited to IMU data, VST data, black and white images (such as eye movement images, lip images, limb images, hand gesture images), depth images, etc. The above-mentioned perception and interaction data can be obtained through real-time images collected by sensors and cameras (also described as cameras) of the XR helmet 201 (which can include real scene images around the user wearing the XR helmet 201 (an example of VST data)).

[0232] The XR helmet 201 can transmit the perception and interaction data to the terminal device 100, and the terminal device 100 can superimpose virtual images generated by the terminal device 100 on the above-mentioned real-time images based on the perception and interaction data, generate virtual-real fusion display data, and send the display data to the XR helmet 201. The display data can include virtual-real fusion image data and audio data, the XR helmet 201 can output the audio data through a loudspeaker and output the virtual-real fusion image data through two display screens of the XR helmet 201.

[0233] In this way, the terminal device 100 of the present application can be compatible with the XR helmet to realize the transmission of perception and interaction data and display data, so as to realize a virtual-real fusion spatial interaction experience.

[0234] 2. Office scenario:

[0235] As shown in FIG. 3b, the terminal device 100 can be connected with the display 202 and the keyboard 203 in a wired (or wireless) manner, wherein the display 202 and the keyboard 203 together serve as an example of the display peripheral 200 of the present application.

[0236] The display screen 202 can be a television, a projector, a vehicle-mounted display screen, a desktop display screen, etc., which is not limited herein.

[0237] The terminal device 100 can receive keyboard input data (an interactive data) from the keyboard 203, and generate display data (e.g., an image) to be displayed based on the keyboard input data, and send the display data to the display 202 for on-screen display.

[0238] In this way, the terminal device 100 of the present application can be compatible with the display screen and the keyboard, so as to realize the effect of expanding the display screen of the terminal device 100.

[0239] 3. Independent shooting scene:

[0240] Returning to FIG. 2b, the terminal device 100 can optionally include N camera modules, microphone modules, audio modules, and display modules.

[0241] The camera modules can be used to capture images, the microphone modules can be used to capture audio, the audio modules (e.g., speakers) can be used to output audio, and the display modules (e.g., display screens) can be used to display multimedia data (e.g., images or videos).

[0242] In combination with FIG. 2b, as shown in FIG. 3c, the terminal device 100 performs multi-purpose three-dimensional (3D) shooting on the user through the camera modules, and performs audio recording in the three-dimensional space through the microphone modules, and finally outputs the 3D shooting image or video through the display modules and the audio modules, so that the audio in the output video has the live recording sense of immediacy.

[0243] In addition, in combination with FIG. 2b, as shown in FIG. 3c, the N camera modules can include camera modules with micro-single shooting capability, so that the terminal device 100 can use the camera modules with micro-single shooting capability to shoot images or videos, so that the terminal device 100 of the present application can be additionally provided with micro-single shooting capability, and the image shooting quality is high.

[0244] 4. Cooperative XR glasses shooting scene:

[0245] Different from the first entertainment scene (for example, FIG. 3a), in the present scene (for example, FIG. 3d), as shown in FIG. 3d, not only the XR helmet 201 can shoot the surrounding environment (here, taking the user as an example) to obtain perception and interaction data and transmit to the terminal device 100, but also the terminal device 100 can shoot the surrounding environment (taking the user as an example) due to the N camera modules of the terminal device 100, so that the perception and interaction data on which the terminal device 100 is based when generating the to-be-displayed data not only comes from the XR helmet 201, but also can come from the image data shot by the terminal device 100 itself; finally, the terminal device 100 can send the to-be-displayed data obtained (generated by the terminal device 100 or obtained from the cloud by being generated by the cloud) to the XR helmet 201 for output.

[0246] In this way, the terminal device 100 of the present application can cooperate with the XR helmet to perform cooperative shooting to meet the shooting requirements of the real environment in the XR scene.

[0247] 5. Application scene of the neck-hung terminal device

[0248] The terminal device in the above FIGS. 3a to 3d is a box-type terminal device, and the terminal device 100 shown in FIG. 3e is a neck-hung terminal device 100.

[0249] As shown in FIG. 3e, the terminal device 100 can interact with the XR helmet 201, and the specific interaction process is the same as that introduced in FIG. 3a, which will not be repeated here.

[0250] As shown in FIG. 3e, the neck-hung terminal device 100 can be worn on the neck of the user 10, so that the user 10 can use the worn terminal device 100 and the XR helmet 201 to immerse in the XR game, video, etc.

[0251] It should be understood that, of course, the neck-hung terminal device 100 can also be applied to the scenes shown in FIGS. 3b to 3d, and the specific implementation principle can be referred to the introduction of FIGS. 3b to 3d, which will not be repeated here.

[0252] The terminal device of the present application provides more space for camera modules and audio modules compared with mobile phones, can layout micro-single level shooting capability, multi-purpose 3D shooting, spatial audio recording, and the terminal device layout positioning perception module capability can realize cooperative shooting of the terminal device and glasses, and solve the core scene requirements of daily work, entertainment and life.

[0253] In order to enable the terminal device 100 shown in FIGS. 2a and 2b and FIGS. 3a to 3d to realize the functions mentioned in the present application, the present application also provides a data processing method to ensure the implementation of the corresponding functions of the terminal device 100.

[0254] FIG. 4 is a process diagram illustrating an example of a data processing method according to the present application.

[0255] Based on FIG. 2b, FIG. 2c further describes the internal structures of the display peripheral 200 and the terminal device 100 and the data interaction therebetween.

[0256] The functions of the modules and structures shown in FIG. 2c will be described below in combination with the process shown in FIG. 4.

[0257] In FIG. 4, the steps indicated by the dashed boxes and dashed arrows are optional steps in the implementation of the method.

[0258] As shown in FIG. 4, the process can include the following steps:

[0259] S100, the terminal device 100 is communicatively connected with the display peripheral 200.

[0260] The connection can be a wired connection or a wireless connection, which is not limited here.

[0261] Optionally, S101, the terminal device 100 determines the device type of the connected display peripheral 200.

[0262] As described above, the device type of the display peripheral 200 can include, but is not limited to, any one of the following: a display, a personal computer (PC), a mobile phone, a television, a projector, a vehicle-mounted display screen, and a wearable display device. The wearable display device can be an XR glasses or an XR helmet, etc.

[0263] The terminal device 100 can read the information indicating the device type from the display peripheral 200.

[0264] S102, the terminal device 100 obtains the capability information based on the device type.

[0265] As shown in FIG. 2c, the storage module of the display peripheral 200 can store the capability information of the sensing and interaction data and the capability information of the display data.

[0266] As shown in FIG. 4, the capability information of the sensing and interaction data can include the data capability information (also referred to as first capability information) of the sensing and interaction data, and the capability information (also referred to as second capability information) of the transmission link of the sensing and interaction data indicated by the dashed arrow. The transmission link is a link used by the display peripheral 200 to transmit the sensing and interaction data.

[0267] The capability information of the display data can include data capability information of the display data (also referred to as third capability information), and capability information of a transmission link of the display data (also referred to as fourth capability information) shown by the dashed arrow. The transmission link is a link used by the display peripheral 200 to transmit the display data.

[0268] The data capability information of the perception and interaction data can provide format information of the perception and interaction data supported by the display device 200. The transmitted perception and interaction data can be collected by the display device 200, or a result obtained by performing calculation on the collected perception and interaction data.

[0269] The data capability information of the display data can provide format information of the display data supported by the display device 200.

[0270] In some embodiments, when it is determined that the device type of the display peripheral 200 is a general display, the general display can include but is not limited to a display, a personal computer (PC), a mobile phone, a television, a projector, and a vehicle-mounted display screen. The perception and interaction data of the display peripheral 200 is generally microphone data of voice input, remote control type, and input data of a keyboard and a mouse. The display peripheral 200 can process the perception and interaction data, and thus the general display does not need to transmit the perception and interaction data to the terminal device 100 for processing by the terminal device 100. As shown in FIG. 4, when the terminal device 100 determines that the display peripheral 200 is a general display, the capability information of the perception and interaction data does not need to be obtained, and only the capability information of the display data needs to be obtained from the display peripheral 200. As shown in FIG. 4, the capability information of the display data can include data capability information of the display data, and optionally capability information of a transmission link of the display data shown by the dashed arrow.

[0271] In some embodiments, when it is determined that the device type of the display peripheral 200 is a wearable display device (for example, an XR glasses or an XR helmet), the terminal device 100 can obtain the capability information of the perception and interaction data from the display peripheral 200, and optionally the capability information of the display data. As shown in FIG. 4, the terminal device 100 can obtain the data capability information of the perception and interaction data from the display peripheral 200, and optionally the capability information of a transmission link of the perception and interaction data shown by the dashed arrow. Optionally, as shown in FIG. 4, the terminal device 100 can obtain the data capability information of the display data from the display peripheral 200, and optionally the capability information of a transmission link of the display data.

[0272] The data capability information of the display data can be used to provide format information of the display data supported by the display device 200.

[0273] The format information of the display data can include, but is not limited to, a data format of the display data, a resolution of the display data, a frame rate of the display data, a number of eyes (for example, two eyes, or more than two eyes) of the display data, and the like.

[0274] For example, the data format of the display data supported by a general display is generally a Red Green Blue (RGB) format, and the resolution and frame rate (unit: hertz (hz)) of the display data supported by the general display are generally 1920*1080*60hz, 3840*2160*120hz, 1280*720*60hz, and the like, which are not limited herein. In this document, the resolution*frame rate is taken as an example for illustration.

[0275] For another example, the data format of the display data supported by the XR glasses, the XR helmet, and the like, can include an image format after virtual image and real image synthesis (also referred to as a directly displayable data format), such as an RGB format and the like, and the data format of the display data can also include an image format of the virtual image (also referred to as a data format to be displayed after virtual-real synthesis), such as an RGBAZ format or an RGBA format and the like.

[0276] The data format to be displayed after virtual-real synthesis is RGBA or RGBAZ. Compared with the RGB format, the RGBAZ format has an additional transparency channel A and a depth channel Z. In virtual-real synthesis, the transparency channel A is used as a mask, and the depth channel Z provides correct occlusion relationship in synthesis to realize virtual-real occlusion. Compared with the RGB format, the RGBA format has an additional transparency channel A, which is used as a mask in virtual-real synthesis.

[0277] The number of eyes of the display data supported by the AR glasses is 2, and the resolution and frame rate of the display data can include 2*1920*1080*60hz and 2*1536*1760*120hz. The number of eyes of the display data supported by the VR glasses is 2, and the data format of the display data can include 2*2000*2000*90hz and 2*3000*3000*90Hz. The number of eyes of the display data supported by the MR glasses is 2, and the data format of the display data can be 2*3840*3600*90Hz. Since the XR glasses have two display screens, the above resolutions are multiplied by 2. Taking 2*1920*1080*60hz as an example, the resolution and frame rate of the display data of each eye are 1920*1080*60hz.

[0278] The AR glasses and VR glasses described above can support two resolutions of display data per purpose, for example, the VR glasses support a resolution of display data per purpose as full high definition, which is 3000*3000*90hz, and the VR glasses also support a resolution of display data per purpose as a gaze point rendering, which is 2000*2000*90hz.

[0279] The resolutions and frame rates of display data supported by the AR glasses, VR glasses and XR glasses described above are only examples, and the resolutions and frame rates of display data supported by the XR glasses are not limited by the present application, and depend on the XR glasses.

[0280] The data capability information of the perception and interaction data can provide format information of the perception and interaction data supported by the display device 200 for transmission. The format information can include but is not limited to: sensor types supported by the display device 200, data formats of the perception and interaction data supported by the sensors, resolutions of the perception and interaction data, frame rates (also described as refresh rates) of the perception and interaction data, etc.

[0281] For example, the XR glasses are near-eye display, and support the realization of the three characteristics of positioning, perception and interaction in use. When the XR glasses realize the positioning characteristic, the Y image stream of the IMU, binocular or four-eye black and white camera can be supported. When the XR glasses realize the perception characteristic, the depth stream using the depth camera can be supported. When the XR glasses realize the interaction characteristic, the camera used for realizing the positioning characteristic can be multiplexed, and two additional black and white cameras are added. The interaction characteristic can include the eye movement detection characteristic. When the XR glasses support the realization of the eye movement detection characteristic, 2 to 4 eye tracking cameras (also black and white cameras) can be supported. If it is MR glasses, two VST cameras can also be supported for collecting color images, and the format of the color images can be YUV or RGB format, etc.

[0282] So in some scenarios, the data capability information of the perception and interaction data acquired by the terminal device 100 from the XR glasses can include the following information: the supported sensor types include: IMU, 6-10 black-and-white cameras, 1 depth camera, and 2 VST color cameras. Among them, the black-and-white camera supports the generation of a luminance (Luminance, Y) image stream, and the resolution and frame rate (unit: FPS) of the Y image stream supported by the black-and-white camera include 1280*1280*60FPS and 640*480*60FPS. The depth camera supports a depth image resolution and frame rate of 640*480*30FPS. The two VST color cameras support the generation of color image streams in YUV ("Y" represents luminance (Luminance, Luma), and "U" and "V" are chrominance, chroma) or RGB format, and the color image stream resolution and frame rate supported by the two VST color cameras are 4000*3000*60FPS.

[0283] In some scenarios, part of the wearable display device (such as MR glasses) is also equipped with a powerful processing module, which can directly complete the entire link closed loop from VST data acquisition to color image display, and can calculate perception and interaction data other than VST data, such as spatial positioning perception and interaction calculation, thereby directly outputting the calculation results of the perception and interaction data. Such a display peripheral 200 does not need to transmit VST data to the terminal device 100, so the data capability information of the perception and interaction data of the display peripheral 200 can include the format information of the calculated perception and interaction data supported by the display peripheral 200, such as the format information can include but not limited to: spatial positioning results, local positioning maps, spatial geometric grids, lighting information, gesture points, gaze points, limb key points, facial expression bases, etc.

[0284] Optionally, when implementing S102, the terminal device 100 can also acquire the capability information of the display data transmission link from the display peripheral 200. The transmission link is the transmission link between the display interfaces at both ends as shown in FIGS. 2b and 2c.

[0285] The capability information of the display data transmission link (also referred to as the fourth capability information) can provide third link information supported by the display peripheral 200 on the second transmission link. Among them, the second transmission link is a link for transmitting display data.

[0286] The third link information can include, but is not limited to, the number of lanes (e.g., maximum number) supported by the display peripheral 200 to be opened for the second transmission link, the transmission rate of the lanes, and the like. In this way, the terminal device 100 can determine the link rate (e.g., maximum rate) supported by the second transmission link based on the fourth capability information.

[0287] For example, the number of lanes of the display data transmission link supported by the display peripheral 200 to be opened includes x1, x2, x4, and x8. Wherein x1, x2, x4, and x8 respectively represent that the display data transmission link of the display peripheral 200 is opened with 1 lane, 2 lanes, 4 lanes, and 8 lanes. Wherein the transmission link supports up to 8 lanes to be opened, that is, the maximum link rate is the product of x8 and the lane transmission rate. The number of lanes of the display data transmission link opened by the display peripheral 200 can affect the link rate and bandwidth of the transmission link, and the more the number of lanes opened, the faster the link rate and the wider the bandwidth.

[0288] Similarly, optionally, when implementing S102, the terminal device 100 can obtain the capability information of the transmission link of the sensing and interaction data from the display peripheral 200. The transmission link is the transmission link between the sensing and interaction interfaces at both ends as shown in FIGS. 2b and 2c.

[0289] The capability information (also referred to as second capability information) of the transmission link of the sensing and interaction data can provide the first link information supported by the display peripheral 200 to the first transmission link. Wherein the first transmission link is a link for transmitting sensing and interaction data.

[0290] The first link information can include, but is not limited to, the number of lanes (e.g., maximum number) supported by the display peripheral 200 to be opened for the first transmission link, the transmission rate of the lanes, and the like. In this way, the terminal device 100 can determine the link rate (e.g., maximum rate) supported by the first transmission link based on the second capability information.

[0291] For example, the number of lanes of the sensing and interaction data transmission link supported by the display peripheral 200 to be opened includes x1 and x2.

[0292] In combination with FIG. 2c, when the terminal device 100 acquires the capability information of the display data of the display peripheral 200, the terminal device 100 can send a capability request of a display module to the display peripheral 200, and the display peripheral 200 can read the capability information of the display data from a storage module in response to the capability request, and send the capability information of the display data to a display interface of the terminal device 100 through a display interface of the display peripheral 200; and a parsing module of the capability information of the display data of the terminal device 100 can read and parse the capability information of the display data from the display interface, to obtain the capability information of the data format of the display data supported by the display peripheral 200, the resolution of the display data supported by the display peripheral 200, the frame rate of the display data supported by the display peripheral 200, and the like.

[0293] Wherein, when the terminal device 100 acquires the data capability information of the display data of the display peripheral 200 and the capability information of the transmission link, the terminal device 100 can send two requests respectively to acquire the two kinds of capability information in sequence, or can acquire the two kinds of capability by sending one request, which is not limited here.

[0294] Similarly, in combination with FIG. 2c, when the terminal device 100 acquires the capability information of the sensing and interaction data of the display peripheral 200, the terminal device 100 can send a sensor capability request to the display peripheral 200, and the display peripheral 200 can read the capability information of the sensing and interaction data from a storage module in response to the capability request, and send the capability information of the sensing and interaction to a sensing and interaction interface of the terminal device 100 through a sensing and interaction interface of the display peripheral 200; and a parsing module of the capability information of the sensing and interaction data of the terminal device 100 can read and parse the capability information of the sensing and interaction data from the sensing and interaction interface, to obtain the information of the sensor type supported by the display peripheral 200, the format of the data supported by the sensor, the resolution, the frame rate (also described as the refresh rate), and the like.

[0295] Wherein, when the terminal device 100 acquires the data capability information of the sensing and interaction data of the display peripheral 200 and the capability information of the transmission link, the terminal device 100 can send two requests respectively to acquire the two kinds of capability information in sequence, or can acquire the two kinds of capability by sending one request, which is not limited here.

[0296] Continuing to FIG. 4, after the terminal device 100 acquires the data capability information of the display data of the display peripheral 200 through S102, S103a and S104a can be executed, and the corresponding display peripheral 200 can execute S201a after S104a. Based on the data capability information of the display data of the display peripheral 200, the data format of the to-be-displayed data to be transmitted between the terminal device 100 and the display peripheral 200 this time is uniformly configured.

[0297] S103a, the terminal device 100 determines the format of the to-be-displayed data to be transmitted based on the data capability information of the display data.

[0298] Optionally, the terminal device 100 can set the format of the to-be-displayed data to be transmitted within the data capability range of the display data supported by the display peripheral 200 in combination with at least one of the application scenario and the computing capability (referred to as computing power) of the terminal device 100 on the to-be-displayed data.

[0299] For example, the display peripheral 200 is a general display, and the data capability information (also referred to as third capability information) of the display data of the display peripheral 200 includes that the data format of the display data supported by the display peripheral 200 is an RGB format, and the resolution and frame rate of the display data supported by the display peripheral 200 are three kinds: 1920*1080*60hz, 3840*2160*120hz, and 1280*720*60hz.

[0300] Then, the format information of the display data supported by the display peripheral 200 provided by the third capability information of the display peripheral 200 can include that the data format is an RGB format, and the resolution and frame rate of the display data are 1920*1080*60hz, 3840*2160*120hz, and 1280*720*60hz. In addition, the format information of the display data also implicitly expresses that the minimum resolution of the display data supported by the display peripheral 200 is 1280*720 among the above three resolutions, the maximum resolution of the display data supported by the display peripheral 200 is 3840*2160, the minimum frame rate of the display data supported by the display peripheral 200 is 60hz, and the maximum frame rate of the display data supported by the display peripheral 200 is 120hz.

[0301] Then, the terminal device 100 can set the data format, resolution, and frame rate of the to-be-displayed data to be transmitted within the above data capability range (for example, the format information of the display data supported by the display peripheral provided by the third capability information) of the display peripheral 200. For example, the set data format is an RGB format, and the set resolution and frame rate can be selected from one of the above three kinds. Alternatively, the set resolution is an arbitrary resolution a*b within the resolution range of the maximum resolution (for example, 3840*2160) and the minimum resolution (for example, 1280*720), wherein a / b=1280 / 720=3840 / 2160=1920 / 1080, and the set frame rate is an arbitrary frame rate c within the frame rate range of the maximum frame rate (for example, 120hz) and the minimum frame rate (60hz), wherein 60hz≤c≤120hz, for example, c=90hz.

[0302] That is, the terminal device 100 can set the data format of the to-be-displayed data to be transmitted between the terminal device 100 and the display device 200 this time in the format information of the display data supported by the display device 200, and the set data format cannot exceed the maximum data capacity (for example, 3840*2160*120hz here) of the display data supported by the display device 200.

[0303] For example, the general display device only supports display data in RGB format, and the terminal device 100 determines that the data format of the to-be-displayed data to be transmitted this time can only be RGB and cannot be other formats.

[0304] When the display device 200 supports multiple formats of the same parameter (for example, resolution) of the display data, for example, the supported resolutions are 1920*1080, 3840*2160, and 1280*720 respectively. Then, the terminal device 100 can determine a resolution from the above three resolutions as the resolution of the to-be-displayed data to be transmitted in combination with at least one of the computing capacity of the terminal device 100 for the to-be-displayed data and the application scenario.

[0305] For example, the computing capacity of the terminal device 100 for the to-be-displayed data is low, and only supports gaze point rendering, so a lower resolution, for example, 1280*720, can be selected.

[0306] For another example, the rendering requirement of the application currently running in the terminal device 100 for the image is full-load rendering, so a higher resolution, for example, 3840*2160, can be selected.

[0307] Similarly, for example, the display device 200 is an XR glasses, and the data capacity information of the display data of the display device 200 is that the data format is RGBA or RGBAZ format, and the supported two resolutions and frame rates are 2000*2000*90hz (for example, corresponding to the resolution of gaze point rendering) and 3000*3000*90hz (for example, corresponding to the resolution of full high definition). Then, the terminal device 100 can set the data format, resolution, and frame rate of the to-be-displayed data to be transmitted this time within the above data capacity range of the display device 200. For example, the set data format is RGBAZ format or RGBA format; the set resolution and frame rate can be selected from one of the two examples above. Or, the set resolution is any resolution e*f within the resolution range of the maximum resolution (for example, 3000*3000) and the minimum resolution (for example, 2000*2000), where e / f=3000 / 3000=2000 / 2000=1 / 1, and the set frame rate g, where g≤90hz, g is a positive integer, for example, g=60hz.

[0308] Then the terminal device 100 can set the data format of the to-be-displayed data to be transmitted between the two ends this time in combination with the application requirement and the format information of the to-be-displayed data supported by the display peripheral 200, and the set data format cannot exceed the maximum data capacity (for example, 3000*3000*90hz here) of the to-be-displayed data supported by the display peripheral 200.

[0309] That is, the terminal device 100 of the present application can reasonably determine the format of the to-be-displayed data to be transmitted through the display interface in the display capability range of the display data supported by the display peripheral 200, in combination with the application requirement and the computing power of the terminal device 100 itself, so that the determined format of the to-be-displayed data is not only within the range of the data format supported by the display peripheral 200, but also can meet the display requirement of the image of the currently running application, and can match the computing power of the terminal device 100 on the to-be-displayed data. In this way, the terminal device 100 of the present application can be compatible with the formats of the display data supported by various display peripherals, so as to be compatible with the data display functions of various display peripherals.

[0310] Optionally, in S104a, the terminal device 100 sends the format of the to-be-displayed data to the display peripheral 200.

[0311] Wherein, S103a can determine what format of to-be-displayed data the terminal device 100 can transmit through the display interface. Then, as shown in FIG. 2c, the terminal device 100 can send the format of the to-be-displayed data determined through S103a to the display peripheral 200 through the display interface.

[0312] S201a, the display peripheral 200 enables and initializes the display module based on the received format of the to-be-displayed data.

[0313] For example, as shown in FIG. 2b, the display peripheral 200 can initialize the display rendering driver of the display module and processing module of the display peripheral 200 according to the format (such as the display format, resolution, frame rate, etc.) of the to-be-displayed data received from the terminal device 100, so that the display peripheral 200 can drive the display module (such as the left eye display screen and the right eye display screen) of the display peripheral 200 to display the display data according to the format (including the data format, resolution and frame rate) of the display data determined by the terminal device 100.

[0314] Continuing to FIG. 4, similar to the principles of S103a, S104a, S201a, after the terminal device 100 obtains the sensing and interaction data capability information of the display peripheral 200 through S102, S103b and S104b can be executed, and the corresponding display peripheral 200 can execute S201b after S104b. Based on the display capability information of the sensing and interaction data of the display peripheral 200, the format of the sensing and interaction data to be transmitted between the terminal device 100 and the display peripheral 200 this time is uniformly configured.

[0315] The present application does not limit the execution order of S103a and S103b, which can be executed in series or in parallel.

[0316] S103b, the terminal device 100 determines the format of the sensing and interaction data to be transmitted based on the sensing and interaction data capability information.

[0317] Optionally, the terminal device 100 can set the format of the sensing and interaction data to be transmitted within the sensing and interaction data capability range supported by the display peripheral 200 in combination with at least one of the application scenario and the computing power (referred to as computing power) of the terminal device 100 for sensing and interaction data.

[0318] As described above, when the device type of the display peripheral 200 is a general display, the terminal device 100 does not need to obtain the sensing and interaction energy information, and does not need to execute S103b, S104b, S201b, and the following S105b, S106, S202b.

[0319] When the display peripheral 200 is a wearable display device such as an XR glasses or an XR helmet, the execution of S103b is triggered.

[0320] The execution principle of S103b is the same as that of S103a described above. The terminal device 100 can set the first format of the sensing and interaction data to be transmitted this time within the format information provided by the data capability (also referred to as first capability information) of the sensing and interaction data supported by the display peripheral 200. The format information is the format information of the sensing and interaction data supported by the display peripheral 200.

[0321] In one possible implementation, the display peripheral 200 is an XR glasses, and the data capability information (also referred to as first capability information) of the sensing and interaction data of the display peripheral 200 can include but is not limited to the following information:

[0322] The supported sensor types include: IMU, 6-10 black and white cameras, 1 depth camera, and 2 VST cameras.

[0323] The black-and-white camera supports the format of the generated Y image stream, and the resolution and frame rate of the generated Y image stream include 1280*1280*60FPS and 640*480*60FPS.

[0324] The resolution and frame rate of the depth image generated by the depth camera are 640*480*30FPS.

[0325] The format of the color image stream (such as a real scene image stream) generated by the two VST cameras is one of YUV and RGB formats, and the resolution and frame rate of the color image stream generated by each color camera are 4000*3000*60FPS.

[0326] Therefore, the format information of the sensing and interaction data supported by the display peripheral 200 provided by the first capability information can include but is not limited to: the number of supported eyes (such as binocular), supporting IMU, supporting starting 6-10 black-and-white cameras, supporting starting at most one depth camera, and supporting starting at most two VST cameras. The data format supported by the black-and-white camera is a Y image stream, and the resolution of the Y image stream can include 1280*1280*60FPS and 640*480*60FPS. In addition, the resolution of the Y image stream can be h*h, where h≤1280, h is a positive integer, and the resolution of the Y image stream can also be j*k, where j / k=640 / 480. The frame rate of the Y image stream is m≤60FPS, and m is a positive integer. The principle of the value range of the resolution and frame rate of the above-mentioned depth image and color image stream provided by the first capability information is similar to that of the value range of the resolution and frame rate of the Y image stream, which will not be described here.

[0327] Therefore, the terminal device 100 can set the format of the sensing and interaction data to be transmitted this time within the data capability range of the sensing and interaction data supported by the display peripheral 200, and optionally, in combination with the application scenario and the calculation capability of the sensing and interaction data, to set the format of the sensing and interaction data to be transmitted this time. The format can specifically include: the type of the sensor to be started, the data format of the sensing and interaction data obtained by starting the sensor (sensor collected or calculated on the sensor collected data), the resolution and frame rate of the sensing and interaction data, etc.

[0328] For example, the format of the sensing and interaction data to be transmitted this time set by the terminal device 100 includes: starting the IMU, starting 8 black-and-white cameras (or starting a number of black-and-white cameras less than 8), starting one depth camera, and starting two VST cameras. The type and number of the started sensors are within the above-mentioned data capability range, for example, 15 black-and-white cameras cannot be driven because the display peripheral 200 has at most 10 black-and-white cameras that can be used to collect sensing and interaction data.

[0329] The format of the perception and interaction data to be transmitted this time set by the terminal device 100 further includes: the data format of the black-and-white camera is Y image stream, and the resolution and frame rate of the Y image stream are 1280*1280*60FPS. The resolution and frame rate of the image collected by the depth camera are 640*480*30FPS. The format of the VST data (color image) collected by the VST camera is YUV, and the resolution and frame rate are 4000*3000*60FPS.

[0330] That is, the terminal device 100 of the present application can reasonably determine the format of the perception and interaction data to be transmitted through the perception and interaction interface within the range of the capability of the perception and interaction data supported by the display peripheral 200, in combination with the application requirement and the computing power of the terminal device 100 for the perception and interaction data, so that the determined format of the perception and interaction data is not only within the range of the data format of the perception and interaction data supported by the display peripheral 200, but also can meet the requirement of the currently running application for the perception and interaction data, and can match the computing power of the terminal device 100 for the perception and interaction data. In this way, the terminal device 100 of the present application can be compatible with the formats of the perception and interaction data supported by various display peripherals, so as to be compatible with the capabilities of the perception and interaction data of various display peripherals.

[0331] In another possible implementation, the display peripheral 200 is an XR glasses, as shown in FIG. 2b, the processing module of the display peripheral 200 has a processing unit with strong computing power, so that the display peripheral 200 can calculate the perception and interaction data collected by the perception module and the interaction module to obtain the calculation result of the perception and interaction data. Then, the data capability of the perception and interaction data obtained by the terminal device 100 from the display peripheral 200 is: the type of the sensor corresponding to the calculation result of the perception and interaction data, the data type, the data format, etc. of the calculation result.

[0332] For example, the data capability of the perception and interaction data includes: spatial geometric grid, illumination information, gesture point, gaze point, limb key point, etc. In addition, each data type can have a respective data format.

[0333] Then, the terminal device 100 can select one or more data types and the corresponding data format from the above data types within the range of the data capability, in combination with the application requirement and the computing power of the terminal device 100.

[0334] S104b, the terminal device 100 sends the format of the perception and interaction data to be transmitted to the display peripheral 200.

[0335] In S103a, the terminal device 100 determines the format of the perception and interaction data to be transmitted through the perception and interaction interface. In S103b, the terminal device 100 transmits the perception and interaction data to be transmitted to the display device 200 through the perception and interaction interface.

[0336] In S201b, the display device 200 enables and initializes the sensors based on the format of the perception and interaction data to be transmitted received from the terminal device 100.

[0337] For example, as shown in FIG. 2b, the display device 200 can enable and initialize the sensors in the least amount based on the format of the perception and interaction data to be transmitted received from the terminal device 100 (e.g., which sensors to start, the data format of the data generated by the started sensors, the resolution and frame rate of the data, etc.). For example, the display device 200 can start the corresponding sensors, initialize the settings of the corresponding sensors, enable the sensors to collect data, and start the corresponding ISP processing of the processing module of the display device 200, etc. The specific process is related to the format of the perception and interaction data to be transmitted received.

[0338] For example, although the display device 200 supports sensor types including IMU, 6-10 black and white cameras, 1 depth camera, and 2 VST cameras, based on the format of the perception and interaction data to be transmitted determined by the terminal device 100, the display device 200 only initializes and starts the IMU and 6 black and white cameras, 1 depth camera, and 2 VST cameras. In addition, the display device 200 can configure the resolution and frame rate of the black and white cameras to be smaller, such as 640*480*60FPS, when initializing the configuration of the above-mentioned sensors based on the format of the perception and interaction data to be transmitted received in S104b.

[0339] Continuing to FIG. 4, after S103a, the terminal device 100 can further perform S105a and S106a, and the corresponding display device 200 can perform S202a after S106a. In this way, the configuration of the display data transmission link of the terminal device 100 and the initialization of the display data transmission link of the display device 200 are realized.

[0340] Optionally, in S105a, the terminal device 100 configures the display data transmission link (also referred to as the second transmission link) based on the capability information (also referred to as the fourth capability information) of the display data transmission link and the format of the display data determined in S103a.

[0341] The capability information of the transmission link of the display data can provide a maximum transmission rate supported by the transmission link of the display device 200 for the display data, for example, a transmission rate corresponding to x4. Then the terminal device 100 can determine a maximum data amount maxl of the display data that can be transmitted by the transmission link based on the maximum transmission rate.

[0342] Similarly, after determining the format of the to-be-displayed data, the terminal device 100 can determine a maximum data amount max2 of the to-be-displayed data sent by the terminal device 100 to the outside through the display interface based on the format of the to-be-displayed data, for example, a product of resolution and frame rate.

[0343] Then the terminal device 100 can determine fourth link information according to the format of the to-be-displayed data within the range of the maximum capability of the transmission link of the display device, and configure the transmission link for transmitting the display data based on the fourth link information.

[0344] The fourth link information can include an indication of whether to start compression encoding, and optionally, at least one of the following: a number of channels (for example, x4) allocated to the second transmission link, a data sending frequency, a data receiving and sending format of the second transmission link, and the like.

[0345] Wherein, when the terminal device 100 determines that the maximum data amount max2 is greater than the maximum data amount maxl that can be transmitted by the transmission link of the display device 200 for the display data, it means that the maximum bandwidth of the transmission link of the display device 200 is difficult to support the transmission of the data amount max2 of the to-be-displayed data from the terminal device 100. Then the terminal device 100 can configure to start compression encoding for the to-be-displayed data transmitted when configuring the transmission link of the display data, so that the data amount of the compressed to-be-displayed data is less than or equal to the maximum data amount maxl.

[0346] In this way, when the maximum capability of the transmission link of the display device 200 for transmitting the to-be-displayed data is less than the maximum data amount max2 of the to-be-displayed data sent by the terminal device 100 through the display interface of the terminal device, the to-be-displayed data transmitted by the terminal device 100 can be compressed and encoded, and the compressed and encoded data can be sent to the display device 200 through the transmission link of the display data, so that the transmission of the XR display data can be realized in the case of low latency and low bandwidth when the data amount of the to-be-displayed data is large.

[0347] On the contrary, when the terminal device 100 determines that the maximum data amount max2 is less than or equal to the maximum data amount maxl that the transmission link of the display device 200 can support, it indicates that the maximum bandwidth of the display device 200 can support the transmission of the data amount of the display data from the terminal device 100. Then, the terminal device 100 can configure the transmission link of the display data to be closed to the compression encoding of the display data to be transmitted.

[0348] In S106a, the terminal device 100 sends the configuration information (for example, the fourth link information) of the transmission link of the display data to the display device 200.

[0349] After S105a, the terminal device 100 can send the configuration information after the configuration of the transmission link of the display data in S105a to the display device 200, which can include but is not limited to: indicating whether to start compression encoding, and optionally, at least one of the following: the number of channels (for example, x4) allocated to the second transmission link, the data transmission frequency, the data transmission format of the second transmission link, and the like. So that the display device 200 can configure the transmission link (also referred to as the second transmission link) of the display data on the display device 200 side based on the configuration information, and the display device 200 can decompress and decode the compressed display data received.

[0350] In one possible implementation, in S202a, the display device 200 configures and initializes the format of the transmission link of the display data on the display device 200 based on the received configuration information of the transmission link of the display data.

[0351] In one possible implementation, in S202a, the display device 200 configures and initializes the format of the transmission link of the display data on the display device 200 based on the received configuration information of the transmission link of the display data.

[0352] In this way, the display device 200 can configure the transmission link of the display data on the display device 200 according to the configuration information of the transmission link of the display data of the terminal device 100, and start the corresponding transmission link according to the corresponding configuration, for example, start the channel number x4.

[0353] Optionally, after S202a, the terminal device 100 can further perform link state confirmation of the display data transmission link with the display peripheral 200, complete the handshake of the data transceiver end, and the implementation can refer to the link state confirmation process of the transmission link in the prior art, which is not limited or described herein.

[0354] In a possible implementation, continuing to return to FIG. 4, after S103b, the terminal device 100 can further perform S105b and S106b, and the corresponding display peripheral 200 can perform S202b after S106b. To realize the configuration of the perception and interaction data transmission link of the terminal device 100 and the initialization of the perception and interaction data transmission link of the display peripheral 200. The implementation principle of the process is the same as that of the setting process of the display data transmission link described above in S105a, S106a and S202a, which is not described herein.

[0355] Optionally, S105b, the terminal device 100 configures the perception and interaction data transmission link (also referred to as the first transmission link) based on the capability information (also referred to as the second capability information) of the perception and interaction data transmission link, and the format (also referred to as the first format) of the perception and interaction data to be transmitted determined by S103b.

[0356] The capability information of the perception and interaction data transmission link can provide the maximum transmission rate supported by the perception and interaction data transmission link of the display peripheral 200, for example, the transmission rate corresponding to x8. Then the terminal device 100 can determine the maximum data amount max3 of the perception and interaction data that can be transmitted by the transmission link based on the maximum transmission rate.

[0357] Similarly, after determining the format of the perception and interaction data, the terminal device 100 can determine the maximum data amount max4 of the perception and interaction data received by the terminal device 100 through the perception and interaction interface based on the format of the perception and interaction data to be transmitted, for example, the product of the resolution and the frame rate corresponding to each started sensor, and the product is accumulated based on the number of started sensors to obtain the maximum data amount max4.

[0358] Then the terminal device 100 can determine the second link information according to the format (the first format described above) of the perception and interaction data within the maximum capability range of the transmission link of the display peripheral, and configure the transmission link for transmitting the perception and interaction data based on the second link information.

[0359] The second link information can include an indication of whether to start decompression decoding, and optionally, can further include at least one of the following: the number of channels (for example, x4) allocated to the first transmission link, the data receiving frequency, the data transmission format of the first transmission link, and the like.

[0360] When the terminal device 100 determines that the maximum data quantity max4 is greater than the maximum data quantity max3 supported by the transmission link of the display peripheral 200 for transmitting the perception and interaction data, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 cannot support the transmission of the perception and interaction data with the data quantity max4 to the terminal device 100, which exceeds the transmission capability of the transmission link of the display peripheral 200 for transmitting the perception and interaction data. Then, the terminal device 100 can configure to start decompression and decoding of the transmitted perception and interaction data when configuring the transmission link of the perception and interaction data at the terminal device 100, wherein the data quantity of the compressed and encoded perception and interaction data is less than or equal to the maximum data quantity max3.

[0361] In this way, when the maximum capability of the transmission link of the display peripheral 200 for transmitting the perception and interaction data is less than the maximum data quantity max4 of the perception and interaction data transmitted by the display peripheral 200 to the terminal device 100, the display peripheral 200 can compress and encode the perception and interaction data to be transmitted, and then transmit the compressed and encoded data to the terminal device 100 through the transmission link of the perception and interaction data at both sides, so as to realize the transmission of the perception and interaction data in the case of low latency and low bandwidth when the data quantity of the perception and interaction data to be transmitted by the display peripheral 200 is large, and further realize the transmission of the display data of XR.

[0362] On the contrary, when the terminal device 100 determines that the maximum data quantity max4 is less than or equal to the maximum data quantity max3 supported by the transmission link of the display peripheral 200 for transmitting the perception and interaction data, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 can support the transmission of the perception and interaction data with the data quantity max4 to the terminal device 100. Then, the terminal device 100 can configure to stop decompression and decoding of the received perception and interaction when configuring the transmission link of the perception and interaction data at the terminal device 100.

[0363] S106b, the terminal device 100 transmits the configuration information of the transmission link of the perception and interaction data to the display peripheral 200.

[0364] After S105b, the terminal device 100 can transmit the configuration information after configuring the transmission link of the perception and interaction data at the terminal device 100 in S105b to the display peripheral 200, for example, whether to start decompression and decoding, and optionally, can allocate the channel (for example, x8) of the transmission link, configure the receiving frequency of the data, configure the data format of the transmission and reception of the transmission link, etc., so that the display peripheral 200 can configure the transmission link of the perception and interaction data at the display peripheral 200 based on the configuration information, and the display peripheral 200 can compress and encode the perception and interaction data to be transmitted.

[0365] In a possible implementation, S202b, the display peripheral 200 configures and initializes the format of the transmission link of the perception and interaction data of the display peripheral 200 based on the configuration information of the transmission link of the received perception and interaction data.

[0366] In which, the display peripheral 200 can configure the format of the transmission link of the local perception and interaction data (such as various sensor data) according to the configuration information of the transmission link of the perception and interaction data sent by the terminal device 100, for example, configure to start compression encoding, allocate the channel (such as x8) of the transmission link, configure the sending frequency of the data, configure the transceiving data format of the transmission link, and initialize the local transmission link.

[0367] In this way, the display peripheral 200 can configure and initialize the transmission link of the perception and interaction data of the display peripheral 200 according to the configuration information of the transmission link of the perception and interaction data of the terminal device 100, and start the corresponding transmission link according to the corresponding configuration, for example, start the channel number x8.

[0368] Optionally, after S202b, the terminal device 100 can also perform link state confirmation of the transmission link of the perception and interaction data with the display peripheral 200, complete the handshake of the data transceiving end, and the specific implementation can refer to the link state confirmation process of the transmission link in the prior art, which is not limited and elaborated here.

[0369] Continue to return to FIG. 2c, the terminal device 100 can generate display data, and based on the configuration of the transmission link of the display data in S105a in FIG. 4, the terminal device 100 can determine whether to perform compression encoding on the display data.

[0370] In a possible implementation, the transmission link configuration of the display data of the terminal device 100 in S105a starts compression encoding. Then, as shown in FIG. 4, the process can further include S107a, S108a and S203a.

[0371] S107a, the terminal device 100 performs compression encoding on the display data.

[0372] As described above, the data amount of the display data generated by the terminal device 100 is max2, which exceeds the transmission capability (for example, the maximum data amount is max1) of the display peripheral 200 supporting the transmission of the display data. As shown in FIG. 2c, the terminal device 100 can use the encoding module of the display data to perform compression encoding on the generated display data to obtain the code stream of the display data.

[0373] S108a, the terminal device 100 sends the code stream of the to-be-displayed data to the display device 200.

[0374] As shown in FIG. 2c, the terminal device 100 can send the compressed code stream to the display device 200 through the display interface of the terminal device 100 and the transmission link of the display data configured by the terminal device 100.

[0375] S203a, the display device 200 decodes the received to-be-displayed data.

[0376] As shown in FIG. 2c, after the transmission link of the display data of the display interface of the display device 200 receives the code stream of the to-be-displayed data, the display device 200 can perform decompression and decoding on the code stream through the display data decoding module to obtain the to-be-displayed data for on-screen display.

[0377] In a possible implementation, when the data amount of the to-be-displayed data generated by the terminal device 100 is max2 and does not exceed the transmission capability (for example, the maximum data amount is max1) of the to-be-displayed data supported by the display device 200 for transmission, the compression and encoding in the above S105a is not started, and then the terminal device 100 does not need to perform the encoding operation of the display data through the display data encoding module when sending the to-be-displayed data to the display device 200. After receiving the code stream of the to-be-displayed data, the display device 200 also does not need to perform the decoding operation through the display data decoding module shown in FIG. 2c. The transmission of the to-be-displayed data can be directly performed according to the transmission link configured by both parties.

[0378] Continuing to FIG. 2c, the display device 200 can generate the perception and interaction data. Based on the configuration of the transmission link of the perception and interaction data in S105b in FIG. 4, the display device 200 can determine whether to perform compression and encoding on the perception and interaction data.

[0379] In a possible implementation, the transmission link configuration of the perception and interaction data of the terminal device 100 in S105b starts the decompression and decoding. Then, as shown in FIG. 4, the process can further include S203b, S204b and S107b.

[0380] S203b, the display device 200 performs compression and encoding on the generated perception and interaction data.

[0381] As described above, the data amount of the perception and interaction data generated by the display device 200 is max4, which exceeds the transmission capability (for example, the maximum data amount is max3) of the perception and interaction data supported by the display device 200 for transmission. As shown in FIG. 2c, the display device 200 can perform compression and encoding on the generated perception and interaction data through the perception and interaction data encoding module to obtain the code stream of the perception and interaction data.

[0382] S204b, the display peripheral 200 sends the code stream of the perception and interaction data to the terminal device 100.

[0383] As shown in FIG. 2c, the display peripheral 200 can send the compressed code stream to the terminal device 100 through the perception and interaction interface of the local terminal device 100 via the transmission link of the perception and interaction data configured by the local terminal device 100.

[0384] S107b, the terminal device 100 decompresses and decodes the received code stream of the perception and interaction data.

[0385] As shown in FIG. 2c, after the transmission link of the perception and interaction interface of the terminal device 100 receives the code stream of the perception and interaction data, the terminal device 100 can decompress and decode the code stream through the decoding module of the perception and interaction data to obtain the perception and interaction data.

[0386] In a possible implementation, when the amount of the perception and interaction data generated by the display peripheral 200 is max4, which does not exceed the transmission capability (for example, the maximum data amount is max2) of the perception and interaction data supported by the display peripheral 200, the decompression and decoding in S105b is not started, and then the display peripheral 200 does not need to perform the encoding operation of the encoding module of the perception and interaction data when sending the perception and interaction data to the terminal device 100. The terminal device 100 also does not need to perform the decoding operation through the decoding module of the perception and interaction data shown in FIG. 2c after receiving the code stream of the perception and interaction data. The transmission of the perception and interaction data can be directly performed according to the transmission link configured by both sides.

[0387] In addition, the order between the two steps of sending the display data to be displayed by the terminal device 100 to the display peripheral 200 and sending the perception and interaction data by the display peripheral 200 to the terminal device 100 is not limited.

[0388] For example, in FIG. 4, S204b can be performed before S108a, that is, the display peripheral 200 can first send the collected perception and interaction data to the terminal device 100, and then the terminal device 100 sends the display data to be displayed to the display peripheral 200.

[0389] In a possible implementation, the terminal device 100 can obtain the display data to be displayed (which can be generated by the terminal device 100 or obtained from the cloud and generated by the cloud) by using the perception and interaction data.

[0390] In a possible implementation, the terminal device 100 can also compress and encode the obtained display data to be displayed by using the perception and interaction data.

[0391] The process that the terminal device 100 utilizes the perception and interaction data to compress and encode the obtained to-be-displayed data will be described below in combination with FIG. 5a.

[0392] The to-be-displayed data can be image or video data. As explained above for the first image sequence, the to-be-displayed data can be monocular image (i.e., one frame of image), binocular image (i.e., two frames of image), or multiocular image (i.e., more than two frames of image).

[0393] In FIG. 5a, the to-be-displayed data is taken as an example of binocular to-be-displayed data, and then the to-be-displayed data can include a first-view image and a second-view image. Taking that the display peripheral 200 has a left-eye display screen and a right-eye display screen as an example, the first-view image can be an image to be displayed on the left-eye display screen, and the second-view image can be an image to be displayed on the right-eye display screen.

[0394] Similarly, the perception and interaction data can also include monocular perception and interaction data, or binocular perception and interaction data, or multiocular perception and interaction data. For example, the perception and interaction data can include first-view data and optionally second-view data, for example, the first-view data is perception and interaction data about the left eye, and the second-view data is perception and interaction data about the right eye.

[0395] Although FIG. 5a describes the compression and encoding process of the to-be-displayed data by taking binocular as an example, the method of the present application is not limited to the binocular scenario. When the to-be-displayed data includes multiocular (the number of views is greater than 2) to-be-displayed data, the method is the same, and thus details are not repeated here.

[0396] In some embodiments, the terminal device 100 can compress and encode the first-view image based on the first-view data, and compress and encode the second-view image based on the second-view data, to obtain a code stream of the to-be-displayed data.

[0397] In this embodiment, in a binocular or multiocular (more than two views) scenario, the corresponding-view to-be-displayed data can be encoded in combination with the perception and interaction data, to reduce the code rate and latency. The principle of the process of utilizing the perception and interaction data to encode the corresponding-view to-be-displayed data can refer to the principle of the implementation process of S3011 in the embodiment of FIG. 5a, and thus details are not repeated here.

[0398] In the embodiment of FIG. 5a, the image data for each of the two purposes in the to-be-displayed data can be compressed in sequence, for example, the first purpose image is compressed first and the second purpose image is compressed later. Since the distance between the binocular cameras (for example, the left eye camera and the right eye camera) on the display peripheral 200 is relatively small, at the same time, the contents of the two images (the first purpose image and the second purpose image respectively) captured by the binocular cameras are relatively similar, therefore, the method of the present application can use the first compressed image as the reference image of the second compressed image, so as to remove the redundant information and reduce the code rate.

[0399] As shown in FIG. 5a, the process can include the following steps:

[0400] S301, compress and encode the first purpose image in the to-be-displayed data to obtain the code stream of the first purpose image and the reconstructed image of the first purpose image.

[0401] Wherein, the reconstructed image is generated by compressing and encoding the first purpose image to obtain the code stream of the first purpose image, and the reconstructed image of the first purpose image is obtained by decompressing and decoding the code stream.

[0402] S302, pre-process the reconstructed image of the first purpose image to obtain a similar image.

[0403] Wherein, the similar image is an image obtained by pre-processing the first purpose image and matching the second purpose image.

[0404] Wherein, the pre-processing can be morphing conversion, projection transformation between two images (for example, projection transformation based on depth information, etc.), and the like, which is not limited here.

[0405] Through S302, the reconstructed image can be processed into an image (hereinafter referred to as a similar image) that is more similar (or said to be matched) to the pixel content of the second purpose image.

[0406] S303, based on the similar image, compress and encode the second purpose image in the to-be-displayed data to obtain the code stream of the second purpose image.

[0407] Wherein, the residual between the second purpose image and the similar image can be calculated, and the residual is compressed and encoded to obtain the code stream of the second purpose image.

[0408] In the embodiment of the present application, when compressing the second purpose image, the reconstructed image of the first purpose image can be processed once (for example, morphing conversion) to process the reconstructed image into a similar image that is more matched to the pixel content of the second purpose image; then the residual is obtained based on the second purpose image by referring to the similar image, and the residual is used for the compression and encoding of the second purpose image, so that the second purpose image can be compressed by the redundant information of the two purposes, so as to improve the compression efficiency of the to-be-displayed data.

[0409] In a specific implementation, the related parameter data involved in the compression and encoding process of the to-be-displayed data of the second purpose can be added to the code stream of the image of the first purpose or the code stream of the image of the second purpose, so that the display peripheral 200 can accurately decode the code stream of the to-be-displayed data.

[0410] Then in the above S301 to S303, any one step can refer to the perception and interaction data to implement the corresponding step.

[0411] As shown in FIG. 5a, in a possible implementation, when performing the above S301, S3011 can be used to implement.

[0412] S3011, based on the first purpose data in the perception and interaction data, compresses and encodes the image of the first purpose in the to-be-displayed data to obtain the code stream of the image of the first purpose and the reconstructed image of the image of the first purpose.

[0413] When compressing and encoding the image of the first purpose, the first purpose data in the perception and interaction data can be referred to to assist in image compression.

[0414] For example, the first purpose data can include a depth image captured by a depth camera for the left eye.

[0415] Then the terminal device 100 can obtain the depth information of the first purpose by using the depth image.

[0416] When using the first purpose data to assist in compression, for example, the above depth information can be used to adjust the compression quality of the image region in the compressed image (here, the image of the first purpose). The adjustment of the compression quality can be implemented by adjusting the encoding parameters of the encoder, such as adjusting the quantization coefficient of the encoder, or adjusting the quantization step length, and the like.

[0417] For example, the image region corresponding to a larger depth can have a lower compression quality (for example, when encoding, the quantization coefficient is set to be larger, and the quantization step length is set to be smaller), and the image region corresponding to a smaller depth can have a higher compression quality (for example, when encoding, the quantization coefficient is set to be smaller, and the quantization step length is set to be larger).

[0418] As shown in FIG. 5a, in a possible implementation, when performing the above S302, S3021 can be used to implement.

[0419] S3021, based on the first purpose data in the perception and interaction data, pre-processes the reconstructed image of the image of the first purpose to obtain the similar image.

[0420] In one example, taking the pre-processing as morphing conversion, the first target data can include a depth image captured by a depth camera for the left eye, and the terminal device 100 can use the depth image to obtain the first target depth information for constructing the morphing function. Then, the morphing function is used to morph the reconstructed image of the first target image to obtain the similar image.

[0421] Compared with the similar image obtained by S302, the morphing function constructed based on the first target depth information is more accurate, and the similar image obtained by morphing the reconstructed image using the more accurate morphing function can be more similar to the second target image.

[0422] Thus, in the embodiment, the reconstructed image can be pre-processed by referring to the perception and interaction data in the process of pre-processing the reconstructed image of the first target image. Since the perception and interaction data provide the first target depth information, the similar image obtained based on the perception and interaction data can be more similar to the second target image, and the code rate can be reduced.

[0423] As shown in FIG. 5a, in one possible embodiment, S303 can be implemented by S3031 when performing the above S303.

[0424] S3031, based on the second target data in the perception and interaction data and the similar image, compresses and encodes the second target image in the to-be-displayed data to obtain the code stream of the second target image.

[0425] The principle of using the perception and interaction data to compress the second target image in S3031 is similar to that of S3011, and details are described in S3011, which will not be repeated here.

[0426] Thus, FIG. 5a exemplarily describes the process of compressing and encoding the to-be-displayed data in S107a in FIG. 4.

[0427] In a binocular or multi-view scenario, the single-view depth information can be used to assist in compressing the multi-view image to improve the code stream and reduce the bandwidth requirement.

[0428] In another scenario, not limited to a binocular or multi-view scenario, the eye movement information calculated based on the eye movement image in the perception and interaction data can be used to compress and encode the to-be-displayed data.

[0429] Specifically, eye movement information can be extracted from eye movement images captured by the eye tracking camera in the perception and interaction data, to determine the region (i.e. the fixation point) of the line of sight focus in the to-be-displayed data, and then a higher compression quality is used for the region in the to-be-displayed data located at the fixation point, and a lower compression quality can be used for the region in the to-be-displayed data outside the fixation point. The adjustment of the compression quality can be realized by adjusting the encoding parameters of the encoder, such as adjusting the quantization coefficients of the encoder, or adjusting the quantization step length, etc. In this way, the image quality of the image region with the line of sight focus after compression is higher than that of the image region outside the image region. Of course, the scheme of using eye movement information to assist in the compression of the to-be-displayed data can also be applied to binocular or multi-purpose scenarios.

[0430] The perception and interaction data used in S3011, S3021, S3031 shown in FIG. 5a can be eye movement information, and then the first target data and the second target data are both the same eye movement information.

[0431] In addition, in FIG. 5a, the perception and interaction data used in S3011, S3021, S3031 can be depth information alone, for example, the first target data is first target depth information and the second target data is second target depth information, can be eye movement information alone, and can also be depth information and eye movement information, which is not limited here.

[0432] Correspondingly, with reference to FIG. 4, when the display peripheral 200 decodes the code stream of the to-be-displayed data to obtain the to-be-displayed data in S203a, the inverse process of the process shown in FIG. 5a is used to realize the decoding process, and the principle of the decoding process is corresponding to the principle of the encoding process.

[0433] For example, the display data decoding module of the display peripheral 200 shown in FIG. 2c can first decode the encoded first target image to obtain the reconstructed image of the first target image; then, the same preprocessing operation (such as morphing conversion) corresponding to the encoding end (such as the display data encoding module shown in FIG. 2c) is performed on the reconstructed image of the first target image to obtain the preprocessed reconstructed image (also referred to as the similar image). In addition, the display data decoding module shown in FIG. 2c can also decode the code stream of the second target image, and obtain the reconstructed image of the second target image based on the decoding result of the second target image and the similar image. Specifically, the code stream of the second target image can be decoded to obtain the residual of the second target image; then, the residual and the similar image are processed (such as superimposed processing) to obtain the reconstructed image of the second target image. In this way, the reconstructed image of the first target image and the reconstructed image of the second target image are obtained. The processing process parameter data for the reconstructed image of the first target image can be obtained by parsing the code stream.

[0434] In the decoding process, similar to S3011, S3021 and S3031, the display device 200 can refer to the first-eye data in the perception and interaction data when pre-processing the reconstructed image of the first-eye image, and can also refer to the first-eye data in the perception and interaction data when decoding the code stream of the first-eye image, and can also refer to the first-eye data in the perception and interaction data when decoding the code stream of the second-eye image.

[0435] In the embodiments of the present application, the terminal device 100 can use the perception and interaction data to compress and encode the to-be-displayed data, and the display device 200 can also use the perception and interaction data to decompress and decode the code stream of the encoded to-be-displayed data, so that the effective perception and interaction information can be reasonably used to assist the compression of the to-be-displayed data, the encoding prediction can be made more accurate, and the compression efficiency can be improved. Moreover, the encoding parameters of the encoder (such as the encoding module of the display data) of the terminal device 100 can be adjusted by using the perception and interaction data, so that the image quality of the encoded image is more suitable for human visual sense. Therefore, the method can obtain higher compression efficiency than the prior art, reduce the code rate and the bandwidth required for transmitting the code stream of the to-be-displayed data, and reduce the time delay. Moreover, the quality of the video generated by decoding is also better, and the user's viewing experience can be improved.

[0436] Based on S203b in FIG. 4, the process of encoding the perception and interaction data by the display device 200 is introduced below.

[0437] For example, the perception and interaction data to be encoded can be binocular data, and the perception and interaction data can include data collected by various sensors of the display device 200. For example, as shown in FIG. 2b, the perception and interaction data can include at least one of the following: black and white images, depth images, IMU data, etc. obtained from the perception module and the interaction module; VST data (such as real scene images, etc.) obtained from the VST module, etc. The perception and interaction data can be a single image or a video obtained based on a period of time. The perception and interaction data can include first-eye data (such as perception and interaction data related to the left eye) and second-eye data (such as perception and interaction data related to the right eye). Alternatively, more than two eyes of data can also be included, which is not limited here.

[0438] Then the perception and interaction data can be divided into two categories: one is VST data, and the other is other perception and interaction data other than VST data.

[0439] First, the encoding process of the VST data is introduced.

[0440] For example, the encoding module of the display peripheral 200 shown in FIG. 2c can compress and encode the VST data according to other perception and interaction data in addition to the VST data.

[0441] The process of the display peripheral 200 compressing and encoding the VST data according to other perception and interaction data in addition to the VST data will be described below in conjunction with FIG. 5b.

[0442] As described above, taking the VST data as an example of visual image data, the visual image data can include a first-view image and a second-view image.

[0443] In this embodiment, the image data of each view in the binocular VST data can be compressed in sequence, for example, the first-view image is compressed first and the second-view image is compressed later. Since the distance between the binocular cameras (for example, the left-eye camera and the right-eye camera) on the display peripheral 200 is relatively small, the contents of the two images (the first-view image and the second-view image) captured by the binocular cameras at the same time are relatively similar. Therefore, the method of the present application can use the first-compressed image as a reference image for the second-compressed image, so as to remove the redundant information and reduce the code rate.

[0444] As shown in FIG. 5b, the process can include the following steps:

[0445] S401, compress and encode the first-view image in the VST data to obtain a code stream of the first-view image and a reconstructed image of the first-view image.

[0446] S402, pre-process the reconstructed image of the first-view image to obtain a similar image.

[0447] The similar image is an image obtained by pre-processing the first-view image.

[0448] S403, compress and encode the second-view image in the VST data based on the similar image to obtain a code stream of the second-view image.

[0449] In the embodiment of the present application, when compressing the second-view image in the VST data, the reconstructed image of the first-view image can be processed once (for example, morphological conversion) to process the reconstructed image into a similar image that is more matched with the pixel content of the second-view image. Then, the residual error is obtained by referring to the similar image based on the second-view image, and the residual error is used for the compression and encoding of the second-view image, so that the second-view image can be compressed by using the binocular redundant information, thereby improving the compression efficiency of the VST data.

[0450] In a specific implementation, the related parameter data involved in the compression and encoding process of the VST data of the two purposes can be added to the code stream of the first purpose image or the code stream of the second purpose image, so that the terminal device 100 can accurately decode the code stream of the VST data.

[0451] Then in the above S401 to S403, any one step can refer to other perception and interaction data other than the VST data to implement the corresponding step.

[0452] As shown in FIG. 5b, in a possible implementation, when performing the above S401, S4011 can be used to implement.

[0453] S4011, compress and encode the first purpose image in the VST data based on the first purpose data in the other perception and interaction data other than the VST data, to obtain the code stream of the first purpose image and the reconstructed image of the first purpose image.

[0454] When compressing and encoding the first purpose image, the first purpose data in the other perception and interaction data other than the VST data can be referred to to assist in image compression.

[0455] Among them, the other perception and interaction data other than the VST data used in S4011, S4021 and S4031 is the original other perception and interaction data other than the VST data without compression processing.

[0456] When obtaining the other perception and interaction data other than the VST data, the original other perception and interaction data other than the VST data can be obtained by decompressing the compressed other perception and interaction data, or the other perception and interaction data other than the VST data can be used to assist in the compression of the VST data before the other perception and interaction data other than the VST data is compressed, which is not limited in the present application.

[0457] As shown in FIG. 5b, in a possible implementation, when performing the above S402, S4021 can be used to implement.

[0458] S4021, pre-process the reconstructed image of the first purpose image based on the first purpose data in the other perception and interaction data other than the VST data, to obtain the similar image.

[0459] As shown in FIG. 5b, in a possible implementation, when performing the above S403, S4031 can be used to implement.

[0460] S4031, compress and encode the second object image in the VST data based on the second object data in the other perception and interaction data than the VST data and the similar image, to obtain a code stream of the second object image.

[0461] Thus, Fig. 5b exemplarily describes the process of compressing and encoding the VST data in the perception and interaction data in S203b in Fig. 4.

[0462] The implementation principle of the process shown in Fig. 5b is the same as that of the process shown in Fig. 5a, except that the objects to be compressed are different (the to-be-displayed data and the VST data, respectively), and the perception and interaction data referred to when compressing the objects can be different. In the embodiment of Fig. 5a, the perception and interaction data based on which can be the VST data or other perception and interaction data than the VST data, while in the embodiment of Fig. 5b, the perception and interaction data based on which is other perception and interaction data than the VST data. The principles of other processes are the same, and thus the specific implementation details of Fig. 5b are not described in detail here, and can be referred to the introduction of Fig. 5a.

[0463] Correspondingly, with reference to Fig. 4, when the terminal device 100 decodes the code stream of the VST data in the perception and interaction data to obtain the VST data in S107b, the inverse process of the process shown in Fig. 5b can be used to implement the decoding process, and the principle of the decoding process is corresponding to that of the encoding process, which will not be described herein again. The specific principle can be referred to the principle of decoding the code stream of the to-be-displayed data by the display peripheral 200 to obtain the to-be-displayed data, which will not be described herein again.

[0464] In the embodiments of the present application, the display peripheral 200 can use other perception and interaction data than the VST data to compress and encode the VST data, and the terminal device 100 can also use other perception and interaction data than the VST data to decompress and decode the code stream of the encoded VST data, so as to reasonably use effective perception and interaction information to assist the compression of the VST data, to make the encoding prediction more accurate and improve the compression efficiency. Moreover, the other perception and interaction data can be used to adjust the encoding parameters of the encoder (for example, the encoding module of the perception and interaction data) of the display peripheral 200, so as to make the quality of the encoded image more suitable for human visual sense. Therefore, the method can obtain higher compression efficiency than the prior art, reduce the code rate and the bandwidth required for transmitting the code stream of the VST data, and reduce the time delay. Moreover, the quality of the video generated by decoding is also better, which can improve the user's viewing experience.

[0465] As mentioned above, the perception and interaction data may include not only VST data, but also other perception and interaction data besides VST data, such as black and white images, depth images, IMU data, etc., corresponding to at least one of spatial data, gesture data, eye movement data, facial data, limb data, depth data, etc.

[0466] In some embodiments, the amount of the other perception and interaction data is relatively small compared to the VST data. Then the display peripheral 200 may not compress and encode the other perception and interaction data, but may directly transmit the other perception and interaction data to the terminal device 100.

[0467] In some embodiments, the display peripheral 200 can also encode the other perception and interaction data based on the encoding process of encoding the VST data shown in Figure 5b. When encoding the other perception and interaction data, it can be implemented through the principles of S401, S402, and S403 shown in Figure 5b (specifically, the compressed data object is replaced from the VST data with the other perception and interaction data), without the need to use the perception and interaction data for auxiliary compression (that is, there is no need to execute S4011, S4021, and S4031).

[0468] Similarly, the terminal device 100 can also decompress the compressed code stream of the other perception and interaction data from the display peripheral 200 according to the principle of the inverse process of S401, S402, and S403 in Figure 5b to obtain the other perception and interaction data.

[0469] The following describes the implementation process of the processing method of this application with reference to specific examples.

[0470] Example 1

[0471] FIG6 exemplarily shows a process diagram of the processing method of the present application.

[0472] In FIG. 6 , the display peripheral device 200 connected to the terminal device 100 is specifically an MR device 301 .

[0473] The MR device 301 may be MR glasses or a MR helmet.

[0474] The two devices shown in FIG6 are a specific schematic diagram of the two devices shown in FIG2b.

[0475] In this example 1, the processing chip of the MR device 301 has strong computing capabilities, and may include a perception and interaction data calculation module, which can be used to calculate other perception and interaction data in addition to the real scene image, and send the calculated perception and interaction data (i.e., the calculation results of the perception and interaction data) to the terminal device 100;

[0476] For example, as shown in FIG6 , the calculated perception and interaction data may include but is not limited to: spatial positioning results, spatial geometric grids, lighting information, gesture points, gaze points, facial geometry, limb key points, local positioning maps, expression bases, etc.

[0477] In addition, since the MR device 301 is equipped with a powerful processing chip, it can directly complete the link closed loop from the VST camera (one or more sensor modules) to the display of the VST image. Therefore, the MR device 301 does not need to send VST data to the terminal device 100.

[0478] In addition, the MR device 301 has a display data processing module, which can be used to perform computational processing such as virtual-real synthesis on the image to be displayed. Therefore, the display data format supported by the MR 301 is a virtual frame in RGBAZ format or RGBA format.

[0479] Specifically, the terminal device 100 can obtain the data capability information of the display data of the MR device 301 and the capability information of its transmission link, the data capability information of the perception and interaction data and the capability information of its transmission link, and determine the format of the display data to be transmitted, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links based on various capability information, and instruct the display peripheral 200 to perform corresponding initialization according to the various determined information.

[0480] For example, in the scenario of Figure 6, the terminal device 100 can determine that the format of the data to be transmitted and displayed is RGBAZ or RGBA format, and determine that the resolution of each image in the data to be displayed supported by the MR device 301 is full HD resolution (or it can also be the resolution of the foveated rendering display).

[0481] In addition, the terminal device 100 may determine that when rendering each image at full HD resolution, the maximum transmission rate of the transmission link of the display data can reach 100Gbps (G stands for gigabytes, bps stands for bits per second), and compression encoding needs to be started.

[0482] For example, in the scenario of Figure 6, the terminal device 100 can determine that the format of the perception and interaction data to be transmitted includes: the format of the calculation results of the perception and interaction data, such as the results of the above-mentioned spatial positioning, spatial geometric grid, lighting information, gesture points, gaze points, Blendershape, limb key points, subMap, positioning local map, expression base, etc.

[0483] In addition, the data volume of the above calculation results is small, so compression encoding does not need to be enabled.

[0484] Then, as shown in FIG6 , the MR device 301 can collect perception and interaction data through a sensor module. The sensor module may include, but is not limited to, two VST cameras, four Mono cameras, two eye-tracking cameras, two downward-facing cameras, a depth camera, an IMU, etc. This application does not impose any restrictions on the type or quantity of sensors.

[0485] Among them, the VST camera can collect VST data (also expressed as VST image, or real scene image), the Mono camera can collect Mono image (also expressed as Y image above), the eye tracking camera can collect eye movement images (since the eye tracking camera is also a Mono camera, it is also called Y image or Mono image, or black and white image), the lower camera can collect lower images (since the lower camera is also a Mono camera, it is also called Y image or Mono image, or black and white image), the depth camera can collect depth images, and the IMU can collect motion information.

[0486] Then, the ISP of the MR device 301 can calculate the collected perception and interaction data, wherein the real scene image from the VST camera can be calculated to obtain the signal of the real scene image, and the signal of the real scene image is sent to the processing module of the display data for fusion with the virtual frame.

[0487] Furthermore, the ISP may pass the calculated perception and interaction data to the perception and interaction data calculation module for further calculation to obtain the perception and interaction data calculation results. This calculation may include, but is not limited to, calculations of the following aspects: spatial geometric grid, lighting, gestures, eye movements, face, body parts, etc. The perception and interaction data may include, but is not limited to, at least one of the following: data from the ISP, motion information from the IMU, depth information from depth calculations performed on depth images captured by the depth camera, etc.

[0488] Here, depth calculation is performed on the depth image to obtain depth information, which can be used to compress and encode the data to be displayed and decompress and decode it (specifically, it has been described in the embodiment of FIG. 5 a).

[0489] 6 , the perception and interaction data calculation module may send the calculated perception and interaction data calculation results to the terminal device 100 through the perception and interaction interface.

[0490] The terminal device 100 can generate a virtual frame in, for example, RGBAZ format or RGBA format based on the calculation results of the perception and interaction data, and optionally use the received calculated perception and interaction data (such as depth information, eye movement information, etc.) to compress the virtual frame, and send the compressed code stream to the MR device 301 through the display interface.

[0491] The MR device 301 may decompress and decode the received code stream of the virtual frame to obtain the virtual frame.

[0492] The display data processing module of the MR device 301 can use the calculation results of the perception and interaction data calculated by the perception and interaction data calculation module, the signal of the real-scene image calculated by the ISP, and the decoded virtual frame to process and calculate the display data. The calculation may include the fusion of the virtual frame and the real-scene image, and optionally may also include anti-distortion processing of the virtual frame and the real-scene image respectively. Finally, the upper screen display is performed through the display driver. The screens here are the left eye display and the right eye display, and the two screens are used to display binocular images.

[0493] Optionally, as shown in FIG6 , the terminal device 100 may include two main cameras for image capture. In the XR scenario, the two main cameras may also be used to collect perception and interaction data on the terminal device 100 side. Accordingly, the processing chip of the terminal device 100 may also implement the algorithms for calculating the main camera's capture and the algorithms for calculating the perception and interaction data collected by the terminal device 100. In addition, the terminal device 100 may also collaborate with the display peripheral 100 for capture.

[0494] As shown in Figure 6, the processing chip can provide a two-dimensional user interface (UI), such as an entrance to two-dimensional buttons and other controls, and a three-dimensional UI (such as an entrance to three-dimensional buttons and other controls). After the user clicks the corresponding control through the display interface (such as a game interface), the processing chip can provide a two-dimensional window or a three-dimensional model, and render the two-dimensional window or three-dimensional model through a rendering engine module (an example of a processing module shown in Figure 2b, such as a CPU, GPU) to generate data to be displayed, such as a virtual frame.

[0495] In some embodiments, the processing chip may also include an AI module (e.g., an AI processing chip). The AI ​​module may generate some or all elements in the virtual frame based on the calculation results of the received perception and interaction data through AI model reasoning, calculation of a large language model, and other methods.

[0496] The terminal device 100 can flexibly choose to generate a virtual frame through one or more modules of the rendering engine module and the AI ​​module according to the application scenario and computing power.

[0497] The positioning module is used to position the terminal device 100 itself.

[0498] The reconstruction module can be used to reconstruct the environment where the terminal device 100 is located into a three-dimensional model.

[0499] As shown in FIG. 6, the terminal device 100 can have an audio module, which can collect audio; also have a microphone module, which can collect audio; and also have a display screen (for example, 720P), which can display media data such as images and videos.

[0500] As shown in FIG. 6, the terminal device 100 can have a Wi-Fi module, which can be connected to various display peripherals 100 through Wi-Fi.

[0501] In the present example 1, the terminal device 100 can be compatible with display peripherals equipped with MR glasses with strong processing modules. Such MR glasses can complete the link closed loop from the VST module of the real scene to the display of the entire real scene at the glasses end, and can display the result of virtual-real synthesis at the glasses end. The MR glasses can also realize the calculation of spatial positioning perception and interaction at the glasses end, and can directly output the calculation result of the perception and interaction data, which can include the calculation result of positioning, perception, and interaction.

[0502] In the present example 1, when generating elements in a virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation result of the perception and interaction data through a rendering engine module, and can also obtain the elements in the to-be-displayed data through model inference by an AI module. The terminal device 100 finally obtains the to-be-displayed data (for example, a virtual frame) synthesized by multiple elements, and sends the to-be-displayed data to the MR device 301. The MR device 301 performs virtual-real synthesis based on the received virtual frame and the real scene image obtained at the MR device 301 side, and displays the image after virtual-real synthesis on the left eye display screen and the right eye display screen.

[0503] Different from the prior art, the display interface of the terminal device 100 of the present application can support the transmission of display data to be virtually-real synthesized, and the perception and interaction interface thereof can support the transmission of the calculated perception and interaction data. The to-be-displayed data supported by the terminal device 100 for transmission can be data in the format of RGBAZ (or RGBA format): one transparency channel A and one depth channel Z are added. The transparency channel is used as a mask in virtual-real synthesis, and the depth channel provides correct occlusion relationship in synthesis to realize virtual-real occlusion. The perception and interaction data supported by the terminal device 100 for transmission is the result calculated by an algorithm: for example, the result of spatial positioning, the local positioning map, the spatial geometric grid, the light information, the gesture point, the gaze point, the limb key point, and the facial expression base. The transmission in this way can reduce bandwidth occupation and latency.

[0504] In addition, the example 1 can obtain information by using the perception and interaction data, and can perform a reference auxiliary processing on the multi-view video to be processed when coding the display data, so that a higher compression efficiency than the prior art can be obtained, the code rate and the required bandwidth for transmission are reduced, the time delay is reduced, the subjective quality of the obtained video is better, and the subjective experience of the user is improved.

[0505] Example 2

[0506] FIG. 7 exemplarily shows a process schematic diagram of the processing method of the application.

[0507] In FIG. 7, the display peripheral 200 connected with the terminal device 100 is specifically an MR device 302.

[0508] The MR device 302 can be an MR glasses or an MR helmet, etc.

[0509] The two devices shown in FIG. 7 are a specific schematic diagram of the two devices shown in FIG. 2b.

[0510] Compared with FIG. 6 in the example 1, please refer to FIG. 7, most of the contents of the example 2 are the same as those of the example 1, and the main difference is as follows:

[0511] Difference 1: As shown in FIG. 7, the terminal device 100 can have a calculation module of the perception and interaction data of the MR device 301 as shown in FIG. 6, which is used to calculate the perception and interaction data from the MR device 302. That is, the terminal device 100 of the application can have the calculation capability of the perception and interaction data of the XR glasses and other devices.

[0512] Difference 2: As shown in FIG. 7, the MR device 302 can compress and encode the perception and interaction data to be sent to the terminal device 100, for example, compress and encode the real scene image (such as the VST image), and optionally compress and encode the depth image, the black and white image, the motion information, etc., and then send to the terminal device 100.

[0513] Difference 3: As shown in FIG. 7, the terminal device 100 can have a processing module of the display data of the MR device 301 as shown in FIG. 6, which is used to calculate and process the display data, for example, to generate the virtual fusion image. That is, the terminal device 100 of the application can have the processing capability of the display data of the XR glasses and other devices.

[0514] Difference 4: As shown in FIG. 7, the terminal device 100 generates the display data by using the processing module of the display data, and the display data is a virtual fusion image in RGB format. Optionally, the terminal device 100 can compress and encode the display data and send it to the MR device 302.

[0515] In this way, the terminal device 100 can integrate the computing capability of the AR glasses, the VR glasses, the MR glasses and the like into the terminal device 100, and perform the calculation on the perception interaction data and the virtual-real synthesis on the to-be-displayed data, so as to reduce the computing requirement of the display peripheral device, and reduce the weight and the power consumption of the display peripheral device.

[0516] In the following, the differences between the process shown in FIG. 7 and the process shown in FIG. 6 will be mainly described, and the same parts will not be described again, and can be referred to the description of Example 1.

[0517] In the embodiment of FIG. 7, the terminal device 100 can determine, by acquiring the data capability information of the display data of the MR glasses 302 and the capability information of the corresponding transmission link, that the display data supported by the MR glasses 302 is in the RGB format, and the resolution includes the full high definition resolution and the gaze point rendering display resolution, and the terminal device 100 can determine that the upper limit of the transmission rate of the to-be-displayed data to be transmitted is 60 Gbps.

[0518] In addition, the terminal device 100 can also determine, by acquiring the data capability information of the perception and interaction data of the MR glasses 302 and the capability information of the corresponding transmission link, that the sensors supported by the MR glasses 302 to be started can include 2 VST cameras, 8 black and white cameras (of which, the eye tracking camera and the downward camera are also black and white cameras), 1 depth camera, and an IMU, and the terminal device 100 can determine that the upper limit of the data amount of the perception and interaction data to be transmitted is close to 30 Gbps.

[0519] Then, the terminal device 100 can determine, based on various capability information, the format of the to-be-displayed data to be transmitted, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links, and instruct the display peripheral device 200 to perform the corresponding initialization according to the determined various information.

[0520] For example, in the scenario of FIG. 7, the terminal device 100 can determine that the format of the to-be-displayed data to be transmitted is RGB, and determine that the resolution of each image in the to-be-displayed data supported by the MR device 302 is the full high definition resolution (or can also be the gaze point rendering display resolution). The transmission rate of the data stream of the full high definition to-be-displayed image can reach 60 Gbps, which exceeds the transmission capability of the display data transmission link, and the display data transmission link can be configured to be compressed and encoded, so as to ensure that the transmission rate of the display data transmission link can meet the upper limit requirement of 60 Gbps.

[0521] For example, in the scenario of FIG. 7, the terminal device 100 can determine that the format of the perception and interaction data to be transmitted includes: starting 2 VST cameras, starting 8 black and white cameras, starting 1 depth camera, starting 1 IMU, and the resolution and frame rate of the images collected by each sensor. Based on the format of the perception and interaction data, the terminal device 100 can determine that the transmission rate of the data stream of the VST images collected by the two VST cameras reaches 30 Gbps, and the transmission link of the perception and interaction data on the terminal device 100 side can be configured to start compression encoding.

[0522] For other perception and interaction data in addition to VST images, such as depth images, black and white images, IMU data, etc., since the data volume is small, compression encoding can be optionally performed (such as motion information shown in FIG. 7).

[0523] Subsequently, as shown in FIG. 7, the MR device 302 can collect perception and interaction data through the sensor module, including real scene images (such as VST images), depth images, black and white images, etc. The MR device 302 can perform compression encoding on the real scene images to obtain the code stream of the real scene images, and send to the terminal device 100 through the perception and interaction interface. The MR glasses 302 can not compress other perception and interaction data in addition to the real scene images, but directly send to the terminal device 100 through the transmission link of the above-mentioned perception and interaction interface.

[0524] Optionally, as shown by the dashed arrows in FIG. 7, in order to compress and decompress encode the VST images by means of the perception and interaction data, and to compress and decompress encode the to-be-displayed data by means of the perception and interaction data. As shown in FIG. 7, the MR device 302 can calculate eye movement information based on the eye movement images collected by the sensor module (such as the eye movement images collected by the eye tracking camera, which are also black and white images). Optionally, the MR device 302 can calculate depth information from the depth images captured by the depth camera. And the eye movement information and the depth information are also sent to the terminal device 100 through the perception and interaction interface.

[0525] Among them, the MR glasses 302 can utilize at least one of the eye movement information and the depth information to compress and decompress encode the VST images to obtain the code stream of the VST data.

[0526] While the data volume of the perception and interaction data such as depth images, black and white images, motion information, etc. can be compressed or not compressed, which is not limited here. These perception and interaction data are other perception and interaction data in addition to the VST data as described above.

[0527] The terminal device 100 can decompress the received perception and interaction data to obtain the perception and interaction data.

[0528] As to the process of compressing and decompressing the perception and interaction data, please refer to the introduction of FIG. 5b and the related solutions, which will not be repeated here.

[0529] Continuing to refer to FIG. 7, the terminal device 100 can utilize the calculation module of the perception and interaction data in the processing chip to calculate the decoded perception and interaction data to obtain the calculated perception and interaction data (for example, the result of spatial positioning, spatial geometric grid, lighting information, gesture point, etc. shown in FIG. 6), and the specific process is the same as the process principle in Example 1, which will not be repeated here.

[0530] In addition, as shown in FIG. 7, the terminal device 100 can generate a virtual frame (RGBA format or RGBA format) based on the calculated perception and interaction data through the rendering engine module; in addition, the terminal device 100 can also generate part or all of the elements in the virtual frame through the AI module. For example, the AI module can generate part or all of the elements in the virtual frame based on the calculated perception and interaction data; and the virtual frame is fused with the decoded real scene image through the display data processing module in the MR real scene rendering synthesis module to obtain a virtual-real fused image (for example, an image in RGB format).

[0531] Then, the terminal device 100 can compress and encode the virtual-real fused image in RGB format (for example, utilizing the above-mentioned depth information and eye movement information), and send the code stream of the image to the MR glasses 302 through the display interface.

[0532] The MR glasses 302 can decode the code stream of the image, and display the decoded virtual-real fused image on the screen through display driving to display the binocular image on the left eye display screen and the right eye display screen.

[0533] In the embodiments of the present application, the display interface transmission of the terminal device 100 is the directly displayable data in RGB format, and the terminal device 100 of the present application can be compatible with the display peripherals such as ordinary MR glasses to realize the calculation, virtual-real synthesis and other processing of the to-be-displayed data. In addition, the perception and interaction interface of the terminal device 100 of the present application can transmit the image data collected by the sensor, and can decompress the compressed perception and interaction data.

[0534] In addition, the display peripheral side perception and interaction interface transmitted data in this example 2 includes real scene image data, and other perception and interaction data (such as depth image, IMU data, etc.) in addition to the real scene image data. This method can use the information obtained from other perception and interaction data for the perception and interaction data, especially the real scene image data, and can perform reference auxiliary processing on the multi-view video to be processed when encoding the real scene image data, thereby obtaining higher compression efficiency than the prior art, reducing the code rate and transmission required bandwidth, reducing the latency, and the subjective quality of the video obtained is better, which can improve the subjective experience of the user.

[0535] In addition, the processing chip of the terminal device can include an AI module. When generating the elements in the virtual frame, the terminal device can render the elements in the to-be-displayed data based on the calculation result of the perception and interaction data through the rendering engine module, and can also obtain the elements in the to-be-displayed data through the AI module in a model inference manner. The terminal device finally obtains the to-be-displayed elements, which can be displayed in the form of AR on the AR device.

[0536] In this example 1, when generating the elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation result of the perception and interaction data through the rendering engine module, and can also obtain the elements in the to-be-displayed data through the AI module in a model inference manner. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI module. The terminal device 100 can fuse the final virtual frame and the real scene image through the MR real scene rendering synthesis module to obtain the image after virtual-real fusion. And send the image after virtual-real fusion to the MR device 302 for display.

[0537] Example 3

[0538] FIG. 8 exemplarily shows a process schematic diagram of the processing method of the present application.

[0539] In FIG. 8, the display peripheral 200 connected with the terminal device 100 is specifically an MR device 303.

[0540] The MR device 303 can be an MR glasses or an MR helmet, etc.

[0541] The two devices shown in FIG. 8 are a specific schematic diagram of the two devices shown in FIG. 2b.

[0542] With reference to FIG. 8, the example 3 is the same as the example 2 in most aspects, and the difference mainly lies in that when the terminal device 100 accesses the MR device 303, the VST images captured by the VST cameras of the MR device 303 can be directly processed and displayed on the MR device 303, and therefore the MR device 303 does not need to transmit the high-definition real scene images (for example, the VST images) to the terminal device 100 through the perception and interaction interface for processing. In addition, the MR device 303 can transmit other perception and interaction data (for example, data captured by various cameras and sensors, such as depth images, black-and-white images, motion information) to the terminal device 100, so that the terminal device 100 can perform calculation and processing on the other perception and interaction data.

[0543] In addition, the difference between the example 3 and the example 2 also includes that the calculation and processing of the display data (for example, the fusion of the virtual frame and the real scene image) is completed on the MR device 303 side, which is the same as the example 1.

[0544] The process of the method of the present application implemented by the system shown in FIG. 8 is briefly described as follows.

[0545] Firstly, the terminal device 100 can acquire the data capability information of the display data of the MR glasses 303 and the capability information of the corresponding transmission link, and the data capability information of the perception and interaction data and the capability information of the corresponding transmission link.

[0546] The terminal device parses the capability information to determine that the format of the display data supported by the MR glasses 303 is a virtual frame (for example, an RGBAZ format or an RGBA format), the resolution and the frame rate are 2*3840*3600*90Hz. Based on this, it can be determined that the upper limit of the transmission rate of the to-be-displayed data is 100Gbps.

[0547] The terminal device 100 parses the capability information to determine the format of the perception and interaction data to be transmitted by the MR glasses 303: the real scene images of the two VST cameras are not included, the format of the perception and interaction data supported by the MR glasses 303 for transmission includes the down-sampled images (for example, the resolution is 640*480) of the two VST cameras, the image data of the eight black-and-white cameras, the image data of the one depth camera, and the IMU data, and then the transmission rate of the data to be transmitted by the perception and interaction interface is only 1Gbps-2Gbps.

[0548] Then, the terminal device 100 can determine the format of the to-be-displayed data to be transmitted, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links based on various capability information, and instruct the display peripheral 200 to perform corresponding initialization according to the determined various information.

[0549] For example, in the scenario of FIG. 8, the terminal device 100 can determine that the format of the to-be-displayed data to be transmitted is RGBAZ or RGBA format, and determine that the resolution of each-eye image in the to-be-displayed data supported by the MR device 303 is full high definition (or can also be the resolution of gaze point rendering display). Then the transmission rate of the data stream of the full high definition to-be-displayed image can reach 100 Gbps, which exceeds the transmission capacity of the display data transmission link, and the display data transmission link can be configured to start compression encoding to ensure that the transmission rate of the display data transmission link can meet the upper limit requirement of 100 Gbps.

[0550] For example, in the scenario of FIG. 8, the terminal device 100 can determine that the format of the perception and interaction data to be transmitted includes starting 2 VST cameras, the down-sampled image resolution of the obtained VST image is, for example, 640*480, starting 8 black and white cameras, starting 1 depth camera, starting one IMU, and the image resolution and frame rate collected by each sensor. Based on the format of the perception and interaction data, the terminal device 100 can ensure that the data amount of the to-be-transmitted perception and interaction data is small, and the perception and interaction data transmission link can meet the requirement. Since the VST data does not need to be transmitted, only the compressed image of the VST image with a small data amount and the data of some sensors need to be transmitted, the terminal device 100 can not start compression encoding when setting the transmission link.

[0551] Then, as shown in FIG. 8, the MR device 303 can collect the perception and interaction data through the sensor module, including VST images, depth images, black and white images, etc. The MR device 302 can perform the calculation and display processing on the VST image, and does not need to transmit the VST image.

[0552] The MR device 303 can transmit other perception and interaction data, such as depth images, 8 black and white images, motion information, etc., to the perception and interaction interface of the terminal device 100 through the perception and interaction interface.

[0553] The terminal device 100 can perform calculation on the perception and interaction data through the perception and interaction data calculation module. The specific calculation content is similar to that in the above examples 1 and 2, which will not be described here again.

[0554] In addition, the terminal device 100 can also generate part or all of the elements in the virtual frame through the AI module. For example, the AI module can generate part or all of the elements in the virtual frame based on the calculated perception and interaction data.

[0555] In addition, the terminal device 100 can generate to-be-displayed data (for example, a virtual frame in RGBAZ or RGBA format) based on the calculated perception and interaction data.

[0556] When the terminal device 100 generates the elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation results of the perception and interaction data through the rendering engine module, and can also obtain the elements in the to-be-displayed data through the AI module in the form of model inference. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI module. The terminal device 100 can send the final virtual frame to the MR device 303, so that the MR device 303 displays the final virtual frame after completing the virtual-real synthesis.

[0557] In some embodiments, the terminal device 100 can compress the to-be-displayed data, for example, using the depth information and eye movement information obtained by the terminal device 100 by calculating the perception and interaction data to compress the to-be-displayed data, and transmitting the compressed virtual frame to the MR glasses 303 through the display interface.

[0558] Then, the MR glasses 303 can decompress and decode the compressed virtual frame to obtain a virtual frame in the RGBAZ format or the RGBA format.

[0559] Next, the display data processing module of the MR glasses 303 can process (for example, virtual-real synthesis) the real scene image and the virtual frame based on the information processed by the ISP, and display the processed image on the screen through the display driver.

[0560] For example, the dual-purpose virtual-real synthesized image can be displayed on the left eye display screen and the right eye display screen, respectively.

[0561] In this example 3, the terminal device 100 of the present application can be compatible with a display peripheral of an MR glasses type, which is a device capable of supporting VST data processing and directly sending to display. The VST data of the MR glasses 303 does not need to be transmitted to the terminal device 100, and can be processed, synthesized and displayed in the MR glasses 303.

[0562] Different from the above example 2, the display interface of the terminal device 100 of the present application transmits data in the RGBAZ format or the RGBA format, which is the same as example 1. Different from the above example 2, in this example 3, the data transmitted by the perception and interaction interface does not include VST data, but image data down-sampled from the VST. In this way, the data transmitted by the perception and interaction interface can include image data of various sensors, the down-sampled image data and IMU data, which has a small amount of data and does not need to be compressed, and can be directly transmitted. Compared with example 1, this example 3 can reduce the time delay of the VST channel.

[0563] In this example 3, the rendering of the image, and the calculation of the perception and interaction data are all deployed on the terminal device 100 side, which also takes into account reducing the power consumption and weight of the glasses-type peripheral device.

[0564] Example 4

[0565] FIG. 9 exemplarily shows a process schematic diagram of the processing method of the present application.

[0566] In FIG. 9, the display peripheral device 200 connected with the terminal device 100 is specifically an AR device 304.

[0567] The AR device 304 can be AR glasses or an AR helmet, etc.

[0568] The two devices shown in FIG. 9 are a specific schematic diagram of the two devices shown in FIG. 2b.

[0569] Comparing with FIG. 8 in example 3, please refer to FIG. 9, most of the content of this example 4 is the same as that of example 3, the difference mainly lies in that the sensor module of the AR glasses 304 does not include a sensor for collecting real scene images, such as a VST camera, so that the perception and interaction data transmitted by the AR glasses 304 to the terminal device 100 does not include real scene images; and the ISP of the AR device 304 also does not need to send VST images to the display data processing module. Other contents of example 4 are the same as those of example 3, which will not be repeated here.

[0570] In addition, the terminal device 100 can also generate part or all of the elements in the virtual frame through an AI module. For example, the AI module can generate part or all of the elements in the virtual frame based on the calculated perception and interaction data;

[0571] In addition, the processing chip of the terminal device 100 can include an AI module, and when the terminal device 100 generates elements in the virtual frame, the terminal device 100 can render the elements in the to-be-displayed data based on the calculation results of the perception and interaction data through the rendering engine module, and also can obtain the elements in the to-be-displayed data through the model inference of the AI module. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI module. The terminal device 100 can send the finally obtained to-be-displayed data (such as the virtual frame) to the AR device 304, and display it on the AR device 304 in the form of AR. In this way, the terminal device 100 of the present application can be compatible with the AR device.

[0572] Example 5

[0573] FIG. 10 exemplarily shows a process schematic diagram of the processing method of the present application.

[0574] In FIG. 10, the display peripheral device 200 connected with the terminal device 100 is specifically a VR device 305.

[0575] The VR device 305 can be a VR glasses or a VR helmet, etc., which is not limited here.

[0576] The two devices shown in FIG. 10 are a specific illustration of the two devices shown in FIG. 2b.

[0577] Most of the content of this example 5 is the same as that of example 4, and can refer to the introduction of example 4 for details, which is not repeated here. The main difference is that the display device connected with the terminal device 100 is a VR device, not an AR device, but the process of interaction between the terminal device 100 and the VR device 305 is the same as the process of interaction between the terminal device 100 and the AR device 304 in example 4. The virtual frame sent to the VR device 305 is displayed on the VR device 304 in the form of full immersive VR. In this way, the terminal device 100 of the present application can be compatible with the VR device.

[0578] In any of the above embodiments, when encoding and decoding the perception and interaction data, or the data to be displayed, a software encoder / decoder or a hardware encoder / decoder can be used, and in addition, a standard encoder / decoder or a non-standard encoder / decoder can also be used. The present application does not limit the implementation form of the encoder / decoder.

[0579] Next, an apparatus provided by an embodiment of the present application is introduced. As shown in FIG. 11:

[0580] FIG. 11 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. As shown in FIG. 11, the apparatus 500 can include a processor 501, optionally a transceiver 505, and optionally further include a memory 502.

[0581] The transceiver 505 can be referred to as a transceiving unit, a transceiver, or a transceiving circuit, etc., and is used to realize the transceiving function. The transceiver 505 can include a receiver and a transmitter. The receiver can be referred to as a receiver or a receiving circuit, etc., and is used to realize the receiving function; the transmitter can be referred to as a transmitter or a transmitting circuit, etc., and is used to realize the transmitting function.

[0582] The transceiver 505 can also be a communication interface.

[0583] The memory 502 can store computer programs or software codes or instructions 504, which can also be referred to as firmware. The processor 501 can implement the data processing method provided by the embodiments of the present application by running computer programs or software codes or instructions 503 therein, or by invoking the computer programs or software codes or instructions 504 stored in the memory 502. The processor 501 can be a central processing unit (CPU), and the memory 502 can be, for example, a read-only memory (ROM) or a random access memory (RAM).

[0584] The processor 501 and the transceiver 505 described in the present application can be implemented on an integrated circuit (IC), an analog IC, an RFIC, a mixed-signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc.

[0585] The apparatus 500 described above can further include an antenna 506. The modules included in the apparatus 500 are merely exemplary, and the present application is not limited thereto.

[0586] The structure of the data processing apparatus can not be limited to FIG. 11. The data processing apparatus can be a standalone device or a part of a larger device. For example, the data processing apparatus can be implemented in the following forms:

[0587] (1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally include a storage component for storing data and instructions; (3) a module that can be embedded in other devices; (4) a vehicle-mounted device, etc.; (5) other forms, etc.

[0588] For the case where the data processing apparatus is a chip or a chip system, the structure of the chip can be seen from FIG. 12. The chip shown in FIG. 12 includes a processor 601 and an interface 602. The number of the processor 601 can be one or more, and the number of the interface 602 can be multiple. Optionally, the chip or chip system can include a memory 603. The processor 601 is configured to invoke and run instructions from the interface 602, and when the processor 601 executes the instructions, the steps of the above method embodiments can be performed.

[0589] All the related content of each step involved in the method embodiments can be cited to the function description of the corresponding function module, and will not be repeated here.

[0590] Based on the same technical concept, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program includes at least one piece of code, which can be executed by a computer to control the computer to implement the method embodiments.

[0591] Based on the same technical concept, the embodiments of the present application also provide a computer program, which, when executed, implements the method embodiments.

[0592] The program can be stored in a storage medium packaged with the processor, or partially or entirely stored in a storage medium not packaged with the processor.

[0593] Based on the same technical concept, the embodiments of the present application also provide a chip including a processor. The processor can implement the method embodiments.

[0594] The steps of the method or algorithm described in combination with the disclosure of the embodiments of the present application can be implemented in the form of hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disk (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0595] Those skilled in the art should be aware that, in the above one or more examples, the functions described in the embodiments of the present application can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or code on the computer readable medium. The computer readable medium includes computer storage medium and communication medium, and the communication medium includes any medium that facilitates transfer of computer program from one place to another. The storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

[0596] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized by, A terminal device is in communication connection with a display device, and the method comprises: The terminal device receives first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; The terminal device receives the perception and interaction data in the first format from the display device; The terminal device obtains multimedia data based on the perception and interaction data.

2. The method of claim 1, wherein, The terminal device obtains multimedia data based on the perception and interaction data, comprising: The terminal device obtains multimedia data by reasoning the perception and interaction data through an artificial intelligence (AI) module.

3. The method of claim 1, wherein, The terminal device obtains multimedia data based on the perception and interaction data, comprising: The terminal device obtains a first multimedia element by reasoning the perception and interaction data through an AI module; The terminal device renders a second multimedia element based on the perception and interaction data; The terminal device obtains multimedia data based on the first multimedia element and the second multimedia element.

4. The method according to any one of claims 1 to 3, characterized in that, The multimedia data comprises at least one of the following: an image, a video, and a model.

5. The method according to any one of claims 1 to 4, characterized in that, Before the terminal device receives the perception and interaction data in the first format from the display device, the method further comprises: The terminal device receives second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; The terminal device determines second link information based on the second capability information and the first format; The terminal device configures the first transmission link based on the second link information.

6. The method of claim 5, wherein, The second link information comprises information indicating to start decompression, and the terminal device receives the perception and interaction data in the first format from the display device, comprising: The terminal device receives a first code stream of the perception and interaction data in the first format from the display device through the configured first transmission link, wherein the first code stream is compressed and encoded; The method further comprises: The terminal device decodes the first code stream based on the second link information to obtain the perception and interaction data in the first format.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: The terminal device receives third capability information, wherein the third capability information provides format information of display data supported by the display device; The terminal device determines a second format of display data to be transmitted based on the third capability information, wherein the second format comprises an image format, and the image format is an image format after virtual image and real scene image are synthesized, or an image format of virtual image; The terminal device obtains multimedia data based on the perception and interaction data, comprising: The terminal device obtains a first image sequence in the second format based on the perception and interaction data.

8. The method of claim 7, wherein, The method further comprises: The terminal device receives fourth capability information, wherein the fourth capability information provides third link information supported by the display device for a second transmission link; The terminal device determines fourth link information based on the fourth capability information and the second format; The terminal device configures the second transmission link based on the fourth link information.

9. The method of claim 8, wherein, The fourth link information includes information indicating starting compression, and the method further includes: The terminal device encodes the first image sequence based on the fourth link information to obtain a second code stream of the first image sequence; The terminal device sends the second code stream to the display device through the configured second transmission link.

10. The method of claim 9, wherein, The method further includes: The terminal device encodes the first image sequence based on the perception and interaction data to obtain a code stream of the first image sequence.

11. The method of claim 10, wherein, The first image sequence includes at least two-eye images, the at least two-eye images include a first-eye image and a second-eye image, and the perception and interaction data includes first-eye data.

12. The method of claim 11, wherein, The perception and interaction data further includes second-eye data, and the terminal device encodes the first image sequence based on the perception and interaction data to obtain a second code stream of the first image sequence, including: The terminal device encodes the first-eye image based on the first-eye data and encodes the second-eye image based on the second-eye data to obtain the second code stream of the first image sequence.

13. The method according to claim 11 or 12, characterized in that, The terminal device encodes the first-eye image to obtain a code stream of the first-eye image and a reconstructed image of the first-eye image; The terminal device pre-processes the reconstructed image based on the first-eye data to obtain a reference image matching the second-eye image; The terminal device encodes the second-eye image based on the reference image to obtain a code stream of the second-eye image. The first format includes at least one of the following:

14. The method according to any one of claims 1 to 13, characterized in that, Data format, resolution, frame rate, and number of eyes. The second format includes at least one of the following:

15. The method according to any one of claims 7 to 13, characterized in that, Data format, resolution, frame rate, and number of eyes. The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, including:

16. The method according to any one of claims 1 to 14, characterized in that, The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information and target information, wherein the target information is at least one of a computing capability of the perception and interaction data and an application scenario. The terminal device determines a second format of display data to be transmitted based on the third capability information, including:

17. The method according to any one of claims 7 to 13 or 15, characterized in that, The terminal device determines a second format of display data to be transmitted based on the third capability information and target information, wherein the target information is at least one of a computing capability of the display data and an application scenario. The first capability information provides format information of perception and interaction data supported by the display device for collection, or format information of perception and interaction data supported by the display device for calculation.

18. The method according to any one of claims 1 to 17, characterized in that, The method includes:

19. A data processing method, characterized by, ​ The display device sends first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; The display device obtains perception and interaction data in a first format, and the format information comprises the first format; The display device sends the perception and interaction data in the first format; The display device receives multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

20. The method of claim 19, wherein, Before the display device obtains the perception and interaction data in the first format, the method further comprises: The display device receives the first format; The display device initializes and sets a sensor for collecting perception and interaction data according to the first format.

21. The method according to claim 19 or 20, characterized in that, The method further comprises: The display device sends second capability information, wherein the second capability information provides first link information supported by the display device for a first transmission link; The display device receives second link information, wherein the second link information is determined based on the second capability information and the first format; The display device configures the first transmission link based on the second link information.

22. The method of claim 21, wherein, The second link information comprises information indicating starting compression or decompression, and the method further comprises: The display device compressively encodes the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; The display device sends the first code stream through the configured first transmission link.

23. The method of claim 22, wherein, The perception and interaction data in the first format comprises a live image and image data other than the live image, and the display device compressively encodes the obtained perception and interaction data in the first format to obtain a first code stream, comprising: The display device compressively encodes the live image based on the image data to obtain a first code stream.

24. A data processing method, characterized by, The terminal device is in communication connection with the display device, and the method comprises: The display device sends first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device determines a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information comprises the first format; The display device sends the perception and interaction data in the first format to the terminal device; The terminal device obtains multimedia data based on the perception and interaction data in the first format; The terminal device sends the multimedia data to the display device; The display device displays an image based on the multimedia data.

25. The method of claim 24, wherein, Before the display device sends the perception and interaction data in the first format to the terminal device, the method further comprises: The terminal device sends the first format to the display device; The display device initializes and sets a sensor for collecting perception and interaction data according to the first format; The display device obtains perception and interaction data in a first format based on the initialized sensor.

26. The method of claim 24 or 25, wherein, The method further comprises: The display device sends second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for the first transmission link; The terminal device determines second link information based on the second capability information and the first format; The terminal device sends the second link information to the display device; The display device configures the first transmission link based on the second link information.

27. The method of claim 26, wherein, The second link information includes information indicating starting decompression, and the method further includes: The display device encodes the obtained perception and interaction data in the first format based on the second link information to obtain a first code stream; The display device sends the first code stream to the terminal device through the configured first transmission link; The terminal device decodes the first code stream based on the second link information to obtain the perception and interaction data in the first format.

28. A data processing system comprising: The system includes a terminal device and a display device in communication connection; The display device is configured to send first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device is configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; The display device is further configured to send the perception and interaction data in the first format to the terminal device; The terminal device is further configured to obtain multimedia data based on the perception and interaction data in the first format; The terminal device is further configured to send the multimedia data to the display device; The display device is further configured to display an image based on the multimedia data.

29. A data processing apparatus, characterized by The data processing apparatus is in communication connection with a display device, and the data processing apparatus includes: A first receiving module configured to receive first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; A first determining module configured to determine a first format of perception and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; A second receiving module configured to receive the perception and interaction data in the first format from the display device; An obtaining module configured to obtain multimedia data based on the perception and interaction data.

30. A data processing apparatus, characterized in that, The apparatus includes: A first sending module configured to send first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; An obtaining module configured to obtain perception and interaction data in a first format, wherein the format information includes the first format; A second sending module configured to send the perception and interaction data in the first format; A first receiving module configured to receive multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

31. A computer readable storage medium, characterized in that, A computer program comprising computer instructions which, when executed on a computer or processor, cause the computer or processor to perform the method of any one of claims 1 to 18, or the method of any one of claims 19 to 27.

32. A data processing apparatus, characterized in that, An apparatus comprising one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory and transmit the signals to the processors, the signals comprising computer instructions stored in the memory; when the processors execute the computer instructions, the processors are configured to perform the method of any one of claims 1 to 18, or the method of any one of claims 19 to 27.

33. A computer program product, characterised in that, The computer program product comprises a software program which, when executed by a computer or processor, causes the steps of the method of any one of claims 1 to 18, or the method of any one of claims 19 to 27 to be performed.

Images