Data processing method, system and apparatus

Abstract

The embodiments of the present application relate to the technical field of media. Provided are a data processing method, system and apparatus. The method comprises: a terminal device being capable of acquiring format information of sensing and interaction data supported by a display device, and on the basis of the format information, determining the format of sensing and interaction data to be transmitted, so that transmission of the sensing and interaction data in said format is realized between the terminal device and the display device. Thus, regardless of the format of the sensing and interaction data supported by the display device, the terminal device can be compatible with the display device to perform the transmission of the sensing and interaction data, and thus realize the acquisition of image data, thereby achieving compatibility of the terminal device with various display devices.

Description

Data processing methods, systems and devices

[0001] This application claims priority to Chinese Patent Application No. 202410476542.X, filed on April 17, 2024, entitled "Data Processing Method, System and Apparatus", the entire contents of which are incorporated herein by reference. It also claims priority to Chinese Patent Application No. 202410808963.8, filed on June 20, 2024, also entitled "Data Processing Method, System and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of media technology, and in particular to a data processing method, system, and apparatus. Background Technology

[0003] With the continuous development of technologies such as AR (Augmented Reality) and VR (Virtual Reality), VR / AR movies, VR / AR games, and VR / AR product displays are gradually entering users' field of vision.

[0004] Current set-top boxes are designed to be used in conjunction with AR and VR devices, making them incompatible with various display devices. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a data processing method and apparatus. In this method, a terminal device can acquire format information of the sensing and interactive data supported by a display device, and determine the format of the sensing and interactive data to be transmitted based on this format information, thereby enabling the transmission of the sensing and interactive data in that format between the terminal device and the display device. Thus, regardless of the format of the sensing and interactive data supported by the display device, the terminal device can transmit sensing and interactive data compatiblely with the display device, thereby acquiring image data and achieving compatibility with various display devices.

[0006] In one possible implementation, this application provides a data processing method. A terminal device is communicatively connected to a display device. The method includes: the terminal device receiving first capability information, wherein the first capability information provides format information of sensing and interactive data supported by the display device; the terminal device determining a first format of sensing and interactive data to be transmitted based on the first capability information, wherein the format information includes the first format; the terminal device receiving sensing and interactive data of the first format from the display device; and the terminal device obtaining multimedia data based on the sensing and interactive data.

[0007] The format information provided by the first capability information may be information carried in the first capability information or information derived from the first capability information. For example, the first capability information carries multiple resolution information of the perception and interaction data supported by the display device. The format information provided by the first capability information is not limited to the multiple resolution information, but may also include information on the resolution range defined by the multiple resolution information, so as to determine at least one of the maximum resolution and minimum resolution of the perception and interaction data supported by the display device.

[0008] The terminal device of this application can obtain first capability information from a display device. This first capability information provides the format information of the sensing and interaction data supported by the display device (which can be the sensing and interaction data that can be collected, or the result of calculation on the collected sensing and interaction data). Based on this format information, within the capability range of the display device, the terminal device determines the format of the sensing and interaction data to be transmitted. Then, the sensing and interaction data received by the terminal device from the display device is also in the determined format. Therefore, regardless of the format of the sensing and interaction data supported by the display device, the terminal device of this application can compatiblely receive the sensing and interaction data from the display device, making the terminal device of this application compatible with various manufacturers and models of display devices for the transmission of sensing and interaction data.

[0009] Thus, the terminal device of this application is compatible with wearable display devices (such as AR glasses, VR glasses, AR helmets, VR helmets, etc.) with various data transmission protocols to achieve immersive virtual-real fusion and spatial interaction to meet VR, AR and other experiences; in addition, the terminal device 100 is also compatible with ordinary display devices (such as monitors, televisions, vehicle displays and projectors, etc.) with various data transmission protocols, so that the terminal device can be used as a portable micro host.

[0010] For example, a user can use a single terminal device to work with display devices of different forms and specifications from different manufacturers (such as displays, projectors, in-vehicle displays, AR, VR devices, etc.). It can also be used with different sensing units (3DOF / 6DOF / mesh / lighting / material) and different interaction units (gestures / eye tracking / handhelds / keyboards, etc.).

[0011] In one possible implementation, the terminal device obtains multimedia data based on the perception and interaction data, including: the terminal device infers from the perception and interaction data through an artificial intelligence (AI) module to obtain multimedia data.

[0012] The terminal device may include an AI module, which can use the AI ​​module to infer multimedia data based on perception and interaction data (or the calculation results of perception and interaction data).

[0013] In some scenarios, terminal devices can also interact with cloud servers. For example, a terminal device can send the perception and interaction data to the cloud to request the cloud to perform AI inference and obtain multimedia data returned by the cloud.

[0014] In other words, terminal devices can use their own AI modules or cloud-based AI modules to perform inference to obtain the multimedia data. The choice between using a single-sided or dual-sided AI module depends on the computing power and complexity of the inference. This can improve the display quality of the obtained multimedia data.

[0015] In one possible implementation, the terminal device obtains multimedia data based on the perception and interaction data, including: the terminal device inferring from the perception and interaction data using an AI module to obtain a first multimedia element; the terminal device rendering a second multimedia element based on the perception and interaction data; and obtaining multimedia data based on the first multimedia element and the second multimedia element.

[0016] Similarly, the AI ​​module can be an AI module on the terminal device side or an AI module on the server side; there are no restrictions here.

[0017] The first and second multimedia elements can be any multimedia element such as images, text, videos, models, and audio.

[0018] The terminal device can process (e.g., synthesize, without limitation) the first and second multimedia elements to obtain multimedia data.

[0019] Thus, the embodiments of this application can obtain multimedia data through rendering and AI inference. Text, images, etc., can be displayed elements in the multimedia data; audio from audio and video can be used as audio in the multimedia data; and models in the first and / or second multimedia elements can be used as two-dimensional or three-dimensional models of the multimedia data.

[0020] In one possible implementation, the multimedia data includes at least one of the following: images, videos, and models.

[0021] In one possible implementation, before the terminal device receives the first format perception and interaction data from the display device, the method further includes: the terminal device receiving second capability information, wherein the second capability information provides first link information supported by the display device for the first transmission link; the terminal device determining second link information based on the second capability information and the first format; and the terminal device configuring the first transmission link based on the second link information.

[0022] The first link information may include the number of supported transmission lanes (e.g., the maximum number) and the transmission rate of the lanes.

[0023] The second capability information can provide the display device with first link information supporting the first transmission link used for transmitting sensing and interactive data. The first link information may include, but is not limited to: the number of transmission channels (lanes) that can be opened (e.g., the maximum number), the transmission rate of the channels, and the maximum transmission rate (e.g., the transmission rate corresponding to x8).

[0024] Then, the terminal device 100 can determine the second link information based on the format of the sensing and interaction data (the first format mentioned above) within the maximum capability of the display device for the first transmission link, and configure the first transmission link based on the second link information.

[0025] The second link information may include an indication of whether to start decompression decoding, and optionally may also include at least one of the following: the number of channels allocated to the first transmission link (e.g., x4), the data reception frequency, the data transmission and reception format of the first transmission link, etc.

[0026] For example, if a display device supports eight transmission channels for sensing and interactive data, and a terminal device supports four reception channels for sensing and interactive data, then according to the first format of the sensing and interactive data to be transmitted, a corresponding number of transmission channels (up to four) can be selected from one to four transmission channels to transmit the sensing and interactive data through the first transmission link, thereby achieving compatibility of the terminal device with the transmission links of sensing and interactive data of various display devices.

[0027] In one possible implementation, the second link information includes information indicating the initiation of decompression, and the terminal device receiving the first format perception and interaction data from the display device includes: the terminal device receiving a first bitstream of compressed and encoded first format perception and interaction data from the display device through the configured first transmission link; the method further includes: the terminal device decoding the first bitstream based on the second link information to obtain the first format perception and interaction data.

[0028] The terminal device can determine the data transmission rate of the sensing and interactive data to be transmitted based on the first format, and determine the maximum transmission rate of the transmission link of the sensing and interactive data of the display peripheral based on the second capability information. When the maximum transmission rate is less than the data transmission rate mentioned above, it indicates that the current bandwidth cannot meet the transmission requirements of a large amount of sensing and interactive data, and compression encoding is required.

[0029] For example, if terminal device 100 determines that the maximum data volume max4 (determined based on the first format) is greater than the maximum data volume max3 (determined based on the first link information) that the transmission link for the perception and interaction data of display peripheral 200 can support, it indicates that the maximum bandwidth of the transmission link of display peripheral 200 is insufficient to support the transmission of perception and interaction data of volume max4 to terminal device 100, as this data volume exceeds the transmission capacity of the transmission link for perception and interaction data of display peripheral 200. Therefore, when configuring the transmission link for its own perception and interaction data, terminal device 100 can configure the initiation of decompression and decoding of the transmitted perception and interaction data, wherein the data volume of the compressed and encoded perception and interaction data is less than or equal to the maximum data volume max3.

[0030] In this way, the display peripheral 200 can compress and encode the sensing and interaction data to be sent, and then send the compressed and encoded data to the terminal device 100 through the sensing and interaction data transmission link. This allows for the transmission of sensing and interaction data with low latency and low bandwidth even when the amount of data to be sent by the display peripheral 200 is large. By encoding and decoding the sensing and interaction data, higher compression efficiency can be achieved, reducing the bit rate and the bandwidth required for transmission, and reducing latency.

[0031] In one possible implementation, the method further includes: the terminal device receiving third capability information, wherein the third capability information provides format information of display data supported by the display device; the terminal device determining a second format of the display data to be transmitted based on the third capability information, wherein the second format includes an image format, the image format being an image format synthesized from virtual images and real-world images, or an image format of a virtual image; the terminal device obtaining multimedia data based on the perception and interaction data, including: the terminal device obtaining a first image sequence of the second format based on the perception and interaction data.

[0032] The multimedia data may include a first image sequence. This first image sequence may be a single frame or multiple frames.

[0033] For example, the first image sequence can be a monocular image (an example of a single frame), or a binocular or multi-view image (an example of multiple frames).

[0034] Furthermore, the first image sequence obtained by the terminal device is also described as data to be displayed. The first image sequence may be generated by the terminal device itself based on the perception and interaction data of the first format, or the first image sequence may be generated by the cloud based on the perception and interaction data of the first format, so that the terminal device can obtain the first image sequence from the cloud.

[0035] Unlike existing terminal devices, the terminal device of this application can receive third capability information from a display device. This third capability information provides format information for display data supported by the display device. This format information can include not only image formats but also the resolution, frame rate, etc., of the data to be displayed. The terminal device can then determine a second format for the display data to be transmitted based on this third capability information. This second format is also within the format information provided by the third capability information. The second format can be a fused image format (e.g., RGB format) or a virtual image format (e.g., RGBAZ or RGBA format). Thus, the terminal device can obtain display data (e.g., first image data) in the second format based on the display data formats supported by the display device. This allows the terminal device of this application to be compatible with sending fused images to the display device for display on the display device side, and also compatible with sending virtual images to the display device for fusion and display of fused and displayed images.

[0036] In one possible implementation, the method further includes: the terminal device receiving fourth capability information, wherein the fourth capability information provides third link information supported by the display device for the second transmission link; the terminal device determining fourth link information based on the fourth capability information and the second format; and the terminal device configuring the second transmission link based on the fourth link information.

[0037] The third link information may include the number of supported transmission lanes (e.g., the maximum number) and the transmission rate of the lanes.

[0038] The fourth capability information can provide the display device with third link information for the second transmission link used to transmit display data. The third link information may include, but is not limited to: the number of transmission channels (lanes) that can be opened (e.g., the maximum number), the transmission rate of the channels, and the maximum transmission rate (e.g., the transmission rate corresponding to x8).

[0039] Then, the terminal device 100 can determine the fourth link information based on the format of the display data (the second format mentioned above) within the maximum capacity of the display device for the second transmission link, and configure the second transmission link based on the fourth link information.

[0040] The fourth link information may include an indication of whether compression decoding is initiated, and optionally may also include at least one of the following: the number of channels allocated to the second transmission link (e.g., x4), the data reception frequency, the data transmission and reception format of the second transmission link, etc.

[0041] For example, if a display device supports 8 channels for sending display data and a terminal device supports 4 channels for receiving display data, then according to the second format of the display data to be transmitted, a corresponding number of transmission channels (up to 4) can be selected from 1 to 4 transmission channels to transmit the display data through the second transmission link, thereby achieving compatibility of the terminal device with the transmission links of display data from various display devices.

[0042] In one possible implementation, the fourth link information includes information indicating the initiation of compression, and the method further includes: the terminal device compressing and encoding the first image sequence based on the fourth link information to obtain a second bitstream of the first image sequence; and the terminal device sending the second bitstream to the display device through the configured second transmission link.

[0043] The terminal device can determine the data transmission rate of the display data to be transmitted based on the second format, and determine the maximum transmission rate of the transmission link of the display data of the display peripheral based on the fourth capability information. When the maximum transmission rate is less than the data transmission rate mentioned above, it indicates that the current bandwidth cannot meet the transmission requirements of a large amount of display data and compression encoding is required.

[0044] For example, the terminal device 100 can determine the fourth link information based on the format of the data to be displayed, within the maximum capability of the display peripheral for the transmission link, and configure the transmission link for transmitting the display data based on the fourth link information.

[0045] The fourth link information may include an indication of whether compression encoding is initiated, and optionally may also include at least one of the following: the number of channels allocated to the second transmission link (e.g., x4), the data transmission frequency, the data transmission and reception format of the second transmission link, etc.

[0046] Specifically, if the terminal device 100 determines that the maximum data size max2 is greater than the maximum data size max1 that the transmission link of the display peripheral 200 can support, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 is insufficient to support the transmission of the data size max2 of the data to be displayed from the terminal device 100. Therefore, when configuring the transmission link for the display data, the terminal device 100 can configure the transmission of the data to be displayed to be compressed and encoded, so that the data size of the compressed and encoded data to be displayed is less than or equal to the maximum data size max1.

[0047] Thus, when the maximum capacity of the transmission link used by the display peripheral 200 to transmit the data to be displayed is less than the maximum data volume max2 of the data to be displayed sent by the terminal device 100 through its local display interface, the terminal device 100 can compress and encode the transmitted data to be displayed, and then send the compressed and encoded data to the display peripheral 200 through the display data transmission link. This allows for the transmission of display data with low latency and low bandwidth even when the amount of data to be displayed is large. By encoding and decoding the display data (the aforementioned first image sequence), higher compression efficiency can be achieved, reducing the bit rate and the bandwidth required for transmission, and reducing latency.

[0048] In one possible implementation, the method further includes: the terminal device compressing and encoding the first image sequence based on the perception and interaction data to obtain the bitstream of the first image sequence.

[0049] In this embodiment, the displayed data can be compressed by combining perception and interaction data to improve compression efficiency, reduce bit rate and transmission bandwidth, and reduce latency.

[0050] In one possible implementation, the first image sequence includes at least two visual images, the at least two visual images including a first visual image and a second visual image, and the perception and interaction data including first visual data.

[0051] In this context, the first-eye image and the first-eye data correspond to the same eye, for example, both being the left eye.

[0052] In one possible implementation, the perception and interaction data further includes second data, and the terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain a second bitstream of the first image sequence, including:

[0053] The terminal device compresses and encodes the first image based on the first image data and compresses and encodes the second image based on the second image data to obtain a second bitstream of the first image sequence.

[0054] In this context, the first-eye image and the first-eye data correspond to the same eye, for example, both being the left eye.

[0055] In this case, the second eye image and the second eye data correspond to the same eye, for example, both are the right eye.

[0056] In this embodiment, in multi-view scenarios with binoculars or more, the corresponding target display data can be compressed and encoded by combining perception and interaction data to reduce bit rate and latency.

[0057] In one possible implementation, the terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain a second bitstream of the first image sequence, including: the terminal device compresses and encodes the first target image to obtain a bitstream of the first target image and a reconstructed image of the first target image; the terminal device preprocesses the reconstructed image based on the first target data to obtain a reference image that matches the second target image; and the terminal device compresses and encodes the second target image based on the reference image to obtain a bitstream of the second target image.

[0058] In this embodiment, the reconstructed image of the first visual image can be preprocessed (e.g., deformable transformation) using perception and interaction data, making the preprocessed reference image closer to the second visual image in pixel content. Then, the second visual image is compressed and encoded based on the reference image, which can improve the compression rate of the second visual image. In this way, the redundant information of the two visual images can be used to compress the second visual image, thereby improving the compression efficiency of the first image sequence.

[0059] In one possible implementation, the first format includes at least one of the following: data format (e.g., the type of supported sensor), resolution (e.g., the resolution of the image data generated by the aforementioned sensor), frame rate (e.g., the frame rate of the image data generated by the aforementioned sensor), and number of pixels (e.g., the number of pixels corresponding to the aforementioned sensor).

[0060] In one possible implementation, the second format includes at least one of the following: data format (e.g., RGB format, RGBA format, RGBAZ format, etc.), resolution (e.g., the resolution of the displayed data), frame rate (e.g., the frame rate of the displayed data), and number of pixels (e.g., the number of pixels corresponding to the displayed data).

[0061] In one possible implementation, the terminal device determines a first format of the sensing and interaction data to be transmitted based on the first capability information, including: the terminal device determines a first format of the sensing and interaction data to be transmitted based on the first capability information and target information, wherein the target information is at least one of the computing capabilities and application scenarios of the sensing and interaction data.

[0062] The terminal device 100 can combine at least one of the application scenario (such as the application's rendering requirements) and the terminal device 100's computing power for perceived and interactive data to set the format of the perceived and interactive data to be transmitted within the format range supported by the display peripheral 200. In this way, the display device can cooperate with terminal devices with different computing powers and application scenarios to realize the transmission of perceived and interactive data in different formats.

[0063] In one possible implementation, the terminal device determines a second format of the display data to be transmitted based on the third capability information, including: the terminal device determines a second format of the display data to be transmitted based on the third capability information and target information, wherein the target information is at least one of the computing power and application scenario of the display data.

[0064] The terminal device 100 can combine at least one of the application scenario and the computing power (hereinafter referred to as computing power) of the terminal device 100 to set the second format of the display data to be transmitted within the format range of the display data supported by the display peripheral 200. In this way, the same display device, such as mixed reality (MR) glasses, can use terminal devices with different computing powers (or in different application scenarios) to drive different display effects (full load rendering / center rendering / focal rendering, etc.) of the first image sequence.

[0065] In one possible implementation, the first capability information provides format information of the perception and interaction data that the display device supports collecting, or format information of the perception and interaction data that the display device supports calculating.

[0066] When the display device has strong computing power and supports the calculation of the collected perception and interaction data, the format information of the calculated perception and interaction data (also known as the calculation result of perception and interaction data) can be provided to the terminal device through the first capability information.

[0067] In one possible implementation, the method may further include: a terminal device collecting sensing and interaction data; the terminal device obtaining a first image sequence based on the collected sensing and interaction data and the received sensing and interaction data in a first format.

[0068] As shown in Figure 3d, not only can the XR helmet 201 capture images of the surrounding environment (taking the user as an example) to obtain perception and interaction data and transmit it to the terminal device 100, but also, since the terminal device 100 has N camera modules, it can also capture images of the surrounding environment (taking the user as an example). Thus, the perception and interaction data that the terminal device 100 uses to generate data to be displayed comes not only from the XR helmet 201, but also from the image data captured by the terminal device 100 itself. Finally, the terminal device 100 can send the data to be displayed (generated by the terminal device 100 or obtained from the cloud) to the XR helmet 201 for output.

[0069] In this way, the terminal device 100 of this application can work in conjunction with the XR helmet to collect perception and interaction data, so as to meet the needs of collecting perception and interaction data of the real environment in AR, VR, MR and other scenarios.

[0070] In one possible implementation, this application provides a data processing method. The method includes: a display device sending first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; the display device acquiring perception and interaction data in a first format, the format information including the first format; the display device sending the perception and interaction data in the first format; and the display device receiving multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

[0071] For example, the display device can send the first capability information to the terminal device, or the display device can send the first capability information to the terminal device through the cloud; there are no restrictions here.

[0072] Similarly, the display device can receive the first image sequence from the terminal device or from the cloud; there are no restrictions on this.

[0073] The display device of this application is compatible with different models of terminal devices from different manufacturers, so as to achieve compatibility between the display device and the terminal device.

[0074] In one possible implementation, before the display device acquires the first format of perception and interaction data, the method further includes: the display device receiving the first format; and the display device initializing the sensor used to collect the perception and interaction data according to the first format.

[0075] The display device can receive the first format from the terminal device or from the cloud; there is no limitation on this.

[0076] In this way, the display device can initialize the sensor on its own side according to the format of the sensing and interaction data set by the terminal device, so that the sensor after initialization can collect the sensing and interaction data in the first format.

[0077] In one possible implementation, the method further includes: the display device sending second capability information, wherein the second capability information provides first link information supported by the display device for the first transmission link; the display device receiving second link information, wherein the second link information is determined based on the second capability information and the first format; and the display device configuring the first transmission link based on the second link information.

[0078] The display device can send the second capability information to the terminal device, or it can send the second capability information to the terminal device via the cloud, or it can send the second capability information to the cloud; there are no restrictions here.

[0079] This method enables compatibility of the transmission links for sensing and interactive data between display devices and terminal devices.

[0080] In one possible implementation, the second link information includes information indicating the initiation of compression or decompression, and the method further includes: the display device compressing and encoding the acquired perception and interaction data in the first format based on the second link information to obtain a first bitstream; and the display device sending the first bitstream through the configured first transmission link.

[0081] In this way, the display peripheral 200 can compress and encode the sensing and interaction data to be sent, and then send the compressed and encoded data through the sensing and interaction data transmission link, for example, to the terminal device 100. This allows for the transmission of sensing and interaction data with low latency and low bandwidth even when the amount of data to be sent by the display peripheral 200 is large. By encoding and decoding the sensing and interaction data, higher compression efficiency can be achieved, reducing the bit rate and the bandwidth required for transmission, and reducing latency.

[0082] In one possible implementation, the first format of perception and interaction data includes real-scene images and image data other than the real-scene images. The display device compresses and encodes the acquired first format of perception and interaction data to obtain a first bitstream, including: the display device compresses and encodes the real-scene images based on the image data to obtain the first bitstream.

[0083] For example, the real-world image could be VST data (VST data acquired by a VST sensor). The image data other than the real-world image could include, but is not limited to, image data acquired by other sensing and interactive sensors. These other sensing and interactive sensors are sensors other than those used to acquire the real-world image (e.g., monochrome cameras, depth cameras, etc.).

[0084] The image data other than the real-scene image can be at least one of eye-tracking images and depth images. The display device can calculate eye-tracking information based on the eye-tracking image and depth information based on the depth image, and use at least one of the eye-tracking information and depth information to compress and encode the real-scene image, thereby improving the compression rate of the real-scene image and reducing the bit rate.

[0085] In one possible implementation, this application provides a data processing method in which a terminal device and a display device are communicatively connected. The method includes: the display device sending first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; the terminal device determining a first format of the perception and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; the display device sending the perception and interaction data of the first format to the terminal device; the terminal device obtaining multimedia data based on the perception and interaction data of the first format; the terminal device sending the multimedia data to a display peripheral; and the display device displaying an image based on the multimedia data.

[0086] In one possible implementation, before the display device sends the first format perception and interaction data to the terminal device, the method further includes: the terminal device sending the first format to the display device; the display device initializing the sensor used to collect perception and interaction data according to the first format; and the display device acquiring the first format perception and interaction data based on the sensor after the initialization settings.

[0087] In one possible implementation, the method further includes: the display device sending second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for the first transmission link; the terminal device determining second link information based on the second capability information and the first format; the terminal device sending the second link information to the display device; and the display device configuring the first transmission link based on the second link information.

[0088] In one possible implementation, the second link information includes information indicating the initiation of decompression, and the method further includes: the display device compressing and encoding the acquired perception and interaction data in the first format based on the second link information to obtain a first bitstream; the display device sending the first bitstream to the terminal device through the configured first transmission link; and the terminal device decoding the first bitstream based on the second link information to obtain the perception and interaction data in the first format.

[0089] The effects of the methods in the above-described embodiments of collaboration between the terminal device and the display device are similar to the effects of the data processing methods executed by the terminal device and the display device in the above-described embodiments, and will not be repeated here.

[0090] In one possible implementation, this application provides a data processing system. The system includes a terminal device and a display device connected in communication; the display device is configured to send first capability information to the terminal device, wherein the first capability information provides format information of sensing and interactive data supported by the display device; the terminal device is configured to determine a first format of the sensing and interactive data to be transmitted based on the first capability information, wherein the format information includes the first format; the display device is further configured to send the sensing and interactive data of the first format to the terminal device; the terminal device is further configured to obtain multimedia data based on the sensing and interactive data of the first format; the terminal device is further configured to send the multimedia data to the display device; and the display device is further configured to display images based on the multimedia data.

[0091] In one possible implementation, the terminal device is further configured to send the first format to the display device; the display device is further configured to initialize the sensor used for collecting perception and interaction data according to the first format; the display device is further configured to acquire perception and interaction data in the first format based on the sensor after initialization and to send the perception and interaction data in the first format to the terminal device.

[0092] In one possible implementation, the display device is further configured to send second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for the first transmission link; the terminal device is further configured to determine second link information based on the second capability information and the first format; the terminal device is further configured to configure the first transmission link based on the second link information; the terminal device is further configured to send the second link information to the display device; and the display device is further configured to configure the first transmission link based on the second link information.

[0093] In one possible implementation, the second link information includes information indicating the initiation of decompression; the display device is specifically configured to compress and encode the acquired perception and interaction data in the first format based on the second link information to obtain a first bitstream; the display device is specifically configured to send the first bitstream to the terminal device through the configured first transmission link; the terminal device is specifically configured to decode the first bitstream based on the second link information to obtain the perception and interaction data in the first format.

[0094] In one possible implementation, the display device is further configured to send third capability information to the terminal device, wherein the third capability information provides format information of display data supported by the display device; the terminal device is further configured to determine a second format of the display data to be transmitted based on the third capability information, wherein the second format includes an image format, the image format being an image format synthesized from virtual images and real-world images, or an image format of a virtual image; specifically, the terminal device is configured to obtain a first image sequence of the second format based on the perception and interaction data.

[0095] In one possible implementation, the terminal device is further configured to send the second format to the display device; the display device is further configured to initialize display settings according to the second format; specifically, the display device is configured to display an image based on the initialized display settings and the first image sequence.

[0096] In one possible implementation, the display device is further configured to send fourth capability information to the terminal device, wherein the fourth capability information provides third link information supported by the display device for the second transmission link; the terminal device is further configured to determine fourth link information based on the fourth capability information and the second format; the terminal device is further configured to configure the second transmission link based on the fourth link information; the terminal device is further configured to send the fourth link information to the display device; and the display device is further configured to configure the second transmission link based on the fourth link information.

[0097] In one possible implementation, the fourth link information includes information indicating the initiation of compression; the terminal device is specifically configured to compress and encode the first image sequence based on the fourth link information to obtain a second bitstream of the first image sequence; the terminal device is specifically configured to send the second bitstream to the display device through the configured second transmission link; the display device is specifically configured to decode the second bitstream based on the fourth link information to obtain a first image sequence in a second format.

[0098] The effects of the systems in the above embodiments are similar to those of the data processing methods executed collaboratively by the terminal device and the display device in the above embodiments, and will not be repeated here.

[0099] In one possible implementation, this application provides a data processing apparatus. The data processing apparatus is communicatively connected to a display device, and includes: a first receiving module for receiving first capability information, wherein the first capability information provides format information of sensing and interactive data supported by the display device; a first determining module for determining a first format of the sensing and interactive data to be transmitted based on the first capability information, wherein the format information includes the first format; a second receiving module for receiving the sensing and interactive data of the first format from the display device; and an acquiring module for obtaining a first image sequence based on the sensing and interactive data.

[0100] The data processing device can be a standalone device (e.g., a terminal device) or part of a larger device. For example, the data processing device can be implemented as follows:

[0101] (1) A standalone integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally including storage components for storing data or instructions; (3) A module that can be embedded in other devices; (4) Vehicle equipment, etc.; (5) Others, etc.

[0102] In one possible implementation, the apparatus further includes: a first receiving module, further configured to receive second capability information, wherein the second capability information provides first link information supported by the display device for the first transmission link; a second determining module, configured to determine the second link information based on the second capability information and the first format; and a first configuration module, configured to configure the first transmission link based on the second link information.

[0103] In one possible implementation, the second link information includes information indicating the initiation of decompression; the second receiving module is configured to receive a first bitstream of compressed and encoded perception and interaction data in the first format from the display device via the configured first transmission link; the device further includes a decoding module configured to decode the first bitstream based on the second link information to obtain the perception and interaction data in the first format.

[0104] In one possible implementation, the device further includes: a first receiving module, further configured to receive third capability information, wherein the third capability information provides format information of display data supported by the display device; a third determining module, configured to determine a second format of the display data to be transmitted based on the third capability information, wherein the second format includes an image format, the image format being an image format synthesized from virtual images and real-world images, or an image format of a virtual image; and an acquiring module, specifically configured to obtain a first image sequence of the second format based on the perception and interaction data.

[0105] In one possible implementation, the apparatus further includes: a first receiving module, further configured to receive fourth capability information, wherein the fourth capability information provides third link information supported by the display device for the second transmission link; a fourth determining module, configured to determine fourth link information based on the fourth capability information and the second format; and a second configuration module, configured to configure the second transmission link based on the fourth link information.

[0106] In one possible implementation, the fourth link information includes information indicating the initiation of compression, and the device further includes: an encoding module for compressing and encoding the first image sequence based on the fourth link information to obtain a second bitstream of the first image sequence; and a sending module for sending the second bitstream to the display device through the configured second transmission link.

[0107] In one possible implementation, the encoding module is further configured to compress and encode the first image sequence based on the perception and interaction data to obtain the bitstream of the first image sequence.

[0108] In one possible implementation, the first image sequence includes at least two visual images, the at least two visual images including a first visual image and a second visual image, and the perception and interaction data including first visual data.

[0109] In one possible implementation, the perception and interaction data further includes second eye data. The encoding module is specifically used to compress and encode the first eye image based on the first eye data, and to compress and encode the second eye image based on the second eye data to obtain a second bitstream of the first image sequence.

[0110] In one possible implementation, the encoding module is specifically configured to: compress and encode the first eye image to obtain a bitstream of the first eye image and a reconstructed image of the first eye image; preprocess the reconstructed image based on the first eye data to obtain a reference image that matches the second eye image; and compress and encode the second eye image based on the reference image to obtain a bitstream of the second eye image.

[0111] In one possible implementation, the first format includes at least one of the following: data format, resolution, frame rate, and number of pixels.

[0112] In one possible implementation, the second format includes at least one of the following: data format, resolution, frame rate, and number of pixels.

[0113] In one possible implementation, the first determining module is specifically used to determine a first format of the sensing and interaction data to be transmitted based on the first capability information and target information, wherein the target information is at least one of the computing capabilities and application scenarios of the sensing and interaction data.

[0114] In one possible implementation, the third determining module is specifically used to determine the second format of the display data to be transmitted based on the third capability information and the target information, wherein the target information is at least one of the computing power and application scenario of the display data.

[0115] In one possible implementation, the first capability information provides format information of the perception and interaction data that the display device supports collecting, or format information of the perception and interaction data that the display device supports calculating.

[0116] The effects of the data processing devices in the above embodiments are similar to the effects of the data processing methods executed by the terminal devices in the above embodiments, and will not be described again here.

[0117] In one possible implementation, this application provides a data processing apparatus. The apparatus includes: a first transmitting module for transmitting first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; an acquiring module for acquiring perception and interaction data in a first format, the format information including the first format; a second transmitting module for transmitting the perception and interaction data in the first format; and a first receiving module for receiving multimedia data, wherein the multimedia data is obtained based on the perception and interaction data in the first format.

[0118] The data processing device can be a standalone device (e.g., a display device) or part of a larger device. For example, the data processing device can be implemented as follows:

[0119] (1) A standalone integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally including storage components for storing data or instructions; (3) A module that can be embedded in other devices; (4) Vehicle equipment, etc.; (5) Others, etc.

[0120] In one possible implementation, the device includes: a second receiving module for receiving the first format; and a first configuration module for initializing the sensor used to collect sensing and interactive data according to the first format.

[0121] In one possible implementation, the apparatus further includes: a first transmitting module, further configured to transmit second capability information, wherein the second capability information provides first link information supported by the display device for the first transmission link; a third receiving module, configured to receive the second link information, wherein the second link information is determined based on the second capability information and the first format; and a second configuration module, configured to configure the first transmission link based on the second link information.

[0122] In one possible implementation, the second link information includes information indicating the initiation of compression or decompression, and the device further includes: an encoding module, configured to compress and encode the acquired first-format perception and interaction data based on the second link information to obtain a first bitstream; and a third sending module, configured to send the first bitstream through the configured first transmission link.

[0123] In one possible implementation, the first format of perception and interaction data includes real-scene images and image data other than the real-scene images. The encoding module is specifically used to compress and encode the real-scene images based on the image data to obtain a first bitstream.

[0124] The effects of the data processing apparatus in the above embodiments are similar to the effects of the data processing methods executed by the display devices in the above embodiments, and will not be described again here.

[0125] In one possible implementation, this application provides a data processing apparatus. The data processing apparatus includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the processor can implement the method executed by a terminal device or a display device in any of the above embodiments.

[0126] The effect of the data processing device in this embodiment is similar to that of the data processing methods in the above embodiments, and will not be described again here.

[0127] In one possible implementation, this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method executed by the terminal device or the method executed by the display device in any of the above embodiments.

[0128] The effect of the computer-readable storage medium in this embodiment is similar to that of the data processing methods in the embodiments described above, and will not be repeated here.

[0129] In one possible implementation, this application provides a computer program product. The computer program product includes a software program that, when executed by a computer or processor, causes the method executed by the terminal device in any of the above embodiments, or the method executed by the display device, to be performed.

[0130] The effect of the computer program product in this embodiment is similar to that of the data processing methods in the above embodiments, and will not be repeated here. Attached Figure Description

[0131] Figure 1 is a schematic diagram of the structure of an exemplary VR split-type machine;

[0132] Figure 2a is an exemplary schematic diagram of device interaction;

[0133] Figure 2b is a schematic diagram of an exemplary internal structure of a device;

[0134] Figure 2c is a schematic diagram of an exemplary internal structure of a device;

[0135] Figure 3a is a schematic diagram illustrating an application scenario;

[0136] Figure 3b is a schematic diagram illustrating an exemplary application scenario;

[0137] Figure 3c is a schematic diagram illustrating an application scenario;

[0138] Figure 3d is a schematic diagram illustrating an example application scenario;

[0139] Figure 3e is a schematic diagram illustrating an exemplary application scenario;

[0140] Figure 4 is a schematic diagram illustrating an exemplary data processing procedure;

[0141] Figure 5a is a schematic diagram illustrating an exemplary encoding process;

[0142] Figure 5b is a schematic diagram illustrating an exemplary encoding process;

[0143] Figure 6 is a schematic diagram illustrating an exemplary data processing procedure.

[0144] Figure 7 is a schematic diagram illustrating an exemplary data processing procedure.

[0145] Figure 8 is a schematic diagram illustrating an exemplary data processing procedure.

[0146] Figure 9 is a schematic diagram illustrating an exemplary data processing procedure.

[0147] Figure 10 is a schematic diagram illustrating an exemplary data processing procedure.

[0148] Figure 11 is a schematic diagram of the structure of a device provided in an embodiment of this application;

[0149] Figure 12 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0150] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0151] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0152] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0153] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0154] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0155] Before introducing the technical solution of this application, the technical terms involved in this application will be explained first:

[0156] XR stands for any of the following: Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MR).

[0157] An XR split-type device refers to a device where the display unit, sensing unit, interaction unit, and computing unit are integrated on separate terminal devices. The two terminal devices interact with each other via wired or wireless connection to achieve XR functionality.

[0158] Simultaneous Localization and Mapping (SLAM) cameras use cameras to perceive the environment and generate images that can be called pose maps.

[0159] Video See-Through (VST) is an AR technology that overlays virtual content onto the real world through a real-time video stream, allowing users to see augmented reality scenes through their cameras.

[0160] A mono camera, a type of black-and-white camera, is a camera with only one image sensor, unlike traditional RGB color cameras. Its principle is to use the image sensor to perceive ambient light, convert the light into electrical signals, and then process them to form monochrome images or videos.

[0161] An eye-tracking camera, which can be a mono camera, refers to a sensor or camera system specifically designed to track human eye movements. This technology provides information such as the user's gaze point, gaze duration, and blink frequency by monitoring and recording eye movements.

[0162] A downward camera is a camera or lens mounted on a device or vehicle that faces downwards. These cameras are typically used to capture images or information from the bottom or beneath the device.

[0163] A depth camera is a camera device that can capture depth information of a scene. Traditional cameras mainly capture surface information of an image, while depth cameras can simultaneously capture the distance of each pixel in the image from the camera, thereby obtaining the depth value or distance value of each pixel.

[0164] An inertial measurement unit (IMU) is used to acquire motion information of a device, such as linear acceleration and angular velocity.

[0165] Figure 1 is a schematic diagram of the structure of a VR split-type machine in a related technology, which is an example shown.

[0166] As shown in Figure 1, the VR split device can be composed of a physically separated computing unit 100 and an integrated sensing, interaction and display unit 2000, wherein the computing unit 1000 and the unit 2000 are connected by a cable.

[0167] As shown in Figure 1, the computing unit 1000 may include a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU). The computing unit 1000 also includes a universal serial bus (USB) interface and a display port (DP).

[0168] As shown in Figure 1, unit 2000 may also include a USB interface and a DP interface. Furthermore, it may include a sensor unit and a camera unit. The sensor unit may include an inertial measurement unit (IMU), a handheld device, a keyboard, a microphone, etc., for acquiring interactive data. The camera unit can be used to acquire sensory data and may include a VST camera, a monochrome camera, etc.

[0169] As shown in Figure 1, unit 2000 may further include a display module, which may include a left-eye display screen and a right-eye display screen for displaying VR images.

[0170] As shown in Figure 1, both the sensor unit and the camera unit are connected to a USB interface, and the display module is connected to a DP interface. Thus, unit 2000 can send the interactive data and the perceived data collected by the sensor unit and camera unit, respectively, to computing unit 1000 via the USB interface. Furthermore, computing unit 1000 can send data to be displayed (e.g., images) to the display module of unit 2000 via the DP interface for display, thereby realizing the display of VR video.

[0171] The aforementioned unit 2000 can be deployed on a wearable display device (such as VR glasses or VR helmet), while the aforementioned computing unit 1000 is deployed on another terminal device (such as a computer).

[0172] The transmission protocol of the USB interface of unit 2000 is customized by the manufacturer of the wearable display device to transmit sensing data and interactive data.

[0173] Therefore, the terminal device containing the same computing unit 1000 in Figure 1 is not compatible with wearable display devices from various manufacturers and models.

[0174] Furthermore, in the XR split-type machine scenario shown in Figure 1, the computing unit 1000 is separated from other sensing, interaction, and display units 2000. In the XR scenario, the amount of data that needs to be transmitted between the two sides is large, which will put great pressure on bandwidth and transmission latency, making it difficult to achieve XR data transmission while meeting the requirements of low latency and low bandwidth.

[0175] Therefore, as shown in FIG2a, this application provides a terminal device 100, a display device (e.g., display peripheral 200), and a system. The system may include the terminal device 100 and the display peripheral 200 that communicates with the terminal device 100 (e.g., wired or wireless connection). The wireless connection method may be Wi-Fi, Bluetooth, etc., which are not limited here.

[0176] The terminal device 100 can acquire capability information for sensing and interactive data from the display peripheral 200, and optionally acquire capability information for display data. The capability information for display data provides information on the display data formats supported by the display peripheral 200 (e.g., display data format, resolution, frame rate, etc.). The capability information for sensing and interactive data provides information on the format of sensing and interactive data supported by the display peripheral 200 (e.g., sensor type, format of data generated by the sensor (e.g., RGB), resolution, frame rate, etc.). Then, the terminal device 100 can determine the format of the sensing and interactive data to be transmitted (also referred to as a first format) based on the capability information for sensing and interactive data; optionally, the terminal device 100 can determine the format of the display data to be transmitted (also referred to as a second format) based on the capability information for display data. The format determined by the terminal device 100 is within the format information provided by the corresponding capability information. The terminal device 100 can receive sensing and interactive data of the first format from the display peripheral 200, and the terminal device 100 can obtain data to be displayed (also referred to as a first image sequence) based on the sensing and interactive data of the first format.

[0177] In some embodiments, the terminal device 100 may generate the data to be displayed based on the perception and interaction data in the first format.

[0178] In other embodiments, the terminal device 100 may send the first format perception and interaction data to a server (e.g., the cloud), and the server may generate the data to be displayed based on the first format perception and interaction data. Then, the terminal device 100 may receive the data to be displayed from the server to obtain the data to be displayed.

[0179] In other words, this application allows the terminal device 100 to generate the data to be displayed using the sensing and interaction data itself, or it can utilize the cloud to generate the data to be displayed and obtain it from the cloud. The specific method used by the terminal device 100 to obtain the data to be displayed can be flexibly selected according to the application scenario, and no restrictions are imposed here.

[0180] Optionally, the terminal device 100 may send the obtained data to be displayed to the display peripheral 200. In this way, the format of the data to be displayed sent by the terminal device 100 to the display peripheral 200 is a display data format supported by the display peripheral 200.

[0181] In this way, the terminal device 100 can receive sensing and interactive data according to the format of sensing and interactive data supported by the display peripheral 200 (e.g., the first format), thereby enabling the terminal device 100 to achieve connection compatibility with display peripherals of any transmission protocol.

[0182] Furthermore, the data to be displayed, i.e. the first image sequence, can be a single image frame or multiple image frames.

[0183] At a transmission moment, the first image sequence can be a single frame or multiple frames corresponding to multi-view (binocular or more).

[0184] For example, when the terminal device 100 is connected to a common display device (such as a vehicle display screen), at one transmission moment, the first image sequence can be a single frame of image, so that the vehicle display screen can receive a single frame of image at the same transmission moment and display it on the screen, so that the user can browse the single frame of image through the vehicle display screen.

[0185] When the terminal device 100 is connected to an XR device (a device involving binocular or multi-lens image display, such as VR glasses), the first image sequence can consist of two or more frames at a given transmission time. For example, if the XR device is VR glasses, the VR glasses can receive the two frames at the same transmission time, one corresponding to the left eye display and the other to the right eye display. The VR glasses can then display the two frames corresponding to both eyes simultaneously on the left and right eye displays. The same principle applies to multi-lens scenarios, and will not be elaborated further here.

[0186] The terminal device 100 can be a desktop device or a portable device. The portable form of the terminal device 100 can be a box-type (such as the terminal device 100 shown in Figures 3a to 3d below), a neck-hanging type (such as the terminal device 100 shown in Figure 3e), a waist-hanging type, etc., and there are no restrictions here.

[0187] The display peripheral 200 can be any device with a display screen, and this application does not impose any restrictions.

[0188] For example, the display peripheral 200 may include, but is not limited to, any of the following: monitor, personal computer (PC), mobile phone, television, projector, in-vehicle display, wearable display device. The wearable display device may be XR glasses or XR helmet, etc.

[0189] Thus, the terminal device 100 of this application is compatible with wearable display devices (such as AR glasses, VR glasses, AR helmets, VR helmets, etc.) with various data transmission protocols to achieve immersive virtual-real fusion and spatial interaction to meet VR, AR and other experiences; in addition, the terminal device 100 is also compatible with ordinary display devices (such as monitors, televisions, vehicle displays and projectors, etc.) with various data transmission protocols, so that the terminal device can be used as a portable micro host.

[0190] Referring to Figure 2a, Figure 2b shows the internal structure and interaction diagram of the display peripheral 200 and the terminal device 100 of this application.

[0191] As shown in Figure 2b, the display peripheral 200 may include, but is not limited to, a display module and an audio module.

[0192] The display module may be, for example, a screen, and the audio module may be, for example, a speaker. The display module and audio module can be used to display or play multimedia data. In this embodiment, the data output by the display module of the display peripheral 200 can be defined as display data. Optionally, the display data may also include data output by the audio module.

[0193] As shown in Figure 2b, the display peripheral 200 may also include: a sensing module, an interaction module, and a microphone module.

[0194] As shown in Figure 2b, the microphone module, perception module (e.g., a depth camera for acquiring depth information), and interaction module (e.g., an eye-tracking camera for acquiring eye-movement images) in the display peripheral 200 can be sensors for acquiring input information. The microphone module can be used to acquire audio (e.g., ambient sound, speech); the perception module can be used for spatial positioning and visual information perception of the surrounding environment; the interaction module can be used to acquire visual information for interaction (e.g., images). Interaction modules are commonly found in eyeglass-type display peripherals. In this embodiment, the data received by the perception module, interaction module, and optionally microphone module in the display peripheral 200 can be defined as perception and interaction data.

[0195] As shown in Figure 2b, the display peripheral 200 may also include a VST module.

[0196] The VST module is a visual sensor, also referred to as a real-world camera, used to capture real-world images. It is commonly found in MR glasses and some VR glasses and can be used to collect visual information in real time to visually reproduce the real world. In this embodiment, the VST data collected by the VST module is also defined as perception and interaction data.

[0197] As shown in Figure 2b, the display peripheral 200 and the terminal device 100 also include their respective sensing and interaction interfaces. The display peripheral 200 and the terminal device 100 can send the sensing and interaction data collected by the display peripheral 200 to the terminal device 100 through their respective sensing and interaction interfaces.

[0198] As shown in Figure 2b, the display peripheral 200 and the terminal device 100 also include their respective display interfaces. The terminal device 100 can generate data to be displayed through its processing module and can send the data to be displayed to the display interface of the display peripheral 200 through the display interface.

[0199] The display interface of the display peripheral 200 can be used to receive data to be displayed.

[0200] Optionally, the display peripheral 200 may also include a processing module and a corresponding storage module.

[0201] In some embodiments, the processing module can be used to calculate the sensing and interaction data to obtain the sensing and interaction data to be transmitted to the terminal device 100.

[0202] In some embodiments, the processing module can be used to perform calculations on the data to be displayed to obtain the display data.

[0203] The processing module of the display peripheral 200 may include, but is not limited to, CPU, GPU, NPU, etc.

[0204] The storage module may include memory and external storage, and can be used to store data, such as the device type of the display peripheral 200, information on the ability to sense and interact with data, information on the ability to display data, algorithm data, user local and temporary data, etc.

[0205] The device types, sensing and interaction data capabilities, and display data capabilities described above will be detailed and explained in the embodiment shown in Figure 3.

[0206] Depending on the specifications of the display peripheral 200, the computing power of its processing module may vary. The processing module of the display peripheral 200 can generally perform basic functions such as driving its own display module and audio module.

[0207] Some display peripherals, mainly wearable devices (such as XR glasses and XR helmets), may also have a microphone module, sensing module, interaction module, and VST module as shown in Figure 2b.

[0208] In some embodiments, the processing module of the display peripheral 200 can be used to manage sensors that acquire input data, as well as perform image signal processing (ISP), etc.

[0209] In some embodiments, the processing module of the display peripheral 200 has strong computing power. This processing module can also perform calculations on the VST data collected by the VST module to obtain displayable real-scene image data, thus realizing a closed loop between the VST module and the display module. In this way, the display peripheral 200 does not need to transmit the VST data collected by the VST module to the terminal device 100.

[0210] In some embodiments, the processing module of the display peripheral 200 has strong computing power. This processing module can also perform calculations on the perception and interaction data collected by the perception module and the interaction module, so as to output the calculated perception and interaction data (also described as the calculation result of the perception and interaction data) to the terminal device 100 through the perception and interaction module. In this way, the perception and interaction data received by the terminal device 100 can be the result of spatial positioning (e.g., pose), local positioning map, spatial geometric grid, lighting information, gesture points, gaze points, limb key points, facial expression base, etc.

[0211] The structure of the terminal device 100 shown in Figure 2b will be described in detail below.

[0212] The terminal device 100 may include a processing module, a parsing and decoding module for interacting with the sensing and interaction interface, a parsing and encoding module for interacting with the display interface, an AI module, a storage module, a battery, and a power interface.

[0213] The parsing and decoding module can be used to decompress and decode the perception and interaction data received by the perception and interaction interface, and can also be used to parse the information from the display peripheral 200 to obtain the capability information of the perception and interaction data.

[0214] This parsing and encoding module can be used to compress and encode the data to be sent to the display interface, and can also be used to parse information from the display peripheral 200 to obtain the capability information of the display data.

[0215] The processing module (e.g., CPU, GPU, NPU, etc.) of the terminal device 100 can be used to implement the data processing method of the terminal device 100, and to render the perceived and interactive data (or the calculation results of the perceived and interactive data) to generate data to be displayed.

[0216] The Artificial Intelligence (AI) module of the terminal device 100 can be used to infer and generate data to be displayed from the received perception and interaction data (or the calculation results of perception and interaction data) through AI models, language models and other models.

[0217] In some embodiments, the processing module and AI module of the terminal device 100 may each generate partial elements of the data to be displayed, so that all elements of the complete data to be displayed can be generated through the collaboration of the processing module and AI module.

[0218] The elements of the data to be displayed can include, but are not limited to: text, audio (e.g., voice), images, videos, models, etc. There are no restrictions here.

[0219] In some embodiments, both the processing module and the AI ​​module of the terminal device 100 can generate all elements of the data to be displayed. The terminal device 100 can select one of the processing module and the AI ​​module to independently generate the data to be displayed, depending on the application scenario and computing power requirements.

[0220] For example, the AI ​​module can generate elements such as text, audio, images, videos, and models based on the received perception and interaction data (or the calculation results of the perception and interaction data) to serve as partial elements in the data to be displayed.

[0221] In some embodiments, the AI ​​module can also communicate with a server (e.g., the cloud) to enable the AI ​​module and the cloud to work together to generate the data to be displayed.

[0222] For example, in scenarios where the process of generating data to be displayed requires high computing power, the AI ​​module can send a request to the cloud for AI processing based on the aforementioned perception and interaction data (or the computational structure of perception and interaction data) to obtain the data to be displayed generated by the cloud.

[0223] The storage module of the terminal device 100 can be used to store data information when implementing the data processing method. The storage module can be memory or external storage.

[0224] Optionally, the terminal device 100 may also include a communication module that supports various wireless communication protocols for communicating with the display peripheral 200.

[0225] The communication module of the terminal device 100 can also be used to communicate with a server (e.g., the cloud). The AI ​​module can then interact with the cloud through the communication module, enabling the terminal device 100 to utilize the cloud's processing capabilities to perform AI model inference and large-scale language model calculations.

[0226] For example, the communication module can be a Wi-Fi module, a Bluetooth module, etc., and there are no restrictions here.

[0227] The battery of the terminal device 100 can be interconnected with the power interface of the display peripheral 200 through the power interface to power the display peripheral 200.

[0228] Referring to Figure 2b, Figures 3a to 3d respectively show schematic diagrams of various application scenarios of the terminal device 100 of this application.

[0229] 1. Entertainment scenarios:

[0230] As shown in Figure 3a, the terminal device 100 can be connected to the XR headset 201 via wired or wireless means, wherein the VR headset is an example of the display peripheral 200 of this application.

[0231] The XR helmet 201 can collect perception and interaction data through the VST module, perception module, and interaction module shown in Figure 2b. This perception and interaction data may include, but is not limited to: IMU data, VST data, black and white images (e.g., eye-tracking images, lip images, limb images, gesture images), depth images, and other perception and interaction data. The aforementioned perception and interaction data can be obtained through real-time images (e.g., including real-world images of the surrounding environment of the user wearing the XR helmet 201 (example of VST data)) captured by the XR helmet 201's sensors and cameras (also described as cameras).

[0232] The XR helmet 201 can transmit perception and interaction data to the terminal device 100. Based on the perception and interaction data, the terminal device 100 can overlay the virtual image generated by the terminal device 100 onto the real-time image to generate virtual-real fusion data to be displayed, and send the virtual-real fusion data to the XR helmet 201. The virtual-real fusion data may include virtual-real fusion image data and audio data. The XR helmet 201 can output the audio data through a speaker and output the virtual-real fusion image data through the two displays on the eyes of the XR helmet 201.

[0233] Thus, the terminal device 100 of this application is compatible with XR helmets to realize the transmission of perception and interaction data, as well as data to be displayed, so as to achieve a spatial interactive experience that blends the virtual and the real.

[0234] 2. Office setting:

[0235] As shown in Figure 3b, the terminal device 100 can be connected to the display 202 and the keyboard 203 via wired (or wireless) connection, wherein the display 202 and the keyboard 203 together serve as an example of the display peripheral 200 of this application.

[0236] The display screen 202 can be a television, projector, vehicle display screen, desktop display screen, etc., and there are no restrictions here.

[0237] Terminal device 100 can receive keyboard input data (a kind of interactive data) from keyboard 203. Based on the keyboard input data, terminal device 100 can generate data to be displayed (e.g., an image) and send the data to be displayed to display 202 for on-screen display.

[0238] Thus, the terminal device 100 of this application is compatible with a display screen and a keyboard, thereby achieving the effect of extending the display screen of the terminal device 100.

[0239] 3. Independent shooting scenes:

[0240] Returning to Figure 2b, the terminal device 100 may optionally include N camera modules, microphone modules, audio modules, and display modules.

[0241] The camera module can be used to capture images, the microphone module can be used to capture audio, the audio module (e.g., a speaker) can be used to output audio, and the display module (e.g., a display screen) can be used to display multimedia data (e.g., images or videos).

[0242] Referring to Figure 2b and as shown in Figure 3c, the terminal device 100 uses a camera module to perform multi-objective 3D photography of the user, and a microphone module to record audio in 3D space on-site. Finally, it outputs the 3D captured images or videos through a display module and an audio module. This gives the audio in the output video a sense of being recorded on-site.

[0243] Furthermore, in conjunction with Figure 2b, as shown in Figure 3c, the N camera modules may include camera modules with micro-single-level shooting capabilities. Then, the terminal device 100 can use the camera modules with micro-single-level shooting capabilities to capture images or videos, so that the terminal device 100 of this application can be equipped with micro-single-level shooting capabilities and the image shooting quality is high.

[0244] 4. Shooting scenarios using XR glasses:

[0245] Unlike the first entertainment scenario mentioned above (e.g., Figure 3a), in this scenario (e.g., Figure 3d), as shown in Figure 3d, not only can the XR helmet 201 capture images of the surrounding environment (taking the user as an example) to obtain perception and interaction data and transmit it to the terminal device 100, but also, since the terminal device 100 has N camera modules, it can also capture images of the surrounding environment (taking the user as an example). Thus, the perception and interaction data that the terminal device 100 uses to generate the data to be displayed comes not only from the XR helmet 201, but also from the image data captured by the terminal device 100 itself. Finally, the terminal device 100 can send the data to be displayed (generated by the terminal device 100 or obtained from the cloud) to the XR helmet 201 for output.

[0246] In this way, the terminal device 100 of this application can work in conjunction with the XR helmet to shoot together, so as to meet the shooting needs of the real environment in the XR scene.

[0247] 5. Application scenarios for neckband-style terminal devices:

[0248] The terminal devices in Figures 3a to 3d are box-type terminal devices, while the terminal device 100 shown in Figure 3e is a neckband-type terminal device 100.

[0249] As shown in Figure 3e, the terminal device 100 can interact with the XR helmet 201. The specific interaction process is the same as that described in Figure 3a, and will not be repeated here.

[0250] As shown in Figure 3e, the neckband-style terminal device 100 can be worn around the neck of user 10, allowing user 10 to immerse themselves in XR games, videos, and more using the terminal device 100 and XR helmet 201.

[0251] It should be understood that the neckband-style terminal device 100 can also be applied to the scenarios shown in Figures 3b to 3d above. The specific implementation principle can be found in the description of Figures 3b to 3d, and will not be repeated here.

[0252] The terminal device of this application provides more space for camera and audio modules compared to mobile phones, allowing for the deployment of mirrorless camera-level shooting capabilities, multi-purpose 3D shooting, spatial audio recording, and positioning and sensing module capabilities. It can also enable collaborative shooting between the terminal device and glasses, addressing core scenario needs in daily work, entertainment, and life.

[0253] In order for the terminal device 100 shown in Figures 2a, 2b and 3a to 3d to implement the functions mentioned in this application, this application also provides a data processing method to ensure the implementation of the corresponding functions of the terminal device 100.

[0254] Figure 4 is a schematic diagram illustrating a data processing method of this application.

[0255] Based on Figure 2b, Figure 2c further describes the internal structure and data interaction of the display peripheral 200 and the terminal device 100.

[0256] The functions of each module and structure shown in Figure 2c will be described below with reference to the process shown in Figure 4.

[0257] In Figure 4, the steps indicated by the dashed boxes and dashed arrows are all optional steps in the implementation process of this method.

[0258] As shown in Figure 4, the process may include the following steps:

[0259] S100, the terminal device 100 establishes a communication connection with the display peripheral 200.

[0260] The connection can be wired or wireless; there are no restrictions here.

[0261] Optionally, in step S101, the terminal device 100 determines the device type of the connected display peripheral 200.

[0262] As mentioned above, the device type of the display peripheral 200 can include, but is not limited to, any of the following: monitor, personal computer (PC), mobile phone, television, projector, in-vehicle display, wearable display device. The wearable display device can be XR glasses or XR helmet, etc.

[0263] The terminal device 100 can read information indicating the device type from the display peripheral 200.

[0264] S102, Terminal device 100 obtains capability information based on device type.

[0265] As shown in Figure 2c, the storage module of the display peripheral 200 can store capability information of sensing and interactive data, and capability information of display data.

[0266] As shown in Figure 4, the capability information of the sensing and interactive data may include the data capability information of the sensing and interactive data (also referred to as the first capability information), and the capability information of the transmission link of the sensing and interactive data, as indicated by the dashed arrow (also referred to as the second capability information). The transmission link is the link used by the display peripheral 200 to transmit the sensing and interactive data.

[0267] The capability information of the display data may include the data capability information of the display data (also referred to as the third capability information), and the capability information of the transmission link of the display data, as indicated by the dashed arrow (also referred to as the fourth capability information). The transmission link is the link used by the display peripheral 200 to transmit the display data.

[0268] The data capability information of the sensing and interaction data can provide the format information of the sensing and interaction data supported by the display device 200. The transmitted sensing and interaction data can be collected by the display device 200, or it can be the result of calculations on the collected sensing and interaction data.

[0269] The data capability information for displaying data can provide information on the format of display data supported by the display device 200.

[0270] In some embodiments, when the device type of the display peripheral 200 is determined to be a general-purpose display, the general-purpose display may include, but is not limited to, a monitor, a personal computer (PC), a mobile phone, a television, a projector, and an in-vehicle display. The perception and interaction data of the display peripheral 200 generally includes microphone data for voice input, remote control type, keyboard and mouse input data, etc. The display peripheral 200 can process this perception and interaction data; therefore, the general-purpose display does not need to transmit the perception and interaction data to the terminal device 100 for processing. Thus, as shown in FIG4, when the terminal device 100 determines that the display peripheral 200 is a general-purpose display, it does not need to obtain the capability information of the perception and interaction data, but only needs to obtain the capability information of the display data from the display peripheral 200. As shown in FIG4, the capability information of the display data may include the data capability information of the display data, and optionally include the capability information of the transmission link of the display data as indicated by the dashed arrow.

[0271] In some embodiments, when the device type of the display peripheral 200 is determined to be a wearable display device (e.g., XR glasses, XR helmet, etc.), the terminal device 100 can obtain capability information for sensing and interaction data from the display peripheral 200, and optionally, it can also obtain capability information for display data. As shown in FIG4, the terminal device 100 can obtain data capability information for sensing and interaction data from the display peripheral 200, and optionally, it can obtain capability information for the transmission link of the sensing and interaction data indicated by the dashed arrow. Optionally, as shown in FIG4, the terminal device 100 can obtain data capability information for display data from the display peripheral 200, and optionally, it can obtain capability information for the transmission link of the display data.

[0272] The data capability information for display data can be used to provide the format information of the display data that the display device 200 supports for transmission.

[0273] The format information of the displayed data may include, but is not limited to: the data format of the displayed data, the resolution of the displayed data, the frame rate of the displayed data, and the number of displays (e.g., binoculars, or more than binoculars).

[0274] For example, the data format supported by general-purpose monitors is typically Red Green Blue (RGB) format, and the supported resolution and frame rate (in Hertz (Hz) are generally: 1920*1080*60Hz, 3840*2160*120Hz, 1280*720*60Hz, etc., without further restrictions. This article uses resolution * frame rate as an example for illustration.

[0275] For example, the display data format supported by display peripherals such as XR glasses and XR helmets 200 may include image formats that can be synthesized from virtual images and real-world images (also described as data formats that can be directly displayed), such as RGB format; the display data format may also include image formats of virtual images (also described as data formats to be displayed after virtual-real synthesis), such as RGBAZ format or RGBA format.

[0276] The data format to be displayed after compositing between real and virtual elements is either RGBA or RGBAZ. Compared to RGB format, RGBAZ format has an additional alpha channel (A) and a depth channel (Z). In compositing, the alpha channel A is used as a mask, and the depth channel Z provides the correct occlusion relationship to achieve real-virtual occlusion. Similarly, the RGBA format, with its additional alpha channel A, is used as a mask in compositing between real and virtual elements.

[0277] AR glasses support a display data display of 2 pixels, with resolutions and frame rates including 2*1920*1080*60Hz and 2*1536*1760*120Hz. VR glasses support a display data display of 2 pixels, with data formats including 2*2000*2000*90Hz and 2*3000*3000*90Hz. MR glasses support a display data display of 2 pixels, with a data format of 2*3840*3600*90Hz. Since XR glasses have two displays, the above resolutions are multiplied by 2. For example, with 2*1920*1080*60Hz, the resolution and frame rate of each pixel's display data are 1920*1080*60Hz.

[0278] The aforementioned AR and VR glasses can support two resolutions for displaying data per target. For example, the VR glasses support a full HD resolution of 3000*3000*90hz for displaying data per target, and the VR glasses also support a foveated rendering resolution of 2000*2000*90hz for displaying data per target.

[0279] The resolution and frame rate of the display data supported by AR glasses, VR glasses, and XR glasses mentioned above are only examples. This application does not limit the resolution and frame rate of the display data supported by XR glasses, which depends on the specific XR glasses.

[0280] The data capability information for sensing and interactive data provides the format information of the sensing and interactive data that the display device 200 supports transmitting. This format information may include, but is not limited to: the sensor types supported by the display peripheral 200, the data format of the sensing and interactive data generated by the sensors, the resolution of the sensing and interactive data, and the frame rate (also described as refresh rate) of the sensing and interactive data.

[0281] For example, XR glasses are near-eye displays that support three features during use: spatial localization, perception, and interaction. When implementing localization, XR glasses can support Y-image streams from IMU, binocular, or quad-lens monochrome cameras. When implementing perception, XR glasses can support depth streams from depth cameras. When implementing interaction, XR glasses can reuse the cameras used for localization and add two additional monochrome cameras. This interaction feature can include eye-tracking detection. When XR glasses support eye-tracking detection, they can support the use of 2 to 4 eye-tracking cameras (also monochrome cameras). If it is MR glasses, it can also support the use of two VST cameras to acquire color images, which can be in YUV or RGB formats, etc.

[0282] In some scenarios, the data capability information of the terminal device 100 acquiring perception and interaction data from XR glasses may include the following: Supported sensor types include: IMU, 6 to 10 monochrome cameras, 1 depth camera, and 2 VST color cameras. The monochrome camera supports generating data in the format of Luminance (Y) image streams, with supported resolutions and frame rates (in FPS) of 1280*1280*60 FPS and 640*480*60 FPS. The depth camera supports generating depth images with a resolution and frame rate of 640*480*30 FPS. The two VST color cameras support generating color image streams in YUV ("Y" represents Luminance, "U" and "V" represent Chrominance, "Chroma") or RGB format, with supported resolutions and frame rates of 4000*3000*60 FPS.

[0283] In some scenarios, wearable display devices (such as MR glasses) are also equipped with powerful processing modules. These modules can directly complete the entire closed-loop process from VST data acquisition to color image display. They can also perform calculations on perception and interaction data beyond VST data, such as spatial positioning and interaction calculations, and directly output the calculation results. Since such a display peripheral 200 does not need to transmit VST data to the terminal device 100, the data capability information of the perception and interaction data of the display peripheral 200 can include the format information of the calculated perception and interaction data supported by the display peripheral 200. This format information may include, but is not limited to: spatial positioning results, local positioning map, spatial geometric grid, lighting information, gesture points, gaze points, limb key points, facial expression bases, etc.

[0284] Optionally, when implementing S102 above, the terminal device 100 can also obtain the capability information of the transmission link for display data from the display peripheral 200. This transmission link is the transmission link between the display interfaces at both ends, as shown in Figures 2b and 2c.

[0285] The capability information of the data transmission link (also referred to as the fourth capability information) can provide the third link information supported by the display peripheral 200 for the second transmission link. The second transmission link is the link used to transmit the data to be displayed.

[0286] The third link information may include, but is not limited to, the number (e.g., maximum number) of channels (lanes) supported by the display peripheral 200 for the second transmission link, and the transmission rate of the channels. Thus, the terminal device 100 can determine the link rate (e.g., maximum rate) supported by the second transmission link based on this fourth capability information.

[0287] For example, the number of channels for the display data transmission link supported by the display peripheral 200 includes x1, x2, x4, and x8. x1, x2, x4, and x8 represent 1, 2, 4, and 8 channels respectively for the display data transmission link of the display peripheral 200. The transmission link supports a maximum of 8 channels, meaning the maximum link rate is the product of x8 and the channel transmission rate. The number of lanes in the display data transmission link of the display peripheral 200 affects the link rate and bandwidth; the more lanes enabled, the faster the link rate and the wider the bandwidth.

[0288] Similarly, optionally, when implementing S102 above, the terminal device 100 can obtain the capability information of the transmission link for sensing and interaction data from the display peripheral 200. This transmission link is the transmission link between the sensing and interaction interfaces at both ends, as shown in Figures 2b and 2c.

[0289] The capability information of the transmission link for the perceived and interactive data (also referred to as the second capability information) can provide the first link information supported by the display peripheral 200 for the first transmission link. The first transmission link is the link used to transmit the perceived and interactive data.

[0290] The first link information may include, but is not limited to, the number (e.g., maximum number) of channels (lanes) supported by the display peripheral 200 for the first transmission link, and the transmission rate of the channels. Thus, the terminal device 100 can determine the link rate (e.g., maximum rate) supported by the first transmission link based on this second capability information.

[0291] For example, the number of channels for the transmission link of the perception and interaction data that the display peripheral 200 supports includes: x1, x2.

[0292] Referring to Figure 2c, when the terminal device 100 obtains the capability information of the display data of the display peripheral 200, the terminal device 100 may send a capability request of the display module to the display peripheral 200. In response to the capability request, the display peripheral 200 may read the capability information of the display data from the storage module and send the capability information of the display data to the display interface of the terminal device 100 through the display interface of the display peripheral 200. The parsing module of the display data capability information of the terminal device 100 may read and parse the capability information of the display data from the display interface to obtain capability information such as the data format of the display data supported by the display peripheral 200, the resolution of the display data supported for display, and the frame rate of the display data supported for display.

[0293] When the terminal device 100 obtains the data capability information and transmission link capability information of the display data of the display peripheral 200, it can send two separate requests to obtain the two capability information sequentially, or it can obtain the two capabilities by sending a single request. There is no restriction here.

[0294] Similarly, referring to Figure 2c, when the terminal device 100 acquires the capability information of the perception and interaction data of the display peripheral 200, the terminal device 100 can send a sensor capability request to the display peripheral 200. In response to the capability request, the display peripheral 200 can read the capability information of the perception and interaction data from the storage module and send the capability information of the perception and interaction data to the perception and interaction interface of the terminal device 100 through the perception and interaction interface of the display peripheral 200. The parsing module of the terminal device 100 can read and parse the capability information of the perception and interaction data from the perception and interaction interface to obtain information such as the sensor type supported by the display peripheral 200, the format of the data supported by the sensor, resolution, and frame rate (also described as refresh rate).

[0295] When the terminal device 100 obtains the data capability information and transmission link capability information of the perception and interaction data of the display peripheral 200, it can send two requests to obtain the two capability information in sequence, or it can obtain the two capabilities by sending a single request. There is no restriction here.

[0296] Returning to Figure 4, after the terminal device 100 obtains the data capability information of the display data from the display peripheral 200 via S102, S103a and S104a can be executed. The corresponding display peripheral 200 can then execute S201a after S104a. Based on the data capability information of the display data from the display peripheral 200, the data format of the data to be transmitted and displayed between the terminal device 100 and the display peripheral 200 is uniformly configured.

[0297] S103a, the terminal device 100 determines the format of the data to be transmitted and displayed based on the data capability information of the display data.

[0298] Optionally, the terminal device 100 may combine at least one of the application scenario and the computing power (hereinafter referred to as computing power) of the terminal device 100 to set the format of the data to be transmitted and displayed within the data capability range supported by the display peripheral 200.

[0299] For example, if the display peripheral 200 is a general-purpose display, the data capability information (also referred to as third capability information) of the display peripheral 200 includes: the display peripheral 200 supports the display data format of RGB format, and supports three display data resolutions and frame rates: 1920*1080*60hz, 3840*2160*120hz, and 1280*720*60hz.

[0300] The third capability information of the display peripheral 200 can provide information on the format of the display data supported by the display peripheral 200, including the data format as RGB, and the resolution and frame rate of the display data as 1920*1080*60hz, 3840*2160*120hz, and 1280*720*60hz respectively. In addition, the above display data format information also implicitly expresses that the minimum resolution of the display data supported by the display peripheral 200 is 1280*720, the maximum resolution of the supported display data is 3840*2160, the minimum frame rate of the supported display data is 60hz, and the maximum frame rate of the supported display data is 120hz.

[0301] Then, the terminal device 100 can set the data format, resolution, and frame rate of the data to be transmitted and displayed within the aforementioned data capability range of the display peripheral 200 (e.g., the display data format information supported by the display peripheral provided by the third capability information). For example, the data format can be set to RGB format, and the resolution and frame rate can be selected from one of the three examples mentioned above. Alternatively, the resolution can be set to any resolution a*b within the range of the maximum resolution (e.g., 3840*2160) and the minimum resolution (e.g., 1280*720), where a / b = 1280 / 720 = 3840 / 2160 = 1920 / 1080, and the frame rate can be set to any frame rate c within the range of the maximum frame rate (e.g., 120Hz) and the minimum frame rate (60Hz), where 60Hz ≤ c ≤ 120Hz, for example, c = 90Hz.

[0302] In other words, the terminal device 100 can set the data format of the data to be transmitted and displayed between the two devices within the display data format information supported by the display peripheral 200, and the set data format cannot exceed the maximum data capacity of the display data supported by the display peripheral 200 (for example, 3840*2160*120hz here).

[0303] For example, if the general display only supports RGB format display data, then the data format of the data to be transmitted and displayed determined by the terminal device 100 can only be RGB, and cannot be other formats.

[0304] When the display peripheral 200 supports multiple formats for the same parameter (e.g., resolution) of the displayed data, for example, the supported resolutions are 1920*1080, 3840*2160, and 1280*720, the terminal device 100 can determine one of the above three resolutions as the resolution of the data to be transmitted and displayed, based on at least one of its computing capabilities for the data to be displayed and the application scenario.

[0305] For example, if the terminal device 100 has a low computing power for the data to be displayed and only supports foveated rendering, a lower resolution can be selected, such as the 1280*720 mentioned above.

[0306] For example, if the application currently running on terminal device 100 requires full-load rendering of images, a higher resolution can be selected, such as the aforementioned 3840*2160.

[0307] Similarly, for example, if the display peripheral 200 is XR glasses, its display data capability information is as follows: the data format is RGBA or RGBAZ, and it supports two resolutions and frame rates of 2000*2000*90Hz (e.g., the resolution corresponding to foveated rendering) and 3000*3000*90Hz (e.g., the resolution corresponding to Full HD). Then, the terminal device 100 can set the data format, resolution, and frame rate of the data to be transmitted and displayed within the aforementioned data capability range of the display peripheral 200. For example, the data format can be set to RGBAZ or RGBA; the resolution and frame rate can be selected from either of the two examples mentioned above. Alternatively, the resolution can be set to any resolution e*f within the range of the maximum resolution (e.g., 3000*3000) and the minimum resolution (e.g., 2000*2000), where e / f = 3000 / 3000 = 2000 / 2000 = 1 / 1, and the frame rate can be set to g, where g ≤ 90 Hz and g is a positive integer, for example, g = 60 Hz.

[0308] Then, the terminal device 100 can set the data format of the data to be transmitted between the two ends in this transaction within the format information of the data to be displayed supported by the display peripheral 200, combined with the application requirements. However, the set data format cannot exceed the maximum data capacity of the data to be displayed supported by the display peripheral 200 (for example, 3000*3000*90hz in this case).

[0309] In other words, the terminal device 100 of this application can, within the display data display capability supported by the display peripheral 200, selectively combine application requirements and the computing power of the terminal device 100 itself to reasonably determine the format of the data to be displayed to be transmitted through the display interface. This ensures that the determined format of the data to be displayed is not only within the data format range supported by the display peripheral 200, but also meets the image display requirements of the currently running application, and is compatible with the computing power of the terminal device 100 for the data to be displayed. Therefore, the terminal device 100 of this application can be compatible with the display data formats supported by various display peripherals, thereby enabling compatibility with the data display functions of various display peripherals.

[0310] Optionally, in S104a, the terminal device 100 sends the format of the data to be displayed to the display peripheral 200.

[0311] In step S103a, it is determined what format of data to be displayed the terminal device 100 can transmit through the display interface. Then, as shown in FIG2c, the terminal device 100 can send the format of the data to be displayed determined by step S103a to the display peripheral 200 through the display interface.

[0312] S201a, the display peripheral 200 enables and initializes the display module based on the format of the received data to be displayed.

[0313] For example, as shown in Figure 2b, the display peripheral 200 can initialize and set the display rendering driver of the display module and processing module of the display peripheral 200 according to the format of the data to be displayed received from the terminal device 100 (e.g., the display format, resolution, frame rate, etc. of the data to be displayed), so that the display peripheral 200 can drive the display module (e.g., the left eye display screen and the right eye display screen) of the display peripheral 200 to display the data according to the format of the display data (including the data format, resolution and frame rate) determined by the terminal device 100.

[0314] Returning to Figure 4, similar to the principles of S103a, S104a, and S201a, after the terminal device 100 obtains the perception and interaction data capability information from the display peripheral 200 via S102, S103b and S104b can be executed. The corresponding display peripheral 200 can then execute S201b after S104b. The display capability information based on the perception and interaction data from the display peripheral 200 is used to uniformly configure the data format for the perception and interaction data to be transmitted between the terminal device 100 and the display peripheral 200.

[0315] This application does not restrict the execution order of S103a and S103b; they can be executed serially or in parallel.

[0316] S103b, the terminal device 100 determines the format of the sensing and interaction data to be transmitted based on the sensing and interaction data capability information.

[0317] Optionally, the terminal device 100 may combine at least one of the application scenario and the terminal device 100's computing power for sensing and interactive data to set the format of the sensing and interactive data to be transmitted within the data capability range of the sensing and interactive data supported by the display peripheral 200.

[0318] As mentioned above, when the device type of the display peripheral 200 is a general display, the terminal device 100 does not need to obtain energy information for perception and interaction, and therefore does not need to execute S103b, S104b, S201b and the following S105b, S106, S202b.

[0319] When the peripheral device 200 is a wearable display device such as XR glasses or XR helmet, the execution of S103b is triggered.

[0320] The execution principle of S103b is the same as that of S103a described above. The terminal device 100 can set the first format of the sensing and interactive data to be transmitted this time within the format information provided by the data capability (also referred to as the first capability information) of the sensing and interactive data supported by the display peripheral 200. This format information refers to the format information of the sensing and interactive data supported by the display peripheral 200.

[0321] In one possible implementation, the display peripheral 200 is an XR glasses, and the data capability information (also referred to as first capability information) of the perception and interaction data of the display peripheral 200 may include, but is not limited to, the following information:

[0322] Supported sensor types include: IMU, 6 to 10 monochrome cameras, 1 depth camera, and 2 VST cameras.

[0323] The black and white camera supports generating data in the format of Y image stream, and the resolution and frame rate of the Y image stream generated by the black and white camera include 1280*1280*60FPS and 640*480*60FPS.

[0324] The depth camera supports a resolution and frame rate of 640*480*30FPS for the generated depth images.

[0325] The two VST cameras mentioned above support the generation of color image streams (e.g., real-world image streams) in either YUV or RGB format. Each color camera supports a color image stream resolution and frame rate of 4000*3000*60FPS.

[0326] The format information of the perception and interaction data supported by the display peripheral 200 provided by the first capability information may include, but is not limited to: supported number of cameras (e.g., binoculars), support for IMU, support for launching 6 to 10 monochrome cameras, support for launching up to one depth camera, and support for launching up to two VST cameras. The data format supported by the monochrome camera is a Y image stream, and the resolution of the Y image stream may include 1280*1280*60FPS and 640*480*60FPS. Furthermore, the resolution of the Y image stream can be h*h, where h≤1280, and h is a positive integer. The resolution of the Y image stream can also be j*k, where j / k=640 / 480, and the frame rate of the Y image stream m≤60FPS, where m is a positive integer. The principle behind the value ranges of the resolution and frame rate of the aforementioned depth image and color image stream provided by the first capability information is similar to the principle behind the value ranges of the resolution and frame rate of the aforementioned Y image stream, and will not be elaborated further here.

[0327] Then, within the data capability range of the aforementioned sensing and interactive data supported by the display peripheral 200, the terminal device 100 may optionally, in combination with the application scenario and the computing capability for the sensing and interactive data, set the format of the sensing and interactive data to be transmitted this time. The format may specifically include: the type of sensor to be activated, the data format of the sensing and interactive data obtained by the activated sensor (either the data collected by the sensor or the data collected by the sensor after calculation), the resolution and frame rate of the sensing and interactive data, etc.

[0328] For example, the format of the sensing and interaction data to be transmitted in this session, as set by terminal device 100, includes: activating the IMU, activating 8 monochrome cameras (or activating fewer than 8 monochrome cameras), activating one depth camera, and activating two VST cameras. The types and number of sensors activated are all within the aforementioned data capability range. For example, it is not possible to drive 15 monochrome cameras, because display peripheral 200 has a maximum of 10 monochrome cameras that can be used to collect sensing and interaction data.

[0329] The formats of the sensing and interactive data to be transmitted, as set by the terminal device 100, also include: the data format of the black-and-white camera is a Y image stream with a resolution and frame rate of 1280*1280*60FPS; the resolution and frame rate of the images acquired by the depth camera are 640*480*30FPS; and the format of the VST data (which is a color image) acquired by the VST camera is YUV with a resolution and frame rate of 4000*3000*60FPS.

[0330] In other words, the terminal device 100 of this application can, within the range of the sensing and interactive data capabilities supported by the display peripheral 200, selectively combine application requirements and the terminal device 100's own computing power for sensing and interactive data to reasonably determine the format of the sensing and interactive data to be transmitted through the sensing and interactive interface. This ensures that the determined format of the sensing and interactive data is not only within the range of sensing and interactive data formats supported by the display peripheral 200, but also meets the needs of the currently running application for sensing and interactive data, and is compatible with the terminal device 100's computing power for sensing and interactive data. Therefore, the terminal device 100 of this application can be compatible with the sensing and interactive data formats supported by various display peripherals, thereby achieving compatibility with the sensing and interactive data capabilities of various display peripherals.

[0331] In another possible implementation, the display peripheral 200 is an XR glasses device, as shown in Figure 2b. The processing module of the display peripheral 200 has a powerful processing unit, enabling it to perform calculations on the perception and interaction data collected by the perception module, interaction module, etc., to obtain the calculation results. Therefore, the data capabilities of the perception and interaction data obtained by the terminal device 100 from the display peripheral 200 include: the type of sensor activated corresponding to the calculation result of the perception and interaction data, and the data type and format of the calculation result.

[0332] For example, the data capabilities of this perception and interaction data include: spatial geometric grids, lighting information, gesture points, gaze points, limb key points, and other data types. In addition, each data type can also have its own data format.

[0333] Then, within the scope of this data capability, the terminal device 100 can select one or more data types and corresponding data formats from the above data types, based on application requirements and its own computing power.

[0334] S104b, the terminal device 100 sends the format of the perception and interaction data to be transmitted to the display peripheral device 200.

[0335] In step S103a, it is determined what format of sensing and interactive data the terminal device 100 can transmit through the sensing and interaction interface. Then, as shown in Figure 2c, the terminal device 100 can send the sensing and interactive data to be transmitted, determined in step S103b, to the display peripheral 200 through the sensing and interaction interface.

[0336] S201b, Display peripheral 200 enables and initializes the sensor based on the format of the received sensing and interaction data to be transmitted.

[0337] For example, as shown in Figure 2b, the display peripheral 200 can perform minimal sensor enabling and initialization according to the format of the sensing and interaction data to be transmitted received from the terminal device 100 (e.g., which sensors are activated, the data format of the data generated by the activated sensors, the resolution and frame rate of the data, etc.). For example, the display peripheral 200 can enable and initialize the settings of the corresponding sensors, start the sensors to collect data, and the processing module of the display peripheral 200 can also start the corresponding ISP processing, etc. The specific process is related to the format of the sensing and interaction data to be transmitted received.

[0338] For example, although the display peripheral 200 supports sensor types including: IMU, 6 to 10 monochrome cameras, 1 depth camera, and 2 VST cameras, according to the format of the sensing and interaction data to be transmitted determined by the terminal device 100, the display peripheral 200 only initializes and starts the IMU and the 6 monochrome cameras, 1 depth camera, and 2 VST cameras. Furthermore, the display peripheral 200 can configure the monochrome camera's resolution and frame rate to a relatively small 640*480*60FPS, etc., when initializing the aforementioned sensors, according to the format of the sensing and interaction data to be transmitted received in S104b.

[0339] Returning to Figure 4, after S103a, the terminal device 100 can also execute S105a and S106a, and the corresponding display peripheral 200 can execute S202a after S106a. This is to configure the transmission link of the display data of the terminal device 100 and initialize the transmission link of the display data of the display peripheral 200.

[0340] Optionally, in S105a, the terminal device 100 configures the transmission link (also referred to as the second transmission link) of the display data based on the capability information of the transmission link of the display data (also referred to as the fourth capability information) and the format of the data to be displayed determined in S103a above.

[0341] The capability information of the display data transmission link can provide the maximum transmission rate supported by the display peripheral 200's display data transmission link, such as the transmission rate corresponding to x4. Then, based on this maximum transmission rate, the terminal device 100 can determine the maximum amount of display data max1 that the transmission link can transmit.

[0342] Similarly, after determining the format of the data to be displayed, the terminal device 100 can determine the maximum data volume max2 of the data to be displayed that the terminal device 100 sends out through the display interface based on the format of the data to be displayed, such as the product of resolution and frame rate.

[0343] Then, the terminal device 100 can determine the fourth link information based on the format of the data to be displayed, within the maximum capability of the display peripheral for the transmission link, and configure the transmission link for transmitting the display data based on the fourth link information.

[0344] The fourth link information may include an indication of whether compression encoding is initiated, and optionally may also include at least one of the following: the number of channels allocated to the second transmission link (e.g., x4), the data transmission frequency, the data transmission and reception format of the second transmission link, etc.

[0345] Specifically, if the terminal device 100 determines that the maximum data size max2 is greater than the maximum data size max1 that the transmission link of the display peripheral 200 can support, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 is insufficient to support the transmission of the data size max2 of the data to be displayed from the terminal device 100. Therefore, when configuring the transmission link for the display data, the terminal device 100 can configure the transmission of the data to be displayed to be compressed and encoded, so that the data size of the compressed and encoded data to be displayed is less than or equal to the maximum data size max1.

[0346] In this way, when the maximum capacity of the transmission link of the display peripheral 200 for transmitting the data to be displayed is less than the maximum data volume max2 of the data to be displayed sent by the terminal device 100 through its local display interface, the terminal device 100 can compress and encode the transmitted data to be displayed, and send the compressed and encoded data to the display peripheral 200 through the display data transmission link. This allows for the transmission of XR display data with low latency and low bandwidth even when the amount of data to be displayed is large.

[0347] Conversely, if the terminal device 100 determines that the maximum data volume max2 is less than or equal to the maximum data volume max1 that the transmission link of the display data of the display peripheral 200 can support, it means that the maximum bandwidth of the transmission link of the display peripheral 200 can support the transmission of the data volume of the data to be displayed from the terminal device 100. Therefore, when configuring the transmission link of its own display data, the terminal device 100 can configure to disable compression encoding of the transmitted data to be displayed.

[0348] S106a, the terminal device 100 sends the configuration information of the transmission link for display data (such as the fourth link information mentioned above) to the display peripheral device 200.

[0349] Following S105a, the terminal device 100 may send configuration information to the display peripheral 200 after configuring the transmission link for display data in S105a. This configuration information may include, but is not limited to, an indication of whether to initiate compression encoding. Optionally, it may also include at least one of the following: the number of channels allocated to the second transmission link (e.g., x4), the data transmission frequency, and the data transmission and reception format of the second transmission link. This allows the display peripheral 200 to configure the transmission link (also referred to as the second transmission link) for display data on its side based on this configuration information, and enables the display peripheral 200 to decompress and decode the received compressed data to be displayed.

[0350] In one possible implementation, S202a, the display peripheral 200 confirms and initializes the format of the transmission link of the display data of the display peripheral 200 based on the configuration information of the transmission link of the received display data.

[0351] The display peripheral 200 can configure the format of the local display data transmission link (also referred to as the second transmission link) according to the configuration information about the transmission link of the display data sent by the terminal device 100. For example, it can configure the start of decompression and decoding, allocate the channel of the transmission link (e.g., x4), configure the data receiving frequency, configure the transmission and receiving data format of the transmission link, and initialize the local transmission link.

[0352] In this way, the display peripheral 200 can configure the transmission link of the display data on its own end according to the configuration information of the terminal device 100 for its own display data transmission link, and start the corresponding transmission link according to the corresponding configuration, such as starting the number of channels as x4.

[0353] Optionally, after S202a, the terminal device 100 may also confirm the link status of the transmission link for display data with the display peripheral device 200, and complete the handshake between the data sending and receiving ends. For specific implementation, please refer to the link status confirmation process of the transmission link in the prior art, which will not be limited or elaborated here.

[0354] In one possible implementation, returning to Figure 4, after S103b, the terminal device 100 can also execute S105b and S106b, and the corresponding display peripheral 200 can execute S202b after S106b. This is to configure the transmission link for the sensing and interaction data of the terminal device 100 and initialize the transmission link for the sensing and interaction data of the display peripheral 200. The implementation principle of this process is the same as the principle of the setting process for the transmission link of display data in S105a, S106a, and S202a described above, and will not be repeated here.

[0355] Optionally, in S105b, the terminal device 100 configures the transmission link (also referred to as the first transmission link) of the sensing and interaction data based on the capability information of the transmission link of the sensing and interaction data (also referred to as the second capability information) and the format of the sensing and interaction data to be transmitted (also referred to as the first format) determined in S103b.

[0356] The capability information of the transmission link for the perceived and interactive data can provide the maximum transmission rate supported by the transmission link for the perceived and interactive data of the display peripheral 200, such as the transmission rate corresponding to x8. Then, based on the maximum transmission rate, the terminal device 100 can determine the maximum amount of perceived and interactive data max3 that the transmission link can transmit.

[0357] Similarly, after determining the format of the sensing and interaction data, the terminal device 100 can determine the maximum data volume max4 of the sensing and interaction data received by the terminal device 100 through the sensing and interaction interface based on the format of the sensing and interaction data to be transmitted. For example, the product of the resolution and frame rate corresponding to each activated sensor, and the product is accumulated based on the number of activated sensors to obtain the maximum data volume max4.

[0358] Then, the terminal device 100 can determine the second link information based on the format of the sensing and interactive data (the first format mentioned above) within the maximum capability of the display peripheral for the transmission link, and configure the transmission link for transmitting the sensing and interactive data based on the second link information.

[0359] The second link information may include an indication of whether to start decompression decoding, and optionally may also include at least one of the following: the number of channels allocated to the first transmission link (e.g., x4), the data reception frequency, the data transmission and reception format of the first transmission link, etc.

[0360] Specifically, if the terminal device 100 determines that the maximum data volume max4 is greater than the maximum data volume max3 that the transmission link for the perception and interaction data of the display peripheral 200 can support, it indicates that the maximum bandwidth of the transmission link of the display peripheral 200 is insufficient to support the transmission of perception and interaction data of volume max4 to the terminal device 100, as this data volume exceeds the transmission capacity of the transmission link for the perception and interaction data of the display peripheral 200. Therefore, when configuring the transmission link for its own perception and interaction data, the terminal device 100 can configure and initiate the decompression and decoding of the transmitted perception and interaction data. The data volume of the compressed and encoded perception and interaction data is less than or equal to the maximum data volume max3.

[0361] In this way, when the maximum capacity of the transmission link of the display peripheral 200 for transmitting sensing and interactive data is less than the maximum data volume max4 of the sensing and interactive data sent by the display peripheral 200 to the terminal device 100, the display peripheral 200 can compress and encode the sensing and interactive data to be sent, and then send the compressed and encoded data to the terminal device 100 through the dual-sided sensing and interactive data transmission links. This allows the transmission of sensing and interactive data with low latency and low bandwidth even when the data volume of the sensing and interactive data to be sent by the display peripheral 200 is large, thereby realizing the transmission of XR display data.

[0362] Conversely, if the terminal device 100 determines that the maximum data volume max4 is less than or equal to the maximum data volume max3 that the transmission link of the display peripheral 200's sensing and interaction data can support, it means that the maximum bandwidth of the transmission link of the display peripheral 200 can support the transmission of sensing and interaction data with a data volume of max4 to the terminal device 100. Therefore, when configuring the transmission link of its own sensing and interaction data, the terminal device 100 can configure to disable the decompression and decoding of received sensing and interaction data.

[0363] S106b, the terminal device 100 sends the configuration information of the transmission link for sensing and interactive data to the display peripheral device 200.

[0364] After S105b, the terminal device 100 can send configuration information to the display peripheral device 200 after configuring the transmission link of the local sensing and interaction data in S105b. This configuration information may include, for example, whether to start decompression and decoding, optionally, the allocation of the transmission link channel (e.g., x8), the configuration of the data receiving frequency, the configuration of the transmission link data sending and receiving format, etc., so that the display peripheral device 200 can configure the transmission link of the sensing and interaction data on the display peripheral device 200 side based on the configuration information, and the display peripheral device 200 can compress and encode the sensing and interaction data to be sent.

[0365] In one possible implementation, S202b, the display peripheral 200 confirms and initializes the format of the transmission link of the received perception and interaction data based on the configuration information of the transmission link of the perception and interaction data received.

[0366] The display peripheral 200 can configure the format of the transmission link of the sensing and interactive data (such as various sensor data) on its own end according to the configuration information of the transmission link of the sensing and interactive data sent by the terminal device 100. For example, it can configure to start compression encoding, allocate the channel of the transmission link (e.g., x8), configure the data transmission frequency, configure the transmission and reception data format of the transmission link, and initialize the transmission link on its own end.

[0367] In this way, the display peripheral 200 can configure the transmission link of its own perception and interaction data according to the configuration information of the terminal device 100 for its own perception and interaction data transmission link, and start the corresponding transmission link according to the corresponding configuration, such as starting the number of channels as x8.

[0368] Optionally, after S202b, the terminal device 100 may also confirm the link status of the transmission link for sensing and interactive data with the display peripheral 200, and complete the handshake between the data sending and receiving ends. For specific implementation, refer to the link status confirmation process of the transmission link in the prior art, which will not be limited or elaborated here.

[0369] Returning to Figure 2c, the terminal device 100 can generate data to be displayed. Based on the configuration of the transmission link for the display data in S105a of Figure 4, the terminal device 100 can determine whether to compress and encode the data to be displayed.

[0370] In one possible implementation, compression encoding is initiated in S105a for the transmission link configuration of the display data of the terminal device 100. Then, as shown in Figure 4, this process may further include S107a, S108a, and S203a.

[0371] S107a, Terminal device 100 compresses and encodes the data to be displayed.

[0372] As mentioned above, the amount of data to be displayed generated by the terminal device 100 is max2, which exceeds the transmission capacity of the data to be displayed supported by the display peripheral 200 (e.g., the maximum data amount is max1). As shown in Figure 2c, the terminal device 100 can use the encoding module of the display data to compress and encode the generated data to be displayed in order to obtain the bitstream of the data to be displayed.

[0373] S108a, the terminal device 100 sends the bit stream of data to be displayed to the display peripheral 200.

[0374] As shown in Figure 2c, the terminal device 100 can send the compressed bitstream to the display peripheral 200 through its local display interface and the display data transmission link configured on the local end.

[0375] S203a, Display peripheral 200 decodes the received data to be displayed.

[0376] As shown in Figure 2c, after the display data transmission link of the display interface of the display peripheral 200 receives the bit stream of the data to be displayed, it can decompress and decode the bit stream through the display data decoding module to obtain the data to be displayed on the screen.

[0377] In one possible implementation, when the amount of data to be displayed generated by the terminal device 100 is max2, which does not exceed the transmission capacity of the display peripheral 200 (e.g., the maximum data amount is max1), compression encoding is not initiated in S105a above. Therefore, when the terminal device 100 sends the data to be displayed to the display peripheral 200, it does not need to undergo encoding by the terminal device 100's display data encoding module. After receiving the bitstream of the data to be displayed, the display peripheral 200 also does not need to perform decoding by the display data decoding module shown in FIG2c. The data to be displayed can be transmitted directly according to the transmission link configured by both parties.

[0378] Returning to Figure 2c, the peripheral device 200 can generate sensing and interaction data. Based on the configuration of the transmission link for sensing and interaction data in S105b of Figure 4, the peripheral device 200 can determine whether to compress and encode the sensing and interaction data.

[0379] In one possible implementation, S105b configures the decompression and decoding of the transmission link for the sensing and interaction data of the terminal device 100. As shown in Figure 4, this process may also include S203b, S204b, and S107b.

[0380] S203b, the display peripheral 200 compresses and encodes the generated perception and interaction data.

[0381] As mentioned above, the amount of perception and interaction data generated by the display peripheral 200 is max4, which exceeds the transmission capacity of the perception and interaction data supported by the display peripheral 200 (e.g., the maximum data amount is max3). As shown in Figure 2c, the display peripheral 200 can use the encoding module of the perception and interaction data to compress and encode the generated perception and interaction data to obtain the bitstream of the perception and interaction data.

[0382] S204b, the display peripheral 200 sends a stream of sensing and interactive data to the terminal device 100.

[0383] As shown in Figure 2c, the display peripheral 200 can send the compressed bitstream to the terminal device 100 through the local sensing and interaction interface and the sensing and interaction data transmission link configured on the local end.

[0384] S107b, the terminal device 100 decompresses and decodes the received sensing and interaction data stream.

[0385] As shown in Figure 2c, after the sensing and interaction data transmission link of the sensing and interaction interface of the terminal device 100 receives the code stream of sensing and interaction data, it can decompress and decode the code stream through the sensing and interaction data decoding module to obtain the sensing and interaction data.

[0386] In one possible implementation, when the amount of perception and interaction data generated by the display peripheral 200 is max4, which does not exceed the transmission capacity of the perception and interaction data supported by the display peripheral 200 (e.g., the maximum data amount is max2), then decompression and decoding are not initiated in S105b above. Therefore, when the display peripheral 200 sends the perception and interaction data to the terminal device 100, it does not need to undergo encoding operations by the encoding module of the perception and interaction data of the display peripheral 200. After receiving the bitstream of the perception and interaction data, the terminal device 100 also does not need to perform decoding operations through the decoding module of the perception and interaction data shown in FIG2c. The perception and interaction data can be transmitted directly according to the transmission link configured by both parties.

[0387] Furthermore, this application does not restrict the order between the two steps: the terminal device 100 sending data to be displayed to the display peripheral 200, and the display peripheral 200 sending perception and interaction data to the terminal device 100.

[0388] For example, in Figure 4, S204b can be executed before S108a. In other words, the display peripheral 200 can first send the collected perception and interaction data to the terminal device 100, and then the terminal device 100 can send the data to be displayed to the display peripheral 200.

[0389] In one possible implementation, the terminal device 100 may use the sensing and interaction data to obtain (either the terminal device 100 generates the data to be displayed, or the data to be displayed is obtained from the cloud and generated by the cloud) the data to be displayed.

[0390] In one possible implementation, the terminal device 100 may also use the sensing and interaction data to compress and encode the obtained data to be displayed.

[0391] The process of how the terminal device 100 uses the sensing and interaction data to compress and encode the obtained data to be displayed will be described below with reference to Figure 5a.

[0392] The data to be displayed can be image or video data. As explained above regarding the first image sequence, the data to be displayed can be a monocular image (i.e., one frame), a binocular image (i.e., two frames), or a multi-view image (i.e., two or more frames).

[0393] In Figure 5a, taking the data to be displayed as binocular data as an example, the data to be displayed may include a first eye image and a second eye image. Taking the display peripheral 200 having a left eye display screen and a right eye display screen as an example, the first eye image may be the image to be displayed on the left eye display screen, and the second eye image may be the image to be displayed on the right eye display screen.

[0394] Similarly, the perception and interaction data may include monocular perception and interaction data, binocular perception and interaction data, or multi-view perception and interaction data (including binocular and multi-view data). For example, the perception and interaction data may include first-view data, and optionally, second-view data, such as first-view data being perception and interaction data about the left eye, and second-view data being perception and interaction data about the right eye.

[0395] Although Figure 5a uses a stereo camera as an example to describe the compression encoding process of the data to be displayed, the method of this application is not limited to stereo scenarios. When the data to be displayed includes multi-camera (more than 2 cameras) data to be displayed, the method is the same, and will not be repeated here.

[0396] In some embodiments, the terminal device 100 may compress and encode the first image based on the first image data, and compress and encode the second image based on the second image data, to obtain the bitstream of the data to be displayed.

[0397] In this embodiment, in multi-view scenarios with binoculars or more, perception and interaction data can be combined to encode the corresponding target data to be displayed, thereby reducing bit rate and latency. The principle of using perception and interaction data to encode the corresponding target data to be displayed can be found in the implementation principle of S3011 in the embodiment of Figure 5a, and will not be elaborated here.

[0398] In the embodiment shown in Figure 5a, the image data of each eye in the binocular display data can be compressed sequentially, for example, the first eye image is compressed first, followed by the second eye image. Since the binocular cameras (e.g., the left-eye camera and the right-eye camera) on the display peripheral 200 are close together, the contents of the two images captured by the binocular cameras at the same time (the first eye image and the second eye image, respectively) are quite similar. Therefore, the method of this application can use the image compressed first as a reference image for the image compressed later, thereby removing redundant information and reducing the bit rate.

[0399] As shown in Figure 5a, the process may include the following steps:

[0400] S301, compress and encode the first image in the data to be displayed to obtain the bitstream of the first image and the reconstructed image of the first image.

[0401] The process of generating the reconstructed image is as follows: the first image is compressed and encoded to obtain the bitstream of the first image, and the bitstream is decompressed and decoded to obtain the reconstructed image of the first image.

[0402] S302, preprocess the reconstructed image of the first image to obtain a similar image.

[0403] The similar image is the image that matches the second image after preprocessing the first image.

[0404] The preprocessing can be deformation transformation, projection transformation between two images (such as projection transformation based on depth information), etc., and there are no restrictions here.

[0405] S302 can process the reconstructed image into an image that is more similar (or matches) to the pixel content of the second image (here named a similar image).

[0406] S303, based on the similar image, compress and encode the second image in the data to be displayed to obtain the bitstream of the second image.

[0407] Specifically, the residual between the second image and the similar image can be calculated, and the residual can be compressed and encoded to obtain the bitstream of the second image.

[0408] In this embodiment of the application, when compressing the second eye image, the reconstructed image of the first eye image can be processed once (e.g., deformation transformation) to process the reconstructed image into a similar image that is more closely matched with the pixel content of the second eye image; then, the residual is obtained by referring to the similar image based on the second eye image, and the residual is used to compress and encode the second eye image, so that the redundant information of the two eyes can be used to compress the second eye image to improve the compression efficiency of the data to be displayed.

[0409] In practical implementation, relevant parameter data involved in the compression encoding process of the dual-view data to be displayed can be added to the bitstream of the first view image or the bitstream of the second view image, so that the display peripheral 200 can accurately decode the bitstream of the data to be displayed.

[0410] Therefore, in S301 to S303 above, any step can be implemented by referring to the perception and interaction data.

[0411] As shown in Figure 5a, in one possible implementation, S301 can be implemented via S3011 when executing S301.

[0412] S3011, based on the first-view data in the perception and interaction data, compresses and encodes the first-view image in the data to be displayed to obtain the bitstream of the first-view image and the reconstructed image of the first-view image.

[0413] When compressing and encoding the first target image, the data about the first target in the perception and interaction data can be used as an aid for image compression.

[0414] For example, the first eye data may include a depth image taken by a depth camera regarding the left eye.

[0415] Then the terminal device 100 can use the depth image to obtain the depth information of the first target.

[0416] When using first-view data for auxiliary compression, the depth information mentioned above can be used to adjust the compression quality of image regions in the image being compressed (here, the first-view image). This adjustment of compression quality can be achieved by adjusting the encoder's encoding parameters, such as adjusting the encoder's quantization coefficients or quantization step size.

[0417] For example, image regions with greater depth can be compressed with lower quality (e.g., by setting a larger quantization factor and a smaller quantization step size during encoding), while image regions with less depth can be compressed with higher quality (e.g., by setting a smaller quantization factor and a larger quantization step size during encoding).

[0418] As shown in Figure 5a, in one possible implementation, S302 can be performed via S3021 when S302 is executed.

[0419] S3021, based on the first-view data in the perception and interaction data, preprocess the reconstructed image of the first-view image to obtain a similar image.

[0420] In one example, taking deformation transformation as an example of preprocessing, the first-eye data may include a depth image taken by a depth camera regarding the left eye. The terminal device 100 can use this depth image to obtain the depth information of the first eye for constructing a deformation function. Then, the deformation function is used to deform the reconstructed image of the first-eye image to obtain a similar image.

[0421] Compared to the similar image obtained through S302, the deformation function constructed based on the depth information of the first target is more accurate. Therefore, the similar image obtained by using the more accurate deformation function to deform the reconstructed image can be closer to the second target image.

[0422] Thus, in this embodiment, during the preprocessing of the reconstructed image of the first target image, the perception and interaction data can be referenced to preprocess the reconstructed image. Since the perception and interaction data provides depth information about the first target, the similar image obtained based on the perception and interaction data can be more similar to the second target image, thereby reducing the bit rate.

[0423] As shown in Figure 5a, in one possible implementation, when executing S303 above, it can be achieved through S3031.

[0424] S3031, based on the second-view data in the perception and interaction data and the similar image, compress and encode the second-view image in the data to be displayed to obtain the bitstream of the second-view image.

[0425] The principle of using the perceptual interaction data to compress the second image in S3031 is similar to that in S3011. For details, please refer to the description of S3011, which will not be repeated here.

[0426] Thus, Figure 5a exemplarily illustrates the compression encoding process of the data to be displayed in S107a of Figure 4.

[0427] In binocular or multi-view scenarios, this application can utilize the depth information of a single view to assist in the compression of multi-view images, thereby improving the bit rate and reducing bandwidth requirements.

[0428] In another scenario, not limited to binocular or multi-view scenarios, eye movement information calculated from eye movement images in perception and interaction data can be used to compress and encode the data to be displayed.

[0429] Specifically, eye-tracking information can be extracted from eye-tracking images captured by an eye-tracking camera in the perception and interaction data to determine the area where the gaze is focused (i.e., the fixation point) in the data to be displayed. Then, the area in the data to be displayed located at the fixation point is compressed with a higher quality, while the area outside the fixation point is compressed with a lower quality. The compression quality can be adjusted by modifying the encoder's encoding parameters, such as adjusting the encoder's quantization coefficients or quantization step size. In this way, the compressed image quality of the area where the gaze is focused is higher than that of the compressed image quality of the area outside that area. Of course, this scheme of using eye-tracking information to assist in the compression of the data to be displayed can also be applied to binocular or multi-view scenarios.

[0430] The perception and interaction data used in S3011, S3021 and S3031 shown in Figure 5a can be eye movement information, so the first eye data and the second eye data are the same eye movement information.

[0431] Furthermore, in Figure 5a, the perception and interaction data used in S3011, S3021, and S3031 can be depth information alone, such as first eye data being first eye depth information and second eye data being second eye depth information. They can also be eye movement information alone, or they can include both depth information and eye movement information. There are no restrictions here.

[0432] Accordingly, referring to Figure 4, when the display peripheral 200 executes S203a to decode the bitstream of the data to be displayed in order to obtain the data to be displayed, it is implemented by the reverse process shown in Figure 5a. The principle of this decoding process corresponds to the principle of the encoding process.

[0433] For example, the decoding module of the display data shown in Figure 2c of the display peripheral 200 can first decode the encoded first eye image to obtain a reconstructed image of the first eye image; then, it performs the same preprocessing operation (e.g., deformation transformation) on the reconstructed image of the first eye image as the encoding end (e.g., the encoding module of the display data shown in Figure 2c) to obtain a preprocessed reconstructed image (also referred to as a similar image). Furthermore, the decoding module of the display data shown in Figure 2c can also decode the bitstream of the second eye image and, based on the decoding result of the second eye image and the aforementioned similar image, obtain a reconstructed image of the second eye image. Specifically, the bitstream of the second eye image can be decoded to obtain the residual of the second eye image; then, the residual and the aforementioned similar image are processed (e.g., superposition processing) to obtain a reconstructed image of the second eye image. In this way, the reconstructed images of the first eye image and the second eye image are decoded. The processing parameters for the reconstructed image of the first eye image can be obtained by parsing the bitstream.

[0434] In the above decoding process, similar to S3011, S3021, and S3031, the display peripheral 200 can refer to the first-view data in the perception and interaction data when preprocessing the reconstructed image of the first view image. In addition, when decoding the bitstream of the first view image, it can refer to the first-view data in the perception and interaction data. Furthermore, when decoding the bitstream of the second view image, it can refer to the first-view data in the perception and interaction data.

[0435] In this embodiment, the terminal device 100 can utilize perception and interaction data to compress and encode the data to be displayed. Similarly, the display peripheral 200 can utilize the same perception and interaction data to decompress and decode the encoded bitstream of the data to be displayed. This allows for the reasonable use of effective perception and interaction information to assist in the compression processing of the data to be displayed, making encoding prediction more accurate and improving compression efficiency. Furthermore, the perception and interaction data can be used to adjust the encoding parameters of the encoder (e.g., the encoding module for display data) of the terminal device 100, making the image quality of the encoded image more suitable for human vision. Therefore, this method achieves higher compression efficiency than existing technologies, reduces the bit rate and bandwidth required to transmit the bitstream of the data to be displayed, and reduces latency. Moreover, the quality of the video generated through decoding is also superior, improving the user's viewing experience.

[0436] Based on S203b in Figure 4, the process of encoding perception and interaction data by the display peripheral 200 is described below.

[0437] For example, the perception and interaction data to be encoded can be binocular data, which may include data collected by various sensors of the display peripheral 200. For example, as shown in Figure 2b, the perception and interaction data may include, but is not limited to, at least one of the following: black and white images, depth images, IMU data, etc., acquired from the perception module and interaction module; VST data (e.g., real-world images, etc.) acquired from the VST module. The perception and interaction data can be a single image or video acquired over a period of time. The perception and interaction data may include first-eye data (e.g., perception and interaction data about the left eye) and second-eye data (e.g., perception and interaction data about the right eye). Optionally, it may also include data from more than one binocular view; this is not limited here.

[0438] The perception and interaction data can be divided into two categories: one is VST data, and the other is other perception and interaction data besides VST data.

[0439] First, the encoding process for VST data will be introduced.

[0440] For example, as shown in Figure 2c, the encoding module for the perception and interaction data of the display peripheral 200 can compress and encode the VST data based on other perception and interaction data besides the VST data.

[0441] The following description, with reference to Figure 5b, illustrates the process by which the display peripheral 200 uses other sensory and interactive data besides the VST data to compress and encode the VST data.

[0442] As mentioned above, taking VST data as visual image data as an example, the visual image data may include a first eye image and a second eye image.

[0443] In this embodiment, the image data of each eye in the binocular VST data can be compressed sequentially, for example, the first eye image is compressed first, and then the second eye image is compressed. Since the binocular cameras (e.g., the left eye camera and the right eye camera) on the display peripheral 200 are close to each other, the contents of the two images (the first eye image and the second eye image, respectively) captured by the binocular cameras at the same time are quite similar. Therefore, the method of this application can use the image compressed first as a reference image for the image compressed later, thereby removing redundant information and reducing the bit rate.

[0444] As shown in Figure 5b, the process may include the following steps:

[0445] S401, compress and encode the first image in the VST data to obtain the bitstream of the first image and the reconstructed image of the first image.

[0446] S402, preprocess the reconstructed image of the first image to obtain a similar image.

[0447] The similar image is the image obtained after preprocessing the first image.

[0448] S403, based on the similar image, compress and encode the second image in the VST data to obtain the bitstream of the second image.

[0449] In this embodiment of the application, when compressing the second eye image in VST data, the reconstructed image of the first eye image can be processed once (e.g., deformation transformation) to process the reconstructed image into a similar image that better matches the pixel content of the second eye image; then, the residual is obtained based on the second eye image by referring to the similar image, and the residual is used to compress and encode the second eye image, so that the redundant information of the two eyes can be used to compress the second eye image to improve the compression efficiency of VST data.

[0450] In practical implementation, relevant parameter data involved in the compression encoding of dual-view VST data can be added to the bitstream of the first view image or the bitstream of the second view image, so that the terminal device 100 can accurately decode the bitstream of the VST data.

[0451] Therefore, in S401 to S403 above, any step can be implemented by referring to other perception and interaction data besides VST data.

[0452] As shown in Figure 5b, in one possible implementation, S401 can be implemented via S4011 when executing S401.

[0453] S4011, based on the first-view data in other perception and interaction data besides VST data, compress and encode the first-view image in VST data to obtain the bitstream of the first-view image and the reconstructed image of the first-view image.

[0454] When compressing and encoding the first target image, data about the first target from other perception and interaction data besides the VST data can be used to assist in image compression.

[0455] Among them, the other perception and interaction data used in S4011, S4021, and S4031 besides the VST data are the original perception and interaction data other than the VST data after uncompressed processing.

[0456] When acquiring other perception and interaction data besides the VST data, the other perception and interaction data can be decompressed to obtain the original other perception and interaction data besides the VST data. Alternatively, the other perception and interaction data besides the VST data can be used to assist in compressing the VST data before compression. This application does not impose any restrictions on this.

[0457] As shown in Figure 5b, in one possible implementation, S402 can be achieved through S4021 when S402 is executed.

[0458] S4021, based on the first-view data from other perception and interaction data besides VST data, preprocess the reconstructed image of the first-view image to obtain a similar image.

[0459] As shown in Figure 5b, in one possible implementation, S403 can be implemented via S4031 when executing S403.

[0460] S4031, based on the second-eye data in other perception and interaction data besides VST data and the similar image, compress and encode the second-eye image in VST data to obtain the bitstream of the second-eye image.

[0461] Thus, Figure 5b exemplarily illustrates the process of compressing and encoding VST data in the perception and interaction data in S203b of Figure 4.

[0462] The implementation principle of the process shown in Figure 5b is the same as that shown in Figure 5a. The only difference is the object being compressed (the data to be displayed and VST data, respectively). Another difference lies in the perceptual and interactive data referenced when compressing this object. In the embodiment of Figure 5a, the based perceptual and interactive data can be VST data or other perceptual and interactive data besides VST data. In the embodiment of Figure 5b, the based perceptual and interactive data is other perceptual and interactive data besides VST data. The principles of other processes are the same. Therefore, the specific implementation details of Figure 5b are not elaborated here; please refer to the description in Figure 5a for details.

[0463] Accordingly, referring to Figure 4, when the terminal device 100 executes S107b to decode the bitstream of the VST data in the perception and interaction data to obtain the VST data, it can be implemented by the reverse process shown in Figure 5b. The principle of this decoding process is the same as the principle of the encoding process, and will not be repeated here. For the specific principle, please refer to the above introduction on the principle of decoding the bitstream of the display peripheral 200 to obtain the data to be displayed, which will not be repeated here.

[0464] In this embodiment, the display peripheral 200 can utilize other sensory and interactive data besides VST data to compress and encode the VST data. Similarly, the terminal device 100 can utilize other sensory and interactive data besides VST data to decompress and decode the encoded VST data stream. This allows for the reasonable use of effective sensory and interactive information to assist in the compression processing of VST data, making encoding prediction more accurate and improving compression efficiency. Furthermore, this other sensory and interactive data can be used to adjust the encoding parameters of the encoder (e.g., the sensory and interactive data encoding module) of the display peripheral 200, making the encoded image quality more suitable for human vision. Therefore, this method achieves higher compression efficiency than existing technologies, reduces the bit rate and bandwidth required to transmit the VST data stream, and reduces latency. Moreover, the quality of the decoded video is also superior, improving the user's viewing experience.

[0465] As mentioned above, the perception and interaction data may include not only VST data, but also other perception and interaction data other than VST data, such as at least one of the spatial data, gesture data, eye-tracking data, facial data, limb data, and depth data involved in black and white images, depth images, IMU data, etc.

[0466] In some embodiments, the amount of other sensing and interaction data is relatively small compared to VST data. Therefore, the display peripheral 200 may not compress and encode the other sensing and interaction data, but may directly transmit the other sensing and interaction data to the terminal device 100.

[0467] In some embodiments, the display peripheral 200 may also encode the other sensing and interactive data based on the encoding process of encoding VST data shown in FIG5b. When encoding the other sensing and interactive data, it can be implemented through the principles of S401, S402, and S403 shown in FIG5b (specifically, the compressed data object is replaced from VST data with the other sensing and interactive data) without the need for auxiliary compression with the sensing and interactive data (i.e., there is no need to execute S4011, S4021, and S4031).

[0468] Similarly, the terminal device 100 can also decompress the compressed bitstream of the other sensing and interaction data from the display peripheral 200 according to the reverse process of S401, S402, and S403 in Figure 5b, so as to obtain the other sensing and interaction data.

[0469] The implementation process of the processing method in this application will be described below with specific examples.

[0470] Example 1

[0471] Figure 6 illustrates a schematic diagram of the processing method of this application.

[0472] In Figure 6, the display peripheral 200 connected to the terminal device 100 is specifically an MR device 301.

[0473] The MR device 301 can be MR glasses or MR helmet, etc.

[0474] The two devices shown in Figure 6 are a specific schematic diagram of the two devices shown in Figure 2b.

[0475] In this example 1, the processing chip of the MR device 301 has strong computing power. It may include a computing module for perception and interaction data, which can be used to calculate other perception and interaction data besides the real scene image, and send the calculated perception and interaction data (i.e. the calculation result of perception and interaction data) to the terminal device 100.

[0476] For example, as shown in Figure 6, the calculated perception and interaction data may include, but is not limited to: spatial positioning results, spatial geometric grids, lighting information, gesture points, gaze points, facial geometry, limb key points, local positioning maps, and expression bases.

[0477] Furthermore, since the MR device 301 is equipped with a powerful processing chip, it can directly complete the closed loop of the link from the VST camera (one or more in the sensor module) to the display of the VST image. Therefore, the MR device 301 does not need to send VST data to the terminal device 100.

[0478] In addition, the MR device 301 has a display data processing module, which can be used for calculation processing such as virtual-real compositing of the image to be displayed. Therefore, the display data format supported by MR301 is virtual frame in RGBAZ or RGBA format.

[0479] Specifically, the terminal device 100 can obtain the data capability information of the display data and its transmission link capability information, the data capability information of the perception and interaction data and its transmission link capability information of the MR device 301, and determine the format of the data to be transmitted and displayed, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links based on the various capability information, and instruct the display peripheral device 200 to perform the corresponding initialization according to the determined information.

[0480] For example, in the scenario of Figure 6, the terminal device 100 can determine that the format of the data to be transmitted and displayed is RGBAZ or RGBA, and determine that the resolution of each image in the data to be displayed supported by the MR device 301 is full HD resolution (or it can be the resolution of foveated rendering display).

[0481] Furthermore, if the terminal device 100 determines that the maximum transmission rate of the display data transmission link can reach 100Gbps (G represents gigabytes and bps represents bits per second) when rendering each image at full HD resolution, then compression encoding needs to be initiated.

[0482] For example, in the scenario of Figure 6, the terminal device 100 can determine the format of the perception and interaction data to be transmitted, including: the format of the calculation result of the perception and interaction data, such as the spatial positioning result, spatial geometric grid, lighting information, gesture points, gaze points, Blendershape, limb key points, subMap, local positioning map, expression base, etc.

[0483] Furthermore, the amount of data in the above calculation results is relatively small, so compression encoding is not required.

[0484] Subsequently, as shown in Figure 6, the MR device 301 can collect perception and interaction data through the sensor module. This sensor module may include, but is not limited to, two VST cameras, four Mono cameras, two eye-tracking cameras, two downward-facing cameras, one depth camera, and one IMU. This application does not impose any restrictions on the type or number of sensors.

[0485] Among them, the VST camera can collect VST data (also referred to as VST images, or real-world images), the Mono camera can collect Mono images (also referred to as Y images above), the eye-tracking camera can collect eye-tracking images (since the eye-tracking camera is also a Mono camera, it is also called Y images or Mono images, or black and white images), the bottom camera can collect bottom images (since the bottom camera is also a Mono camera, it is also called Y images or Mono images, or black and white images), the depth camera can collect depth images, and the IMU can collect motion information.

[0486] Then, the ISP of the MR device 301 can perform calculations on the collected perception and interaction data. Specifically, it can calculate the real-scene image from the VST camera to obtain the signal of the real-scene image, and send the signal of the real-scene image to the display data processing module for fusion with the virtual frame.

[0487] Furthermore, the ISP can pass the calculated perception and interaction data to the perception and interaction data calculation module for further calculation to obtain the calculation results of the perception and interaction data. This calculation may include, but is not limited to, calculations of spatial geometric meshes, lighting, gestures, eye movements, faces, and limbs. The perception and interaction data may include, but is not limited to, at least one of the following: data from the ISP, motion information from the IMU, and depth information derived from depth calculations on depth images acquired by a depth camera.

[0488] Here, depth calculation is performed on the depth image to obtain depth information, which can be used for compression encoding and decompression decoding of the data to be displayed (as illustrated in the embodiment of Figure 5a).

[0489] Referring to Figure 6, the calculation module for perception and interaction data can send the calculation results of the perception and interaction data to the terminal device 100 through the perception and interaction interface.

[0490] The terminal device 100 can generate virtual frames in, for example, RGBAZ or RGBA format based on the calculation results of the perception and interaction data, and optionally use the received calculated perception and interaction data (such as depth information, eye movement information, etc.) to compress the virtual frame, and send the compressed bitstream to the MR device 301 through the display interface.

[0491] The MR device 301 can decompress and decode the received virtual frame bitstream to obtain the virtual frame.

[0492] The display data processing module of the MR device 301 can use the calculation results of the perception and interaction data calculated by the perception and interaction data calculation module, the signal of the real scene image calculated by the ISP, and the decoded virtual frame to process and calculate the display data. The calculation may include the fusion of virtual frame and real scene image, and optionally may also include anti-distortion processing of virtual frame and real scene image respectively. Finally, the display is performed on the screen through the display driver. The screens are the left eye display screen and the right eye display screen, and the two screens are used to display the binocular image.

[0493] Optionally, as shown in Figure 6, the terminal device 100 may include two main cameras for image capture. In XR scenarios, the two main cameras can also be used to collect perception and interaction data on the terminal device 100 side. Correspondingly, the processing chip of the terminal device 100 can also perform algorithm calculations for the main camera's capture and the algorithms for calculating the perception and interaction data it collects. In addition, the terminal device 100 can also perform collaborative shooting with the display peripheral 100.

[0494] As shown in Figure 6, the processing chip can provide a two-dimensional user interface (UI), such as the entry point of two-dimensional buttons and other controls, and a three-dimensional UI (such as the entry point of three-dimensional buttons and other controls). After the user clicks the corresponding control through the display interface (such as the game interface), the processing chip can provide a two-dimensional window or a three-dimensional model, and render the two-dimensional window or three-dimensional model through the rendering engine module (an example of the processing module shown in Figure 2b, such as CPU or GPU) to generate data to be displayed, such as virtual frames.

[0495] In some embodiments, the processing chip may also include an AI module (e.g., an AI processing chip), which can generate some or all of the elements in the virtual frame based on the calculation results of the received perception and interaction data through AI model inference, language large model calculation, etc.

[0496] Terminal device 100 can flexibly select one or more modules from the rendering engine module and AI module to generate virtual frames according to the application scenario and computing power.

[0497] The positioning module is used to locate the terminal device 100 itself.

[0498] The reconstruction module can be used to reconstruct a three-dimensional model of the environment where the terminal device 100 is located.

[0499] As shown in Figure 6, the terminal device 100 may have an audio module for collecting audio; a microphone module for collecting audio; and a display screen (e.g., 720P) for displaying media data such as images and videos.

[0500] As shown in Figure 6, the terminal device 100 may have a Wi-Fi module to communicate with various display peripherals 100 via Wi-Fi.

[0501] In Example 1, the terminal device 100 can be compatible with display peripherals such as MR glasses equipped with powerful processing modules. These MR glasses can complete the closed-loop link from the VST module of the real scene to the display of the entire real scene at the glasses end, and complete the virtual-real synthesis and display at the glasses end; they can also perform spatial positioning perception and interaction calculations at the glasses end, and can directly output the calculation results of perception and interaction data, such as the calculation results of positioning, perception, and interaction.

[0502] Furthermore, in Example 1, when generating elements in a virtual frame, the terminal device 100 can render the elements in the data to be displayed based on the calculation results of the perception and interaction data through the rendering engine module, or it can obtain the elements in the data to be displayed through model inference by the AI ​​module. The data to be displayed (e.g., a virtual frame) synthesized from multiple elements obtained by the terminal device 100 is sent to the MR device 301, where the MR device 301 performs virtual-real synthesis based on the received virtual frame and the real-world image acquired by the MR device 301, and displays the synthesized image on the left-eye display screen and the right-eye display screen.

[0503] Unlike existing technologies, the terminal device 100 of this application has a display interface that supports the transmission of display data to be composited in a virtual-real format, and a perception and interaction interface that supports the transmission of calculated perception and interaction data results. Specifically, the terminal device 100 supports transmitting display data in RGBAZ (or RGBA) format, which includes an additional transparency channel (A) and a depth channel (Z). In virtual-real compositing, the transparency channel is used as a mask, and the depth channel provides the correct occlusion relationship to achieve virtual-real occlusion. The perception and interaction data that the terminal device 100 supports transmitting is the result of algorithmic calculation, such as spatial positioning results, local maps, spatial geometric grids, lighting information, gesture points, gaze points, limb key points, facial expression bases, etc. This transmission method reduces bandwidth consumption and latency.

[0504] Furthermore, this Example 1 can utilize the information obtained from perception and interaction data to perform reference-based auxiliary processing on the multi-view video to be processed when encoding and decoding the data to be displayed. This can achieve higher compression efficiency than existing technologies, reduce bit rate and transmission bandwidth, and reduce latency. The resulting video has better subjective image quality, which can improve the user's subjective experience.

[0505] Example 2

[0506] Figure 7 illustrates a schematic diagram of the processing method of this application.

[0507] In Figure 7, the display peripheral 200 connected to the terminal device 100 is specifically an MR device 302.

[0508] The MR device 302 can be MR glasses or MR helmet, etc.

[0509] The two devices shown in Figure 7 are a specific schematic diagram of the two devices shown in Figure 2b.

[0510] Comparing this example to Figure 6 in Example 1, and referring to Figure 7, most of the content in Example 2 is the same as that in Example 1. The main differences are as follows:

[0511] Difference 1: As shown in Figure 7, the terminal device 100 may have a calculation module for perception and interaction data deployed in the MR device 301, as shown in Figure 6, for calculating perception and interaction data from the MR device 302. In other words, the terminal device 100 of this application may have the capability to calculate perception and interaction data from devices such as XR glasses.

[0512] Difference 2: As shown in Figure 7, the MR device 302 can compress and encode the perception and interaction data to be sent to the terminal device 100. For example, it can compress and encode real-world images (e.g., VST images), and optionally compress and encode depth images, black and white images, motion information, etc., before sending them to the terminal device 100.

[0513] Difference 3: As shown in Figure 7, the terminal device 100 may have a display data processing module, such as the one shown in Figure 6, deployed in the MR device 301, for calculating and processing the data to be displayed, for example, generating a virtual-real fused image. In other words, the terminal device 100 of this application may have the display data processing capability of devices such as XR glasses.

[0514] Difference 4: As shown in Figure 7, the terminal device 100 uses the display data processing module to generate data to be displayed, which is a virtual fused image in RGB format. Optionally, the terminal device 100 can compress and encode the data to be displayed and then send it to the MR device 302.

[0515] In this way, the computing power of AR glasses, VR glasses, MR glasses and other glasses or helmet devices can be integrated on the terminal device 100 side. The terminal device 100 calculates the perceived interaction data and performs virtual-real synthesis on the data to be displayed, so as to reduce the computing power requirements of the display peripherals and reduce the device weight and power consumption on the display peripheral side.

[0516] The following description, in conjunction with Figure 7, mainly focuses on the differences between the process shown in Figure 6 and the process shown in Figure 7. The similarities will not be repeated here; please refer to the description in Example 1.

[0517] In the embodiment shown in Figure 7, the terminal device 100 can determine that the display data supported by the MR glasses 302 is in RGB format and the resolution includes full HD resolution and foveated rendering resolution by acquiring the data capability information of the display data of the MR glasses 302 and the corresponding transmission link capability information. The terminal device 100 can thus determine that the upper limit of the transmission rate of the data to be transmitted and displayed is 60Gbps.

[0518] Furthermore, by acquiring the data capability information of the perception and interaction data of the MR glasses 302 and the corresponding transmission link capability information, the terminal device 100 can also determine that the sensors supported by the MR glasses 302 may include: 2 VST cameras, 8 black and white cameras (of which the eye-tracking camera and the downward camera are also black and white cameras), 1 depth camera, and IMU. Then the terminal device 100 can determine that the upper limit of the data volume of the perception and interaction data to be transmitted is close to 30Gbps.

[0519] Then, the terminal device 100 can determine the format of the data to be transmitted and displayed, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links based on various capability information, and instruct the display peripheral device 200 to perform the corresponding initialization according to the determined information.

[0520] For example, in the scenario shown in Figure 7, the terminal device 100 can determine that the format of the data to be transmitted and displayed is RGB, and that the resolution of each image in the data to be displayed supported by the MR device 302 is full HD (or it could be a resolution for foveated rendering). The data stream transmission rate of this full HD image to be displayed can reach 60Gbps. This transmission rate exceeds the transmission capacity of the display data transmission link. Therefore, compression encoding can be enabled on the display data transmission link to ensure that the transmission rate of the display data transmission link meets the upper limit requirement of 60Gbps.

[0521] For example, in the scenario shown in Figure 7, the terminal device 100 can determine that the format of the sensing and interactive data to be transmitted includes: activating two VST cameras, activating eight monochrome cameras, activating one depth camera, activating one IMU, and the resolution and frame rate of the images acquired by each sensor. Based on the format of this sensing and interactive data, the terminal device 100 can determine that the data stream transmission rate of the VST images acquired by the two VST cameras reaches 30Gbps, and can configure compression encoding to be enabled on the transmission link of this sensing and interactive data on the terminal device 100 side.

[0522] For other perceptual and interactive data besides VST images, such as depth images, black and white images, IMU data, etc., since their data volume is small, they can be optionally compressed and encoded (e.g., motion information as shown in Figure 7).

[0523] Subsequently, as shown in Figure 7, the MR device 302 can collect perception and interaction data through the sensor module, including real-scene images (e.g., VST images), depth images, black and white images, etc. The MR device 302 can compress and encode the real-scene images to obtain the bitstream of the real-scene images, and send it to the terminal device 100 through the perception and interaction interface. The MR glasses 302 can send other perception and interaction data besides the real-scene images directly to the terminal device 100 through the aforementioned transmission link of the perception and interaction interface without compression.

[0524] Optionally, as shown by the dashed arrow in Figure 7, in order to compress and decompress the VST image using perception and interaction data, and to compress and decompress the data to be displayed using perception and interaction data, as shown in Figure 7, the MR device 302 can calculate eye movement information based on eye movement images acquired by the sensor module (e.g., eye movement images acquired by an eye-tracking camera, which are also black and white images). Optionally, the MR device 302 can calculate depth information from depth images captured by a depth camera. This eye movement information and depth information are also sent to the terminal device 100 through the perception and interaction interface.

[0525] The MR glasses 302 can use at least one of eye-tracking information and depth information to compress and encode the VST image to obtain the VST data stream.

[0526] The data volume of perception and interaction data such as depth images, black and white images, and motion information is relatively small and can be compressed or not. There is no restriction here. These perception and interaction data are the other perception and interaction data mentioned above, excluding VST data.

[0527] Terminal device 100 can decompress the received sensing and interaction data to obtain sensing and interaction data.

[0528] For details on the process of compressing and decompressing the perceived and interactive data, please refer to Figure 5b and the introduction of related solutions; they will not be repeated here.

[0529] Referring again to Figure 7, the terminal device 100 can use the calculation module for sensing and interaction data in the processing chip to calculate the decoded sensing and interaction data to obtain the calculated sensing and interaction data (such as the spatial positioning result, spatial geometric grid, lighting information, gesture points, etc. shown in Figure 6). The specific process is the same as the process principle in Example 1, and will not be repeated here.

[0530] Furthermore, as shown in Figure 7, the terminal device 100 can generate virtual frames (RGBAZ or RGBA format) based on calculated perception and interaction data through the rendering engine module. Additionally, the terminal device 100 can generate some or all elements of the virtual frame through the AI ​​module. For example, the AI ​​module can generate some or all elements of the virtual frame based on calculated perception and interaction data; and the virtual frame can be fused with the decoded real-world image through the display data processing module in the MR real-scene rendering and compositing module to obtain a fused virtual-real image (e.g., an RGB format image).

[0531] Then, the terminal device 100 can compress and encode the RGB format virtual-real fusion image (for example, using the aforementioned depth information and eye-tracking information), and send the image bitstream to the MR glasses 302 through the display interface.

[0532] The MR glasses 302 can decode the image stream and display the decoded image after virtual-real fusion on the screen through the display driver, so as to display binocular images on the left eye display screen and the right eye display screen.

[0533] In this embodiment, the display interface of the terminal device 100 transmits data to be displayed in RGB format that can be directly displayed. The terminal device 100 of this application is compatible with display peripherals such as ordinary MR glasses to realize processing such as calculation and virtual-real synthesis of the data to be displayed. In addition, the sensing and interaction interface of the terminal device 100 of this application can transmit image data collected by sensors and can decompress compressed sensing and interaction data.

[0534] Furthermore, the data transmitted by the perception and interaction interface on the display peripheral side in Example 2 includes real-scene image data and other perception and interaction data (such as depth images, IMU data, etc.). This method can utilize information obtained from other perception and interaction data, especially real-scene image data, to perform reference-based auxiliary processing on the multi-view video to be processed during the encoding and decoding of real-scene image data. This achieves higher compression efficiency than existing technologies, reduces bit rate and transmission bandwidth, and lowers latency. The resulting video has superior subjective image quality, enhancing the user's subjective experience.

[0535] Additionally, the processing chip of the terminal device may include an AI module. When generating elements in a virtual frame, the terminal device can use a rendering engine module to render the elements in the data to be displayed based on the calculation results of the perception and interaction data. It can also use the AI ​​module to perform model inference to obtain the elements in the data to be displayed. The final elements to be displayed obtained by the terminal device can be displayed on the AR device in an AR format.

[0536] Furthermore, in Example 1, when generating elements in the virtual frame, the terminal device 100 can use the rendering engine module to render the elements in the data to be displayed based on the calculation results of the perception and interaction data, and can also use the AI ​​module to perform model inference to obtain the elements in the data to be displayed. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI ​​module. The terminal device 100 can then use the MR real-scene rendering and compositing module to fuse the final virtual frame with the real-scene image to obtain a fused virtual-real image. This fused virtual-real image is then sent to the MR device 302 for display.

[0537] Example 3

[0538] Figure 8 illustrates a schematic diagram of the processing method of this application.

[0539] In Figure 8, the display peripheral 200 connected to the terminal device 100 is specifically an MR device 303.

[0540] The MR device 303 can be MR glasses or MR helmet, etc.

[0541] The two devices shown in Figure 8 are a specific schematic diagram of the two devices shown in Figure 2b.

[0542] Comparing this example to Figure 7 in Example 2, and referring to Figure 8, most of the content of this Example 3 is the same as that of Example 2. The main difference is that when the terminal device 100 connects to the MR device 303, the VST images captured by the VST camera of the MR device 303 can be directly processed and displayed on the MR device 303. Therefore, the MR device 303 does not need to transmit high-definition real-scene images (such as VST images) to the terminal device 100 for processing through the perception and interaction interface. In addition, the MR device 303 can send other perception and interaction data (such as data collected by various cameras and sensors, such as depth images, black and white images, and motion information) to the terminal device 100, so that the terminal device 100 can perform calculations and processing on the other perception and interaction data.

[0543] Furthermore, the difference between Example 3 and Example 2 also includes that the calculation and processing of display data (e.g., the fusion of virtual frames and real-world images) is completed on the MR device 303 side, which is the same as in Example 1.

[0544] The process of implementing the method of this application by the system shown in Figure 8 is briefly described below.

[0545] First, the terminal device 100 can acquire data capability information of the display data of the MR glasses 303 and the corresponding transmission link capability information, as well as data capability information of the perception and interaction data and the corresponding transmission link capability information.

[0546] By analyzing the capability information, the terminal device can determine that the MR glasses 303 support virtual frame format (e.g., RGBAZ or RGBA format), with a resolution and frame rate of 2*3840*3600*90Hz. Based on this, the upper limit of the transmission rate of the data to be displayed can be determined to be 100Gbps.

[0547] The terminal device 100 can analyze the capability information to determine the format of the perception and interaction data to be transmitted by the MR glasses 303: it does not include the real-scene images of the two VST cameras. The format of the perception and interaction data that the MR glasses 303 supports for transmission includes downsampled images of the two VST cameras (e.g., with a resolution of 640*480), image data of eight black and white cameras, image data of one depth camera, and IMU data. Therefore, the transmission rate of the data that the perception and interaction interface needs to transmit is only 1Gbps to 2Gbps.

[0548] Then, the terminal device 100 can determine the format of the data to be transmitted and displayed, the format of the perception and interaction data to be transmitted, and the link configuration of the two corresponding transmission links based on various capability information, and instruct the display peripheral device 200 to perform the corresponding initialization according to the determined information.

[0549] For example, in the scenario shown in Figure 8, the terminal device 100 can determine that the format of the data to be transmitted and displayed is RGBAZ or RGBA, and that the resolution of each image in the data to be displayed supported by the MR device 303 is full HD (or it could be a resolution for foveated rendering). The data stream transmission rate of this full HD image to be displayed can reach 100Gbps. This transmission rate exceeds the transmission capacity of the display data transmission link, so compression encoding can be enabled on the display data transmission link to ensure that the transmission rate of the display data transmission link meets the 100Gbps upper limit requirement.

[0550] For example, in the scenario shown in Figure 8, the terminal device 100 can determine that the format of the sensing and interaction data to be transmitted includes: activating two VST cameras, with the downsampled image resolution of the obtained VST images being, for example, 640*480; activating eight monochrome cameras, one depth camera, and one IMU; and the resolution and frame rate of the images acquired by each sensor. Based on this format of the sensing and interaction data, the terminal device 100 can determine that the amount of sensing and interaction data to be transmitted is relatively small, and the transmission link of the sensing and interaction data can meet the requirements. Since it is not necessary to transmit VST data, only compressed images of VST images with a small data volume and some sensor data need to be transmitted; therefore, compression encoding can be omitted when setting up this transmission link.

[0551] Subsequently, as shown in Figure 8, the MR device 303 can collect perception and interaction data through the sensor module, including VST images, depth images, and black and white images. The MR device 302 can process the VST images for calculation and display without transmitting the VST images.

[0552] The MR device 303 can transmit other perception and interaction data besides real-world images, such as depth images, eight black and white images, motion information, etc., to the perception and interaction interface of the terminal device 100 through the perception and interaction interface.

[0553] The terminal device 100 can perform calculations on the perceived and interactive data through the calculation module of the perceived and interactive data. The specific calculation content is similar to that in Examples 1 and 2 above, and will not be repeated here.

[0554] Furthermore, the terminal device 100 can also generate some or all of the elements in the virtual frame through the AI ​​module. For example, the AI ​​module can generate some or all of the elements in the virtual frame based on the calculated perception and interaction data.

[0555] In addition, the terminal device 100 can generate data to be displayed (e.g., virtual frames in RGBAZ or RGBA format) based on the calculated perception and interaction data.

[0556] When generating elements in a virtual frame, the terminal device 100 can use a rendering engine module to render the elements in the data to be displayed based on the calculation results of perception and interaction data, and can also use an AI module to perform model inference to obtain the elements in the data to be displayed. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI ​​module. The terminal device 100 can then send the final virtual frame to the MR device 303, so that the MR device 303 can perform virtual-real compositing and display the virtual frame on the MR device 303.

[0557] In some embodiments, the terminal device 100 may compress the data to be displayed, for example, by using the depth information and eye movement information obtained by the terminal device 100 from the perception and interaction data to compress the data to be displayed, and then transmitting the compressed virtual frame to the MR glasses 303 through the display interface.

[0558] Then, the MR glasses 303 can decompress and decode the compressed virtual frame to obtain a virtual frame in RGBAZ or RGBA format.

[0559] Next, the display data processing module of the MR glasses 303 can process the real-world image and the virtual frame based on the information processed by the ISP (e.g., virtual-real compositing), and display it on the screen through the display driver.

[0560] For example, the composite image of binoculars can be displayed on the left and right eye displays respectively.

[0561] In Example 3, the terminal device 100 of this application is compatible with a display peripheral device of the MR glasses type, which is a device that can support VST data processing and direct delivery to the display. The VST data of the MR glasses 303 does not need to be transmitted to the terminal device 100; it can be processed, synthesized, and displayed within the MR glasses 303.

[0562] Unlike Example 2 above, but similar to Example 1, the display interface of the terminal device 100 in this application transmits data in RGBAZ or RGBA format. Unlike Example 2 above, in this Example 3, the data transmitted by the sensing and interaction interface does not include VST data, but rather image data downsampled from the VST. Thus, the data transmitted by the sensing and interaction interface can include image data from various sensors, the downsampled image data, and IMU data. The data volume is small, and no compression is required, allowing for direct transmission. Compared to Example 1, this Example 3 can reduce the latency of the VST path.

[0563] In this example 3, the rendering of the image, as well as the calculation of perception and interaction data, are all deployed on the terminal device 100 side, which also reduces the power consumption and weight of glasses-like peripherals.

[0564] Example 4

[0565] Figure 9 illustrates a schematic diagram of the processing method of this application.

[0566] In Figure 9, the display peripheral 200 connected to the terminal device 100 is specifically an AR device 304.

[0567] The AR device 304 can be AR glasses or AR helmets, etc.

[0568] The two devices shown in Figure 9 are a specific schematic diagram of the two devices shown in Figure 2b.

[0569] Comparing this example to Figure 8 in Example 3, and referring to Figure 9, most of the content of Example 4 is the same as that of Example 3. The main difference is that the sensor module of the AR glasses 304 does not include a sensor for acquiring real-world images, such as a VST camera. Therefore, the perception and interaction data transmitted by the AR glasses 304 to the terminal device 100 does not include real-world images. Also, the ISP of the AR device 304 does not need to send VST images to the display data processing module. Other aspects of Example 4 are the same as those of Example 3 and will not be repeated here.

[0570] Furthermore, the terminal device 100 can also generate some or all of the elements in the virtual frame through an AI module. For example, the AI ​​module can generate some or all of the elements in the virtual frame based on calculated perception and interaction data;

[0571] Furthermore, the processing chip of the terminal device 100 may include an AI module. When generating elements in a virtual frame, the terminal device 100 can use a rendering engine module to render the elements in the data to be displayed based on the calculation results of the perception and interaction data, and can also use the AI ​​module to perform model inference to obtain the elements in the data to be displayed. In this way, the terminal device 100 can obtain the final virtual frame through the rendering engine module and the AI ​​module. The terminal device 100 can send the final data to be displayed (e.g., the virtual frame) to the AR device 304 and display it on the AR device 304 in an AR format. Thus, the terminal device 100 of this application is compatible with AR devices.

[0572] Example 5

[0573] Figure 10 illustrates a schematic diagram of the processing method of this application.

[0574] In Figure 10, the display peripheral 200 connected to the terminal device 100 is specifically a VR device 305.

[0575] The VR device 305 can be VR glasses or VR headsets, etc., and there are no restrictions here.

[0576] The two devices shown in Figure 10 are a specific schematic diagram of the two devices shown in Figure 2b.

[0577] This Example 5 is largely the same as Example 4, and details can be found in the description of Example 4, which will not be repeated here. The main difference is that the display peripheral connected to the terminal device 100 is a VR device, not an AR device. However, the interaction process between the terminal device 100 and the VR device 305 is the same as the interaction process between the terminal device 100 and the AR device 304 in Example 4. The virtual frames sent to the VR device 305 are displayed on the VR device 304 in a fully immersive VR format. In this way, the terminal device 100 of this application is compatible with VR devices.

[0578] In any of the above embodiments, when encoding and decoding the perceived and interactive data, or the data to be displayed, a software codec or a hardware codec can be used. Alternatively, a standard codec or a non-standard codec can be used. This application does not limit the implementation form of the codec.

[0579] The following describes an apparatus provided by an embodiment of this application. As shown in Figure 11:

[0580] Figure 11 is a schematic diagram of a data processing device provided in an embodiment of this application. As shown in Figure 11, the device 500 may include: a processor 501, optionally a transceiver 505, and optionally a memory 502.

[0581] The transceiver 505, also known as a transceiver unit, transceiver, or transceiver circuit, is used to implement transceiver functions. The transceiver 505 may include a receiver and a transmitter. The receiver, also known as a receiver circuit, is used to implement the receiving function; the transmitter, also known as a transmitter or transmitting circuit, is used to implement the transmitting function.

[0582] The transceiver 505 can also be a communication interface.

[0583] The memory 502 may store computer programs, software code, or instructions 504, which may also be referred to as firmware. The processor 501 may implement the data processing methods provided in the embodiments of this application by running the computer programs, software code, or instructions 503 therein, or by calling the computer programs, software code, or instructions 504 stored in the memory 502. The processor 501 may be a central processing unit (CPU), and the memory 502 may be, for example, a read-only memory (ROM) or a random access memory (RAM).

[0584] The processor 501 and transceiver 505 described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits (RFICs), mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc.

[0585] The aforementioned device 500 may also include an antenna 506. The modules included in the device 500 are merely illustrative examples, and this application does not impose any limitations on them.

[0586] For example, the structure of the data processing device is not limited to that shown in Figure 11. The data processing device can be a standalone device or part of a larger device. For example, the implementation of the data processing device can be:

[0587] (1) A standalone integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally including storage components for storing data or instructions; (3) A module that can be embedded in other devices; (4) Vehicle equipment, etc.; (5) Others, etc.

[0588] When the data processing device is implemented as a chip or chip system, please refer to the schematic diagram of the chip structure shown in Figure 12. The chip shown in Figure 12 includes a processor 601 and an interface 602. The number of processors 601 can be one or more, and the number of interfaces 602 can be multiple. Optionally, the chip or chip system may include a memory 603. The processor 601 is used to call and execute instructions from the interface 602. When the processor 601 executes the instructions, it can perform the steps of the above-described method embodiments.

[0589] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0590] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer program containing at least one piece of code that can be executed by a computer to control the computer to implement the above-described method embodiments.

[0591] Based on the same technical concept, this application also provides a computer program, which, when executed, is used to implement the above-described method embodiments.

[0592] The program may be stored, in whole or in part, on a storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.

[0593] Based on the same technical concept, embodiments of this application also provide a chip, including a processor. The processor can implement the above-described method embodiments.

[0594] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0595] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0596] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The terminal device and the display device are communicatively connected, and the method includes: The terminal device receives first capability information, wherein the first capability information provides format information of the perception and interaction data supported by the display device; Based on the first capability information, the terminal device determines a first format of the sensing and interaction data to be transmitted, wherein the format information includes the first format; The terminal device receives perception and interaction data in the first format from the display device; The terminal device obtains multimedia data based on the perceived and interactive data.

2. The method according to claim 1, characterized in that, The terminal device obtains multimedia data based on the perceived and interactive data, including: The terminal device uses an artificial intelligence (AI) module to reason about the perceived and interactive data to obtain multimedia data.

3. The method according to claim 1, characterized in that, The terminal device obtains multimedia data based on the perceived and interactive data, including: The terminal device uses an AI module to perform reasoning based on the perception and interaction data to obtain the first multimedia element; The terminal device renders a second multimedia element based on the perceived and interactive data; Multimedia data is obtained based on the first multimedia element and the second multimedia element.

4. The method according to any one of claims 1 to 3, characterized in that, The multimedia data includes at least one of the following: images, videos, and models.

5. The method according to any one of claims 1 to 4, characterized in that, Before the terminal device receives the first-format perception and interaction data from the display device, the method further includes: The terminal device receives second capability information, wherein the second capability information provides first link information supported by the display device for the first transmission link; The terminal device determines the second link information based on the second capability information and the first format; The terminal device configures the first transmission link based on the second link information.

6. The method according to claim 5, characterized in that, The second link information includes information indicating the initiation of decompression. The terminal device receives perception and interaction data in the first format from the display device, including: The terminal device receives a first bitstream of compressed and encoded perception and interaction data in the first format from the display device via the configured first transmission link; The method further includes: The terminal device decodes the first bitstream based on the second link information to obtain the first format perception and interaction data.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The terminal device receives third capability information, wherein the third capability information provides format information of display data supported by the display device; Based on the third capability information, the terminal device determines the second format of the display data to be transmitted, wherein the second format includes an image format, which is an image format synthesized from virtual images and real-world images, or an image format of virtual images; The terminal device obtains multimedia data based on the perceived and interactive data, including: The terminal device obtains a first image sequence in the second format based on the perception and interaction data.

8. The method according to claim 7, characterized in that, The method further includes: The terminal device receives fourth capability information, wherein the fourth capability information provides information on the third link supported by the display device for the second transmission link; The terminal device determines the fourth link information based on the fourth capability information and the second format; The terminal device configures the second transmission link based on the fourth link information.

9. The method according to claim 8, characterized in that, The fourth link information includes information indicating the initiation of compression, and the method further includes: The terminal device compresses and encodes the first image sequence based on the fourth link information to obtain a second bitstream of the first image sequence. The terminal device sends the second bitstream to the display device through the configured second transmission link.

10. The method according to claim 9, characterized in that, The method further includes: The terminal device compresses and encodes the first image sequence based on the perception and interaction data to obtain the bitstream of the first image sequence.

11. The method according to claim 10, characterized in that, The first image sequence includes at least two-view images, which include a first-view image and a second-view image, and the perception and interaction data includes first-view data.

12. The method according to claim 11, characterized in that, The perception and interaction data further includes second data. Based on the perception and interaction data, the terminal device compresses and encodes the first image sequence to obtain a second bitstream of the first image sequence, including: The terminal device compresses and encodes the first image based on the first image data and compresses and encodes the second image based on the second image data to obtain a second bitstream of the first image sequence.

13. The method according to claim 11 or 12, characterized in that, The terminal device compresses and encodes the first image sequence based on the perceived and interactive data to obtain a second bitstream of the first image sequence, including: The terminal device compresses and encodes the first target image to obtain the bitstream of the first target image and the reconstructed image of the first target image. The terminal device preprocesses the reconstructed image based on the first image data to obtain a reference image that matches the second image. The terminal device compresses and encodes the second target image based on the reference image to obtain the bitstream of the second target image.

14. The method according to any one of claims 1 to 13, characterized in that, The first format includes at least one of the following: Data format, resolution, frame rate, number of images.

15. The method according to any one of claims 7 to 13, characterized in that, The second format includes at least one of the following: Data format, resolution, frame rate, number of images.

16. The method according to any one of claims 1 to 14, characterized in that, Based on the first capability information, the terminal device determines a first format for the sensing and interaction data to be transmitted, including: The terminal device determines the first format of the sensing and interaction data to be transmitted based on the first capability information and the target information, wherein the target information is at least one of the computing capabilities and application scenarios of the sensing and interaction data.

17. The method according to any one of claims 7 to 13 or 15, characterized in that, Based on the third capability information, the terminal device determines the second format of the display data to be transmitted, including: The terminal device determines the second format of the display data to be transmitted based on the third capability information and the target information, wherein the target information is at least one of the computing power and application scenario of the display data.

18. The method according to any one of claims 1 to 17, characterized in that, The first capability information provides the format information of the perception and interaction data that the display device supports collecting, or the format information of the perception and interaction data that the display device supports calculating.

19. A data processing method, characterized in that, The method includes: The display device sends first capability information, wherein the first capability information provides format information of the perception and interaction data supported by the display device; The display device acquires perception and interaction data in a first format, wherein the format information includes the first format; The display device sends perception and interaction data in the first format; The display device receives multimedia data, wherein the multimedia data is obtained based on the first format of perception and interaction data.

20. The method according to claim 19, characterized in that, Before the display device acquires the perception and interaction data in the first format, the method further includes: The display device receives the first format; The display device initializes the sensors used for collecting perception and interaction data according to the first format.

21. The method according to claim 19 or 20, characterized in that, The method further includes: The display device sends second capability information, wherein the second capability information provides first link information supported by the display device for the first transmission link; The display device receives second link information, wherein the second link information is determined based on the second capability information and the first format; The display device configures the first transmission link based on the second link information.

22. The method according to claim 21, characterized in that, The second link information includes information indicating whether to initiate compression or decompression, and the method further includes: The display device compresses and encodes the acquired perception and interaction data in the first format based on the second link information to obtain a first bitstream; The display device sends the first bitstream through the configured first transmission link.

23. The method according to claim 22, characterized in that, The first format of perception and interaction data includes real-scene images and image data other than the real-scene images. The display device compresses and encodes the acquired first format of perception and interaction data to obtain a first bitstream, including: The display device compresses and encodes the real-scene image based on the image data to obtain a first bitstream.

24. A data processing method, characterized in that, The terminal device and the display device are communicatively connected, and the method includes: The display device sends first capability information to the terminal device, wherein the first capability information provides format information of the perception and interaction data supported by the display device; Based on the first capability information, the terminal device determines a first format of the sensing and interaction data to be transmitted, wherein the format information includes the first format; The display device sends the first format of perception and interaction data to the terminal device; The terminal device obtains multimedia data based on the perception and interaction data in the first format; The terminal device sends the multimedia data to the display peripheral; The display device displays images based on the multimedia data.

25. The method according to claim 24, characterized in that, Before the display device sends the sensing and interaction data of the first format to the terminal device, the method further includes: The terminal device sends the first format to the display device; The display device initializes the sensors used for collecting perception and interaction data according to the first format; The display device acquires perception and interaction data in a first format based on the sensor after the initialization settings.

26. The method according to claim 24 or 25, characterized in that, The method further includes: The display device sends second capability information to the terminal device, wherein the second capability information provides first link information supported by the display device for the first transmission link; The terminal device determines the second link information based on the second capability information and the first format; The terminal device sends the second link information to the display device; The display device configures the first transmission link based on the second link information.

27. The method according to claim 26, characterized in that, The second link information includes information indicating the initiation of decompression, and the method further includes: The display device compresses and encodes the acquired perception and interaction data in the first format based on the second link information to obtain a first bitstream; The display device sends the first bitstream to the terminal device through the configured first transmission link; The terminal device decodes the first bitstream based on the second link information to obtain the first format perception and interaction data.

28. A data processing system, characterized in that, The system includes terminal equipment and display equipment that are connected in communication; The display device is configured to send first capability information to the terminal device, wherein the first capability information provides format information of perception and interaction data supported by the display device; The terminal device is configured to determine a first format of the sensing and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; The display device is also used to send the first format of perception and interaction data to the terminal device; The terminal device is also used to obtain multimedia data based on the first format of perception and interaction data; The terminal device is also used to send the multimedia data to the display device; The display device is also used to display images based on the multimedia data.

29. A data processing apparatus, characterized in that, The data processing device is communicatively connected to the display device, and the data processing device includes: A first receiving module is configured to receive first capability information, wherein the first capability information provides format information of the perception and interaction data supported by the display device; The first determining module is used to determine a first format of the sensing and interaction data to be transmitted based on the first capability information, wherein the format information includes the first format; The second receiving module is used to receive the first format perception and interaction data from the display device; The acquisition module is used to obtain multimedia data based on the perception and interaction data.

30. A data processing apparatus, characterized in that, The device includes: A first sending module is used to send first capability information, wherein the first capability information provides format information of perception and interaction data supported by the display device; An acquisition module is used to acquire perception and interaction data in a first format, wherein the format information includes the first format; The second sending module is used to send the perception and interaction data in the first format; A first receiving module is configured to receive multimedia data, wherein the multimedia data is obtained based on perception and interaction data in the first format.

31. A computer-readable storage medium, characterized in that, Includes a computer program that, when running on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 1 to 18, or the method as described in any one of claims 19 to 27.

32. A data processing apparatus, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory and send the signals to the processors, the signals including computer instructions stored in the memory; when the processors execute the computer instructions, the processors are configured to perform the method as described in any one of claims 1 to 18, or the method as described in any one of claims 19 to 27.

33. A computer program product, characterized in that, The computer program product includes a software program that, when executed by a computer or processor, causes the steps of the method according to any one of claims 1 to 18, or the method according to any one of claims 19 to 27, to be performed.

Images