Image transmission and display method, related device and system

Abstract

The application discloses an image transmission and display method, related equipment and system. The parallel processing of splitting a whole set of images into multiple parts for transmission, rendering and display can optimize the display path, thereby reducing the waiting time in the image transmission and display process, and further reducing the delay time of image transmission and display. Further, in the AR / VR or MR scene, the MTP delay can be reduced, the adverse reactions such as motion sickness of user dizziness and vomiting are effectively alleviated, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image transmission and display method, related equipment and system. Background Technology

[0002] With the development of terminal display technology, the application scenarios of augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies are becoming increasingly diverse. AR devices can overlay virtual images onto users while they view real-world scenes, allowing users to interact with these virtual images to achieve augmented reality effects. VR devices can simulate a three-dimensional (3D) virtual world scene, providing simulated experiences through sight, hearing, touch, and other senses, making users feel as if they are actually there. Furthermore, users can interact with this simulated virtual world scene. MR combines AR and VR, providing users with a view that merges the real and virtual worlds.

[0003] Head-mounted displays, or head-mounted displays for short, are devices worn on a user's head that provide a new visual environment. These displays can emit optical signals to present AR, VR, MR, and other display effects. A crucial factor affecting the user experience of AR / VR or MR head-mounted displays is motion-to-photons (MTP) latency. MTP latency refers to the total delay occurring within one cycle from when the user's head moves until a corresponding new image is displayed on the device screen.

[0004] Higher MTP latency negatively impacts the user experience, potentially causing motion sickness. Motion sickness is a physiological reaction that occurs when users experience AR / VR or MR systems. Because the visuals are dynamic, the brain interprets these changes as body movement, but the vestibular system, responsible for balance and coordination, doesn't send a signal back to the brain. This disharmony between perceived visual information and actual movement leads to dizziness, nausea, and other symptoms. Currently, an industry-recognized MTP latency below 20 milliseconds (ms) significantly reduces the occurrence of motion sickness.

[0005] By minimizing MTP latency as much as possible, motion sickness can be avoided and the user experience can be improved. Summary of the Invention

[0006] This application provides an image transmission and display method and related equipment, which can optimize the display path by dividing a whole image into multiple parts for parallel processing of transmission, rendering and display, thereby reducing the waiting time in the image transmission and display process and further reducing the latency of image transmission and display.

[0007] The aforementioned and other objectives will be achieved through the features described in the independent claims. Further implementations are illustrated in the dependent claims, the specification, and the drawings.

[0008] In a first aspect, this application provides an image transmission and display method for display on a first device, the first device including a display apparatus. The method may include: between a first vertical synchronization signal and a second vertical synchronization signal, the first device transmitting a first image signal to the display apparatus; and between the first and second vertical synchronization signals, the first device transmitting a second image signal to the display apparatus, the first image signal and the second image signal being asynchronous.

[0009] The first aspect of the method involves dividing a whole image into multiple parts and processing these parts for parallel image display. This optimizes the display path, thereby reducing the waiting time during image transmission and display, further reducing the latency of image transmission and display, and enabling faster image display.

[0010] In conjunction with the first aspect, in one possible implementation, the first device transmits a first image signal to the display device, which is also located between the first display signal and the second display signal. The first device also transmits a second image signal to the display device, which is also located between the first display signal and the second display signal.

[0011] In conjunction with the first aspect, in one possible implementation, the time interval between the first vertical synchronization signal and the second vertical synchronization signal is T1, and the time interval between the first display signal and the second display signal is T2, where T1 and T2 are equal.

[0012] In conjunction with the first aspect, in one possible implementation, between the first vertical synchronization signal and the second vertical synchronization signal, the display device displays a black-inserted image frame with a period of T3, where T3 is less than T1.

[0013] In conjunction with the first aspect, in one possible implementation, in response to a first vertical synchronization signal and a second vertical synchronization signal, the display device begins displaying a black-indented image frame. In response to either the first display signal or the second display signal, the display device ends displaying the black-indented image frame and displays a third image, the third image comprising the first image and the second image.

[0014] In conjunction with the first aspect, in one possible implementation, the first image signal is the image signal of the rendered first image, and the second image signal is the image signal of the rendered second image. That is, before displaying the first and second images in slices, the first and second images are rendered in slices respectively.

[0015] In conjunction with the first aspect, in one possible implementation, when the first device transmits a first image signal to the display device, the first device renders a second image. That is, the step of sending the first image to the display device and the step of rendering the second image are performed in parallel. Here, the first time is the time consumed by the first device from the start of rendering the first image to the end of transmitting the second image to the display device, and the second time is the time consumed by the first device alone to complete rendering the third image and transmitting the third image to the display device. The third image includes both the first and second images, and the first time is less than the second time.

[0016] In conjunction with the first aspect, in one possible implementation, the first device renders the first image before transmitting the first image signal to the display device.

[0017] In conjunction with the first aspect, in one possible implementation, the first device further includes an image rendering module for rendering the first image and the second image, and the display device sends a feedback signal to the image rendering module for indicating the vertical synchronization information of the display device.

[0018] In conjunction with the first aspect, in one possible implementation, the first device receives the first image transmitted by the second device before rendering the first image. Similarly, the first device receives the second image transmitted by the second device before rendering the second image. That is, before the first device renders the first and second images in chunks, it receives the first and second images chunked from the second device.

[0019] In conjunction with the first aspect, in one possible implementation, when the first device renders the first image, the first device receives the second image transmitted by the second device. That is, the rendering step of the first image and the transmission step of the second image are performed in parallel. The third time is the time consumed from the start of the first device receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the fourth time is the time consumed from the start of the first device receiving the third image transmitted by the second device to the end of transmitting the third image to the display device; the third time is less than the fourth time.

[0020] In conjunction with the first aspect, in one possible implementation, the first device receives the first image transmitted by the second device before transmitting the first image signal to the display device. The first device also receives the second image transmitted by the second device before transmitting the second image signal to the display device. That is, before the first device displays the first and second images in segments, it receives the first and second images transmitted in segments from the second device to the first device.

[0021] In conjunction with the first aspect, in one possible implementation, when the first device transmits a first image signal to the display device, the first device also receives a second image transmitted by the second device. That is, the step of sending the first image to the display device and the step of transmitting the second image are performed in parallel. The fifth time is the time consumed from the start of receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the sixth time is the time consumed from the start of receiving the third image transmitted by the second device to the end of transmitting the third image to the display device; the fifth time is less than the sixth time.

[0022] In conjunction with the first aspect, in one possible implementation, both the first image and the second image are rendered by the second device. That is, before the first device receives the first image and the second image from the second device in slices, the second device performs slice rendering on the first image and the second image respectively.

[0023] In conjunction with the first aspect, in one possible implementation, when the second image is rendered by the second device, the first device receives the first image transmitted by the second device. That is, the transmission step of the first image and the rendering step of the second image are performed in parallel. Here, the seventh time is the time consumed from the start of rendering the first image by the second device to the end of transmitting the second image to the display device, and the eighth time is the time consumed from the start of rendering the third image by the second device to the end of transmitting the third image to the display device; the seventh time is less than the eighth time.

[0024] In conjunction with the first aspect, in one possible implementation, in response to a second display signal, the first device displays a third image, the third image comprising the first image and the second image.

[0025] In conjunction with the first aspect, in one possible implementation, the first device displays a first image in response to a first image signal. In response to a second image signal, the first device displays a second image.

[0026] In conjunction with the first aspect, in one possible implementation, the display device further includes a frame buffer for storing pixel data of the first image and the second image.

[0027] In conjunction with the first aspect, in one possible implementation, the first device reads the pixel data of the first and second images from the frame buffer, and then displays the third image. That is, after acquiring the pixel data of the first and second images, the display device refreshes and displays a complete frame of the third image.

[0028] In conjunction with the first aspect, in one possible implementation, the first device reads the pixel data of the first image from the frame buffer and displays the first image. The first device then reads the pixel data of the second image from the frame buffer and displays the second image. That is, after obtaining the pixel data of the first image, the first device displays the first image; after obtaining the pixel data of the second image, the first device displays the second image. The first image and the second image together form a complete frame of the third image.

[0029] Secondly, this application provides an image transmission and display method for display on a first device, the first device including a display apparatus. The method may include: the first device transmitting a first image to the display apparatus; the first device transmitting a second image to the display apparatus; and the display apparatus displaying a third image, wherein the third image includes the first image and the second image.

[0030] The second approach involves dividing a whole image into multiple parts and processing these parts for parallel image display. This optimizes the display path, reduces the waiting time during image transmission and display, and further reduces the latency of image transmission and display, allowing the image to be displayed faster.

[0031] In conjunction with the second aspect, in one possible implementation, the first device renders the first image before transmitting it to the display device. The first device also renders the second image before transmitting it to the display device. That is, before displaying the first and second images in segments, the first and second images are rendered separately in segments.

[0032] In conjunction with the second aspect, in one possible implementation, when the first device transmits a first image signal to the display device, the first device renders a second image. That is, the step of sending the first image to the display device and the step of rendering the second image are performed in parallel. Here, the first time is the time consumed by the first device from the start of rendering the first image to the end of transmitting the second image to the display device, and the second time is the time consumed by the first device alone to complete rendering the third image and transmitting the third image to the display device. The third image includes both the first and second images, and the first time is less than the second time.

[0033] In conjunction with the second aspect, in one possible implementation, the first device receives the first image transmitted by the second device before the first device renders the first image. The first device also receives the second image transmitted by the second device before the first device renders the second image in chunks. That is, before the first device renders the first and second images in chunks, it receives the first and second images that the second device chunks transmits to the first device.

[0034] In conjunction with the second aspect, in one possible implementation, when the first device renders the first image, the first device also receives the second image transmitted by the second device. That is, the rendering step of the first image and the transmission step of the second image are performed in parallel. The third time is the time consumed from the start of the first device receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the fourth time is the time consumed from the first device receiving the third image transmitted by the second device to the end of transmitting the third image to the display device; the third time is less than the fourth time.

[0035] In conjunction with the second aspect, in one possible implementation, the first device receives the first image transmitted by the second device before the first device transmits the first image to the display device. Similarly, the first device receives the second image transmitted by the second device before the first device transmits the second image to the display device. That is, before the first device displays the first and second images in segments, it receives the first and second images transmitted in segments from the second device to the first device.

[0036] In conjunction with the second aspect, in one possible implementation, when the first device transmits a first image to the display device, the first device also receives a second image transmitted by the second device. That is, the step of sending the first image to the display device and the step of transmitting the second image are performed in parallel. The fifth time is the time consumed from the start of receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the sixth time is the time consumed from the start of receiving the third image transmitted by the second device to the end of transmitting the third image to the display device; the fifth time is less than the sixth time.

[0037] In conjunction with the second aspect, in one possible implementation, the first image and the second image are rendered by the second device. That is, before the first device receives the first image and the second image from the second device in slices, the second device performs slice rendering on the first image and the second image respectively.

[0038] In conjunction with the second aspect, in one possible implementation, when the second image is rendered by the second device, the first device receives the first image transmitted by the second device. That is, the transmission step of the first image and the rendering step of the second image are performed in parallel. Here, the seventh time is the time consumed from the start of rendering the first image by the second device to the end of transmitting the second image to the display device, and the eighth time is the time consumed from the start of rendering the third image by the second device to the end of transmitting the third image to the display device; the seventh time is less than the eighth time.

[0039] In conjunction with the second aspect, in one possible implementation, the display device further includes a frame buffer that stores pixel data of the first image and the second image.

[0040] In conjunction with the second aspect, in one possible implementation, the first device reads the pixel data of the first and second images from the frame buffer, and then the display device displays the third image. That is, after acquiring the pixel data of the first and second images, the display device refreshes and displays a complete frame of the third image.

[0041] In conjunction with the second aspect, in one possible implementation, the method further includes: in response to a first image signal, the first device displays a first image; in response to a second image signal, the first device displays a second image.

[0042] In conjunction with the second aspect, in one possible implementation, the first device reads the pixel data of the first image from the frame buffer and displays the first image. The first device then reads the pixel data of the second image from the frame buffer and displays the second image. That is, after obtaining the pixel data of the first image, the first device displays the first image; after obtaining the pixel data of the second image, the first device displays the second image. The first image and the second image together form a complete frame of the third image.

[0043] Thirdly, this application provides an image transmission and display method, which may include: establishing a connection between a first device and a second device; the second device transmitting a first image to the first device through the connection, and the second device transmitting a second image to the first device through the connection; the first device including a display device; the first device transmitting the first image to the display device; and the first device transmitting the second image to the display device; the display device displaying a third image, wherein the third image includes the first image and the second image.

[0044] The third aspect of the method involves the second device dividing a whole image into multiple parts and then transmitting them in parallel to the first device. The first device then processes these multiple parts for parallel image display, thereby optimizing the display path, reducing the waiting time during image transmission and display, further reducing the latency of image transmission and display, and enabling the image to be displayed faster.

[0045] In conjunction with the third aspect, in one possible implementation, when the first device transmits the first image to the display device, the second device transmits the second image to the first device via a connection. That is, the step of sending the first image to the display device and the step of transmitting the second image are performed in parallel. The fifth time is the time consumed from the start of the second device transmitting the first image to the end of the first device transmitting the second image to the display device, and the sixth time is the time consumed from the start of the second device transmitting the third image to the first device to the end of the first device transmitting the third image to the display device; the fifth time is less than the sixth time.

[0046] In conjunction with the third aspect, in one possible implementation, the second device renders the first image before transmitting it to the first device via the connection. Similarly, the second device renders the second image before transmitting it to the first device via the connection. That is, before the first device receives the fragmented transmission of the first and second images from the second device, the second device performs fragmented rendering of both the first and second images respectively.

[0047] In conjunction with the third aspect, in one possible implementation, when the first device receives the first image transmitted by the second device, the second device renders the second image. That is, the transmission step of the first image and the rendering step of the second image are performed in parallel. Here, the seventh time is the time consumed from the second device starting to render the first image to the first device transmitting the second image to the display device, and the eighth time is the time consumed from the second device starting to render the third image to the first device transmitting the third image to the display device; the seventh time is less than the eighth time.

[0048] In conjunction with the third aspect, in one possible implementation, the method further includes: rendering the first image by the first device after the second device transmits the first image to the first device via the connection and before the first device transmits the first image to the display device; and rendering the second image by the first device after the second device transmits the second image to the first device via the connection and before the first device transmits the second image to the display device.

[0049] In conjunction with the third aspect, in one possible implementation, when the second device transmits the second image to the first device via a connection, the first device renders the first image. That is, the rendering step of the first image and the transmission step of the second image are performed in parallel. Here, the third time is the time consumed from the second device starting to send the first image to the first device until the first device finishes transmitting the second image to the display device, and the fourth time is the time consumed from the second device starting to transmit the third image to the first device until the first device finishes transmitting the third image to the display device; the third time is less than the fourth time.

[0050] In conjunction with the third aspect, in one possible implementation, the display device further includes a frame buffer that stores pixel data of the first image and the second image.

[0051] In conjunction with the third aspect, in one possible implementation, the first device reads the pixel data of the first and second images from the frame buffer, and then displays the third image. That is, after acquiring the pixel data of the first and second images, the display device refreshes and displays a complete frame of the third image.

[0052] In conjunction with the third aspect, in one possible implementation, the method further includes: in response to a first image signal, the first device displays a first image; and in response to a second image signal, the first device displays a second image.

[0053] In conjunction with the third aspect, in one possible implementation, the first device reads the pixel data of the first image from the frame buffer and displays the first image. The first device then reads the pixel data of the second image from the frame buffer and displays the second image. That is, after obtaining the pixel data of the first image, the first device displays the first image; after obtaining the pixel data of the second image, the first device displays the second image. The first image and the second image together form a complete frame of the third image.

[0054] Fourthly, embodiments of this application provide an electronic device that may include: a communication device, a display device, a memory, and a processor coupled to the memory, multiple application programs, and one or more programs. The memory stores computer-executable instructions, the display device is used to display images, and the processor, when executing the instructions, enables the electronic device to perform any of the functions possessed by the first device in the first or second aspect.

[0055] Fifthly, embodiments of this application provide a computer storage medium storing a computer program, the computer program including executable instructions, which, when executed by a processor, cause the processor to perform operations corresponding to the methods provided in the first or second aspect.

[0056] In a sixth aspect, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform any possible implementation as described in the first or second aspect.

[0057] In a seventh aspect, embodiments of this application provide a chip system that can be applied to an electronic device. The chip includes one or more processors, which are used to invoke computer instructions to cause the electronic device to implement any possible implementation as described in the first or second aspect.

[0058] Eighthly, embodiments of this application provide a communication system, which includes a first device and a second device, wherein the first device can perform some of the functions of the first device as described in the first or second aspect.

[0059] By implementing the aspects provided in the embodiments of this application, the latency during image transmission and display can be reduced, allowing images to be displayed faster. Furthermore, in AR / VR or MR scenarios, MTP latency can be reduced, effectively alleviating adverse reactions such as dizziness and vomiting, and improving the user experience. It is understood that the methods provided in this application are applicable to more scenarios, such as image projection from electronic devices, image display in vehicle-mounted devices, and other image transmission and display scenarios. Through parallel processing of segmented transmission, rendering, and display, the end-to-end display process is accelerated, latency is reduced, and the user's viewing experience is improved. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the structure of a head-mounted display device;

[0061] Figure 2 This is a schematic diagram illustrating a usage scenario for a head-mounted display device.

[0062] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0063] Figure 4 A schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0064] Figure 5 A schematic diagram of a communication system provided in an embodiment of this application;

[0065] Figure 6 A schematic diagram of a process module provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of an image display process;

[0067] Figure 8 A schematic diagram of module time consumption analysis provided in an embodiment of this application;

[0068] Figure 9 A schematic diagram of module time consumption analysis provided in an embodiment of this application;

[0069] Figure 10 This is a schematic diagram of an image fragmentation transmission process provided in an embodiment of this application;

[0070] Figure 11 This application provides a schematic diagram of image segmentation as an embodiment.

[0071] Figure 12 This application provides an embodiment of an image segmentation display intent.

[0072] Figure 13 This is a schematic diagram of an image fragmentation transmission process provided in an embodiment of this application;

[0073] Figure 14 A schematic diagram of a rendering process provided in an embodiment of this application;

[0074] Figure 15 This is a schematic diagram of an image fragmentation transmission process provided in an embodiment of this application;

[0075] Figure 16 A schematic diagram of a communication system provided in an embodiment of this application;

[0076] Figure 17 This is a schematic diagram of an image fragmentation transmission process provided in an embodiment of this application;

[0077] Figure 18 This is a schematic diagram of an image fragmentation transmission process provided in an embodiment of this application;

[0078] Figure 19 A flowchart illustrating an image transmission and display method provided in an embodiment of this application;

[0079] Figure 20 A flowchart illustrating an image transmission and display method provided in an embodiment of this application;

[0080] Figure 21 A flowchart illustrating an image transmission and display method provided in an embodiment of this application;

[0081] Figure 22 This is a schematic diagram illustrating the process of an image transmission and display method provided in an embodiment of this application;

[0082] Figure 23 This is a schematic diagram illustrating the process of an image transmission and display method provided in an embodiment of this application;

[0083] Figure 24 This is a schematic diagram illustrating the process of an image transmission and display method provided in an embodiment of this application;

[0084] Figure 25 This is a schematic diagram illustrating the process of an image transmission and display method provided in an embodiment of this application;

[0085] Figure 26 This is a schematic diagram illustrating the process of an image transmission and display method provided in an embodiment of this application. Detailed Implementation

[0086] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in the text describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0087] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0088] The electronic device involved in this application embodiment can be a head-mounted display device, worn on the user's head, which can realize display effects such as AR, VR, and MR. The appearance of the head-mounted display device can be glasses, helmet, goggles, etc., and this application embodiment does not limit it. Not limited to a head-mounted display device as an example, the electronic device involved in this application embodiment can also be other devices including a display screen, such as mobile phones, personal computers (PCs), tablet computers (PADs), smart TVs, or in-vehicle devices, etc., and this application embodiment does not limit the type of electronic device.

[0089] Virtual objects displayed on electronic devices can interact with users. In some embodiments, users can interact with virtual objects directly through motion-sensing interactions such as hand / arm movements, head movements, and eye movements. In other embodiments, the electronic device can be used in conjunction with a handheld device, allowing users to interact with virtual objects displayed on the electronic device by controlling the handheld device. This handheld device can be, for example, a gamepad, controller, gyroscope mouse, stylus, or other handheld computing device. The handheld device can be equipped with various sensors such as accelerometers, gyroscopes, and magnetometers to detect and track its own movement. The handheld device can communicate with other electronic devices via short-range technologies such as Wi-Fi, Bluetooth, NFC, and ZigBee, or via wired connections such as a Universal Serial Bus (USB) interface or a custom interface.

[0090] See Figure 1 , Figure 1 A schematic diagram of the structure of a head-mounted display device 200 is shown.

[0091] like Figure 1 As shown, the head-mounted display device 200 may include some or all of the following: left lens 201, right lens 202, left display screen 203, right display screen 204, left camera 205, right camera 206, and inertial measurement unit (IMU) 207.

[0092] The left lens 201 and right lens 202 can be convex lenses, Fresnel lenses, or one or more other types of transparent lenses. By refracting light through the lenses, the image on the display screen can be brought closer to the user's retina, allowing the user's eyes to clearly see the image on the screen that is almost touching their eyes. The left display screen 203 and right display screen 204 are used to display images of the real world, the virtual world, or a combination of both. The left camera 205 and right camera 206 are used to capture images of the real world. The IMU 207 is a sensor used to detect and measure acceleration and rotational motion, and can include accelerometers, angular velocity meters (or gyroscopes), etc. An accelerometer is a sensor that senses axial acceleration and converts it into a usable output signal, while a gyroscope is a sensor that senses the angular velocity of a moving body relative to inertial space.

[0093] See Figure 2 , Figure 2 This is a schematic diagram illustrating a usage scenario of a head-mounted display device 200 according to an embodiment of this application. When a user wears the head-mounted display device 200, the user's eyes can see the image 210 displayed on the screen of the head-mounted display device 200. The image 210 displayed on the screen of the head-mounted display device 200 is a visualized virtual environment, which can be a virtual world image, a real world image, or an image combining the virtual and real worlds. For VR head-mounted display devices, by blocking the user's vision and hearing from the outside world, and displaying images from the left and right eye perspectives on the left and right screens respectively, there will be a positional deviation between these two images. Therefore, there is a difference in the image information obtained by the user's left and right eyes, i.e., parallax. The user's brain will subconsciously calculate the distance of objects from the body based on parallax, thereby making the image seen by the user have a sense of three-dimensionality and depth, guiding the user to feel as if they are in a virtual environment, forming a hyper-realistic immersive experience.

[0094] In this embodiment, providing a visualized virtual environment by an electronic device means that the electronic device uses AR / VR or MR display technologies to render and display a virtual image composed of one or more virtual objects, or an image combining real objects and virtual objects. The virtual object can be generated by the electronic device itself using computer graphics technology, computer simulation technology, etc., or it can be generated by other electronic devices using computer graphics technology, computer simulation technology, etc., and then sent to the electronic device. Other electronic devices can be servers, or mobile phones, computers, etc., connected or paired with the electronic device. Virtual objects can also be called virtual images or virtual elements. Virtual objects can be two-dimensional or three-dimensional. Virtual objects are fake and not real objects in the physical world. Virtual objects can be virtual objects that mimic objects existing in the real physical world, thereby providing users with an immersive experience. Virtual objects can include virtual animals, virtual characters, virtual trees, virtual buildings, virtual labels, icons, pictures, or videos, etc. The corresponding real objects refer to objects existing in the real physical environment or physical space where the user and the electronic device are currently located. Real objects can include animals, characters, trees, buildings, etc.

[0095] VR devices can be either all-in-one or split-type devices.

[0096] An all-in-one machine is a device with an independent processor, capable of independent calculation, input, and output functions, free from the constraints of cables, and offering greater freedom.

[0097] A split-system PC refers to a PC where the display device and the main unit are separate. The display device is primarily used for displaying images, while the main unit is mainly used for processing data. Because the main unit's processing system is separate from the head-mounted display device, the main unit can use a higher-performance processor and cooling system. Therefore, the advantage of a split-system PC lies in the more rational allocation of functions between the display device and the main unit, stronger processing performance, and richer resources. However, split-system PCs need to consider compatibility across devices and platforms, such as hardware platforms, software systems, operating systems, and application software.

[0098] The hardware structure of the electronic device 100 provided in the embodiments of this application is described below as an example.

[0099] The electronic device 100 provided in this application embodiment can be the aforementioned head-mounted display device 200, or the host 310 or other electronic devices described below. The electronic device 100 can include, but is not limited to, mobile phones, desktop computers, laptops, tablets, smart screens (smart TVs), desktop computers, laptops, handheld computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, game consoles, smart home devices, Internet of Things (IoT) or vehicle-to-everything (V2X) devices, etc. This application embodiment does not impose any limitations on the specific type of the electronic device 100.

[0100] Figure 3 This is a schematic diagram of the hardware structure of the electronic device 100 provided in an embodiment of this application. Figure 3 In this example, the electronic device 100 is a head-mounted display device 200. When the electronic device 100 is a mobile phone or other devices, some hardware structures can be added or removed.

[0101] Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display device 194, and an eye-tracking module 195, etc. The sensor module 180 can be used to acquire the user's posture and may include a pressure sensor 180A, a gyroscope sensor 180B, an accelerometer sensor 180C, a proximity sensor 180D, a touch sensor 180E, etc.

[0102] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0103] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the wearable device 100. In other embodiments of this application, the wearable device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0104] Processor 110 is typically used to control the overall operation of electronic device 100 and may include one or more processing units. For example, processor 110 may include a central processing unit (CPU), application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), video processing unit (VPU), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0105] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0106] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0107] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0108] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0109] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0110] In some embodiments, the processor 110 can render different objects based on different frame rates, for example, using a high frame rate to render near objects and a low frame rate to render distant objects.

[0111] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180E, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180E through the I2C interface, enabling the processor 110 and the touch sensor 180E to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0112] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of listening to audio through Bluetooth headphones.

[0113] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of listening to audio through Bluetooth headphones. Both the I2S interface and the PCM interface can be used for audio communication.

[0114] The UART interface is a general-purpose serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication.

[0115] In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable audio playback via Bluetooth headphones.

[0116] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display device 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display device 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0117] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display device 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0118] USB interface 130 is a USB standard compliant interface, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as mobile phones, computers, VR / AR or MR devices, etc. The USB interface can be USB 3.0, used for compatibility with high-speed display port (DP) signal transmission, enabling the transmission of high-speed audio and video data.

[0119] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the wearable device 100. In other embodiments of this application, the wearable device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0120] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0121] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display device 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0122] Electronic device 100 may include wireless communication capabilities. For example, head-mounted display device 200 may receive rendered images from other electronic devices (such as VR hosts or VR servers) for display, or it may receive unrendered images and then have processor 110 render and display them. The wireless communication capability may be implemented through an antenna (not shown), a mobile communication module 150, a wireless communication module 160, a modem processor (not shown), and a baseband processor (not shown).

[0123] Antennas are used to transmit and receive electromagnetic wave signals. Electronic device 100 may include multiple antennas, each capable of covering one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antennas can be multiplexed as diversity antennas for a wireless local area network. In some other embodiments, antennas can be used in conjunction with tuning switches.

[0124] The mobile communication module 150 can provide solutions for wireless communication applications on the electronic device 100, including 2G, 3G, 4G, and 5G networks. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via an antenna, filter and amplify the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and radiate it as electromagnetic waves via the antenna. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0125] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through a display device 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0126] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via the antenna.

[0127] In some embodiments, the antenna of the electronic device 100 is coupled to the mobile communication module 150 and the wireless communication module 160, enabling the electronic device 100 to communicate with networks and other devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0128] Electronic device 100 implements display functions through a GPU, display device 194, and application processor. The GPU is a microprocessor for image processing, connecting the display device 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0129] In this embodiment, the display device 194 is used to display images, videos, etc. The display device 194 can be used to present one or more virtual objects, thereby enabling the electronic device 100 to provide a virtual reality scene for the user. In some embodiments, the electronic device 100 may include one or N display devices 194, where N is a positive integer greater than 1.

[0130] The display device 194 may present virtual objects in one or more of the following ways:

[0131] 1. In some embodiments, the display device 194 may include a display screen, which may include a display panel. The display panel may be used to display physical objects and / or virtual objects, thereby presenting a three-dimensional virtual environment to the user. The user can see the virtual object on the display panel and experience the virtual reality scene. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0132] 2. In some embodiments, the display device 194 may include an optical device for projecting optical signals (e.g., a light beam) directly onto the user's retina. The display device 194 may use one or more optical devices such as mirrors, transmissive mirrors, or optical waveguides to convert a real-pixel image display into a near-eye projected virtual image display. The user can directly see virtual objects through the optical signals projected by this optical device, experience a three-dimensional virtual environment, and achieve a virtual interactive experience, or an interactive experience combining virtual and reality. In one example, the optical device may be a miniature projector, etc.

[0133] The electronic device may have two display devices 194, each corresponding to one of the user's two eyes. The content displayed on these two display devices can be displayed independently. These two display devices can display images with parallax to enhance the stereoscopic effect of the image. In some possible embodiments, the electronic device may also have only one display device 194, with both of the user's eyes viewing the same image.

[0134] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display device 194 and application processor.

[0135] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0136] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. Camera 193 may include, but is not limited to, conventional color cameras (RGB cameras), depth cameras (RGB depth cameras), dynamic vision sensor (DVS) cameras, etc. In some embodiments, camera 193 can be a depth camera. A depth camera can acquire spatial information of the real environment.

[0137] In some embodiments, the camera 193 can capture images including real objects, and the processor 110 can fuse the images of real objects captured by the camera 193 with the images of virtual objects, and display the fused image through the display device 194.

[0138] In some embodiments, the camera 193 can capture images of the user's hand or body, and the processor 110 can analyze the images captured by the camera 193 to identify the hand or body movements input by the user.

[0139] In some embodiments, the camera 193 can be used in conjunction with an infrared device (such as an infrared emitter) to detect the user's eye movements, such as eye gaze direction, blinking, gazing, etc., thereby achieving eye tracking.

[0140] In some embodiments, the electronic device 100 may further include an eye-tracking module 195, which tracks the movement of the human eye to determine the gaze point. For example, the pupil position can be located and the pupil center coordinates can be obtained using image processing technology, thereby calculating the person's gaze point.

[0141] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.).

[0142] In some embodiments of this application, the internal memory 121 may be used to store applications for one or more applications, including instructions. When the application is executed by the processor 110, it causes the electronic device 100 to generate content for presentation to a user. For example, the application may include an application for managing the head-mounted display device 200, a game application, a conferencing application, a video application, a desktop application, or other applications, etc.

[0143] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0144] Random Access Memory (RAM) is characterized by its high read / write speeds and volatility. Volatility means that the data stored in RAM is lost when power is off. Typically, RAM has extremely low static power consumption but relatively high operating power consumption. The data in RAM, also known as memory data, can be read at any time but is lost when power is off.

[0145] Non-volatile memory (NVM) is characterized by its non-volatile nature and stable data storage. Non-volatility means that the stored data will not disappear after a power outage, allowing for long-term data preservation even without power. Data in NVM includes application data, which can be stably stored in NVM for extended periods. Application data refers to content written during the execution of applications or service processes, such as photos or videos captured by camera applications, or text edited by users in document applications.

[0146] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.

[0147] Non-volatile memory can include magnetic disk storage, flash memory, etc.

[0148] Disk storage devices are storage devices that use disks as storage media. They are characterized by large storage capacity, high data transfer rate, and long-term data preservation.

[0149] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0150] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0151] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0152] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0153] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0154] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0155] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0156] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0157] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0158] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0159] Electronic device 100 may include one or more buttons 190, which can control the electronic device and provide users with access to functions on the electronic device 100. The buttons 190 may be mechanical keys such as buttons, switches, or dials, or they may be touch or proximity sensing devices (such as touch sensors). Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of the electronic device 100. Buttons 190 may include a power button, volume buttons, etc.

[0160] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the electronic device 100. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0161] Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, or to indicate messages, notifications, etc.

[0162] Electronic device 100 may also include other input / output interfaces, through which other devices can be connected to electronic device 100. Components may include, for example, audio / video jacks, data connectors, etc.

[0163] The electronic device 100 is equipped with one or more sensors, including but not limited to a pressure sensor 180A, a gyroscope sensor 180B, an acceleration sensor 180C, a distance sensor 180D, a touch sensor 180E, etc.

[0164] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. When a touch operation is applied to electronic device 100, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0165] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can also be used for navigation, motion-sensing game scenarios, camera image stabilization, etc.

[0166] The accelerometer 180C can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to motion-sensing games, screen orientation switching, pedometers, and other applications.

[0167] In some embodiments of this application, the electronic device 100 can track the movement of the user's head based on an accelerometer, a gyroscope, or the like.

[0168] A distance sensor 180D is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, during a shooting scene, the electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0169] Touch sensor 180E, also known as a "touch device," is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display device 194.

[0170] In this embodiment, the processor 110 can be used to determine the content seen by a user through the display device of the electronic device 100 based on data acquired by the sensor from the electronic device 100. The GPU is used to perform mathematical and geometric operations based on data obtained from the processor 110 (e.g., data provided by an application), and to render images using computer graphics technology, computer simulation technology, etc., to determine the image to be displayed on the electronic device 100. In some embodiments, the GPU can add correction or pre-distortion to the image rendering process to compensate for or correct distortion caused by the optical components of the electronic device 100.

[0171] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture. Taking the system as an example, the software structure of electronic device 100 is illustrated.

[0172] Figure 4 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0173] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, this can be... The system is divided into five layers, from top to bottom: applications, applications framework, native libraries, and the Android runtime. Hardware Abstraction Layer (HAL) and Kernel.

[0174] The application layer can include a series of application packages. Application packages can include applications such as camera, gallery, games, WLAN, Bluetooth, music, and video.

[0175] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0176] The application framework layer may include a window manager, an activity manager, a display manager, a resource manager, an input manager, a notification manager, and a view system, etc.

[0177] The window manager is used to manage window programs. It can be used to draw the size and position of windows, control the display or hiding of windows, manage the display order of multiple windows, obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen.

[0178] The Activity Manager is used to manage the lifecycle of application activities, such as managing the creation, background, and destruction of activities.

[0179] The display manager manages the lifecycle of an application's display. It can determine how to control its logical display based on the currently connected physical display device, and send notifications to the system and applications when the state changes, etc.

[0180] The input manager is used to listen for and manage input events in a unified manner. For example, when it detects that a user is inputting using a handheld controller, or when a sensor detects the user's motion data, the input manager can listen for system calls and further forward or process the input events it has listened for.

[0181] The resource manager provides applications with access to various non-code resources, such as localized strings, icons, images, layout files, video files, and so on. The resource management interface class `ResourceImpl` serves as the external resource management interface, allowing applications to update their resources.

[0182] The notification manager allows applications to display notification information in the status bar. It can be used to convey informational messages and can disappear automatically after a short time without user interaction.

[0183] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a notification's display interface may include views for displaying text and views for displaying images.

[0184] Runtime includes core libraries and a virtual machine. The runtime is responsible for scheduling and management of the Android system.

[0185] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0186] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0187] Native libraries can include multiple functional modules. Examples include: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL).

[0188] The Surface Manager manages the display subsystem and provides the blending of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications. The Surface Manager accesses the SurfaceFlinger service, the core of the graphical user interface (GUI), which is responsible for sequentially blending and outputting the graphical data from all applications to a buffered stream.

[0189] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0190] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0191] A 2D graphics engine is a graphics engine for 2D drawing.

[0192] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and the hardware circuitry. Its purpose is to abstract the hardware, encapsulating Linux kernel drivers, providing a standard interface, and hiding the implementation details of lower-level drivers. The HAL can contain hardware composers (HWC), framebuffers, sensor interfaces, Bluetooth interfaces, WiFi interfaces, audio / video interfaces, and more.

[0193] The hardware composer (HWC) is used to perform the final composite display on the buffered stream of graphics data composed by SurfaceFlinger.

[0194] The framebuffer is a graphics buffer stream synthesized by the SurfaceFlinger service. The SurfaceFlinger service draws images by writing content to the framebuffer.

[0195] The sensor interface can be used to acquire relevant data from the sensor driver.

[0196] The kernel layer is the layer between hardware and software, providing core system services such as security, memory management, process management, network protocol stack, and driver models. The kernel layer can contain display controller drivers, sensor drivers, camera drivers, audio drivers, video drivers, and so on. Drivers communicate with hardware devices via the bus, controlling the hardware to enter various operating states and obtaining values ​​from relevant device registers to determine the device's status. For example, drivers can acquire user operation events, such as sensor inputs and camera inputs, and convert these events into data.

[0197] In this embodiment, software and hardware can work together to generate and display corresponding images when the head-mounted display device 200 detects user movement. Specifically, in one example, when the gyroscope sensor 180B and the accelerometer sensor 180C detect an input event of user head rotation, a corresponding hardware interrupt is sent to the kernel layer. The sensor driver in the kernel layer obtains the sensor input data. The input manager in the application framework layer obtains the sensor input event from the kernel layer. Then, the window manager may need to update the size or position of the application window, and the activity manager updates the lifecycle of the display activity. After the window manager adjusts the size and position of the window and the active display interface, the display manager refreshes the image in the window area. After the window activity is updated, the surface manager uses the SurfaceFlinger service to sequentially mix the GPU-rendered graphics data to generate a graphics buffer stream and output it to the frame buffer. The frame buffer then sends the graphics buffer stream to the hardware compositor in the hardware abstraction layer, where the hardware compositor performs the final compositing of the graphics buffer stream. The hardware compositor sends the final graphics data to the display driver in the kernel layer for display by the display device 194.

[0198] The above description of the software architecture is merely illustrative. It is understood that the software architecture illustrated in this embodiment does not constitute a specific limitation on this application. In other embodiments of this application, the software architecture of the electronic device 100 may include more or fewer modules than illustrated, or combine certain modules, or split certain modules, or have different architectural arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0199] Figure 5 A communication system 30 is shown; in some embodiments, this communication system 30 can be a split-type system. The communication system 30 may include multiple smart terminal devices, and these multiple terminal devices establish communication connections with each other. For example... Figure 3 As shown, in some embodiments, the communication system 30 includes a head-mounted display device 200 and a host 310. The host 310 can be, for example, a device with strong image processing capabilities such as a computer A or a mobile phone A, and is not limited to these. Figure 3 In the example, computer A or mobile phone A, host 310 can be one or more other devices with strong image processing capabilities, such as a server. Host 310 can also be one or more cloud devices, such as a cloud host / cloud server; this embodiment does not limit this. A first connection is established between head-mounted display device 200 and host 310. Head-mounted display device 200 is mainly used to display images, and host 310 mainly provides image processing functions, then sends the images to head-mounted display device 200 for display through the first connection. This first connection can be a wired connection or a wireless connection; this embodiment does not limit this. In the following embodiments, terminal device can also be simply referred to as terminal. Terminal device is typically an intelligent electronic device that can provide a user interface, interact with the user, and provide business functions to the user.

[0200] Each terminal device in the communication system 30 can be equipped with system, system, system, The operating systems of the various terminal devices in the communication system 30 can be the same or different, whether they are HarmonyOS (HOS) or other types of operating systems. This application does not impose any restrictions on this. In some embodiments, multiple terminals in the communication system 30 are equipped with a system... A system, then a system composed of multiple terminals can be called a Super virtual device (also known as) Super terminals refer to the integration of the capabilities of multiple terminals through distributed technology, storing them in a virtual hardware resource pool. Based on business needs, the terminal capabilities are uniformly managed, scheduled, and integrated to provide services to the outside world, enabling rapid connection, capability mutual assistance, and resource sharing between different terminals.

[0201] The first connection can include wired connections, such as High Definition Multimedia Interface (HDMI) connections, DisplayPort (DP) connections, etc. It can also include wireless connections, such as Bluetooth (BT) connections, Wireless Fidelity (Wi-Fi) connections, hotspot connections, etc., enabling communication between the head-mounted display device 200 and the host 310 under the same account, no account, or different account conditions. The first connection can also be an Internet connection. In some embodiments, the head-mounted display device 200 and the host 310 can log in to the same account, thereby connecting and communicating via the Internet. Of course, multiple terminals can also log in to different accounts, but connect through a binding method. For example, the head-mounted display device 200 and the host 310 can log in to different accounts, and the host 310 can bind the head-mounted display device 200 to itself in a device management application, and then connect through the device management application. This application embodiment does not limit the type of the first connection; various terminals in the communication system 300 can transmit and interact with data through various communication connection types. Furthermore, the various terminals can also connect and communicate using any of the above methods, and this application embodiment does not impose any restrictions on this.

[0202] Correspondingly, each terminal device can be configured with a mobile communication module and a wireless communication module for communication. The mobile communication module can provide wireless communication solutions including 2G / 3G / 4G / 5G for terminal applications. The wireless communication module can include a Bluetooth (BT) module and / or a wireless local area network (WLAN) module. The Bluetooth module can provide solutions for one or more Bluetooth communication methods, including Bluetooth Classic (Bluetooth 2.1) or Bluetooth Low Energy (BLE). The WLAN module can provide solutions for one or more WLAN communication methods, including Wireless Fidelity Peer-to-Peer (Wi-Fi P2P), Wireless Fidelity Local Area Networks (Wi-Fi LAN), or Wireless Fidelity Software Access Point (Wi-Fi softAP). In some embodiments, Wi-Fi P2P refers to allowing devices in a wireless network to connect to each other in a point-to-point manner without needing a wireless router. This system is also known as Wireless Fidelity Direct (Wi-Fi Direct). Devices establishing a Wi-Fi P2P connection can exchange data directly via Wi-Fi (must be on the same frequency band) without being connected to a network or hotspot, enabling point-to-point communication such as transferring files, pictures, and videos. Compared to Bluetooth, Wi-Fi P2P has advantages such as faster search and transmission speeds, and longer transmission distances.

[0203] It should be noted that, Figure 3 The communication system 30 shown is only used to assist in describing the technical solutions provided in the embodiments of this application and does not constitute a limitation on the embodiments of this application. In actual business scenarios, the communication system 30 may include more or fewer terminal devices, such as handheld controllers. This application does not impose any limitations on terminal type, number of terminals, connection method, etc. For example, in a scenario of PC screen mirroring to a mobile phone, the communication system 30 includes a PC and a mobile phone.

[0204] Head-mounted display devices inevitably experience MTP latency during use. MTP latency refers to the total delay in one cycle from the moment the user's head moves until the optical signal of the corresponding new image displayed on the head-mounted display device reaches the human eye. This includes the time it takes for the sensor to detect the user's movement, for the central processing unit (CPU) to run the application, for the graphics processing unit (GPU) to calculate the image, render it, and send it to the display.

[0205] For example, in AR / VR or MR scenarios, the process from user movement to the user seeing the corresponding image can involve multiple modules, such as... Figure 6 As shown:

[0206] (1) Sensor detection module.

[0207] Sensors are used to detect user motion information and / or external environmental information, ensuring that the viewing angle tracks body movements and switches constantly. When the user moves, sensors (such as cameras, gyroscopes, accelerometers, magnetometers, etc.) on the head-mounted display or handheld motion controller can sensitively capture the user's movement and generate sensor data in real time, such as image data, acceleration data, angular velocity data, etc.

[0208] (2) Motion tracking algorithm module.

[0209] In some embodiments, the processor can use the sensor data generated by the previous sensor as input, and obtain the user's real-time pose data through data fusion and computation. Pose refers to the position and orientation of a person or object, including posture and orientation.

[0210] When a user's pose changes, their field of view changes accordingly. Specifically, pose can be the user's head pose. Pose can be acquired through sensors and / or cameras in a head-mounted display device.

[0211] If the head-mounted display device is a 3-DOF (degree of freedom, DOF) device, then the output pose data includes quaternion data corresponding to rotation (rotation along the X, Y, and Z axes). If the head-mounted display device is a 6DOF device, then the output pose data includes quaternion data corresponding to rotation (rotation along the X, Y, and Z axes) and three-axis position data (up / down, forward / backward, and left / right movement). Quaternions can be used to represent rotation and orientation in three-dimensional space. For example, the quaternion q = ((vx, vy, vz), w) = (v, w), where v is a vector, w is a real number, and q can represent a rotation operation of an angle w about the vector v = (vx, vy, vz) (using the right-hand rule).

[0212] (3) Image rendering module.

[0213] Based on the pose data and the image to be rendered in the corresponding pose, the rendering module processes the image by performing anti-distortion and other methods before sending it to a processor such as a GPU for rendering. This involves coordinate transformation, view transformation, layer rendering, texture compositing, shading, cropping, and rasterization. For VR wearable devices, two separate images can be rendered for the left and right eyes.

[0214] In one embodiment, when a user wears a VR headset, they may move or turn their head. To make the virtual environment more realistic, the image needs to be processed accordingly when the VR headset moves or turns its head, providing the user with a realistic experience. Therefore, in the VR field, image rendering includes rendering the image's color and transparency, as well as rendering the image by rotating and / or translating it based on the human pose data detected by the VR headset. The pose detected by the VR headset includes multiple degrees of freedom, such as rotation angle and / or translation distance. The rotation angle includes yaw angle, pitch angle, and roll angle, and the translation distance includes the translation distance relative to the three axes (X-axis, Y-axis, Z-axis). Therefore, image rendering includes rotating the image based on the VR headset's rotation angle and / or translating the image based on the VR headset's translation distance.

[0215] For example, when a user wears a VR headset facing forward, objects in the rendered image (such as mountains or water) are positioned directly in front of them; when the user's head rotates to the right (e.g., 40 degrees), the objects in the image (such as mountains or water) rotate to the left by 40 degrees. In this way, the image seen by the user is synchronized with the user's movement and perspective, resulting in a better experience.

[0216] (4) Image display module.

[0217] The image delivery module is used to send image data (multiple pixel data) rendered by a processor such as a GPU to a display device. In some embodiments, the display device includes a framebuffer. The framebuffer, also known as video memory, is used to store image rendering data processed by the GPU or about to be extracted. Sending rendered data to the display device by the electronic device can mean sending it to the framebuffer in the display device. In other embodiments, the display device may not have a framebuffer; in this case, sending rendered data directly to the display device can mean sending the image data (multiple pixel data) for display.

[0218] After the GPU submits the image rendering data to the frame buffer, in conjunction with the vertical sync (Vsync) signal for screen refresh, the video controller retrieves the image data from the frame buffer at a specified time point after the Vsync signal. Then, after receiving the display signal, the display screen and other optical display devices retrieve the frames from the buffer and display them on the screen.

[0219] When a display screen is used to display the screen image, it scans the screen line by line from left to right and from top to bottom to display the pixels sequentially. After scanning a line, a horizontal synchronization (Hsync) signal is sent. After scanning a page, a frame is displayed and a Vsync signal is sent, and the scanning of the next page begins.

[0220] The Vsync signal is a pulse signal, typically generated by a hardware clock, but it can also be simulated by software (such as a hardware synthesizer HWC).

[0221] Therefore, normally, when the Vsync mechanism is enabled, electronic devices wait for the Vsync signal to be sent before rendering a new frame and updating the frame buffer. This is to resolve screen tearing and increase screen smoothness. If the data in the current frame buffer is not fully updated and still retains data from the previous frame, then when the screen refreshes, the frames fetched from the frame buffer are from different frames, resulting in screen tearing.

[0222] The display principles of different types of displays are not entirely the same. Common display types include: liquid crystal displays (LCDs), organic light-emitting diodes (OLEDs), and plasma display panels (PDPs), etc. For example, for LCD displays, changing the electric field (voltage) alters the arrangement of liquid crystal molecules, thereby changing the transmittance of external light sources (incident light beams) through the liquid crystals. Color display is then achieved by adjusting color filters (red, green, and blue primary color filters). For OLED displays, an organic light-emitting layer is sandwiched between positive and negative electrodes. Under the influence of an electric field, holes generated at the anode and electrons generated at the cathode move and are injected into the hole transport layer and electron transport layer, respectively, migrating to the light-emitting layer. When holes and electrons meet in the light-emitting layer, they generate excitons, which excite the light-emitting molecules to ultimately produce visible light. Adding color filters completes the color display. The display principles and hardware structures of different types of displays are not entirely the same, and will not be elaborated further here. Different types of displays do not constitute a limitation on the embodiments of this application.

[0223] The following is combined Figure 7 Here is a brief introduction to the image display process in some embodiments.

[0224] like Figure 7 As shown, drawing an image typically requires the collaboration of the CPU and GPU. For example, the CPU can be responsible for calculating the display content related to the image, such as view creation, layout calculation, image decoding, text drawing, pose data calculation, and texture data calculation. The CPU then passes the calculated content to the GPU, which performs transformation, compositing, and rendering operations.

[0225] The GPU can be used to handle layer rendering, texture composition, shader rendering, etc., which can include: vertex shaders processing vertex data, such as translation, scaling, and rotation, performing various model transformations, view transformations, and projection transformations, and converting it into standardized coordinates; tessellation shaders and geometry shaders depicting the shape of objects, processing the geometry of the model to make it smoother; assembling primitives from the processed data, turning the data into primitives; clipping primitives according to the viewport, removing areas not visible to the viewport; rasterizing primitives to generate display coordinates and pixel fragments; fragment shaders coloring the pixel fragments; and rendering processes such as texture blending after fragment shader processing, finally obtaining the pixel data of the image.

[0226] The GPU sends the pixel data of the image to the frame buffer.

[0227] The video controller extracts the pixel data of the image frame from the frame buffer at a specified time based on the Vsync signal, and then displays the image frame on the display screen.

[0228] Typically, the drawing, rendering, and displaying of each frame of an image requires coordination with the Vsync signal. Within each Vsync cycle, a complete frame of an image needs to be rendered and sent to the frame buffer; otherwise, display stuttering, frame drops, and screen tearing will occur.

[0229] (5) Data transmission module.

[0230] In some embodiments, when the AR / VR or MR device has a split structure, such as in the communication system 30, it may also include a data transmission module. The data transmission module can be used to send or receive image-related data via a wired or wireless connection. For example, in one example, when a user wears the head-mounted display device 200 and moves their head, the sensor module captures the user's head movement, generates sensor data, and sends it to the host 310 via the data transmission module. The motion tracking algorithm module in the host 310 processes the sensor data, generates pose and other data, and then sends it to the head-mounted display device 200 via the data transmission module. The image rendering module in the head-mounted display device 200 can render the pose and other data, then send the rendered data to the frame buffer, and finally display it on the screen. In another example, the rendering module can be located on the host 310. When the user wears the head-mounted display device 200, their head moves. The sensor module captures the user's head movement and generates sensor data, which is then sent to the host 310 by the data transmission module. The motion tracking algorithm module in the host 310 processes the sensor data and generates pose data. The image rendering module in the host 310 can render the pose data, and then send the rendered data to the head-mounted display device 200 through the data transmission module. Finally, the display module extracts the rendered image data and displays it on the screen.

[0231] Therefore, from the moment a user moves until they finally see the corresponding image, multiple stages are involved, including sensor data acquisition, image acquisition, image transmission, image rendering, and image display, each of which consumes a certain amount of time. When the MTP latency is high, the user's visual perception of their own body state is inconsistent with the vestibular system in the middle ear responsible for perceiving that state. In other words, the visually observed image is slower than the body's perception, creating a conflict that causes dizziness. Industry-recognized research shows that when the MTP latency is controlled below 20ms, it can significantly reduce dizziness in VR / AR or MR scenarios.

[0232] Therefore, by reducing the MTP latency of head-mounted display devices, human-computer interaction performance can be improved, thereby reducing adverse reactions such as dizziness and vomiting, and enhancing the user experience.

[0233] Some solutions for reducing MTP latency optimize each step, reducing the time spent at each stage. Examples include increasing the frame rate of image acquisition, providing predicted poses, improving GPU rendering performance, and increasing the display refresh rate to 75Hz or higher. These solutions are typically implemented based on a serial workflow, with each step processing a complete image as the data unit.

[0234] based on Figure 6 The process module shown in this example can decompose the MTP latency into various stages. Figure 8 The time consumption analysis for each module is shown. For example... Figure 8 As shown:

[0235] (1) Sensor Detection Module: The sensor detection module is used to detect user motion information and / or external environmental information. Here, we take a camera and an IMU as examples of sensors. Generally, images are generated at a frequency of 30Hz. Assuming the exposure time of one frame is 10ms, and the image timestamp is placed in the middle of the exposure, there is a 5ms delay between the timestamp and the final image. For the IMU sensor, its frequency is 1000Hz, so the IMU generation can be considered to have almost no delay. However, due to the large amount of image data, there is also a certain delay from image generation to transmission to the processor, assumed to be 5ms. Therefore, the total time from sensor data generation to transmission completion is 10ms.

[0236] (2) Motion tracking algorithm module: The processor can calculate the user's pose data based on sensor data. Assume that this step takes 5ms to complete the calculation. Of course, in this module, we can reduce the calculation time through some optimization methods, and even reduce the overall MTP latency by predicting the pose. However, for now, we only consider the real-time pose calculation, that is, there is a 5ms delay in this module.

[0237] (3) Image rendering module: Based on the pose data and the image to be rendered in the corresponding pose, the GPU and other processors can render the image. Assuming the final screen refresh rate is 90Hz, the rendering time for each frame must be less than 1s / 90Hz = 11.1ms. Here we assume the rendering time is 8ms.

[0238] (4) Image Sending and Display Module: The image sending and display module is used to send the rendered image frame data to the display device. Since the final image data to be sent and displayed is large, it is assumed that the transmission and display will take 8ms.

[0239] In one embodiment, the total time for the above four steps to be performed sequentially is 31ms, that is, the MTP latency from the user's head movement to the final display of the image is 31ms. Table 1 shows the time consumption analysis of each step of the MTP latency in the corresponding AR scenario.

[0240]

[0241] Table 1

[0242] The generation of sensor data and the calculation of pose, i.e., the work done by the sensor detection module and the motion tracking algorithm module, are collectively referred to as the image acquisition stage. After image acquisition, image rendering and image display are performed. Here, the previous steps can be considered as follows: Figure 8 The steps described in Table 1 are simplified into three stages: image acquisition, image rendering, and image display, with corresponding time consumption of 15ms, 8ms, and 8ms respectively. Figure 9 As shown.

[0243] In other embodiments, such as in the application scenario of a split-type machine, data transmission between the host and the head-mounted display device will also take a certain amount of time. In some embodiments, the time spent on data transmission can also be included in the time spent on the image acquisition stage.

[0244] In the above embodiments, the process from image acquisition, image rendering to image display is implemented serially, and the smallest image unit of each step is a complete image. There is redundancy in time, which can be further optimized.

[0245] In other embodiments, this application provides an image transmission and display method that optimizes the display path by dividing a whole image into multiple parts, performing parallel image rendering and / or parallel image display processing on these multiple parts, and / or transmitting these multiple image parts in parallel among multiple devices, thereby reducing the waiting time in the image transmission and display process and further reducing the latency of image transmission and display. See below for a detailed description.

[0246] Implementing the method provided in this application can reduce latency during image transmission and display, allowing images to be displayed faster. Furthermore, in AR / VR or MR scenarios, it can reduce MTP latency, effectively alleviating adverse reactions such as motion sickness and vomiting, and improving the user experience. It is understood that the method provided in this application can be applied to more scenarios, such as PC screen projection and in-vehicle device image display. Through parallel processing of segmented transmission, rendering, and display, it accelerates the end-to-end display process, reduces latency, and improves the user's viewing experience.

[0247] This application also provides related electronic devices or systems that can apply the image transmission and display methods provided in this application. The electronic devices may include the aforementioned head-mounted display devices, which can achieve AR, VR, MR, and other display effects. However, this application is not limited to these; the electronic devices involved in the embodiments of this application may also be other devices including a display screen, such as mobile phones, tablets, PCs, smart TVs, in-vehicle devices, and other terminal devices. This application does not impose any restrictions on the specific type of electronic device. The communication system 300 in the foregoing example does not constitute any limitation on other embodiments of this application.

[0248] It is understood that the embodiments provided in this application are mainly described using head-mounted display devices as examples in VR scenarios. However, the image transmission and display methods provided in the embodiments of this application are not limited to head-mounted display devices and VR scenarios. More generally, the image transmission and display methods provided in this application can be universally applied to the transmission and display of images by various types of electronic devices (such as mobile phones, PCs, PADs, in-vehicle devices, game consoles, smart wearable devices, smart home devices, IoT devices, etc.).

[0249] The image transmission and display method provided in this application can be applied to various business scenarios, including but not limited to:

[0250] (1) AR, VR, MR scenarios

[0251] As mentioned above Figure 2 In the scenario depicted, users can wear head-mounted display devices. These devices can utilize VR, AR, and MR technologies to display images, allowing users to experience a virtual environment and providing a VR / AR / MR experience. In traditional serial image transmission, the MTP latency is significant, easily causing adverse reactions such as dizziness and vomiting. Using the segmented image transmission and display method provided in this application, VR / AR / MR images can be displayed faster, MTP latency is reduced, effectively alleviating motion sickness symptoms such as dizziness and vomiting, and improving the user experience.

[0252] (2) Screen mirroring scenarios between mobile phones and PCs or large screens

[0253] For example, when a mobile phone is connected to a PC or large screen, and the mobile phone screen is projected onto the PC or large screen, the segmented image transmission and display method provided in this application allows the PC or large screen to display the mobile phone screen more quickly, providing users with a smoother screen projection experience.

[0254] (3) Display scenarios of in-vehicle equipment

[0255] For example, a smart in-vehicle display screen can generate new images based on user operations, vehicle sensors, or other input information. Using the segmented image transmission and display method provided in this application, the in-vehicle display screen can display the corresponding images faster, providing users with a faster and more comfortable driving experience.

[0256] The scenarios described above are merely illustrative and do not constitute any limitation on other embodiments of this application. Not limited to the scenarios described above, the image transmission and display method provided in this application can be applied to any scenario requiring image display, such as other social scenarios, office scenarios, or shopping scenarios.

[0257] The following describes the specific implementation process of the image transmission and display technology solution provided in the embodiments of this application. The following solution can be applied to the various business scenarios mentioned above.

[0258] The image transmission and display technology solution provided in this application optimizes the display path by dividing a whole image into multiple parts for parallel processing of transmission, rendering, and display. This reduces waiting time during image transmission and display, further lowering latency. In AR / VR or MR scenarios, it can reduce MTP latency, effectively alleviating adverse reactions such as dizziness and vomiting, and improving the user experience.

[0259] Compared to Figure 9 The image transmission and display process shown in this application's technical solution maintains the same time for the image acquisition stage. However, in the image rendering and image display stages, the smallest processing unit is no longer a whole image, but rather the image is divided into multiple parts, which are rendered and displayed in parallel and staggered.

[0260] Figure 10 The diagram illustrates the parallel rendering and display process provided in some embodiments.

[0261] After acquiring sensor data and processing pose data, the head-mounted electronic device 200 obtains image data of an entire image for rendering. For example... Figure 11 As shown, suppose the entire image is divided horizontally into four parts from top to bottom: Slice1, Slice2, Slice3, and Slice4. In one example, the entire image is divided equally into Slice1, Slice2, Slice3, and Slice4. The rendering time for the entire image is 8ms, and the display time is 8ms. Therefore, the rendering time for each of the equally divided Slices1, Slice2, Slice3, and Slice4 is 2ms, and the display time is 2ms.

[0262] The GPU and other processors first render Slice1, which takes 2ms. After Slice1 is rendered, the rendered image pixel data is sent to the display device (such as a display screen), which takes 2ms. If the display device includes a frame buffer, then sending the rendered image pixel data to the frame buffer can refer to sending the rendered image pixel data to the frame buffer. While Slice1 is being rendered and sent, the GPU and other processors render Slice2, which takes 2ms, and then send it to the display, which takes 2ms. While Slice2 is being rendered and sent, the GPU and other processors render Slice3, which takes 2ms, and then send it to the display, which takes 2ms. While Slice3 is being rendered and sent, the GPU and other processors render Slice4, which takes 2ms, and then send it to the display, which takes 2ms.

[0263] In some embodiments, the display screen includes a frame buffer. After Slice1, Slice2, Slice3, and Slice4 have been displayed, the display screen extracts the complete data of the first image from the frame buffer and displays it in conjunction with the Vsync signal. Figure 12 As shown, after Slice1 is displayed, the frame buffer is updated with the image data of Slice1; after Slice2 is displayed, the frame buffer is updated with the image data of Slice1 and Slice2; after Slice3 is displayed, the frame buffer is updated with the image data of Slice1, Slice2, and Slice3; after Slice4 is displayed, the frame buffer is updated with the image data of Slice1, Slice2, Slice3, and Slice4, which is the complete data of the first image. In conjunction with the display signal, the display screen extracts the pixel data of the first image from the frame buffer and displays the first image completely.

[0264] It's important to note that all segments of the image (Slice1, Slice2, Slice3, and Slice4) must be sent to the display before the screen refreshes. This allows the display to extract the complete image data of the first image from the frame buffer for display. If only the image data for Slice1, Slice2, and Slice3 is updated in the frame buffer before the screen refreshes, then during the refresh, the areas corresponding to Slice1, Slice2, and Slice3 will display the updated images, while the Slice4 area will display a remnant of the previous frame image that hasn't been updated, resulting in screen stuttering and tearing.

[0265] In other embodiments, the display screen does not have a frame buffer, such as... Figure 12As shown, after Slice1 is displayed, the screen directly displays the image of Slice1; after Slice2 is displayed, the screen displays the images of Slice1 and Slice2; after Slice3 is displayed, the screen displays the images of Slice1, Slice2, and Slice3; after Slice4 is displayed, the screen displays the images of Slice1, Slice2, Slice3, and Slice4, that is, the first image is displayed completely.

[0266] After the parallel processing of image rendering and image display in the above-described manner, as follows: Figure 10 As shown, the entire image took 10ms from rendering to full display, compared to... Figure 9 The serial processing method for image rendering and image display shown takes 16ms, which can save 6ms of time.

[0267] This embodiment does not impose any restrictions on how each image is divided. It can be divided into four, six, or any other number of slices, horizontally, vertically, or in any direction, and can be divided evenly or unevenly. Generally, evenly dividing the image can make better use of the parallel processing capabilities of each module and save more time than unevenly dividing the image. When dividing evenly, each slice is approximately the same size, while when dividing unevenly, the slices can be of different sizes.

[0268] refer to Figure 13 Here is a comparison Figure 10 This example illustrates the time consumption when an image is not evenly divided. Assume the rendering time for a complete image is 8ms and the display time is 8ms. In one example, the image is unevenly divided into five slices (Slice 5, Slice 6, Slice 7, and Slice 8) in a 2:3:1:2 ratio. The rendering times for each slice are 2ms, 3ms, 1ms, and 2ms, respectively, and the display times are 2ms, 3ms, 1ms, and 2ms, respectively.

[0269] The GPU and other processors first render Slice 5, which takes 2ms. After Slice 5 is rendered, it is sent to the display, which takes 2ms. While Slice 5 is being rendered and sent to the display, the GPU and other processors render Slice 6, which takes 3ms, and then send it to the display, which takes 3ms. While Slice 6 is being rendered and sent to the display, the GPU and other processors render Slice 7, which takes 1ms. After Slice 6 is being sent to the display, Slice 7 is sent to the display, which takes 1ms. After Slice 7 is being rendered, the GPU and other processors render Slice 8, which takes 2ms. After Slice 7 is being sent to the display, Slice 8 is sent to the display, which takes 2ms.

[0270] After the parallel processing of image rendering and image display in the above-described manner, as follows: Figure 13 As shown, the entire image took 11ms from rendering to full display, compared to... Figure 9 The serial processing method for image rendering and image display shown takes 16ms, which can save 5ms of time, but is still faster than... Figure 10 The parallel processing method shown, which involves evenly dividing the image, takes an extra 1ms. This is because evenly dividing the image allows for more efficient parallel utilization of the rendering and display modules, reducing unnecessary idle time. When the image rendering processing capacity or the display speed is mismatched, an unevenly divided method can also be used.

[0271] The specific method of image partitioning can be customized by developers according to the actual situation. The best partitioning scheme is the one that takes the shortest total time for image rendering and delivery to the display when the GPU is operating at its maximum efficiency, that is, when power consumption and performance are in a good balance.

[0272] In some embodiments, in a VR scenario, after acquiring an image, the VR glasses render and display the image.

[0273] In VR scenarios, common image rendering processes include image pre-distortion and time warp.

[0274] Image distortion is a phenomenon caused by the inherent characteristics of optical lenses (such as convex lenses) in VR glasses, resulting in a loss of image quality. This occurs because light rays bend more far from the lens center than near it. The distortion is distributed along the lens radius and includes barrel distortion and pincushion distortion, among others.

[0275] Image pre-distortion is a preprocessing technique used to reverse the image distortion effect of the optical lenses in VR glasses, so that the image seen by the user is the normal image.

[0276] For example, the image distortion caused by the optical lenses in VR glasses can create a pincushion distortion effect on a normal image. Therefore, during the image rendering stage, pre-distortion processing can be performed to transform the normal image into a barrel distortion image. After the barrel distortion image undergoes the pincushion distortion effect through the optical lenses in the VR glasses, the image displayed to the user's eyes is the normal image.

[0277] Time warp is an image frame correction technique. When users wear VR glasses, rapid head movements can cause rendering delays—meaning the user's head has turned before the image is rendered, or the previous frame is being rendered. If the rendering time is too long, a frame is lost, resulting in image jitter. Time warp mitigates this delay by warping an image that was previously sent to the display. The most basic time warp is a direction-based warp that corrects image jitter caused by changes in head posture, and it can generate a new image frame with relatively few computational resources.

[0278] Time warp can generate an image to replace a frame that has not yet been rendered when the image rendering frame is not synchronized with the head movement, thus automatically filling the image frame and making the transition between the preceding and following image frames smooth.

[0279] In one example, such as Figure 14 In the VR image rendering process shown, a whole image is divided into Slice1, Slice2, Slice3, and Slice4 from top to bottom, as follows: Figure 14 The original image texture is shown in (a). In the slice rendering process, assuming the image is divided into a 4×4 grid for texture shading, then each slice is a 1×4 grid, such as... Figure 14 As shown in (b), each slice requires pre-distortion processing. For each image vertex in a slice, the distorted pixel position can be calculated using the original image's pre-distortion pixel position and the distortion formula. Pixels outside the vertices can have their distorted pixel positions calculated using interpolation methods (such as linear interpolation, bilinear interpolation, cubic spline interpolation, etc.), ultimately generating... Figure 14 The pre-distortion effect shown in (b) is shown in the image.

[0280] exist Figure 14 The VR image rendering process shown also includes time-warping processing. By predicting the direction of head movement, the acquired image is rotated accordingly to obtain a new image frame as the final display image, such as... Figure 14 The image shown in (c) is the pre-distorted and temporally warped image. Similarly, during the temporally warped rendering process, for each image vertex in a slice, the predicted pixel position of that vertex can be calculated. For pixels outside the vertices, the predicted pixel positions can be generated using interpolation methods (such as linear interpolation, bilinear interpolation, cubic spline interpolation, etc.).

[0281] Ultimately, the image presented to the user's eyes through the VR glasses' display and optical lenses provides... Figure 14 The final image shown in (d) is the final image.

[0282] Once each image is rendered within the GPU, it is sent to the display device; this process is called sending to the display.

[0283] When rendering and displaying sliced ​​images in parallel, it is necessary to pay attention to the hardware Vsync signal of the display screen; otherwise, image tearing and other phenomena may occur.

[0284] The hardware Vsync signal is a pulse signal emitted after the display has refreshed a complete frame of image. If the image rendering and display process in segments does not match the Vsync signal, the display may only refresh a portion of the image, while the unrefreshed portion still displays the previous frame, resulting in screen tearing.

[0285] For example, in In the system, SurfaceFlinger can composite image data in the buffer and then send it to the display device, or the hardware compositor (HWC) can use hardware to composite the image data and send it to the display. The Vsync signal is matched to the screen's refresh rate; when a Vsync signal arrives, the screen begins to refresh pixels from top to bottom and from left to right. Furthermore, the refresh of each line can be coordinated with the horizontal sync (Hsync) signal, and the pixels of each line can be transmitted in conjunction with the pixel clock (PCLK) signal.

[0286] When the display receives a pixel signal, the LCD screen writes each pixel's data from left to right and top to bottom, completing one screen refresh within one Vsync cycle. During the image fragmentation and display process, the electronic device can control the display time, retrieving pixels from fixed positions in the cache at fixed time points within a cycle and displaying them. For example, if the electronic device determines that the GPU can complete image rendering before a specified time point, then when that time point arrives, the electronic device's HWC will send the image pixels sequentially to fixed positions on the display for display.

[0287] Meanwhile, black frame insertion is also involved in the display process. Black frame insertion is used to prevent ghosting caused by visual persistence. Generally, it can be achieved by inserting a pure black image frame, turning off the display, or reducing the display brightness to zero.

[0288] In a specific example, such as Figure 15 As shown, within the interval between two vertical synchronization signals, the electronic device synchronously completes the rendering, transmission, and display of an entire frame of image. In some embodiments, in order to complete the transmission and display in a timely manner before the display signal arrives, image rendering can be performed in advance.

[0289] In some embodiments, upon receiving a Vsync signal, the electronic device can adjust the brightness of the entire display to zero to achieve black insertion, which takes approximately 80% of a Vsync cycle. For example, if the display refresh rate is 90Hz and its Vsync cycle is 11.1ms, then the black insertion time is approximately 8.9ms.

[0290] When the blackout is complete and the display is turned on, the electronic device needs to ensure that all the image frames to be displayed have been refreshed on the screen so that the user can see a new and complete image.

[0291] Therefore, as Figure 15 As shown, before black insertion ends, the electronic device needs to render and display all the segmented images (i.e., Slice1, Slice2, Slice3, and Slice4). After Slice1 is rendered, it is displayed; simultaneously, Slice2 is rendered. After Slice2 is rendered, it is displayed; simultaneously, Slice3 is rendered. After Slice3 is rendered, it is displayed; simultaneously, Slice4 is rendered. After Slice4 is rendered, it is displayed. At this point, the rendering and display of the entire image is complete, and the display process is in the black insertion phase. Once Slice1, Slice2, Slice3, and Slice4 have all been displayed, black insertion ends, the display lights up, and the screen shows the complete image composed of Slice1, Slice2, Slice3, and Slice4. The time cycles for image rendering and image display are both less than one Vsync cycle.

[0292] The following describes how to turn off the display and how to fix the focus of the display.

[0293] (1) Turn off the display screen

[0294] In this embodiment of the application, turning off the display screen can include any of the following: ① turning off the backlight power supply of the display screen. ② turning off the backlight power supply and the display panel power supply of the display screen. ③ turning off the backlight power supply, the display panel, the integrated circuit (IC), and the backlight driver IC of the display screen.

[0295] When the backlight driver IC is turned off, the processor still sends display data to the display panel through the screen driver IC and the backlight driver IC. However, because the backlight is off, the display screen cannot show an image. Since display data is continuously sent to the display panel, the recovery speed is fast when the backlight power is restored.

[0296] When the power supply to the display panel is turned off, the display panel cannot receive display data sent by the screen driver IC, and the initial configuration data of the display panel is lost. Re-energizing the display requires initializing the configuration of each pixel (such as assigning initial potential values). Therefore, the display recovery speed is slow, which saves power consumption of the display panel. When the response speed for display recovery is not critical, the display panel can be turned off to further save power.

[0297] When the backlight driver IC is turned off, it cannot receive backlight data from the processor. It also cannot receive color data from the screen driver IC. Similarly, restoring power to the backlight driver IC requires initialization. Likewise, when the screen driver IC is turned off, it cannot receive display data from the processor or send color data to the backlight driver IC. Restoring power to the screen driver IC also requires initialization. Therefore, restoring the display is slow.

[0298] In this embodiment of the application, turning off the display of an OLED screen can include any of the following: ① Turning off the power supply to the OLED display panel. ② Turning off the power supply to both the OLED display panel and the screen driver IC.

[0299] When the processor shuts off the power supply to the OLED display panel, it continues to send display data to the screen driver IC. However, because the power supply to the OLED display panel is off, the OLED screen cannot display images. Restoring power to the display requires initializing the configuration of each pixel (e.g., assigning initial potential values). Since display data is continuously sent to the OLED display panel, restoring power to the OLED display panel is fast.

[0300] When the screen driver IC is shut down, it cannot receive display data from the processor, nor can it send display data to the OLED display panel. Similarly, restoring power to the screen driver IC requires initialization configuration. Restoring power to the screen driver IC is a slow process.

[0301] Additionally, the processor can control the power supply to pixels in certain areas of the OLED display panel to be turned off. This prevents images from being displayed in those areas, thus enabling the shutdown of display areas on the screen.

[0302] (2) Fixed focus of the display screen

[0303] The image displayed on the screen can be perceived as a virtual image by the user's eyes. This virtual image can be focused at a certain distance in front of the user's eyes through the optical design of the screen, such as 2 meters or 4 meters. This distance can also be a range, such as 2-4 meters. Therefore, the image displayed on the screen appears to be focused at a fixed point in front of the user's eyes.

[0304] The above embodiments can be applied to situations where electronic devices display images independently, or to situations where multiple devices cooperate to display images.

[0305] In conjunction with the aforementioned communication system 30, Figure 16 A wireless communication system 40 is shown. The communication system 40 includes a computer A and a head-mounted display device 200, which transmit data via a wireless connection 41. Image rendering can be performed by either the computer A or the head-mounted display device 200.

[0306] In some embodiments, image rendering is performed by computer A. The head-mounted display device 200 transmits sensor data detected by sensors to computer A via wireless connection 41. Computer A performs pose calculations and other processing to obtain a frame of image to be rendered. Computer A then divides the entire frame into slices and renders each slice in parallel. After each slice is rendered, it is encoded and then transmitted to the head-mounted display device 200 via wireless connection 41. The head-mounted display device 200 receives the image data slices from computer A via wireless connection 41, decodes each slice in parallel, and then displays it.

[0307] Combination Figure 17 Explain the process of parallel fragmented image transmission in the above wireless transmission scenario.

[0308] In one example, the process from generating sensor data and calculating pose to obtaining a frame of image to be rendered is combined into image acquisition. After obtaining the first frame of image to be rendered, computer A can divide the first frame of image into four parts, namely Slice1, Slice2, Slice3, and Slice4.

[0309] Computer A first renders Slice1, and then encodes it. While Slice1 is being rendered and encoded, Computer A simultaneously renders Slice2 and encodes it. After encoding Slice1, Computer A transmits it to the head-mounted display device 200 via wireless connection 41. While transmitting Slice1 to the head-mounted display device 200, Computer A also encodes Slice2. While encoding Slice2, Computer A simultaneously renders and encodes Slice3. This process continues, with Computer A encoding Slice2 and transmitting it to the head-mounted display device 200 while simultaneously encoding Slice3. While encoding Slice3, Computer A simultaneously renders and encodes Slice4. After computer A completes the encoding of Slice3 and transmits Slice3 to head-mounted display device 200, computer A encodes Slice4 and then transmits it to head-mounted display device 200.

[0310] Similarly, after receiving Slice1, the head-mounted display device 200 decodes Slice1. After decoding Slice1, the head-mounted display device 200 displays Slice1. Simultaneously with decoding and displaying Slice1, the head-mounted display device 200 decodes the received Slice2. This process continues; after decoding Slice2, it displays Slice2, and simultaneously with displaying Slice2, the head-mounted display device 200 decodes the received Slice3. After decoding Slice3, it displays Slice3, and simultaneously with displaying Slice3, the head-mounted display device 200 decodes the received Slice4. After decoding Slice4, it displays Slice4.

[0311] After Slice1, Slice2, Slice3, and Slice4 have all completed their display transmissions, the head-mounted display device 200 can display the complete first image on the screen in conjunction with the display signal.

[0312] In other embodiments, image rendering is performed by a head-mounted display device 200. The head-mounted display device 200 transmits sensor data detected by sensors to computer A via wireless connection 41. Computer A performs pose calculations and other processing to obtain a frame of image to be rendered. Computer A then divides the entire frame of image into segments, encodes each segment, and transmits these segments to the head-mounted display device 200 via wireless connection 41. After receiving the segments from computer A via wireless connection 41, the head-mounted display device 200 decodes, renders, and displays each segment in parallel, ultimately displaying the final image.

[0313] Combination Figure 18 Explain the process of parallel fragmented image transmission in the above wireless transmission scenario.

[0314] In one example, the process of generating sensor data, calculating pose, and other steps leading to an unrendered first frame image is synthesized into an image acquisition. After acquiring a complete first frame image, computer A can divide the first frame image into four parts: Slice1, Slice2, Slice3, and Slice4.

[0315] Computer A first encodes Slice1, and then transmits it to the head-mounted display device 200 via wireless connection 41. While transmitting Slice1 to the head-mounted display device 200, Computer A simultaneously encodes Slice2. This process continues; while transmitting Slice2 to the head-mounted display device 200, Computer A simultaneously encodes Slice3. While transmitting Slice3 to the head-mounted display device 200, Computer A simultaneously encodes Slice4 and then transmits it to the head-mounted display device 200.

[0316] Similarly, after receiving Slice1, the head-mounted display device 200 decodes Slice1. After decoding Slice1, the head-mounted display device 200 renders and displays Slice1. While rendering Slice1, the head-mounted display device 200 decodes the received Slice2. This process continues; after decoding Slice2, it renders and displays Slice2, and while rendering Slice2, the head-mounted display device 200 decodes the received Slice3. After decoding Slice3, it renders and displays Slice3, and while rendering Slice3, the head-mounted display device 200 decodes the received Slice4. After decoding Slice4, it renders and displays Slice4.

[0317] After Slice1, Slice2, Slice3, and Slice4 have all completed their display transmissions, the head-mounted display device 200 can display the complete first image on the screen in conjunction with the display signal.

[0318] The above embodiments do not impose any limitations on image encoding and decoding technologies. In some embodiments, image encoding and decoding may employ technologies such as H.265, aiming to compress image size and accelerate image transmission. The encoding step, transmission step, and decoding step may also be combined into a single transmission step.

[0319] Throughout the transmission process, both the sending end (computer A) and the receiving end (head-mounted display device 200) process each stage of the image fragments (encoding, transmission, decoding, rendering, and display) in parallel, greatly reducing the latency during image transmission and display, allowing images to be displayed faster. For a detailed description of image rendering and image display, please refer to the preceding descriptions; they will not be repeated here.

[0320] It should be noted that, Figure 16 , Figure 17 , Figure 18 The description is merely illustrative and does not constitute a limitation on other embodiments of this application. The transmitting and receiving ends shown in communication system 40 can also be any other type of electronic device. The connection type between the transmitting and receiving ends is not limited to wireless connection; it can also be a wired connection or other connections. The content transmitted between the transmitting and receiving ends is not limited to images; it can also be files, videos, etc. The method of image segmentation is not limited to this example. Other scenarios based on the same scheme are all within the protection scope of this application.

[0321] Based on the foregoing embodiments, an image transmission and display method provided by this application is described below.

[0322] Example 1

[0323] This embodiment uses a communication system consisting of a first device and a second device as an example for illustration. The first device and the second device cooperate to display images, wherein the second device is the image sending end and the first device is the image receiving end. The first device includes a display device for displaying images. In Embodiment 1, image rendering is performed by the first device.

[0324] This application does not limit the type of device in any way, and the devices can be mobile phones, PCs, smart screens, head-mounted display devices, etc. It is understood that the method provided in this embodiment is merely an example and does not constitute any limitation on other embodiments of this application. Actual business scenarios may include more or fewer terminal devices. For example, the first device is the aforementioned computer A, and the second device is the aforementioned head-mounted display device 200. The first and second devices form a communication system 40, which can utilize VR, AR, MR, and other technologies to display images, allowing users to experience a virtual environment and providing VR / AR / MR experiences. This embodiment does not limit the operating systems of the first and second devices. The software systems of the first or second device include, but are not limited to, those of the second device. Or other operating systems.

[0325] Figure 19 This is a flowchart of the image transmission and display method provided in Embodiment 1, which specifically includes the following steps:

[0326] S101, the first device establishes a first connection with the second device.

[0327] In this embodiment, the first connection established between the first device and the second device may include a wired connection, such as an HDMI connection, a DP connection, or a USB connection; or a wireless connection, such as a Bluetooth connection, a Wi-Fi connection, or a hotspot connection; or an Internet connection, enabling communication between the first device and the second device under the same account, without an account, or with different accounts. This embodiment does not limit the type of the first connection. Furthermore, the first connection may also be a communication connection combining any of the above methods; this embodiment does not impose any restrictions on this.

[0328] In one implementation, the first device can establish a first connection with the second device based on Wi-Fi near-field networking communication, such as a Wi-Fi P2P connection.

[0329] After the first device and the second device establish a first connection, they can send data to each other. In this embodiment, the transmitted data is the data corresponding to multiple frames of images. Of course, this embodiment does not impose any restrictions on the type of data transmitted between the first device and the second device; in addition to images, video, audio, text, etc., can also be transmitted.

[0330] S102, the second device acquires the third image and divides the third image into the first image and the second image.

[0331] The third image acquired by the second device can be generated by the second device based on the sensor data fed back by the first device, or it can be an image generated by the second device itself. This embodiment does not limit the source or process of acquiring the third image.

[0332] After acquiring the third image, the second device can segment the third image before transmitting it, for example, by dividing the entire third image region into smaller, non-overlapping regions.

[0333] In this embodiment, the third image is divided into two parts, namely the first image and the second image, as an example for illustration. In other embodiments of this application, the third image may be divided into three or more parts, and the size of the segmented regions is not limited, nor is the basis for segmentation limited. The only requirement is that the segmented images can be combined to form a complete third image.

[0334] S103, the second device encodes the first image.

[0335] In order to improve transmission efficiency and reduce image distortion during image transmission, encoding and decoding can be performed before and after image transmission. The first device and the second device can negotiate the encoding and decoding format according to the type of the first connection and the content to be transmitted. This embodiment does not impose any restrictions on the encoding and decoding method, and in some cases, there may not even be an encoding and decoding process.

[0336] S104, the second device sends the first image to the first device.

[0337] After the second device finishes encoding the first image, it sends the first image to the first device.

[0338] S105, the second device encodes the second image.

[0339] After the first image is encoded, the second device encodes the second image while simultaneously sending the first image to the first device. Steps S104 and S105 occur simultaneously, meaning that the sending of the first image by the second device to the first device and the encoding of the second image by the second device are processed in parallel.

[0340] S106, the second device sends a second image to the first device.

[0341] After the second device finishes encoding the second image, it sends the second image to the first device.

[0342] S107, the first device decodes the first image.

[0343] After the first device receives the first image from the second device, it decodes the first image. Simultaneously, the second device also sends a second image to the first device. Steps S106 and S107 occur simultaneously; that is, the decoding of the first image by the first device and the sending of the second image by the second device are processed in parallel.

[0344] S108, the first device renders the first image.

[0345] After decoding the first image, the first device renders the first image.

[0346] S109, the first device decodes the second image.

[0347] After the first device finishes decoding the first image, it begins decoding the second image. Simultaneously, the first device renders the first image. Steps S108 and S109 occur simultaneously; that is, the rendering of the first image and the decoding of the second image by the first device are processed in parallel.

[0348] S110, the first device sends the first image to the display device.

[0349] After rendering the first image, the first device sends the first image to the display device, which can also be referred to as sending to display. If the display device includes a frame buffer, this step may refer to the first device sending the first image to the frame buffer.

[0350] S111, the first device renders the second image.

[0351] After rendering the first image, the first device renders the second image. Simultaneously, the first device displays the first image. Steps S110 and S111 occur simultaneously; that is, the first device displaying the first image and the first device rendering the second image are processed in parallel.

[0352] S112, the first device sends the second image to the display device.

[0353] After rendering the second image, the first device sends the second image to the display device. If the display device includes a frame buffer, this step may refer to the first device sending the second image to the frame buffer.

[0354] S113, the first device acquires image data of the first image and the second image, and displays a third image, the third image including the first image and the second image.

[0355] In some embodiments, the display device includes a frame buffer. After the first image and the second image have been displayed, the first device waits for black insertion to finish, acquires the image pixel data of the first image and the second image from the frame buffer, and the display screen lights up to display the complete third image, i.e., the third image includes the first image and the second image. Examples of image rendering and image display can be found in the foregoing embodiments and will not be repeated here.

[0356] In other embodiments, the display device does not have a frame buffer. The first device receives and responds to the first image signal of the first image and directly displays the first image. The first device receives and responds to the second image signal of the second image and then displays the second image. At this time, both the first image and the second image have been displayed, that is, the third image has been fully displayed.

[0357] In this embodiment, the third image is described by dividing it into two parts: a first image and a second image. Other embodiments that divide the image into more parts will not be described in detail. It is understood that regardless of how many parts the third image is divided into, the idea of ​​parallel processing is the same. At the same time, different stages (encoding, transmission, decoding, rendering, display, etc.) of different parts of the image can be processed in parallel, thereby reducing the latency in the image transmission and display process and allowing the image to be displayed faster.

[0358] Example 2

[0359] This embodiment uses a communication system consisting of a first device and a second device as an example for illustration. The first device and the second device cooperate to display images, wherein the second device is the image transmitting end and the first device is the image receiving end. The first device includes a display device for displaying images. Unlike Embodiment 1, in Embodiment 2, image rendering is performed by the second device.

[0360] This application does not limit the type of device in any way, and the devices can be mobile phones, PCs, smart screens, head-mounted display devices, etc. It is understood that the method provided in this embodiment is merely an example and does not constitute any limitation on other embodiments of this application. Actual business scenarios may include more or fewer terminal devices. For example, the first device is the aforementioned computer A, and the second device is the aforementioned head-mounted display device 200. The first and second devices form a communication system 40, which can utilize VR, AR, MR, and other technologies to display images, allowing users to experience a virtual environment and providing VR / AR / MR experiences. This embodiment does not limit the operating systems of the first and second devices. The software systems of the first or second device include, but are not limited to, those of the second device. Or other operating systems.

[0361] Figure 20 This is a flowchart of the image transmission and display method provided in Embodiment 2, which specifically includes the following steps:

[0362] S201, the first device establishes a first connection with the second device.

[0363] In this embodiment, the first connection established between the first device and the second device may include a wired connection, such as an HDMI connection, a DP connection, or a USB connection; or a wireless connection, such as a Bluetooth connection, a Wi-Fi connection, or a hotspot connection; or an Internet connection, enabling communication between the first device and the second device under the same account, without an account, or with different accounts. This embodiment does not limit the type of the first connection. Furthermore, the first connection may also be a communication connection combining any of the above methods; this embodiment does not impose any restrictions on this.

[0364] In one implementation, the first device can establish a first connection with the second device based on Wi-Fi near-field networking communication, such as a Wi-Fi P2P connection.

[0365] After the first device and the second device establish a first connection, they can send data to each other. In this embodiment, the transmitted data is the data corresponding to multiple frames of images. Of course, this embodiment does not impose any restrictions on the type of data transmitted between the first device and the second device; in addition to images, video, audio, text, etc., can also be transmitted.

[0366] S202, the second device acquires the third image and divides the third image into the first image and the second image.

[0367] The third image acquired by the second device can be generated by the second device based on the sensor data fed back by the first device, or it can be an image generated by the second device itself. This embodiment does not limit the source or process of acquiring the third image.

[0368] After acquiring the third image, the second device can segment the third image before transmitting it, for example, by dividing the entire third image region into smaller, non-overlapping regions.

[0369] In this embodiment, the third image is divided into two parts, namely the first image and the second image, as an example for illustration. In other embodiments of this application, the third image may be divided into three or more parts, and the size of the segmented regions is not limited, nor is the basis for segmentation limited. The only requirement is that the segmented images can be combined to form a complete third image.

[0370] S203, the second device renders the first image.

[0371] The specific rendering process can be found in the aforementioned embodiments, and will not be repeated here.

[0372] S204, the second device encodes the first image.

[0373] In order to improve transmission efficiency and reduce image distortion during image transmission, encoding and decoding can be performed before and after image transmission. The first device and the second device can negotiate the encoding and decoding format according to the type of the first connection and the content to be transmitted. This embodiment does not impose any restrictions on the encoding and decoding method, and in some cases, there may not even be an encoding and decoding process.

[0374] S205, The second device renders the second image.

[0375] While the first image is being encoded, the second device renders the second image. Steps S204 and S205 occur simultaneously, that is, the encoding of the first image by the second device and the rendering of the second image by the second device are processed in parallel.

[0376] S206, the second device sends the first image to the first device.

[0377] After the second device finishes encoding the first image, it sends the first image to the first device.

[0378] S207, the second device encodes the second image.

[0379] After the first image is encoded, the second device encodes the second image while simultaneously sending the first image to the first device. Steps S206 and S207 occur simultaneously, that is, the second device sending the first image to the first device and the second device encoding the second image are processed in parallel.

[0380] S208, the second device sends a second image to the first device.

[0381] After the second device finishes encoding the second image, it sends the second image to the first device.

[0382] S209, the first device decodes the first image.

[0383] After the first device receives the first image from the second device, it decodes the first image. Simultaneously, the second device also sends a second image to the first device. Steps S208 and S209 occur simultaneously; that is, the decoding of the first image by the first device and the sending of the second image by the second device are processed in parallel.

[0384] S210, the first device sends the first image to the display device.

[0385] After rendering the first image, the first device sends the first image to the display device, which can also be referred to as sending to display. If the display device includes a frame buffer, this step may refer to the first device sending the first image to the frame buffer.

[0386] S211, the first device decodes the second image.

[0387] After the first device finishes decoding the first image, it begins decoding the second image. Simultaneously, the first device displays the first image. Steps S210 and S211 occur simultaneously; that is, the first device displays the first image, and the first device decodes the second image—these processes are parallel.

[0388] S212, the first device sends the second image to the display device.

[0389] After rendering the second image, the first device sends the second image to the display device. If the display device includes a frame buffer, this step may refer to the first device sending the second image to the frame buffer.

[0390] S213, the first device acquires image data of the first image and the second image, and displays a third image, the third image including the first image and the second image.

[0391] In some embodiments, the display device includes a frame buffer. After the first image and the second image have been displayed, the first device waits for black insertion to finish, acquires the image pixel data of the first image and the second image from the frame buffer, and the display screen lights up to display the complete third image, i.e., the third image includes the first image and the second image. Examples of image rendering and image display can be found in the foregoing embodiments and will not be repeated here.

[0392] In other embodiments, the display device does not have a frame buffer. The first device receives and responds to the first image signal of the first image and directly displays the first image. The first device receives and responds to the second image signal of the second image and then displays the second image. At this time, both the first image and the second image have been displayed, that is, the third image has been fully displayed.

[0393] In this embodiment, the third image is described by dividing it into two parts: a first image and a second image. Other embodiments that divide the image into more parts will not be described in detail. It is understood that regardless of how many parts the third image is divided into, the idea of ​​parallel processing is the same. At the same time, different stages (encoding, transmission, decoding, rendering, display, etc.) of different parts of the image can be processed in parallel, thereby reducing the latency in the image transmission and display process and allowing the image to be displayed faster.

[0394] Example 3

[0395] This embodiment illustrates the use of a first device independently displaying an image as an example. This application does not limit the type of the first device; it can be a mobile phone, PC, smart screen, head-mounted display device, etc. The first device includes a display unit for displaying images. It is understood that the method provided in this embodiment is merely an example and does not constitute any limitation on other embodiments of this application. For example, the first device could be the aforementioned head-mounted display device 200 all-in-one machine, which can utilize VR, AR, MR, and other technologies to display images, allowing users to experience a virtual environment and providing VR / AR / MR experiences. This embodiment does not limit the operating system running on the first device, including but not limited to... Or other operating systems.

[0396] Figure 21 This is a flowchart of the image transmission and display method provided in Embodiment 3, which specifically includes the following steps:

[0397] S301, the first device acquires the third image and divides the third image into the first image and the second image.

[0398] The third image acquired by the first device can be a first image generated by the first device based on sensor data fed back by each sensor, or it can be a third image received from a cloud server. This embodiment does not limit the source or process of acquiring the third image.

[0399] After the GPU of the first device acquires the third image, it can segment the third image, that is, divide the entire third image region into smaller, non-overlapping regions.

[0400] In this embodiment, the third image is divided into two parts, namely the first image and the second image, as an example for illustration. In other embodiments of this application, the third image may be divided into three or more parts, and the size of the segmented regions is not limited, nor is the basis for segmentation limited. The only requirement is that the segmented images can be combined to form a complete third image.

[0401] This step is not necessarily performed by the GPU; it can also be performed by the CPU or other processor of the first device.

[0402] S302, the first device renders the first image.

[0403] For examples of image rendering in VR scenes, please refer to the aforementioned embodiments, which will not be repeated here.

[0404] S303, the first device sends the first image to the display device.

[0405] After rendering the first image, the first device sends the first image to the display device, also known as image delivery. If the display device includes a frame buffer, this step may refer to the first device sending the first image to the frame buffer. For an example description of image delivery in a VR scene, please refer to the foregoing embodiments, which will not be repeated here.

[0406] S304, the first device renders the second image.

[0407] After rendering the first image, the first device renders the second image. Simultaneously, the first device displays the first image. Steps S303 and S304 occur simultaneously; that is, the first device displaying the first image and the first device rendering the second image are processed in parallel.

[0408] S305, the first device sends the second image to the display device.

[0409] After the second image is rendered, the first device sends the second image to the display. If the display device includes a frame buffer, this step may refer to the first device sending the second image to the frame buffer.

[0410] S306, the first device acquires image data of the first image and the second image.

[0411] S307, the first device displays a third image, which includes the first image and the second image.

[0412] In some embodiments, the display device includes a frame buffer. After the first image and the second image are both displayed, the first device waits for the black insertion to end, acquires the image pixel data of the first image and the second image from the frame buffer, and the display screen lights up to display the complete third image, that is, the third image includes the first image and the second image.

[0413] In other embodiments, the display device does not have a frame buffer. The first device receives and responds to the first image signal of the first image and directly displays the first image. The first device receives and responds to the second image signal of the second image and then displays the second image. At this time, both the first image and the second image have been displayed, that is, the third image has been fully displayed.

[0414] In this embodiment, the third image is described by dividing it into two parts: a first image and a second image. Other embodiments that divide the image into more parts will not be described in detail. It is understood that regardless of how many parts the third image is divided into, the idea of ​​parallel processing is the same. At the same time, different stages (rendering, display, etc.) of different parts of the image can be processed in parallel, thereby reducing the latency in the image transmission and display process and allowing the image to be displayed faster.

[0415] Example 4

[0416] Example 4 provides an image transmission and display method for use in a first device, the first device including a display apparatus. The method may include: between a first vertical synchronization signal and a second vertical synchronization signal, the first device transmitting a first image signal to the display apparatus; and between the first and second vertical synchronization signals, the first device transmitting a second image signal to the display apparatus, the first image signal and the second image signal being asynchronous. The schematic process is as follows: Figure 22 As shown.

[0417] By implementing this fourth embodiment, a whole image is divided into multiple parts, and these multiple parts are processed for parallel image display. This optimizes the display path, thereby reducing the waiting time in the image transmission and display process, further reducing the latency of image transmission and display, and allowing the image to be displayed faster.

[0418] In conjunction with Embodiment 4, in some embodiments, the first device transmits a first image signal to the display device, which is also located between the first display signal and the second display signal. The first device also transmits a second image signal to the display device, which is also located between the first display signal and the second display signal.

[0419] In conjunction with Embodiment 4, in some embodiments, the time interval between the first vertical synchronization signal and the second vertical synchronization signal is T1, and the time interval between the first display signal and the second display signal is T2, where T1 and T2 are equal.

[0420] In conjunction with Embodiment 4, in some embodiments, between the first vertical synchronization signal and the second vertical synchronization signal, the display device displays a black-inserted image frame, the period of which is T3, and T3 is less than T1.

[0421] In conjunction with Embodiment 4, in some embodiments, in response to the first vertical synchronization signal and the second vertical synchronization signal, the display device begins to display a black-inserted image frame. In response to the first display signal or the second display signal, the display device ends the display of the black-inserted image frame and displays a third image, which includes the first image and the second image.

[0422] In conjunction with Embodiment 4, in some embodiments, the first image signal is the image signal of the rendered first image, and the second image signal is the image signal of the rendered second image. That is, before displaying the first and second images in segments, the first and second images are rendered in segments respectively. The schematic process is as follows... Figure 23 As shown.

[0423] In conjunction with Embodiment 4, in some embodiments, when the first device transmits a first image signal to the display device, the first device renders a second image. That is, the step of sending the first image to the display device and the step of rendering the second image are performed in parallel. Here, the first time is the time consumed by the first device from the start of rendering the first image to the end of transmitting the second image to the display device, and the second time is the time consumed by the first device alone to complete rendering the third image and transmitting the third image to the display device. The third image includes the first image and the second image, and the first time is less than the second time. The schematic process is as follows: Figure 23 As shown.

[0424] In conjunction with Embodiment 4, in some embodiments, the first device renders the first image before transmitting the first image signal to the display device.

[0425] In conjunction with Embodiment 4, in some embodiments, the first device further includes an image rendering module, which is used to render the first image and the second image, and the display device sends a feedback signal to the image rendering module, which is used to indicate the vertical synchronization information of the display device.

[0426] In conjunction with Embodiment 4, in some embodiments, before the first device renders the first image, the first device receives the first image transmitted by the second device. Before the first device renders the second image, the first device receives the second image transmitted by the second device. That is, before the first device renders the first and second images in segments, it receives the first and second images transmitted in segments from the second device to the first device. The schematic process is as follows: Figure 24 As shown.

[0427] In conjunction with Embodiment 4, in some embodiments, when the first device renders the first image, the first device receives the second image transmitted by the second device. That is, the rendering step of the first image and the transmission step of the second image are performed in parallel. The third time is the time consumed from the first device starting to receive the first image transmitted by the second device to the end of transmitting the second image to the display device, and the fourth time is the time consumed from the first device receiving the third image transmitted by the second device to the end of transmitting the third image to the display device; the third time is less than the fourth time. The schematic process is as follows: Figure 24 As shown.

[0428] In conjunction with Embodiment 4, in some embodiments, before the first device transmits the first image signal to the display device, the first device receives the first image transmitted by the second device. Before the first device transmits the second image signal to the display device, the first device receives the second image transmitted by the second device. That is, before the first device displays the first and second images in segments, it receives the first and second images transmitted in segments from the second device to the first device. The schematic process is as follows: Figure 25 As shown.

[0429] In conjunction with Embodiment 4, in some embodiments, when the first device transmits a first image signal to the display device, the first device also receives a second image transmitted by the second device. That is, the step of sending the first image to the display device and the step of transmitting the second image are performed in parallel. The fifth time is the time consumed from the start of receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the sixth time is the time consumed from the start of receiving the third image transmitted by the second device to the end of transmitting the third image to the display device. The fifth time is less than the sixth time. The schematic process is as follows: Figure 25 As shown.

[0430] In conjunction with Embodiment 4, in some embodiments, both the first image and the second image are rendered by the second device. That is, before the first device receives the fragmented transmission of the first and second images from the second device, the second device performs fragmented rendering of the first and second images respectively. The schematic process is as follows: Figure 26 As shown.

[0431] In conjunction with Embodiment 4, in some embodiments, when the second image is rendered by the second device, the first device receives the first image transmitted by the second device. That is, the transmission step of the first image and the rendering step of the second image are performed in parallel. The seventh time is the time consumed from the start of rendering the first image by the second device to the end of transmitting the second image to the display device by the first device, and the eighth time is the time consumed from the start of rendering the third image by the second device to the end of transmitting the third image to the display device by the first device; the seventh time is less than the eighth time. The schematic process is as follows: Figure 26 As shown.

[0432] In conjunction with Embodiment 4, in some embodiments, in response to a second display signal, the first device displays a third image, which includes the first image and the second image.

[0433] In conjunction with Embodiment 4, in some embodiments, the first device displays a first image in response to a first image signal. In response to a second image signal, the first device displays a second image.

[0434] In conjunction with Embodiment 4, in some embodiments, the display device further includes a frame buffer for storing pixel data of the first image and the second image.

[0435] In conjunction with Embodiment 4, in some embodiments, the first device reads the pixel data of the first and second images from the frame buffer, and then displays the third image. That is, after acquiring the pixel data of the first and second images, the display device refreshes and displays a complete frame of the third image.

[0436] In conjunction with Embodiment 4, in some embodiments, the first device reads the pixel data of the first image from the frame buffer and displays the first image. The first device reads the pixel data of the second image from the frame buffer and displays the second image. That is, after the first device obtains the pixel data of the first image, it displays the first image; after obtaining the pixel data of the second image, it displays the second image. The first image and the second image together form a complete frame of the third image.

[0437] Example 5

[0438] Example 5 provides an image transmission and display method for use in a first device, the first device including a display apparatus. The method may include: the first device transmitting a first image to the display apparatus; the first device transmitting a second image to the display apparatus; and the display apparatus displaying a third image, wherein the third image includes the first image and the second image. A schematic diagram is shown below. Figure 22 As shown.

[0439] By implementing this fifth embodiment, a whole image can be divided into multiple parts and these parts can be processed for parallel image display, thereby optimizing the display path, reducing the waiting time in the image transmission and display process, further reducing the latency of image transmission and display, and allowing the image to be displayed faster.

[0440] In conjunction with Embodiment 5, in some embodiments, the first device renders the first image before transmitting it to the display device. The first device also renders the second image before transmitting it to the display device. That is, before displaying the first and second images in segments, the first and second images are rendered separately in segments. The schematic process is as follows: Figure 23 As shown.

[0441] In conjunction with Embodiment 5, in some embodiments, when the first device transmits a first image signal to the display device, the first device renders a second image. That is, the step of sending the first image to the display device and the step of rendering the second image are performed in parallel. Here, the first time is the time consumed by the first device from the start of rendering the first image to the end of transmitting the second image to the display device, and the second time is the time consumed by the first device alone to complete rendering the third image and transmitting the third image to the display device. The third image includes the first image and the second image, and the first time is less than the second time. The schematic process is as follows: Figure 23 As shown.

[0442] In conjunction with Embodiment 5, in some embodiments, before the first device renders the first image, the first device receives the first image transmitted by the second device. Before the first device renders the second image, the first device receives the second image transmitted by the second device. That is, before the first device renders the first and second images in segments, it receives the first and second images transmitted in segments from the second device to the first device. The schematic process is as follows: Figure 24 As shown.

[0443] In conjunction with Embodiment 5, in some embodiments, when the first device renders the first image, the first device receives the second image transmitted by the second device. That is, the rendering step of the first image and the transmission step of the second image are performed in parallel. The third time is the time consumed from the start of the first device receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the fourth time is the time consumed from the start of the first device receiving the third image transmitted by the second device to the end of transmitting the third image to the display device; the third time is less than the fourth time. The schematic process is as follows: Figure 24 As shown.

[0444] In conjunction with Embodiment 5, in some embodiments, before the first device transmits the first image to the display device, the first device receives the first image transmitted by the second device. Before the first device transmits the second image to the display device, the first device receives the second image transmitted by the second device. That is, before the first device displays the first and second images in segments, it receives the first and second images transmitted in segments from the second device to the first device. The schematic process is as follows: Figure 25 As shown.

[0445] In conjunction with Embodiment 5, in some embodiments, when the first device transmits a first image to the display device, the first device also receives a second image transmitted by the second device. That is, the step of sending the first image to the display device and the step of transmitting the second image are performed in parallel. The fifth time is the time consumed from the start of receiving the first image transmitted by the second device to the end of transmitting the second image to the display device, and the sixth time is the time consumed from the start of receiving the third image transmitted by the second device to the end of transmitting the third image to the display device. The fifth time is less than the sixth time. The schematic process is as follows: Figure 25 As shown.

[0446] In conjunction with Embodiment 5, in some embodiments, the first image and the second image are rendered by the second device. That is, before the first device receives the fragmented transmission of the first image and the second image from the second device, the second device performs fragmented rendering of the first image and the second image respectively. The schematic process is as follows: Figure 26 As shown.

[0447] In conjunction with Embodiment 5, in some embodiments, when the second image is rendered by the second device, the first device receives the first image transmitted by the second device. That is, the transmission step of the first image and the rendering step of the second image are performed in parallel. The seventh time is the time consumed from the start of rendering the first image by the second device to the end of transmitting the second image to the display device by the first device, and the eighth time is the time consumed from the start of rendering the third image by the second device to the end of transmitting the third image to the display device by the first device; the seventh time is less than the eighth time. The schematic process is as follows: Figure 26 As shown.

[0448] In conjunction with Embodiment 5, in some embodiments, the display device further includes a frame buffer that stores pixel data of the first image and the second image.

[0449] In conjunction with Embodiment 5, in some embodiments, the first device reads the pixel data of the first and second images from the frame buffer, and then the display device displays the third image. That is, after acquiring the pixel data of the first and second images, the display device refreshes and displays a complete frame of the third image.

[0450] In conjunction with Embodiment 5, in some embodiments, the method further includes: in response to a first image signal, the first device displays a first image; in response to a second image signal, the first device displays a second image.

[0451] In conjunction with Embodiment 5, in some embodiments, the first device reads the pixel data of the first image from the frame buffer and displays the first image. The first device reads the pixel data of the second image from the frame buffer and displays the second image. That is, after the first device obtains the pixel data of the first image, it displays the first image; after obtaining the pixel data of the second image, it displays the second image. The first image and the second image together form a complete frame of the third image.

[0452] Example 6

[0453] Example 6 provides an image transmission and display method, which may include: establishing a connection between a first device and a second device. The second device transmits a first image to the first device through the connection, and the second device transmits a second image to the first device through the connection. The first device includes a display device, and the first device transmits the first image to the display device, and the first device transmits the second image to the display device. The display device displays a third image, wherein the third image includes the first image and the second image. The schematic process is as follows: Figure 25 As shown.

[0454] In implementing this sixth embodiment, the second device divides a whole image into multiple parts and then transmits them to the first device in parallel. The first device then processes these multiple parts for parallel image display, thereby optimizing the display path, reducing the waiting time during image transmission and display, further reducing the latency of image transmission and display, and enabling the image to be displayed faster.

[0455] In conjunction with Embodiment Six, in some embodiments, when the first device transmits the first image to the display device, the second device transmits the second image to the first device via a connection. That is, the step of sending the first image to the display device and the step of transmitting the second image are performed in parallel. The fifth time is the time taken from the start of the second device transmitting the first image to the first device until the first device finishes transmitting the second image to the display device, and the sixth time is the time taken from the start of the second device transmitting the third image to the first device until the first device finishes transmitting the third image to the display device. The fifth time is less than the sixth time. The schematic process is as follows: Figure 25 As shown.

[0456] In conjunction with Embodiment Six, in some embodiments, the second device renders the first image before transmitting it to the first device via the connection. Similarly, the second device renders the second image before transmitting it to the first device via the connection. That is, before the first device receives the fragmented transmission of the first and second images from the second device, the second device performs fragmented rendering of both the first and second images respectively. The schematic process is as follows: Figure 26 As shown.

[0457] In conjunction with Embodiment Six, in some embodiments, when the first device receives the first image transmitted by the second device, the second device renders the second image. That is, the transmission step of the first image and the rendering step of the second image are performed in parallel. The seventh time is the time consumed from the second device starting to render the first image to the first device transmitting the second image to the display device, and the eighth time is the time consumed from the second device starting to render the third image to the first device transmitting the third image to the display device. The seventh time is less than the eighth time. The schematic process is as follows: Figure 26 As shown.

[0458] In conjunction with Embodiment Six, in some embodiments, the method further includes: rendering the first image after the second device transmits the first image to the first device via the connection and before the first device transmits the first image to the display device; and rendering the second image after the second device transmits the second image to the first device via the connection and before the first device transmits the second image to the display device. The schematic process is as follows: Figure 24 As shown.

[0459] In conjunction with Embodiment Six, in some embodiments, when the second device transmits the second image to the first device via a connection, the first device renders the first image. That is, the rendering step of the first image and the transmission step of the second image are performed in parallel. The third time is the time consumed from the second device starting to send the first image to the first device until the first device finishes transmitting the second image to the display device, and the fourth time is the time consumed from the second device starting to transmit the third image to the first device until the first device finishes transmitting the third image to the display device; the third time is less than the fourth time. The schematic process is as follows: Figure 24 As shown.

[0460] In conjunction with Embodiment Six, in some embodiments, the display device further includes a frame buffer that stores pixel data of the first image and the second image.

[0461] In conjunction with Embodiment Six, in some embodiments, the first device reads the pixel data of the first and second images from the frame buffer, and then displays the third image. That is, after acquiring the pixel data of the first and second images, the display device refreshes and displays a complete frame of the third image.

[0462] In conjunction with Embodiment Six, in some embodiments, the method further includes: in response to a first image signal, the first device displays a first image; in response to a second image signal, the first device displays a second image.

[0463] In conjunction with Embodiment Six, in some embodiments, the first device reads the pixel data of the first image from the frame buffer and displays the first image. The first device reads the pixel data of the second image from the frame buffer and displays the second image. That is, after the first device obtains the pixel data of the first image, it displays the first image; after obtaining the pixel data of the second image, it displays the second image. The first image and the second image together form a complete frame of the third image.

[0464] This application also provides an electronic device, which may include: a communication device, a display device, a memory, and a processor coupled to the memory, multiple application programs, and one or more programs. The memory stores computer-executable instructions, the display device is used to display images, and when the processor executes the instructions, the electronic device can perform any of the functions of the first device in Embodiment 4 or Embodiment 5.

[0465] This application also provides a computer storage medium storing a computer program, which includes executable instructions. When executed by a processor, the executable instructions cause the processor to perform operations corresponding to the methods provided in Embodiment 4 or Embodiment 5.

[0466] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform any possible implementation as described in Embodiment 4 or Embodiment 5.

[0467] This application also provides a chip system that can be applied to an electronic device. The chip includes one or more processors, which are used to invoke computer instructions to cause the electronic device to implement any of the possible implementations in Embodiment 4 or Embodiment 5.

[0468] This application also provides a communication system, which includes a first device and a second device. The first device can perform some of the functions of the first device as described in Embodiment 4 or Embodiment 5.

[0469] Implementing the method provided in this application can reduce latency during image transmission and display, allowing images to be displayed faster. Furthermore, in AR / VR or MR scenarios, it can reduce MTP latency, effectively alleviating adverse reactions such as motion sickness and vomiting, and improving the user experience. It is understood that the method provided in this application is applicable to more scenarios, such as PC screen projection and in-vehicle device image display. Through parallel processing of segmented transmission, rendering, and display, it accelerates the end-to-end display process, reduces latency, and improves the user's viewing experience.

[0470] The implementation methods described in the above embodiments are merely illustrative and do not constitute any limitation on other embodiments of this application. The specific internal implementation methods may vary depending on the type of electronic device, the operating system it runs, the program used, and the interfaces called. This application embodiment does not impose any limitations, as long as the feature functions described in this application embodiment can be implemented.

[0471] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0472] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0473] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0474] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image transmission and display method, characterized by, The method is used for display on a first device, the first device including a display unit, the method comprising: Between the first vertical synchronization signal and the second vertical synchronization signal, the first device sends the rendered first image to the display device after the first image is rendered. Between the first vertical synchronization signal and the second vertical synchronization signal, while sending the rendered first image to the display device, the second image is rendered in parallel. Between the first vertical synchronization signal and the second vertical synchronization signal, after the second image is rendered, the rendered second image is sent to the display device for display. Wherein, the second image and the first image are multiple non-overlapping image portions of the third image; the second vertical synchronization signal is the next vertical synchronization signal after the first vertical synchronization signal; The first device displays the third image via the display device.

2. The method according to claim 1, characterized in that, The first device sends the first image to the display device, and is located between the first display signal and the second display signal; The first device sends the second image to the display device, and is located between the first display signal and the second display signal.

3. The method of claim 2, wherein, The time interval between the first vertical synchronization signal and the second vertical synchronization signal is T1, and the time interval between the first display signal and the second display signal is T2, where T1 and T2 are equal.

4. The method of claim 1, wherein, The first device further includes an image rendering module, which is used to render the first image and the second image. The display device sends a feedback signal to the image rendering module, which is used to indicate the vertical synchronization information of the display device.

5. The method according to claim 4, characterized in that, Before the first device renders the first image, the method further includes: The first device receives the first image transmitted by the second device; Before the first device renders the second image, the method further includes: The first device receives the second image transmitted by the second device.

6. The method of claim 5, wherein, The first device receives the second image transmitted by the second device, specifically including: When the first device renders the first image, the first device receives the second image transmitted by the second device.

7. The method according to claim 1, characterized in that, Before the first device sends the rendered first image to the display device, the method further includes: The first device receives the first image transmitted by the second device; Before the first device sends the rendered second image to the display device, the method further includes: The first device receives the second image transmitted by the second device.

8. The method of claim 7, wherein, The first device receives the second image transmitted by the second device, specifically including: When the first device sends the rendered first image to the display device, the first device receives the second image transmitted by the second device.

9. The method according to claim 7 or 8, characterized in that, Both the first image and the second image are rendered by the second device.

10. The method of claim 9, wherein, The first device receives the first image transmitted by the second device, specifically including: When the second image is rendered by the second device, the first device receives the first image transmitted by the second device.

11. The method of claim 2, wherein, The first device displays the third image through the display device, specifically including: In response to the second display signal, the first device displays the third image.

12. The method according to claim 1, characterized in that, The first device displays the third image through the display device, specifically including: After the rendered first image is sent and displayed, the first device displays the first image; After the rendered second image is sent and displayed, the first device displays the second image.

13. The method according to claim 11, characterized in that, The display device further includes a frame buffer for storing pixel data of the first image and the second image; The first device displays the third image, specifically including: The first device reads the pixel data of the first image and the second image from the frame buffer; The first device displays the third image.

14. The method according to claim 12, characterized in that, The display device further includes a frame buffer for storing pixel data of the first image and the second image; The first device displays the first image, specifically including: The first device reads the pixel data of the first image from the frame buffer; The first device displays the first image; The first device displays the second image, specifically including: The first device reads the pixel data of the second image from the frame buffer; The first device displays the second image.

15. An image transmission and display method, characterized in that, The method is used for display on a first device, the first device including a display unit, the method comprising: After rendering the first image, the first device sends the rendered first image to the display device for display. While the first image is being displayed to the display device, the second image is being rendered in parallel. The first device sends the rendered second image to the display device; the second image and the first image are multiple non-overlapping image portions of the third image. The display device displays the third image.

16. The method according to claim 15, characterized in that, Before the first device renders the first image, the method further includes: The first device receives the first image transmitted by the second device; Before the first device renders the second image, the method further includes: The first device receives the second image transmitted by the second device.

17. The method according to claim 16, characterized in that, The first device receives the second image transmitted by the second device, specifically including: When the first device renders the first image, the first device receives the second image transmitted by the second device.

18. An image transmission and display method, characterized in that, The method includes: The first device establishes a connection with the second device; The second device transmits the first image to the first device via the connection; The second device transmits the second image to the first device through the connection; the second image and the first image are multiple non-overlapping image portions of the third image. The first device includes a display device, and after rendering the first image, the first device sends the rendered first image to the display device; While the rendered first image is being displayed to the display device, the second image is being rendered in parallel. The first device sends the rendered second image to the display device; The display device displays the third image.

19. The method according to claim 18, characterized in that, The second device transmits the second image to the first device through the connection, specifically including: When the first device transmits the first image to the display device, the second device transmits the second image to the first device through the connection.

20. The method according to claim 18 or 19, characterized in that, Before the second device transmits the first image to the first device via the connection, the method further includes: The second device renders the first image; Before the second device transmits the second image to the first device via the connection, the method further includes: The second device renders the second image.

21. The method according to claim 20, characterized in that, The second device renders the second image, specifically including: When the first device receives the first image transmitted by the second device, the second device renders the second image.

22. The method according to claim 18, characterized in that, The first device renders the first image, specifically including: When the second device transmits the second image to the first device through the connection, the first device renders the first image.

23. The method according to claim 18, characterized in that, The method further includes: In response to the first image signal, the first device displays the first image; In response to the second image signal, the first device displays the second image.

24. An electronic device, characterized in that, The electronic device includes: a communication device, a display device, a memory, and a processor coupled to the memory, multiple application programs, and one or more programs; the memory stores computer-executable instructions, the display device is used to display images, and the processor, when executing the instructions, causes the electronic device to implement the method as claimed in any one of claims 1 to 14 or 15 to 17.

25. A storage medium storing a computer program, the computer program including executable instructions that, when executed by a processor, cause the processor to perform an operation corresponding to the method provided in any one of claims 1 to 14 or 15 to 17.

26. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as claimed in any one of claims 1 to 14 or 15 to 17.

Images