Display method, electronic device, chip, and storage medium

Abstract

The present application is applicable to the technical field of display, and provides a display method, an electronic device, a chip, and a storage medium. The method comprises: a first chip acquires first image data of a first image by means of a first interface, wherein the first image is a real-scene image obtained by means of a camera module, and the first image data is part of data or all data of the first image; the first chip acquires second image data of a second image by means of a second interface; the first chip generates a synthesis image on the basis of the first image data and the second image data; and the first chip processes the synthesis image by means of an image processing algorithm corresponding to each output channel to generate an output image corresponding to the output channel, wherein each output channel corresponds to one display module, and the output image is displayed by means of the display module corresponding to the output channel. According to the technical solution provided by the present application, the waiting duration required between VR image synthesis can be shortened, and the time delay is reduced, thereby improving the smoothness of VR visuals, and reducing the dizziness of a user during viewing.

Description

Display method, electronic device, chip and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 25, 2023, with application number 202311813482.8 and application name “A display method, electronic device, chip and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of device management technology, and in particular relates to a display method, electronic device, chip and storage medium. Background Art

[0003] Virtual Reality (VR) technology, as one of the mainstream display technologies today, has an increasingly wide range of applications. Compared with traditional two-dimensional display technology, VR display technology is highly immersive. Therefore, how VR display devices can display smooth images directly affects the user's sense of immersion when viewing multimedia data through VR display devices. Video See Through (VST), as one of the important branches of VR display technology, uses a camera module to capture real-life images of the user's scene, and uses VR technology to overlay virtual content on the real-life images, allowing users to simultaneously view virtual content and a real environment consistent with reality, further enhancing the user's sense of immersion in the viewing process.

[0004] In order to achieve the above-mentioned combination of real-scene images and virtual images, the VR display device needs to perform multiple steps of image processing on the actual image to generate a VR composite image. Different image processing steps require the processor to execute them in sequence according to the processing sequence. Some processing steps have a large delay, resulting in an unsmooth final output image, which reduces the user's immersive viewing experience and affects the user's viewing experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a display method, electronic device, chip, and computer-readable storage medium, which can solve the problem of existing VR display technology that, when generating VR composite images, different image processing steps need to be executed sequentially by the same processor according to the processing timing, and the processing time is too long, resulting in an unsmooth picture and affecting the user's viewing experience.

[0007] In a first aspect, an embodiment of the present application provides a display method, which is applied to an electronic device, wherein the electronic device includes a first chip, and the display method includes:

[0008] The first chip acquires first image data of a first image through a first interface; the first image is a real scene image acquired by a camera module; the first image data is part of or all of the data of the first image;

[0009] The first chip acquires second image data of the second image through the second interface; the second image data is part of or all of the data of the second image;

[0010] The first chip generates a composite image based on the first image data and the second image data;

[0011] The first chip processes the synthesized image through the image processing algorithm corresponding to each of the output channels to generate the output image corresponding to each of the output channels; the output image is displayed through the display module corresponding to the output channel.

[0012] The implementation of the embodiment of the present application has the following beneficial effects: when generating a VR composite image, image synthesis and image processing are completed by an independent first chip, so that the exposure operation of the real-scene image can be performed synchronously with later operations such as image synthesis and image processing, so that when the VR composite image needs to be output continuously, the waiting time between each frame of the composite image can be shortened, thereby improving the smoothness of the VR display screen. Compared with existing display technologies, since the embodiment of the present application performs the acquisition of the real-scene image and the subsequent image synthesis and image processing operations by different chips when displaying the VR composite image, the logical operations of the two parts can be executed synchronously, thereby shortening the waiting time required between VR image synthesis, reducing latency, and improving the smoothness of the VR screen, reducing the dizziness of the user when watching, and improving the user experience.

[0013] In a possible implementation of the first aspect, the method further includes:

[0014] The first chip obtains first depth information of the first image and second depth information of the second image data through the first interface;

[0015] The first chip generates a composite image based on the first image data and the second image data, including:

[0016] The first chip constructs a first depth image according to the first depth information and the first image data;

[0017] The first chip constructs the second depth image according to the second image data and the second depth information;

[0018] The first chip generates the composite image according to the first depth image and the second depth image.

[0019] In a possible implementation manner of the first aspect, the image processing algorithm includes: an anti-distortion processing algorithm and an anti-color cast processing algorithm.

[0020] In a possible implementation of the first aspect, the first chip acquiring first image data of a first image through a first interface includes:

[0021] The first chip receives the first image data of several rows in the first image sent by the second chip through the first interface.

[0022] In a possible implementation of the first aspect, before the first chip generates output images corresponding to the respective output channels based on the first image data and the second image data, the method further includes:

[0023] The first chip receives a first synchronization signal sent by the second chip through the first interface;

[0024] The first chip generates an output image corresponding to each output channel according to the first image data and the second image data, including:

[0025] The first chip generates output images corresponding to respective output channels according to the first image data and the second image data in response to the first synchronization signal.

[0026] In a possible implementation of the first aspect, after the first chip receives the first synchronization signal through the first interface, the method further includes:

[0027] The first chip sends the first synchronization signal to the third chip through the second interface, so that the third chip sends the second image data according to the first synchronization signal.

[0028] In a possible implementation of the first aspect, the electronic device further includes a second chip, and a third interface of the second chip is electrically connected to the first interface of the first chip; and the display method includes:

[0029] The second chip receives the first image data obtained by exposure by the camera module;

[0030] The second chip sends the first image data to the first chip through a third interface.

[0031] In a possible implementation of the first aspect, the second chip is connected to at least two camera modules to obtain a first image with depth information.

[0032] In a possible implementation manner of the first aspect, the camera module is a depth camera module to obtain a first image with depth information.

[0033] In a possible implementation of the first aspect, the second chip includes a graphics signal processor and a storage unit;

[0034] The graphic signal processor receives the first image data obtained by exposure of the camera module and sets the value of the storage unit according to the first image data; the value of the storage unit is used to determine the data volume of the first image data;

[0035] The second chip sends the first image data corresponding to the data volume to the first chip through a third interface according to the value of the storage unit.

[0036] In a possible implementation of the first aspect, the first interface and the third interface are electrically connected to perform data transmission of the first image data; the first image data is obtained by performing slice processing on the first image.

[0037] In some implementations, the first interface and the third interface may be substrate-like multi-chip package input / output (SMIO) interfaces.

[0038] In a possible implementation of the first aspect, the method further includes:

[0039] The second chip receives second depth information of the second image data sent by the third chip through a fourth interface;

[0040] The second chip determines first depth information of the first image data; the first depth information and the second depth information are used to generate the output image;

[0041] The second chip sends the first image data to the first chip through the third interface, including:

[0042] The second chip sends the first image data, the first depth information, and the second depth information to the first chip through the third interface.

[0043] In a possible implementation manner of the first aspect, the fourth interface and the sixth interface are interfaces based on a high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIE) protocol.

[0044] In a possible implementation of the first aspect, before the first chip generates output images corresponding to the respective output channels based on the first image data and the second image data, the method further includes:

[0045] The second chip sends a first synchronization signal to the first chip through the third interface, and sends a second synchronization signal to the third chip through the fourth interface.

[0046] In a possible implementation of the first aspect, the electronic device further includes a third chip; a fifth interface of the third chip is electrically connected to the second interface; and the display method includes:

[0047] The third chip generates the second image data;

[0048] The third chip sends the second image data to the first chip through the fifth interface.

[0049] In some implementations, the second interface and the fifth interface are interfaces based on the Mobile Industry Processor Interface (MIPI) protocol.

[0050] In a possible implementation of the first aspect, the third chip sending the second image data to the first chip through the fifth interface includes:

[0051] The third chip receives the first synchronization signal sent by the first chip through the fifth interface or receives the second synchronization signal sent by the second chip through the sixth interface;

[0052] The third chip sends the second image data to the first chip through the fifth interface in response to the first synchronization signal or the second synchronization signal.

[0053] In a possible implementation of the first aspect, the sixth interface of the third chip is electrically connected to the fourth interface of the second chip; and the display method includes:

[0054] The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the output image.

[0055] In a possible implementation of the first aspect, the method further includes:

[0056] The third chip sends a third synchronization signal to the first chip through the fifth interface, and sends a fourth synchronization signal to the second chip through the sixth interface.

[0057] In a second aspect, an electronic device is provided, characterized in that the electronic device includes a first chip; the first chip includes a first interface, a second interface, and at least one seventh interface; the first interface is electrically connected to a third interface of the second chip; the second interface is electrically connected to a fifth interface of the third chip; the seventh interface is electrically connected to a display module; the second chip and the third chip are the same chip or different chips;

[0058] The first chip is used to obtain first image data of a first image sent by the second chip through a first interface; the first image is a real scene image obtained by a camera module; and the first image data is part of or all of the data of the first image;

[0059] The first chip is used to obtain second image data of the second image sent by the third chip through the second interface; the second image data is part of or all of the data of the second image;

[0060] The first chip is used to generate output images corresponding to respective output channels according to the first image data and the second image data;

[0061] The first chip is configured to send the output image to the display module of the output channel corresponding to the seventh interface through the seventh interface, so that the output image is displayed through the display module.

[0062] In a possible implementation of the second aspect, the first chip is configured to generate an output image corresponding to each output channel based on the first image data and the second image data, including:

[0063] The first chip is used to generate a composite image based on the first image data and the second image data;

[0064] The first chip is used to process the synthesized image using image processing algorithms corresponding to the output channels to generate the output images corresponding to the output channels.

[0065] In a possible implementation of the second aspect, the first chip is further configured to: acquire, through the first interface, first depth information of the first image and second depth information of the second image data;

[0066] The first chip is configured to generate a composite image based on the first image data and the second image data, including:

[0067] The first chip is configured to construct a first depth image according to the first depth information and the first image data;

[0068] The first chip is configured to construct the second depth image according to the second image data and the second depth information;

[0069] The first chip is configured to generate the composite image according to the first depth image and the second depth image.

[0070] In a possible implementation manner of the second aspect, the image processing algorithm includes: an anti-distortion processing algorithm and an anti-color cast processing algorithm.

[0071] In a possible implementation manner of the second aspect, the first chip is configured to receive, through the first interface, several rows of first image data in the first image sent by the second chip.

[0072] In a possible implementation of the second aspect, the first chip is further configured to:

[0073] The first chip is configured to receive a first synchronization signal sent by the second chip through a first interface;

[0074] The first chip is configured to generate output images corresponding to respective output channels based on the first image data and the second image data, including:

[0075] The first chip generates output images corresponding to respective output channels according to the first image data and the second image data in response to the first synchronization signal.

[0076] In a possible implementation of the second aspect, the first chip is further configured to: send the first synchronization signal to the third chip through the second interface, so that the third chip sends the second image data according to the first synchronization signal.

[0077] In a possible implementation of the second aspect, the electronic device further includes a second chip; an eighth interface of the second chip is electrically connected to the camera module;

[0078] The second chip is used to receive the first image data obtained by exposure of the camera module;

[0079] The second chip is used to send the first image data to the first chip through a third interface.

[0080] In a possible implementation of the second aspect, the second chip includes a graphics signal processor and a storage unit; the graphics signal processor is electrically connected to the storage unit; the graphics signal processor is electrically connected to the eighth interface; and the storage unit is electrically connected to the third interface;

[0081] The graphics signal processor is used to receive the first image data obtained by exposure of the camera module and set the value of the storage unit according to the first image data; the value of the storage unit is used to determine the data volume of the first image data;

[0082] The second chip is configured to send the first image data corresponding to the data volume to the first chip through a third interface according to the value of the storage unit.

[0083] In a possible implementation of the second aspect, the first interface and the third interface are electrically connected to perform data transmission of the first image data; the first image data is obtained by performing slice processing on the first image.

[0084] In a possible implementation of the second aspect, the second chip includes a fourth interface; the fourth interface is electrically connected to the sixth interface of the third chip;

[0085] The second chip is configured to receive, through a fourth interface, second depth information of the second image data sent by the third chip;

[0086] The second chip is used to determine first depth information of the first image data; the first depth information and the second depth information are used to generate the output image;

[0087] The second chip is configured to send the first image data, the first depth information, and the second depth information to the first chip through the third interface.

[0088] In a possible implementation manner of the second aspect, the second chip is further configured to: send a first synchronization signal to the first chip through a third interface, and send a second synchronization signal to the third chip through the fourth interface.

[0089] In a possible implementation of the second aspect, the electronic device further includes a third chip;

[0090] The third chip is used to generate the second image data;

[0091] The third chip is configured to send the second image data to the first chip via a fifth interface.

[0092] In some implementations, the second interface and the fifth interface are interfaces based on the Mobile Industry Processor Interface (MIPI) protocol.

[0093] In a possible implementation of the second aspect, the third chip is configured to send the second image data to the first chip through the fifth interface, including:

[0094] The third chip is configured to receive a first synchronization signal sent by the first chip through the fifth interface or receive a second synchronization signal sent by the second chip through the sixth interface;

[0095] The third chip is configured to send the second image data to the first chip through the fifth interface in response to the first synchronization signal or the second synchronization signal.

[0096] In a possible implementation of the second aspect, the sixth interface of the third chip is electrically connected to the fourth interface of the second chip;

[0097] The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the output image.

[0098] In a possible implementation manner of the second aspect, the third chip is further configured to: send a third synchronization signal to the first chip through the fifth interface, and send a fourth synchronization signal to the second chip through a sixth interface.

[0099] In a third aspect, an embodiment of the present application provides a chip comprising: a memory, a processor, and a program stored in the memory. When the processor executes the program, the steps of the first chip, the steps of the second chip, or the steps of the third chip in the display method described in any one of the first aspects above are implemented.

[0100] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores a program. When the program is executed by a processor, it implements the steps of the first chip, the steps of the second chip, or the steps of the third chip in the display method described in any one of the first aspects above.

[0101] In a fifth aspect, an embodiment of the present application provides a program product. When the program product is run on a device, the device executes the steps of the first chip, the steps of the second chip, or the steps of the third chip in the display method described in any one of the first aspects above.

[0102] In a sixth aspect, an embodiment of the present application provides a chip system, including a first chip, a second chip, and a third chip;

[0103] The first chip includes a processor, the processor is coupled to a memory, and the processor executes a program stored in the memory to implement the steps of the first chip in the display method according to any one of the first aspects;

[0104] The second chip includes a processor, the processor is coupled to a memory, and the processor executes a program stored in the memory to implement the steps of the second chip in the display method according to any one of the first aspects;

[0105] The third chip includes a processor, which is coupled to a memory. The processor executes a program stored in the memory to implement the steps of the third chip in the display method as described in any one of the first aspects.

[0106] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] FIG1 is a schematic diagram of an electronic device displaying a VR composite image according to an embodiment of the present application;

[0108] FIG2 is a schematic diagram of a screen based on VST display technology provided by an embodiment of the present application;

[0109] FIG3 is a data flow diagram of a VR synthetic image based on VST technology provided by an embodiment of the present application;

[0110] FIG4 is a schematic structural diagram of a first electronic device provided in an embodiment of the present application;

[0111] FIG5 is a flow chart of a display method according to an embodiment of the present application;

[0112] FIG6 is a partial timing diagram of a VR composite image after the entire real scene image is exposed, provided by an embodiment of the present application;

[0113] FIG7 is a partial timing diagram of a VR synthesized image when a real scene image is sliced ​​and processed according to an embodiment of the present application;

[0114] FIG8 is a schematic diagram of a VR display system provided in one embodiment of the present application;

[0115] FIG9 is a schematic diagram of a process for generating a composite image according to an embodiment of the present application;

[0116] FIG10 is a schematic structural diagram of a second electronic device provided in one embodiment of the present application;

[0117] FIG11 is a flow chart of a display method according to another embodiment of the present application;

[0118] FIG12 is a schematic diagram of a VR display system provided by another embodiment of the present application;

[0119] FIG13 is a schematic structural diagram of a third electronic device provided in one embodiment of the present application;

[0120] FIG14 is a flow chart of a display method according to another embodiment of the present application;

[0121] FIG15 is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0122] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0123] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0124] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0125] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0126] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0127] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0128] The display method provided in the embodiments of the present application can be applied to augmented reality (AR) / virtual reality (VR) display devices, smart phones, tablet computers, and other electronic devices that can implement VR display. In particular, the display method can be applied to electronic devices that can implement VR display, or electronic devices with external VR display devices. The embodiments of the present application do not impose any restrictions on the specific type of electronic device.

[0129] For example, FIG1 shows a schematic diagram of an implementation of displaying a VR composite screen by an electronic device provided in an embodiment of the present application.

[0130] As shown in (a) in Figure 1, the electronic device can be a wearable VR display device, which can have a built-in chip system 11 and a camera module 12. The chip system 11 includes: a second chip and a first chip, etc. The second chip is used to obtain the first image data of the real scene image from the camera module 12. The first chip processes and synthesizes the first image data. The camera module 12 can be used to obtain the environmental image of the scene where the wearer (i.e., the user) is located. The chip system 11 can synthesize the environmental image with the virtual picture to generate a VR synthetic image based on VST technology.

[0131] As shown in (b) of FIG1 , the electronic device may be a smart phone, and the smart phone 13 includes a first chip, wherein the smart phone 13 can establish a communication connection with a pair of smart glasses 14. The communication connection may be a wired communication connection or a wireless communication connection; for example, the smart phone 13 can be connected to the smart glasses 14 via a serial port; if the smart glasses 14 are equipped with a wireless communication module, such as a Bluetooth module or a WIFI module, the smart phone 13 can establish a communication connection with the smart glasses 14 via the wireless communication module. The smart glasses 14 may also be equipped with a camera module, which captures an environmental image and feeds it back to the smart phone 13. The smart phone 13 can synthesize the environmental image with the virtual image through a built-in processor to generate a VR composite image based on VST technology, and feed it back to the wearable glasses, and output the VR composite image through the wearable glasses.

[0132] With the continuous advancement of display technology, VR display technology has become one of the mainstream display technologies today. Traditional VR display technology provides users with an immersive viewing experience and has been applied in various fields, including games and movies. However, because traditional VR display technology uses the GPU in the processor to construct one or more virtual graphics, the combination of these virtual graphics generates a purely fictitious virtual image, which has low realism and reduces the user's sense of immersion. To further enhance the user's viewing experience and immersion, VST technology has emerged. Compared to traditional VR display technology that outputs purely virtual images, VST display technology uses a camera module configured on the VR display device to capture real-world images of the user's scene. It then overlays a synthetic virtual image on top of the real-world image, creating a VR composite image based on VST technology. Because the background image in the VR composite image is often generated based on the real-world image, this can improve the image's realism, thereby enhancing the user's sense of immersion when viewing the VR composite image.

[0133] For example, FIG2 shows a schematic diagram of a screen based on VST display technology provided by an embodiment of the present application. Referring to FIG2 (a), the user wears a VR display device, which may be the VR display device shown in FIG1 (a). The VR display device is equipped with a camera module, which can obtain an environmental image within the user's field of view, such as a television and a clock, and the VR display device can synthesize the required synthesized virtual image into the above-mentioned environmental image. As shown in FIG2 (b). By comparing FIG2 (a) with FIG2 (b), it can be found that the TV is in the off state in the real scene, that is, no screen content is output. When the image is synthesized by the VR display device to generate the corresponding VR synthesized image, the specified video image frame can be added to the area where the TV is located in the environmental image, thereby realizing the combination of the virtual image and the real image, improving the immersiveness of the output image, and then improving the user's viewing experience.

[0134] While VST display technology can improve the realism of VR composite images and thus enhance the user's sense of immersion, it also introduces new challenges for VR display devices, namely the large amount of data required to be processed and the lengthy processing flow. VR composite images require a combination of real and virtual, requiring the use of a camera module to capture real-world images of a real scene, which involves processes such as image exposure and image transmission. The combination of virtual images requires the generation of virtual images and synthesis, and then the synthesis of real and virtual images, requiring multiple steps to be executed sequentially according to the processing order.

[0135] For example, Figure 3 shows a data flow diagram of a VR composite image based on VST technology provided by an embodiment of the present application. Referring to Figure 3, the process of generating a VR composite image by an electronic device specifically includes the following multiple stages:

[0136] Stage 1: Image exposure stage

[0137] Since it is necessary to obtain the real-life image of the scene where the user is located, it is necessary to capture the image through the camera module. When capturing the real-life image through the camera module, it includes exposure (such as 11 milliseconds), and the first image data reading (8.5ms), wherein the above two steps are mainly completed by the image signal processor (Image Signal Processing, ISP) in the chip. Among them, the image signal processing stage can be divided into a processing stage through the image front-end processor (Image Front End, IFE) (such as 0.5ms) and a processing stage through the image processor (Image processing engine, IPE) (such as 8.5ms). The above multiple processing stages all take time, which can be seen to bring a certain processing delay to the process of generating VR synthetic images.

[0138] Phase 2: Image transmission phase

[0139] When the ISP obtains the real scene image captured by the camera module, it needs to transmit the first image data of the real scene image to the relevant thread at the system layer for processing, which consumes a certain transmission time, such as 6.5ms.

[0140] Phase 3: One-shot rendering and video parsing

[0141] When an electronic device generates a VR composite image, it must use a rendering application at the application layer to render virtual objects, such as a virtual keyboard or cartoon character. This requires a single rendering phase. This single rendering phase can be performed in parallel with the aforementioned image exposure phase. This means the electronic device can perform a single rendering process while the image is being exposed and transmitted. This phase also introduces a certain processing delay.

[0142] Similarly, if it is necessary to insert already generated video data into the picture, the video decoder in the application framework layer needs to parse the video data frame by frame, and the above process also requires a certain amount of processing time.

[0143] Stage 4: Virtual Image Fusion

[0144] Since there are virtual objects generated in a rendering phase and video image frames obtained in the video phase, the virtual objects need to be fused with the video image frames. The fusion of the virtual first image data also requires a processing time, such as 0.6ms.

[0145] Stage 5: Second rendering stage

[0146] This stage requires the fusion of virtual and real scenes, combining real-world images with virtual ones. This involves image reprojection (e.g., requiring 10.2ms) and virtual-reality fusion (e.g., requiring 21.3ms) to generate a composite VR image. This stage has a relatively long latency; in some implementations, the average latency of the secondary rendering stage can reach 21.3ms.

[0147] Stage 6: Image processing stage for synthesized images

[0148] The VR synthesized image needs to be processed, such as anti-distortion processing and anti-dispersion processing, to adapt to the corresponding display module.

[0149] Stage 7: Image display stage

[0150] When the display module obtains the VR composite image of the corresponding channel, it needs to scan row by row or column by column to output the image.

[0151] It can be seen that when generating a VR composite image based on VST, the process from generating the first image data to the final VR composite image needs to go through multiple stages, and the various stages are basically in a serial execution relationship. Therefore, the processing of the VR composite image of the previous frame can only be performed after the processing of the VR composite image of the previous frame is completed. As a result, the interval between VR composite images of different frames is too long, affecting the smoothness of the overall picture. Excessive inter-frame delay will also cause users to feel dizzy when viewing the picture, which in turn affects the user experience.

[0152] Therefore, in order to solve the problems of the above-mentioned VR display technology, an embodiment of the present application provides a display method, the execution subject of the display method can be an electronic device, and the electronic device can include but is not limited to: a VR display device, a smart phone, and other electronic devices capable of synthesizing VR synthetic images based on VST technology. The electronic device can have a built-in VR display module, or can be externally connected to a wearable VR display device, and output the above-mentioned VR synthetic image through the external wearable VR display device. Among them, the VR display device includes a first chip, and the first chip processes the synthesis of virtual and real images and the image processing of the output adaptation of the display module to improve the smoothness of the output image, reduce the dizziness of the user when watching, and enhance the user's viewing experience.

[0153] Figure 4 shows a schematic structural diagram of the first electronic device provided in an embodiment of the present application. As shown in Figure 4, the electronic device may include a first chip 40. The first chip may include a first interface 41, a second interface 42 and at least one seventh interface 43. Among them, the first chip 40 can obtain the first image data of the first image (i.e., the real image) through the first interface 41, can obtain the second image (i.e., the virtual image) through the second interface 42, and can output the VR composite image to the display module through the seventh interface 43. It should be noted that if the electronic device is a VR display device, then in addition to the first chip 40, the electronic device may also include other chips, camera modules and display modules other than the first chip 40. Other chips can be used to process the exposure stage of the real image and the synthesis stage of the virtual image. The number of other chips can be one or more, which is not limited here.

[0154] Exemplarily, the first chip may be one of the chips in the chip system 11 as shown in (a) of FIG1 . The chip system 11 may also include other chips in addition to the above-mentioned first chip, and cooperate with other chips to achieve the purpose of displaying VR synthetic images.

[0155] In some implementations, the aforementioned other chips may be chips with image processing functions used by other electronic devices, such as chips used in smartphones that are already in use.

[0156] Exemplarily, the first chip can also be a chip in the smart glasses 14 as shown in (b) of Figure 1, and other chips can be configured in the smart phone 13 or the smart glasses 14. When the other chips are configured in the smart phone 13, the first interface 41 and the second interface 42 corresponding to the first chip can be wireless communication interfaces or wired interfaces, so as to receive real-life images and / or virtual images sent by the smart phone 13. Of course, the number of other chips can also be multiple. For example, the second chip for obtaining real-life images can be set in the smart glasses 14, and the third chip for generating synthetic images can be set in the smart phone 13. The specific settings can be made according to actual conditions and are not limited here. As mentioned above, the second chip and the third chip can be chips in other electronic devices, thereby improving the utilization rate of the chips and reducing the production cost of VR display devices.

[0157] Exemplarily, the first chip can also be a chip in a smart phone 13 as shown in (b) in Figure 1, and other chips can be configured in the smart phone 13 or the smart glasses 14. In this case, the output image finally generated by the first chip can be transmitted to the smart glasses 14 wirelessly or by wire, so that the corresponding VR synthetic picture can be displayed by the smart glasses 14. It should be noted that the second chip for generating the real-scene image can be set on the smart glasses 14. In this case, the first interface can be a wireless communication interface or a wired interface, so as to receive the first image data of the real-scene image collected by the smart glasses 14. The specific setting method of other chips can be set according to actual conditions and is not limited here.

[0158] Specifically, FIG5 shows a flow chart of a display method provided by an embodiment of the present application. In combination with the electronic device of FIG4 and the flow chart of FIG5, the display method provided by the embodiment of the present application includes:

[0159] In S501, the first chip obtains first image data of a first image through a first interface; the first image may be a real scene image obtained through a camera module; the first image data may be partial data or all data of the first image.

[0160] In S502 , the first chip obtains part or all of the data of the second image through the second interface.

[0161] In an embodiment of the present application, the first chip can obtain virtual and real images respectively through the above two interfaces, that is, the first image data corresponding to the real scene image captured by the camera module, and the virtual image that can be synthesized by software. This display method can be applied to the display field of VST technology, that is, there is a real scene image corresponding to the scene where the user is located in the display screen, and this part of the image can be obtained by capturing the scene where the user is located by the camera module; at the same time, some virtual objects can be added to the above scene, such as a synthesized video screen, or virtual objects can be added to the scene. The virtual object can be an interactive virtual object, such as a virtual user, or a non-interactive virtual object, such as a virtual photo frame. The first chip can perform image synthesis on the above two types of images, that is, complete the operation of combining virtual and real, such as the relevant steps of stage 3 in Figure 3.

[0162] In an embodiment of the present application, the first interface may be used to receive first image data of a real-scene image. The chip that acquires the real-scene image may be a different chip from the first chip. This chip may perform exposure and preprocessing operations on the real-scene image, and may also transmit the processed first image data of the real-scene image to the first chip via the first interface, thereby enabling the real-scene image exposure operation to be performed in parallel with the virtual-reality fusion operation and the display operation.

[0163] In some implementations, the first image data may be all the data of the first image, that is, another chip different from the first chip may transmit the first image data to the first chip after completing the exposure operation of the entire real-scene image, so as to facilitate subsequent operations of the first chip. For example, FIG6 shows a partial timing diagram of a VR composite image after completing the exposure of the entire real-scene image. Referring to FIG6 (a), which is a timing flow chart of the existing VR display technology, the VR device needs to process the exposure operation of a real-scene image and the virtual-real synthesis operation of the real-scene image before executing the exposure operation of the next real-scene image, that is, the time delay between the composite images of each frame must be greater than the sum of the exposure time T1, the data transmission time T2 and the processing time T3 required for the above-mentioned virtual-real combination. Different from the existing VR display technology, as shown in (b) in Figure 6, since the exposure operation of the real-scene image and the virtual-reality fusion operation can be completed by different chips, for example, another chip different from the first chip (such as the second chip) can perform the exposure operation of the next real-scene image after completing the exposure operation of the entire real-scene image and transmitting it to the first chip. While the second chip is performing the exposure operation of the next real-scene image, the first chip can perform the subsequent steps. The two stages are carried out simultaneously, so that the time delay between each frame of VR synthetic image is reduced from T1+T2+T3 to T1+T2, thereby increasing the smoothness of image display.

[0164] In some implementations, the above-mentioned first image data may be partial data of a real-scene image. In this case, the first interface is an interface that supports data transmission of sliced ​​images, such as an interface that supports pipeline processing. In this case, the first chip can receive the first image data sent line by line or in multiple lines in the real-scene image, and perform subsequent virtual-real fusion and display operations based on the first image data sent line by line or in multiple lines. Exemplarily, FIG7 shows a partial timing diagram of a VR synthetic image when the real-scene image is sliced. Referring to FIG7 (a), as described above, the time delay between the synthetic images of each frame must be greater than the sum of the exposure time T1, the data transmission time T2 and the processing time T3 required for the above-mentioned virtual-real combination. Different from the existing VR display technology, as shown in (b) in Figure 7, the real-scene image can be divided into two segments, that is, for the first frame of the real-scene image, image 1 can be divided into segments 11 and 12. After the exposure of segment 11 is completed, segment 11 can be sent to the first chip for virtual-reality fusion and display operations. At the same time, the first chip can continue to expose part of segment 12, and after the exposure of segment 12 is completed, it can be transmitted to the first chip for subsequent processing. After the data is segmented, the transmission time required for each segment is reduced, thereby reducing the delay between each frame of VR synthetic image from T1+T2+T3 to T1+T2.

[0165] In some implementations, if the chip performing real-scene image exposure completes the exposure operation of the first segment in the first frame of real-scene image and then continues to capture the next frame of real-scene image, that is, the exposure operation of the first segment of the subsequent frame of real-scene image can be synchronized with the exposure operation of the second segment of the previous frame of real-scene image, then the delay between each frame of VR synthetic image will be further shortened, as shown in (c) of Figure 7. After the exposure of segment 11 of the first frame of real-scene image is completed, the exposure operation of segment 12 will be performed. At the same time, the exposure operation of segment 21 of the subsequent frame of real-scene image (i.e., image 2) can also be performed, thereby reducing the delay between each frame of VR synthetic image to T1 / 2+T2 / 2.

[0166] In an embodiment of the present application, the transmission of image fragment data is supported between the chip that processes the exposure process of the real-scene image and the first chip, which can further reduce the delay between different frames of the VR synthetic image, thereby improving the smoothness of the picture, avoiding dizziness caused by excessive picture delay, and improving the user's viewing experience.

[0167] In an embodiment of the present application, the second interface can be specifically used to receive a second image, which can be an image synthesized by software, that is, a virtual image. The chip that generates the virtual image can be another chip different from the first chip (such as a third chip). If it is necessary to add content that is not in the real image to the VR synthesized image, the corresponding virtual image (i.e., the second image) can be synthesized by the third chip, and then the virtual image can be sent to the first chip through the second interface so that the first chip can synthesize the virtual image with the real image.

[0168] In some implementations, the virtual image can also support image slice transmission. That is, the first chip can obtain all first image data of the second image through the second interface. For example, after the third chip generates the entire virtual image, it sends it to the first chip through the second interface. Optionally, upon receiving a synchronization signal from the first or second chip, the third chip can send part of the first image data of the virtual image to the first chip through the second interface. The first chip can perform image fusion of part of the real scene image and part of the virtual image, and then perform subsequent display operations, thereby achieving the purpose of image slice processing and improving display smoothness.

[0169] In the embodiment of the present application, the chip for generating the real scene image (ie, the second chip) and the chip for generating the virtual image (ie, the third chip) can be the same chip or different chips, and can be specifically configured according to actual conditions.

[0170] Exemplarily, FIG8 shows a schematic diagram of a VR display system provided in an embodiment of the present application. The VR display system may include a VR display device, and in some scenarios, may also include other electronic devices, such as smart phones. Referring to FIG8 , the VR display device may include a first chip 81. In some scenarios, the VR display device may also include another chip 82. Of course, the chip 82 may also be provided on other electronic devices, such as smart phones. The chip 82 may be used to implement operations of generating real-life images and generating virtual images. The first chip 81 may include an interface 811 and an interface 812. The interface 811 may be electrically connected to the interface 821 of the chip 82, and may be used to receive first image data of a real-life image sent by the chip 82; the interface 812 may be electrically connected to the interface 822 of the chip 82, and may be used to receive virtual images sent by the chip 82.

[0171] Among them, interface 811 and interface 812 can be electrically connected to transmit image slicing data. Optionally, the above-mentioned interface 811 and interface 821 can be SMIO interfaces. Since the SMIO interface provides a high-speed, low-latency communication channel and can support a bandwidth of up to 1TB / s, it can reduce the time required for transmission when transmitting real-scene data, thereby shortening the transmission delay between chips. It should be noted that the SMIO interface can support image slicing transmission technology. Among them, the first chip and the second chip can be chips packaged based on chiplet technology, and the interface between the above-mentioned two chips can be an SMIO interface, so that high-speed mutual transmission of data between the two chips can be achieved to achieve the purpose of collaborative work. The interface between the above-mentioned two chips can be other interfaces that support high-speed data transmission.

[0172] The interface 812 and the interface 822 can be electrically connected to transmit the virtual image. Optionally, the interface 812 and the interface 822 can be MIPI interfaces.

[0173] In some implementations, the chip 81 may include a fusion module 814 and image processing modules 815 having the same number as the output channels. For example, if the VR display device can output images through two screens, the number of the image processing modules 815 may be 2. The fusion module 814 may be used to implement reprojection of virtual images and reprojection of real-scene images, and to fuse the reprojected virtual and real images. The image processing module may be used to perform image processing on the synthesized image, such as preprocessing, anti-distortion processing, and anti-dispersion processing, so that the synthesized image meets the display requirements of the screen of the corresponding output channel. Preprocessing may include image enhancement operations such as adjusting image contrast, color cast, and color difference.

[0174] In some implementations, the chip 82 may include an ISP 823, a graphics processing unit (GPU) 824, a data processing unit (DPU) 825, and a central processing unit (CPU) 826. The ISP 823 may be electrically connected to an interface 827, which may be connected to the camera module 83, and may receive an image signal transmitted by the camera module, perform exposure based on the image signal, convert the image data into corresponding first image data, and send the first image data to the first chip 81 through the interface 821. At the same time, the chip 82 may generate a virtual image through the GPU 824, send the virtual image to the DPU 825 for rendering, and send the processed virtual image to the first chip through the interface 822.

[0175] In S503, the first chip generates output images corresponding to each output channel according to the first image data and the second image data; each output channel corresponds to a display module; and the output image can be displayed by the display module corresponding to the output channel.

[0176] In an embodiment of the present application, after acquiring the real scene image and the virtual image, the first chip can perform virtual-real fusion to obtain a VR composite image, that is, the above-mentioned output image, and can send the output image to the display module through the output channel.

[0177] In an embodiment of the present application, the first chip can receive a synchronization signal sent by other chips through the first interface or the second interface, generate an output image according to the synchronization signal, and send the output image to the corresponding display module for display.

[0178] Furthermore, in the embodiment of the present application, the above S503 may further include S5031 and S5032:

[0179] In S5031 , the first chip generates a composite image based on the first image data and the second image data.

[0180] In an embodiment of the present application, after obtaining the first image data of the real-world image and the virtual image, the first chip may synthesize the two images. Optionally, each virtual image may correspond to a display coordinate, and based on the display coordinate, the virtual image may be added to the corresponding position in the real-world image, thereby generating a virtual-real VR image.

[0181] Furthermore, in the embodiment of the present application, FIG9 shows a schematic diagram of a process for generating a composite image provided by the embodiment of the present application. Referring to FIG9 , compared with the embodiment shown in FIG5 , the embodiment of the present application may further include S901 before S5031, and the aforementioned S5031 may further include: S902 to S904, which are specifically described as follows:

[0182] In S901 , the first chip obtains first depth information of a first image and second depth information of a second image through a first interface.

[0183] In an embodiment of the present application, when generating a VR composite image, the first chip can perform a two-step operation: a reprojection operation and a synthesis operation. Since VR composite images are generally three-dimensional images with a depth of field effect, after obtaining the first image data, a three-dimensional projection can be performed to form a three-dimensional image with a stereoscopic viewing effect. Based on this, the first chip can obtain the first depth information of the real image and the second depth information of the virtual image through the first interface.

[0184] In S902 , the first chip constructs a first depth image according to the first depth information and the first image data.

[0185] In an embodiment of the present application, the first chip can reproject the first image data of the real-scene image according to the first depth information corresponding to the real-scene image, that is, the two-dimensional real-scene image can be projected into a three-dimensional image according to the first depth information, that is, a first depth image is obtained.

[0186] In some implementations, if the real-scene image sends the first image data by image segmentation, that is, the first image data is partial data of the entity image, then the depth information corresponding to the partial image can be obtained from the first depth information, and the first depth image of the partial image can be reconstructed.

[0187] In S903 , the first chip constructs a second depth image according to the second image and the second depth information.

[0188] In an embodiment of the present application, similarly, the first chip can also construct a three-dimensional image of the virtual image in the corresponding three-dimensional coordinate system based on the second depth information corresponding to the virtual image, that is, the above-mentioned second depth image. The specific process of constructing the second depth image can be found in the description of S902 and will not be repeated here.

[0189] It should be noted that the first chip may use the same three-dimensional coordinate system when re-projecting the first image data and the virtual image, thereby facilitating subsequent synthesis operations of the real scene image and the virtual image.

[0190] In S904 , the first chip generates a composite image according to the first depth image and the second depth image.

[0191] In the embodiment of the present application, the first chip can perform image fusion on the re-projected first depth image and the second depth image to form a three-dimensional image with a depth of field effect, that is, the above-mentioned composite image.

[0192] In an embodiment of the present application, the first chip can construct a depth image with a depth of field effect by receiving depth information about the real scene image and the virtual image, and then can fuse the two depth images into virtual and real images, thereby improving the consistency between the synthesized image and the real scene, and thus improving the user's immersion and authenticity when watching the VR screen.

[0193] In S5032, the first chip may process the synthesized image using image processing algorithms corresponding to each output channel to generate an output image corresponding to the output channel; each output channel may correspond to a display module; and the output image may be displayed by the display module corresponding to the output channel.

[0194] In the embodiment of the present application, the first chip may further include a seventh interface that can be connected to the display module. Continuing with FIG8 , the first chip 81 may further include an interface 813 that can be connected to the display module 84 and can transmit the generated composite image to the display module via the interface 813 for displaying the VR composite image on the display module.

[0195] Exemplarily, the VR display device may be a head-mounted display device, with each eye of the user corresponding to a display module. In this case, the number of the above-mentioned interfaces 813 may be two, for outputting a composite image of the display module corresponding to the left eye and a composite image of the display module corresponding to the right eye.

[0196] In an embodiment of the present application, the first chip can perform image processing on the above-mentioned composite image through an image processing algorithm, thereby generating an output image corresponding to a subsequent display module. It should be noted that if there are multiple display modules connected to the first chip, different display modules can correspond to different channels, such as a left eye channel and a right eye channel. The first chip can process the composite image according to the output requirements corresponding to the different channels through an image processing algorithm corresponding to the output channel, thereby obtaining an output image corresponding to the output channel, such as a right eye output image and a left eye output image, and displaying the corresponding output image through the display module corresponding to the output channel.

[0197] In some implementations, the image processing algorithm may include an anti-distortion algorithm and an anti-dispersion algorithm.

[0198] From the above, it can be seen that in a display method provided by an embodiment of the present application, when generating a VR composite image, the image synthesis and image processing and other contents can be handed over to an independent first chip for completion, so that the exposure operation of the real-scene image can be performed synchronously with the subsequent operations such as image synthesis and image processing, so that when the VR composite image needs to be output continuously, the waiting time required between each frame of the composite image can be reduced, thereby improving the smoothness of the VR display screen. Compared with existing display technologies, since the embodiment of the present application can perform the acquisition of the real-scene image and the subsequent image synthesis and image processing operations when displaying the VR composite image, the logical operations of the two parts can be performed synchronously, thereby shortening the waiting time required between VR image synthesis, reducing the time delay, and thereby improving the smoothness of the VR screen, reducing the dizziness of the user when watching, and improving the user's experience.

[0199] FIG10 shows a schematic diagram of the structure of the second electronic device provided in an embodiment of the present application. Referring to FIG10 , compared with the embodiment shown in FIG4 , the electronic device provided in an embodiment of the present application may include, in addition to the first chip 10, a second chip 20 for processing the exposure operation of the real scene image. The first chip may include a first interface 101, a second interface 102, and at least one seventh interface 107; the second chip may include a third interface 103, a fourth interface 104, and an eighth interface 108. The functions implemented by each interface in the first chip 10 can be found in the relevant description of the first embodiment and will not be repeated here.

[0200] In the embodiment of the present application, the first interface 101 of the first chip 10 is electrically connected to the third interface 103 of the second chip 20 .

[0201] In some implementations, the first interface 101 and the third interface 103 may be interfaces that support image slice transmission.

[0202] In some implementations, the first interface 101 and the third interface 103 may be SMIO interfaces.

[0203] In an embodiment of the present application, the fourth interface 104 of the second chip 20 can be electrically connected to the interfaces of other chips used to generate virtual images, for example, it can be electrically connected to the sixth interface of the third chip 30, and data sent by the chip that generates the virtual image can be received through the sixth interface.

[0204] In some implementations, the chip 20 may include an ISP 201 and a storage unit. In particular, the storage unit may include a register 202, and the amount of the written first image data may be recorded through the register 202 in the storage unit. The eighth interface 108 may be electrically connected to the input end of the ISP, the ISP 201 may be electrically connected to the register 202, and the register 202 may be electrically connected to the third interface 103. The ISP 201 may write the first image data of the real scene image obtained by exposure into the storage unit, and then set the value of the register 202 according to the amount of data stored in the storage unit. For example, the value of the register may be set according to the number of rows and columns written. The second chip 20 may control the third interface 103 according to the value of the register to obtain the first image data of the corresponding number of rows from the storage unit and send it to the first chip 10, so as to achieve the purpose of fragmented transmission of the real scene image.

[0205] Exemplarily, the chip system composed of the first chip and the second chip can be the chip system 11 in Figure 1 (a). In addition to the above-mentioned first chip and second chip, the chip system 11 can also include other chips to achieve the purpose of displaying VR synthetic images.

[0206] For example, the chip system composed of the first chip and the second chip can also be a chip in the smart glasses 14 shown in Figure 1(b), and the other chip can be configured in the smartphone 13 or the smart glasses 14. When the other chip is configured in the smartphone 13, the second interface 102 in the first chip 10 and the fourth interface 104 of the second chip can be a wireless communication interface or a wired interface, thereby being able to receive data related to the virtual image sent by the smartphone 13. The specific configuration can be based on actual conditions and is not limited here.

[0207] For example, the chip system composed of the first chip and the second chip can also be a chip in a smartphone 13 as shown in (b) of Figure 1, and the other chips can be configured in the smartphone 13 or the smart glasses 14. In this case, the output image finally generated by the first chip can be transmitted to the smart glasses 14 via wireless or wired means, so that the corresponding VR composite image can be displayed by the smart glasses 14.

[0208] Specifically, FIG11 shows a flow chart of a display method provided by another embodiment of the present application. In combination with the electronic device of FIG10 and the flow chart of FIG11, the display method provided by the embodiment of the present application includes:

[0209] In S1101 , the second chip receives first image data obtained by exposure of the camera module.

[0210] In S1102 , the second chip sends the first image data to the first chip through the third interface.

[0211] In an embodiment of the present application, the second chip can be connected to the camera module through the eighth interface, so that it can receive the image signal of the real-scene image transmitted by the camera module. The second chip can convert the image signal of the real-scene image into first image data and write it into the storage unit. When the preset transmission timing is met, the stored first image data can be sent to the first chip through the third interface to complete subsequent synthesis operations through the first chip.

[0212] In some implementations, the second chip can also receive a synchronization signal sent by the third chip through the sixth interface, and can send the corresponding first image data to the first chip according to the synchronization signal, thereby maintaining synchronization between multiple chips, improving the accuracy of subsequent VR display images, and avoiding screen tearing.

[0213] In some implementations, the second chip may send all data of the real scene image to the first chip after exposing the entire real scene image.

[0214] In some implementations, when image fragment transmission is supported between the third interface and the first interface, the second chip can send several rows of first image data of the real-scene image to the first chip to implement fragment transmission of the real-scene image, so as to reduce the inter-frame delay of the VR synthetic image.

[0215] Correspondingly, when image fragment transmission is supported between the third interface and the first interface, the second chip may further include an ISP and a read-write control unit. Specifically, the above S1101 and S1102 may be: the graphics signal processor may receive the first image data exposed by the camera module, and may write the first image data into the storage unit, and may set the value of the register according to the amount of data written into the storage unit; the second chip may send the first image data of the corresponding data amount to the first chip through the third interface according to the value of the register.

[0216] In an embodiment of the present application, the second chip may include an ISP and a register. The ISP may be connected to the eighth interface, may receive the image signal sent by the camera module through the eighth interface, convert the image signal into the first image data, and write it into the corresponding storage unit, thereby completing the operation of the above-mentioned stage 1. The process of the above-mentioned ISP writing the first image data may be writing the acquired first image data into the storage unit row by row. The ISP may set the value in the register according to the amount of data written into the storage unit, for example, may set the value of the register according to the number of rows or columns of the first image data written.

[0217] In an embodiment of the present application, the second chip can determine the value of the above-mentioned register when a preset read trigger condition is met (for example, based on a locally generated synchronization signal, or based on a synchronization signal sent by a third chip) to determine the amount of data that has been written, and obtain the first image data of the corresponding data amount from the storage unit. For example, the value of the register can be 10, that is, the first image data of 10 rows in the storage unit is obtained, and can be sent to the first chip through the third interface.

[0218] In some implementations, the first chip can generate a synchronization signal and send the synchronization signal to the second chip through the first interface. The second chip can determine the number of rows of first image data recorded in the storage unit in response to the synchronization signal, and then send the corresponding amount of first image data to the first chip through the third interface.

[0219] In an embodiment of the present application, a register can be set in the second chip to determine the amount of first image data that has been written, and then the first image data of the corresponding amount can be obtained and sent, so as to improve the reading accuracy of the first image data segment transmission process, and then reduce the delay of the entire VR composite image generation.

[0220] Furthermore, in an embodiment of the present application, the fourth interface of the second chip can be electrically connected to the sixth interface of the third chip. The third chip can be a chip for generating a virtual image. The second chip can receive a depth image of the virtual image sent by the third chip:

[0221] In S1103 , the first chip obtains second image data through the second interface; the second image may be a virtual image.

[0222] In S1104 , the first chip generates a composite image based on the first image data and the second image data.

[0223] In S1105, the first chip processes the synthesized image using the image processing algorithms corresponding to the respective output channels to generate output images corresponding to the output channels; each output channel corresponds to a display module; and the output image can be displayed by the display module corresponding to the output channel.

[0224] Specifically, the implementation method of steps S1103 to S1105 can be exactly the same as the implementation method of Example 1. The specific description of S1103 can refer to the relevant description of S502, the specific description of S1104 can refer to the relevant description of S5031, and the description of S1105 can refer to the relevant description of S5032. No further details will be given here.

[0225] For example, FIG12 shows a schematic diagram of a VR display system provided by another embodiment of the present application. The VR display system may include a VR display device. In some scenarios, the VR display system may also include other electronic devices in addition to the VR display device, such as a smart phone. Referring to FIG12 , the VR display device may include a first chip 121 and a second chip 122. In addition, the VR display device may also include a third chip 123. Of course, in some implementation scenarios, the third chip may be provided on other electronic devices, such as a smart phone. Among them, the functions of the various interfaces and modules of the first chip 121 can refer to the relevant description of the chip 81 in the embodiment of FIG8 , and the functions of the various interfaces and modules of the second chip 122 can refer to the relevant description of the chip 82 in the embodiment of FIG8 .

[0226] Different from the embodiment shown in FIG8 , the second chip 122 may include an ISP 1221 and a register 1222 . The implementation functions of the above two modules may be described in relation to S1101 and S1102 and are not limited here.

[0227] In an embodiment of the present application, the third chip 123 can be used to synthesize a virtual image and a depth image corresponding to the virtual image. The third chip 123 can send the virtual image to the first chip 121 for virtual-real fusion, and can also send the depth image of the virtual image to the second chip 122, so that the second depth information corresponding to the depth image can be extracted through the second chip 122, so as to facilitate the subsequent reprojection by the fusion module 1211 in the first chip 121.

[0228] From the above, it can be seen that in a display method provided by an embodiment of the present application, when generating a VR composite image, image synthesis and image processing and other contents can be handed over to an independent first chip for completion, so that the exposure operation of the real-scene image can be performed synchronously with the subsequent operations such as image synthesis and image processing, so that when the VR composite image needs to be output continuously, the waiting time required between each frame of the composite image can be reduced, thereby improving the smoothness of the VR display screen. Compared with existing display technologies, since the embodiment of the present application can perform the acquisition of the real-scene image and the subsequent image synthesis and image processing operations by different chips when displaying the VR composite image, the logical operations of the two parts can be executed synchronously, thereby shortening the waiting time required between VR image synthesis, reducing the time delay, and thereby improving the smoothness of the VR screen, reducing the dizziness of the user when watching, and improving the user's experience.

[0229] FIG13 shows a schematic diagram of the structure of a third electronic device provided in an embodiment of the present application. Compared to the embodiments shown in FIG4 and FIG10 , FIG13 shows that the electronic device provided in an embodiment of the present application includes, in addition to a first chip 131 and a second chip 132, a third chip 133 for generating a virtual image.

[0230] The first chip may include a first interface 1301, a second interface 1302, and at least a seventh interface 1307; the second chip may include a third interface 1303, a fourth interface 1304, and an eighth interface 1308. The functions implemented by the various interfaces in the first chip 131 and the second chip 132 can be found in the descriptions of the first and second embodiments and are not repeated here.

[0231] The third chip may include a fifth interface 1305 and a sixth interface 1306. The fifth interface 1305 may be electrically connected to the second interface 1302 and may be used to transmit the virtual image. In some implementation scenarios, it may also be used to receive a synchronization signal sent by the first chip 131 or to send a synchronization signal to the first chip 131. The sixth interface 1306 may be electrically connected to the fourth interface 1304 and may be used to transmit second depth information of the virtual image. In some implementation scenarios, it may also be used to receive a synchronization signal sent by the second chip 132 or to send a synchronization signal to the second chip 132.

[0232] In some implementations, the fourth interface 1304 of the second chip 132 and the sixth interface 1306 of the third chip 133 may be interfaces based on the PCIE protocol, and in particular, may be interfaces based on the PCIE 4.0 protocol.

[0233] In some implementations, the second interface 1302 of the first chip 131 and the fifth interface 1305 of the third chip 133 may be interfaces based on the MIPI protocol.

[0234] In some implementations, the third chip 133 may include a GPU 1331 and a DPU 1332 .

[0235] The DPU 1332 may be electrically connected to the fifth interface 1305, and may be used to generate a virtual image and send the virtual image to the first chip 1331 via the fifth interface 1305. The DPU 1332 may be electrically connected to the GPU 1331, and may transmit the virtual image to the GPU 1331.

[0236] The GPU 1331 may be electrically connected to the sixth interface 1306 , and the GPU 1331 may be configured to generate a depth image corresponding to the virtual image, and send the depth image to the second chip 132 through the sixth interface.

[0237] In some implementations, the third chip 133 may further include a CPU 1333. The CPU 1333 may be used to control the GPU 1331 and the DPU 1332 to perform corresponding operations, such as generating a virtual image and a depth image of the virtual image.

[0238] Specifically, FIG14 shows a flow chart of a display method provided by another embodiment of the present application. In combination with the electronic device of FIG13 and the flow chart of FIG14, the display method provided by the embodiment of the present application includes:

[0239] In S1401 , the second chip receives first image data obtained by exposure of the camera module.

[0240] In S1406 , the second chip sends the first image data, the first depth information, and the second depth information to the first chip through the third interface.

[0241] Specifically, the implementation method of steps S1401 to S1402 is exactly the same as that of Example 2. The specific description of S1401 can refer to the relevant description of S1101, and the specific description of S1402 can refer to the relevant description of S1102, which will not be repeated here.

[0242] In S1402 , the third chip generates second image data.

[0243] In S1403 , the third chip sends the second image data to the first chip through the fifth interface.

[0244] In the embodiment of the present application, the third chip can generate a corresponding virtual image based on the user's viewing needs. The virtual image can be a video screen such as the video screen in Figure 2(b) or a virtual control such as a virtual keyboard. The specific generation can be based on actual conditions. The number of virtual images required to be generated for each frame can be one or more, and is not limited here.

[0245] In some implementations, the first chip can send a synchronization signal to the third chip, and the third chip can receive the synchronization signal sent by the first chip via a fifth interface. The synchronization signal can carry a frame number, and the third chip can send the corresponding virtual image to the first chip via the fifth interface based on the frame number, so that the first chip can complete operations such as virtual-reality fusion and display.

[0246] In some implementations, the third chip may be a chip that controls synchronization between chips. The third chip may generate a synchronization signal and send the synchronization signal to the first chip and the second chip through the fifth interface and the sixth interface, thereby ensuring operational synchronization between the chips.

[0247] In some implementations, the second chip can be a signal that controls synchronization between chips. In this case, the third chip can receive the synchronization signal sent by the second chip through the sixth interface, or it can receive the synchronization signal sent by the second chip forwarded by the first chip through the fifth interface, that is, the second chip can first send a synchronization signal to the first chip, and then the first chip can forward the synchronization signal to the third chip.

[0248] In S1404 , the third chip generates second depth information corresponding to the second image.

[0249] In S1405 , the third chip sends the second depth information to the second chip through the sixth interface.

[0250] In an embodiment of the present application, the third chip can also generate second depth information corresponding to the virtual image through the GPU, and can send the second depth information to the second chip, so that the second depth information can be sent to the first chip through the second chip. The first chip can reproject the virtual image in the three-dimensional coordinate system according to the second depth information to achieve a VR synthetic image with a depth of field effect.

[0251] It should be noted that if the second chip does not receive the second depth information sent by the third chip when sending the first image data to the first chip, the second depth information corresponding to the virtual image of the previous frame can be used as the second depth information of the virtual image of the current frame, thereby avoiding the output delay of the entire VR composite image due to the delay in obtaining the second depth information, thereby improving the smoothness of the display.

[0252] In S1407 , the first chip generates a composite image according to the first image data, the second image, the first depth information, and the second depth information.

[0253] In S1408, the first chip processes the synthesized image using the image processing algorithms corresponding to the respective output channels to generate output images corresponding to the output channels; each output channel corresponds to a display module; and the output image can be displayed by the display module corresponding to the output channel.

[0254] From the above, it can be seen that in a display method provided by an embodiment of the present application, when generating a VR composite image, the image synthesis and image processing and other contents can be handed over to an independent first chip for completion, so that the exposure operation of the real-scene image can be performed synchronously with the subsequent operations such as image synthesis and image processing, so that when the VR composite image needs to be output continuously, the waiting time required between each frame of the composite image can be reduced, thereby improving the smoothness of the VR display screen. Compared with existing display technologies, since the embodiment of the present application can perform the acquisition of the real-scene image and the subsequent image synthesis and image processing operations when displaying the VR composite image, the logical operations of the two parts can be performed synchronously, thereby shortening the waiting time required between VR image synthesis, reducing the time delay, and thereby improving the smoothness of the VR screen, reducing the dizziness of the user when watching, and improving the user's experience.

[0255] Figure 15 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in Figure 15, the electronic device 15 of this embodiment includes: at least one processor 150 (only one processor is shown in Figure 15, and the number of processors can match the number of chips actually included in the electronic device in the embodiment), a memory 151, and a program 152 stored in the memory 151 and executable on the at least one processor 150. When the processor 150 executes the program 152, the steps of any of the above-mentioned electronic device setting method embodiments are implemented.

[0256] The electronic device 15 may be a VR display device, a smartphone, etc. The electronic device may include, but is not limited to, a processor 150 and a memory 151. Those skilled in the art will appreciate that FIG15 is merely an example of the electronic device 15 and does not limit the electronic device 15. The electronic device 15 may include more or fewer components than shown, or may combine certain components or different components. For example, it may also include input and output electronic devices, network access electronic devices, etc.

[0257] The processor 150 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0258] In some embodiments, the memory 151 may be an internal storage unit of the electronic device 15, such as a hard disk or memory of the electronic device 15. In other embodiments, the memory 151 may also be an external storage electronic device of the electronic device 15, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 15. Furthermore, the memory 151 may also include both an internal storage unit of the electronic device 15 and an external storage electronic device. The memory 151 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the program. The memory 151 may also be used to temporarily store data that has been output or is to be output.

[0259] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0260] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0261] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0262] An embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores a program, and when the program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0263] An embodiment of the present application provides a program product. When the program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments when executing the program product.

[0264] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0265] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0266] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A display method, characterized in that, Applied to an electronic device, the electronic device includes a first chip, and the display method includes: The first chip obtains first image data of a first image through a first interface; the first image is a real scene image obtained by a camera module; the first image data is part or all of the data of the first image; The first chip obtains second image data of a second image through a second interface; the second image data is part or all of the data of the second image; The first chip generates a composite image based on the first image data and the second image data; The first chip processes the composite image through image processing algorithms corresponding to respective output channels to generate output images corresponding to the output channels; each output channel corresponds to a display module; the output images are displayed through the display modules corresponding to the output channels.

2. The method according to claim 1, characterized in that It further includes: The first chip obtains first depth information of the first image and second depth information of the second image through the first interface; The first chip generates a composite image based on the first image data and the second image data, including: The first chip constructs a first depth image based on the first depth information and the first image data; The first chip constructs the second depth image based on the second image data and the second depth information; The first chip generates the composite image based on the first depth image and the second depth image.

3. The method according to claim 1, wherein The image processing algorithms include: an anti-distortion processing algorithm and an anti-color cast processing algorithm.

4. The method according to any one of claims 1 to 3, characterized in that, The first chip obtains first image data of a first image through a first interface, including: The first chip receives, through the first interface, the first image data of several rows in the first image sent by a second chip.

5. The method according to any one of claims 1-4, characterized in that, The electronic device further includes a second chip, and there is an electrical connection between a third interface of the second chip and the first interface of the first chip; the display method includes: The second chip receives the first image data obtained by the camera module through exposure; The second chip sends the first image data to the first chip through the third interface.

6. The method according to claim 5, characterized in that, The second chip includes a graphics signal processor and a storage unit; The graphics signal processor receives the first image data obtained by the camera module through exposure and sets the value of the storage unit according to the first image data; The value of the storage unit is used to determine the data volume of the first image data; The second chip sends the first image data corresponding to the data volume to the first chip through the third interface.

7. The method according to claim 5, wherein The first interface and the third interface perform data transmission of the first image data through an electrical connection; the first image data is obtained by fragmenting the first image.

8. The method according to claim 5, characterized in that, It further includes: The second chip receives second depth information of the second image sent by a third chip through a fourth interface; The second chip determines first depth information of the first image data; The first depth information and the second depth information are used to generate the composite image; The second chip sends the first image data to the first chip through a third interface, including: The second chip sends the first image data, the first depth information, and the second depth information to the first chip through the third interface.

9. The method according to any one of claims 1 to 8, characterized in that, The electronic device further includes a third chip; there is an electrical connection between a fifth interface of the third chip and the second interface; the display method includes: The third chip generates the second image data; The third chip sends the second image data to the first chip through the fifth interface.

10. The method according to claim 9, characterized in that, The third chip sends the second image data to the first chip through the fifth interface, including: The third chip receives a synchronization signal sent by the first chip through the fifth interface; In response to the synchronization signal, the third chip sends the second image data to the first chip through the fifth interface.

11. The method according to claim 9, wherein There is an electrical connection between a sixth interface of the third chip and a fourth interface of the second chip; the display method includes: The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the output image.

12. An electronic device, characterized in that, The electronic device includes a first chip; the first chip includes a first interface, a second interface, and at least one seventh interface; the first interface is electrically connected to a third interface of a second chip; the second interface is electrically connected to a fifth interface of a third chip; the seventh interface is electrically connected to a display module; the second chip and the third chip are the same chip or different chips; The first chip is configured to obtain the first image data of the first image sent by the second chip through the first interface; the first image is a real scene image obtained by a camera module; the first image data is part or all of the data of the first image; The first chip is configured to obtain the second image data of the second image sent by the third chip through the second interface; the second image data is part or all of the data of the second image; The first chip is configured to generate a composite image based on the first image data and the second image data; The first chip is configured to process the composite image through an image processing algorithm corresponding to each output channel to generate an output image corresponding to the output channel; The first chip is configured to send the output image to the display module of the output channel corresponding to the seventh interface through the seventh interface, so as to display the output image through the display module.

13. The electronic device according to claim 12, characterized in that, The first chip is further configured to: obtain the first depth information of the first image and the second depth information of the second image data through the first interface; The first chip is configured to generate a composite image based on the first image data and the second image data, including: The first chip is configured to construct a first depth image based on the first depth information and the first image data; The first chip is configured to construct the second depth image based on the second image data and the second depth information; The first chip is configured to generate the composite image based on the first depth image and the second depth image.

14. The electronic device according to claim 12, characterized in that, The image processing algorithm includes: an undistortion processing algorithm and an anti-color cast processing algorithm.

15. The electronic device according to any one of claims 12-14, characterized in that, The first chip is configured to receive, through the first interface, the first image data of several rows in the first image sent by the second chip.

16. The electronic device according to any one of claims 12-15, characterized in that, The electronic device further includes a second chip; an eighth interface of the second chip is electrically connected to the camera module; The second chip is configured to receive the first image data obtained by the exposure of the camera module. The second chip is configured to send the first image data to the first chip through the third interface.

17. The electronic device according to claim 16, wherein The second chip includes a graphics signal processor and a storage unit; the graphics signal processor is electrically connected to the storage unit; the graphics signal processor is electrically connected to the eighth interface; the storage unit is electrically connected to the third interface; The graphics signal processor is configured to receive the first image data obtained by the exposure of the camera module and set the value of the storage unit according to the first image data. The value of the storage unit is used to determine the data volume of the first image data. The second chip is configured to, according to the value of the storage unit, send the first image data corresponding to the data volume through the third interface to the first chip.

18. The electronic device according to claim 16, wherein The first interface and the third interface perform data transmission of the first image data through electrical connection; the first image data is obtained by fragmenting the first image.

19. The electronic device according to claim 16, characterized in that, The second chip includes a fourth interface; the fourth interface is electrically connected to a sixth interface of the third chip; The second chip is configured to receive, through the fourth interface, the second depth information of the second image sent by the third chip. The second chip is configured to determine the first depth information of the first image data. The first depth information and the second depth information are used to generate the composite image. The second chip is configured to send the first image data, the first depth information, and the second depth information to the first chip through the third interface.

20. The electronic device according to any one of claims 12-19, characterized in that, The electronic device further includes a third chip; The third chip is configured to generate the second image data. The third chip is configured to send the second image data to the first chip through the fifth interface.

21. The electronic device according to claim 20, wherein The third chip is configured to send the second image data to the first chip through the fifth interface, including: The third chip is configured to receive, through the fifth interface, a synchronization signal sent by the first chip. The third chip is configured to, in response to the synchronization signal, send the second image data to the first chip through the fifth interface.

22. The electronic device according to claim 20, characterized in that, A sixth interface of the third chip is electrically connected to the fourth interface of the second chip; The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the composite image.

23. A chip, characterized in that, The chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps performed by the first chip in the method according to any one of claims 1 to 4, or the steps performed by the second chip in the method according to any one of claims 5 to 8, or the steps performed by the third chip in the method according to any one of claims 9 to 11.

24. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 11.

Images