Display method, electronic equipment, chip and storage medium

Abstract

The invention is suitable for the technical field of display, and provides a display method, electronic equipment, a chip and a storage medium, the method comprises the following steps: a first chip obtains first image data of a first image through a first interface; the first image is a live-action image acquired through a camera module; the first image data is partial data or all data of the first image; the first chip obtains second image data of the second image through the second interface; the first chip generates a composite image according to the first image data and the second image data; the first chip processes the synthesized image through an image processing algorithm corresponding to each output channel to generate an output image corresponding to the output channel; each output channel corresponds to one display module; and the output image is displayed through a display module corresponding to the output channel. According to the technical scheme provided by the invention, the waiting time required for VR image synthesis can be shortened, and the time delay is reduced, so that the fluency of the VR image is improved, and the dizzy feeling when a user watches the VR image is reduced.

Description

Technical Field

[0001] This application belongs to the technical field of device management, and particularly relates to a display method, an electronic device, a chip, and a storage medium. Background Art

[0002] Virtual Reality (VR) technology, as one of the mainstream display technologies today, has a wider and wider application field. Compared with traditional two-dimensional display technology, VR display technology has the characteristic of high immersion. Therefore, how the VR display device can display a smooth picture directly affects the immersion of users in the process of watching multimedia data through the VR display device. As an important branch of VR display technology, Video See Through (VST) captures the real scene image of the user's location through a camera module, and superimposes virtual content on the real scene image through VR technology, allowing users to view virtual content and the real environment consistent with reality at the same time, further improving the immersion of users in the viewing process.

[0003] In order to combine the above-mentioned real scene image and virtual image, the VR display device needs to perform multiple steps of image processing on the actual image to generate a VR composite image. Different image processing steps need to be sequentially executed by the processor according to the processing timing. Some processing steps have a large time delay, resulting in an unsmooth final output picture, reducing the immersion of users in watching, and affecting the viewing experience of users. Summary of the Invention

[0004] The embodiments of this application provide a display method, an electronic device, a chip, and a computer-readable storage medium, which can solve the problem that in the existing VR display technology, when generating a VR composite image, different image processing steps need to be sequentially executed by the same processor according to the processing timing, and the processing duration is too long, resulting in an unsmooth picture and affecting the viewing experience of users.

[0005] In a first aspect, the embodiments of this application provide a display method, which is applied to an electronic device. The electronic device includes a first chip. The display method includes:

[0006] The first chip obtains the first image data of the first image through a first interface; the first image is a real scene image obtained through a camera module; the first image data is part or all of the data of the first image;

[0007] The first chip obtains the second image data of the second image through a second interface; the second image data is part or all of the data of the second image;

[0008] The first chip generates a composite image according to the first image data and the second image data;

[0009] The first chip processes the composite image through the image processing algorithms corresponding to each of the output channels to generate the output images corresponding to each of the output channels; the output images are displayed through the display modules corresponding to the output channels.

[0010] Implementing the embodiments of the present application has the following beneficial effects: When generating a VR composite image, the content such as image synthesis and image processing is handed over to an independent first chip, so that the exposure operation of the real scene image can be executed synchronously with the subsequent operations such as image synthesis and image processing. Therefore, when continuous output of VR composite images is required, the waiting time required between each frame of composite images can be reduced, thereby improving the smoothness of the VR display screen. Compared with the existing display technologies, since in the embodiments of the present application, when displaying a VR composite image, the acquisition of the real scene image and the subsequent operations of image synthesis and image processing are handed over to different chips, the logical operations of the two parts can be executed synchronously, thereby shortening the waiting time required for VR image synthesis, reducing the time delay, improving the smoothness of the VR screen, reducing the dizziness of the user when watching, and improving the user experience.

[0011] In a possible implementation manner of the first aspect, it further includes:

[0012] The first chip obtains the first depth information of the first image and the second depth information of the second image data through the first interface;

[0013] The first chip generates a composite image according to the first image data and the second image data, including:

[0014] The first chip constructs a first depth image according to the first depth information and the first image data;

[0015] The first chip constructs the second depth image according to the second image data and the second depth information;

[0016] The first chip generates the composite image according to the first depth image and the second depth image.

[0017] In a possible implementation manner of the first aspect, the image processing algorithms include: an anti-distortion processing algorithm and an anti-color deviation processing algorithm.

[0018] In a possible implementation manner of the first aspect, the first chip obtains the first image data of the first image through the first interface, including:

[0019] The first chip receives the first image data of several rows in the first image sent by the second chip through the first interface.

[0020] In a possible implementation of the first aspect, before the first chip generates output images corresponding to respective output channels based on the first image data and the second image data, it further includes:

[0021] The first chip receives a first synchronization signal sent by a second chip through a first interface;

[0022] The first chip generates output images corresponding to respective output channels based on the first image data and the second image data, including:

[0023] The first chip, in response to the first synchronization signal, generates output images corresponding to respective output channels based on the first image data and the second image data.

[0024] In a possible implementation of the first aspect, after the first chip receives the first synchronization signal through the first interface, it further includes:

[0025] The first chip sends the first synchronization signal to a third chip through the second interface, so that the third chip sends the second image data according to the first synchronization signal.

[0026] In a possible implementation of the first aspect, the electronic device further includes a second chip, and there is an electrical connection between a third interface of the second chip and the first interface of the first chip; the display method includes:

[0027] The second chip receives the first image data obtained by the exposure of the camera module;

[0028] The second chip sends the first image data to the first chip through the third interface.

[0029] In a possible implementation of the first aspect, the second chip is connected to at least two camera modules to obtain a first image with depth information.

[0030] In a possible implementation of the first aspect, the camera module is a depth camera module to obtain a first image with depth information.

[0031] In a possible implementation of the first aspect, the second chip includes a graphics signal processor and a storage unit;

[0032] The graphics signal processor receives the first image data obtained by the exposure of the camera module and sets the value of the storage unit according to the first image data; the value of the storage unit is used to determine the data volume of the first image data;

[0033] The second chip sends the first image data corresponding to the data volume to the first chip through a third interface according to the value of the storage unit.

[0034] In a possible implementation manner of the first aspect, the first interface and the third interface perform data transmission of the first image data through an electrical connection; the first image data is obtained by slicing a first image.

[0035] In some implementation manners, the first interface and the third interface may be Substrate-like Multi-chip Package Input / Output (SMIO) interfaces.

[0036] In a possible implementation manner of the first aspect, it further includes:

[0037] The second chip receives second depth information of the second image data sent by a third chip through a fourth interface;

[0038] The second chip determines first depth information of the first image data; the first depth information and the second depth information are used to generate the output image;

[0039] The second chip sending the first image data to the first chip through the third interface includes:

[0040] The second chip sends the first image data, the first depth information, and the second depth information to the first chip through the third interface.

[0041] In a possible implementation manner of the first aspect, the fourth interface and the sixth interface are interfaces based on the Peripheral Component Interconnect Express (PCIE) protocol.

[0042] In a possible implementation manner of the first aspect, before the first chip generates an output image corresponding to each output channel according to the first image data and the second image data, it further includes:

[0043] The second chip sends a first synchronization signal to the first chip through the third interface, and sends a second synchronization signal to the third chip through the fourth interface.

[0044] In a possible implementation manner of the first aspect, the electronic device further includes a third chip; there is an electrical connection between a fifth interface of the third chip and the second interface; the display method includes:

[0045] The third chip generates the second image data;

[0046] The third chip sends the second image data to the first chip through a fifth interface.

[0047] In some implementation manners, the second interface and the fifth interface are interfaces based on the Mobile Industry Processor Interface (MIPI) protocol.

[0048] In a possible implementation manner of the first aspect, the third chip sending the second image data to the first chip through the fifth interface includes:

[0049] The third chip receives a first synchronization signal sent by the first chip through the fifth interface or receives a second synchronization signal sent by the second chip through a sixth interface;

[0050] In response to the first synchronization signal or the second synchronization signal, the third chip sends the second image data to the first chip through the fifth interface.

[0051] In a possible implementation manner of the first aspect, the sixth interface of the third chip is electrically connected to the fourth interface of the second chip; the display method includes:

[0052] The third chip sends second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the output image.

[0053] In a possible implementation manner of the first aspect, it further includes:

[0054] The third chip sends a third synchronization signal to the first chip through the fifth interface and sends a fourth synchronization signal to the second chip through the sixth interface.

[0055] In a second aspect, an electronic device, characterized in that the electronic device includes a first chip; the first chip includes a first interface, a second interface, and at least one seventh interface; the first interface is electrically connected to the third interface of a second chip; the second interface is electrically connected to the fifth interface of a third chip; the seventh interface is electrically connected to a display module; the second chip and the third chip are the same chip or different chips;

[0056] The first chip is configured to obtain first image data of a first image sent by the second chip through the first interface; the first image is a real scene image obtained by an imaging module; the first image data is part of or all of the first image;

[0057] The first chip is configured to obtain second image data of a second image sent by the third chip through a second interface; the second image data is partial data or all data of the second image;

[0058] The first chip is configured to generate an output image corresponding to each output channel according to the first image data and the second image data;

[0059] The first chip is configured to send the output image to a display module of the output channel corresponding to the seventh interface through the seventh interface, so as to display the output image through the display module.

[0060] In a possible implementation manner of the second aspect, the first chip is configured to generate an output image corresponding to each output channel according to the first image data and the second image data, including:

[0061] The first chip is configured to generate a composite image according to the first image data and the second image data;

[0062] The first chip is configured to process the composite image through an image processing algorithm corresponding to each output channel to generate the output image corresponding to each output channel.

[0063] In a possible implementation manner of the second aspect, the first chip is further configured to: obtain first depth information of the first image and second depth information of the second image data through the first interface;

[0064] The first chip is configured to generate a composite image according to the first image data and the second image data, including:

[0065] The first chip is configured to construct a first depth image according to the first depth information and the first image data;

[0066] The first chip is configured to construct the second depth image according to the second image data and the second depth information;

[0067] The first chip is configured to generate the composite image according to the first depth image and the second depth image.

[0068] In a possible implementation manner of the second aspect, the image processing algorithm includes: an undistortion processing algorithm and an anti-color cast processing algorithm.

[0069] In a possible implementation manner of the second aspect, the first chip is configured to receive the first image data of several rows in the first image sent by the second chip through the first interface.

[0070] In a possible implementation of the second aspect, the first chip is further configured to:

[0071] The first chip is configured to receive a first synchronization signal sent by a second chip through a first interface;

[0072] The first chip is configured to generate an output image corresponding to each output channel according to the first image data and the second image data, including:

[0073] In response to the first synchronization signal, the first chip generates an output image corresponding to each output channel according to the first image data and the second image data.

[0074] In a possible implementation of the second aspect, the first chip is further configured to: send the first synchronization signal to a third chip through the second interface, so that the third chip sends the second image data according to the first synchronization signal.

[0075] In a possible implementation of the second aspect, the electronic device further includes a second chip; an eighth interface of the second chip is electrically connected to a camera module;

[0076] The second chip is configured to receive the first image data obtained by the camera module during exposure;

[0077] The second chip is configured to send the first image data to the first chip through a third interface.

[0078] In a possible implementation of the second aspect, the second chip includes a graphics signal processor and a storage unit; the graphics signal processor is electrically connected to the storage unit; the graphics signal processor is electrically connected to the eighth interface; the storage unit is electrically connected to the third interface;

[0079] The graphics signal processor is configured to receive the first image data obtained by the camera module during exposure and set the value of the storage unit according to the first image data; the value of the storage unit is used to determine the data volume of the first image data;

[0080] The second chip is configured to send the first image data corresponding to the data volume to the first chip through the third interface according to the value of the storage unit.

[0081] In a possible implementation of the second aspect, the first interface and the third interface perform data transmission of the first image data through electrical connection; the first image data is obtained by slicing a first image.

[0082] In a possible implementation of the second aspect, the second chip includes a fourth interface; the fourth interface is electrically connected to the sixth interface of the third chip;

[0083] The second chip is configured to receive, through the fourth interface, the second depth information of the second image data sent by the third chip;

[0084] The second chip is configured to determine the first depth information of the first image data; the first depth information and the second depth information are used to generate the output image;

[0085] The second chip is configured to send the first image data, the first depth information, and the second depth information to the first chip through the third interface.

[0086] In a possible implementation of the second aspect, the second chip is further configured to: send a first synchronization signal to the first chip through the third interface, and send a second synchronization signal to the third chip through the fourth interface.

[0087] In a possible implementation of the second aspect, the electronic device further includes a third chip;

[0088] The third chip is configured to generate the second image data;

[0089] The third chip is configured to send the second image data to the first chip through the fifth interface.

[0090] In some implementations, the above-mentioned second interface and fifth interface are interfaces based on the Mobile Industry Processor Interface (MIPI) protocol.

[0091] In a possible implementation of the second aspect, the third chip is configured to send the second image data to the first chip through the fifth interface, including:

[0092] The third chip is configured to receive, through the fifth interface, the first synchronization signal sent by the first chip or receive, through the sixth interface, the second synchronization signal sent by the second chip;

[0093] The third chip is configured to, in response to the first synchronization signal or the second synchronization signal, send the second image data to the first chip through the fifth interface.

[0094] In a possible implementation of the second aspect, the sixth interface of the third chip is electrically connected to the fourth interface of the second chip;

[0095] The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the output image.

[0096] In a possible implementation manner of the second aspect, the third chip is further configured to: send a third synchronization signal to the first chip through the fifth interface, and send a fourth synchronization signal to the second chip through the sixth interface.

[0097] In a third aspect, an embodiment of the present application provides a chip, including: a memory, a processor, and a program stored in the memory. When the processor executes the program, the steps of the first chip, the second chip, or the third chip in any one of the display methods in the first aspect described above are implemented.

[0098] In a fourth aspect, an embodiment of the present application provides a readable storage medium. The readable storage medium stores a program, and when the program is executed by a processor, the steps of the first chip, the second chip, or the third chip in any one of the display methods in the first aspect described above are implemented.

[0099] In a fifth aspect, an embodiment of the present application provides a program product. When the program product runs on a device, the device is caused to execute the steps of the first chip, the second chip, or the third chip in any one of the display methods in the first aspect described above.

[0100] In a sixth aspect, an embodiment of the present application provides a chip system, including a first chip, a second chip, and a third chip;

[0101] The first chip includes a processor, the processor is coupled to a memory, and the processor executes a program stored in the memory to implement the steps of the first chip in any one of the display methods in the first aspect;

[0102] The second chip includes a processor, the processor is coupled to a memory, and the processor executes a program stored in the memory to implement the steps of the second chip in any one of the display methods in the first aspect;

[0103] The third chip includes a processor, the processor is coupled to a memory, and the processor executes a program stored in the memory to implement the steps of the third chip in any one of the display methods in the first aspect.

[0104] It can be understood that the beneficial effects of the second aspect to the sixth aspect can refer to the relevant descriptions in the first aspect above, and will not be elaborated here. Description of the Drawings

[0105] Figure 1It is a schematic diagram showing the implementation of the VR composite image displayed by the electronic device provided in an embodiment of the present application;

[0106] Figure 2 It is a schematic diagram of the screen based on the VST display technology provided in an embodiment of the present application;

[0107] Figure 3 It is a data flow diagram of the VR composite image based on the VST technology provided in an embodiment of the present application;

[0108] Figure 4 It is a schematic structural diagram of the first electronic device provided in an embodiment of the present application;

[0109] Figure 5 It is a schematic flowchart of the display method provided in an embodiment of the present application;

[0110] Figure 6 It is a partial timing diagram of the VR composite image after the exposure of the entire real scene image is completed, provided in an embodiment of the present application;

[0111] Figure 7 It is a partial timing diagram of the VR composite image when the real scene image is segmented, provided in an embodiment of the present application;

[0112] Figure 8 It is a schematic diagram of the VR display system provided in an embodiment of the present application;

[0113] Figure 9 It is a schematic flowchart of generating a composite image provided in an embodiment of the present application;

[0114] Figure 10 It is a schematic structural diagram of the second electronic device provided in an embodiment of the present application;

[0115] Figure 11 It is a schematic flowchart of the display method provided in another embodiment of the present application;

[0116] Figure 12 It is a schematic diagram of the VR display system provided in another embodiment of the present application;

[0117] Figure 13 It is a schematic structural diagram of the third electronic device provided in an embodiment of the present application;

[0118] Figure 14 It is a schematic flowchart of the display method provided in yet another embodiment of the present application;

[0119] Figure 15 It is a schematic structural diagram of the electronic device provided in an embodiment of the present application. Detailed implementation manners

[0120] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present application.

[0121] It should be understood that when used in the specification and claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0122] It should also be understood that the term "and / or" as used in the specification and claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0123] As used in the specification and claims of the present application, the term "if" can be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0124] In addition, in the description of the specification and claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0125] The reference to "one embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0126] The display method provided by the embodiments of the present application can be applied to augmented reality (AR) / virtual reality (VR) display devices, smartphones, tablets, and other electronic devices that can implement VR display. In particular, the display method can be applied to electronic devices that can implement VR display, or electronic devices externally connected with VR display devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0127] Exemplarily, Figure 1 FIG. shows a schematic diagram of the implementation of the VR composite image displayed by the electronic device provided by an embodiment of the present application.

[0128] As Figure 1 shown in (a) of, the electronic device can be a wearable VR display device. The VR display device can be internally provided with a chip system 11 and a camera module 12. The chip system 11 includes: a second chip, a first chip, etc. The first image data of the real scene image is obtained from the camera module 12 through the second chip. Through the processing and synthesis of the first chip on the first image data, the camera module 12 can be used to obtain the environmental image of the scene where the wearer (i.e., the user) is located. The chip system 11 can synthesize the environmental image and the virtual picture to generate a VR composite image based on the VST technology.

[0129] As Figure 1 shown in (b) of, the electronic device can be a smartphone. The smartphone 13 includes a first chip. The smartphone 13 can establish a communication connection with a smart glasses 14. The communication connection can be a wired communication connection or a wireless communication connection. For example, the smartphone 13 can be connected to the smart glasses 14 through a serial port. If the smart glasses 14 are configured with a wireless communication module, such as a Bluetooth module or a WIFI module, the smartphone 13 can establish a communication connection with the smart glasses 14 through the wireless communication module. The smart glasses 14 can also be configured with a camera module to collect the environmental image and feedback it to the smartphone 13. The smartphone 13 can synthesize the environmental image and the virtual picture through the built-in processor to generate a VR composite image based on the VST technology, and feedback it to the wearable glasses to output the VR composite image.

[0130] With the continuous development of display technology, VR display technology has become one of the mainstream display technologies today. Traditional VR display technology provides users with an immersive viewing experience, and the above-mentioned traditional VR display technology has been applied in many fields such as games and movies. However, since traditional VR display technology constructs one or more virtual graphics through the GPU in the processor and generates a purely fictional virtual picture through the combination of virtual graphics, its authenticity is relatively low, thus reducing the user's sense of immersion. In order to further improve the user's viewing experience and enhance the sense of immersion, VST technology has emerged. Compared with the traditional VR display technology that outputs a purely virtual virtual picture, the VST display technology can collect the real environment images of the scene captured by the user through the camera module configured on the VR display device, and superimpose the virtual picture based on synthesis on the real environment image, that is, obtain the VR composite image based on VST technology. Since the background picture in the VR composite image is often generated based on the real environment image, it can improve the authenticity of the picture and then enhance the sense of immersion of the user during the process of viewing the VR composite picture.

[0131] Exemplarily, Figure 2 FIG. shows a schematic diagram of a picture based on VST display technology provided by an embodiment of the present application. Refer to Figure 2 as shown in (a) in. The user wears a VR display device, and this VR display device can be Figure 1 the VR display device shown in (a) in. This VR display device is built-in with a camera module, and through the camera module, the environmental images within the user's line of sight can be obtained. For example, objects such as a TV set and a clock can be photographed. The VR display device can synthesize the virtual image to be synthesized into the above environmental image. Such as Figure 2 the picture shown in (b) in. Comparing Figure 2 (a) in with Figure 2 (b) in, it can be found that in the real scene, the TV set is in the off state, that is, it does not output any picture content. When image synthesis is performed through the VR display device to generate the corresponding VR composite image, the specified video image frame can be added to the area where the TV set is located in the environmental image, realizing the combination of the virtual picture and the real picture, improving the immersion of the output picture, and then enhancing the user's viewing experience.

[0132] Then, although the VST display technology can improve the authenticity of VR synthetic images and then enhance the immersion of users' viewing experience, it also introduces new challenges to VR display devices, namely, a large amount of data needs to be processed and the processing process is relatively long. Since the VR synthetic images need to combine virtual and real scenes, that is, real scene images need to be captured by a camera module, which involves processes such as image exposure and image transmission. The combination of virtual images requires operations such as generating virtual images and synthesis. Then, the real scene images and virtual images need to be synthesized, that is, multiple steps need to be executed sequentially according to the processing order.

[0133] Exemplarily, Figure 3 FIG. shows a data flow diagram of a VR synthetic image based on the VST technology provided by an embodiment of the present application. Refer to Figure 3 As shown, in the process of generating a VR synthetic image by an electronic device, it specifically includes the following multiple stages:

[0134] Stage 1: Image exposure stage

[0135] Since real scene images of the user's location need to be obtained, image capture needs to be performed through a camera module. When capturing real scene images through the camera module, it includes exposure (such as 11 milliseconds, ms) and the first image data reading (8.5 ms). Among them, the above two steps are mainly completed by the Graphics Signal Processor (ISP) in the chip. Among them, the image signal processing stage can be divided into the processing stage through the Image Front End (IFE) (such as 0.5 ms) and the processing stage through the Image Processing Engine (IPE) (such as 8.5 ms). The above multiple processing stages all take time, so it can be seen that it will bring a certain processing delay to the process of generating VR synthetic images.

[0136] Stage 2: Image transmission stage

[0137] When the ISP obtains the real scene images captured by the camera module, the first image data of the real scene images needs to be transmitted to relevant threads at the system layer for processing, and this will consume a certain transmission time, such as 6.5 ms.

[0138] Stage 3: Primary rendering stage and video parsing stage

[0139] When an electronic device generates a VR composite image, it needs to use a rendering application at the application layer to complete the task of rendering virtual objects, such as rendering a virtual keyboard, a virtual cartoon image, etc., which requires a rendering stage. Among them, this rendering stage can be performed in parallel with the above-mentioned image exposure stage, that is, while the image is exposed and transmitted, the electronic device can perform a rendering process, which also introduces a certain processing delay.

[0140] Similarly, if it is necessary to insert the generated video data into the picture, the video decoder in the application framework layer needs to parse the video data frame by frame, and the above process also requires a certain amount of processing time.

[0141] Stage 4: Virtual Image Fusion

[0142] Since there are virtual objects generated in a rendering phase and video image frames obtained in the video phase, the virtual objects need to be fused with the video image frames. The virtual first image data fusion also needs to introduce a processing time, for example, 0.6ms.

[0143] Stage 5: Second rendering stage

[0144] This stage needs to complete the fusion of virtual and real, that is, the fusion of real images and virtual images, which involves the image reprojection stage (such as 10.2ms) and the virtual and real fusion stage (such as 21.3ms), so as to generate VR synthetic images. The delay of this stage is relatively long. In some implementation scenarios, the average delay of the secondary rendering stage can reach 21.3ms.

[0145] Stage 6: Image processing stage for synthesized images

[0146] The VR synthesized image needs to be processed, such as anti-distortion processing and anti-dispersion processing, to adapt to the corresponding display module.

[0147] Stage 7: Image display stage

[0148] When the display module obtains the VR composite image of the corresponding channel, it needs to scan row by row or column by column to output the image.

[0149] It can be seen that when generating a VR synthetic image based on VST, the process from generating the first image data to the final VR synthetic image needs to go through multiple stages, and the various stages are basically in a serial execution relationship. Therefore, the processing of the VR synthetic image of the previous frame can be completed before the processing of the VR synthetic image of the next frame can be executed, resulting in the interval between VR synthetic images of different frames being too long, affecting the smoothness of the overall picture, and excessive inter-frame delay will also cause users to feel dizzy when viewing the picture, which in turn affects the user experience.

[0150] Therefore, to solve the problems of the above VR display technology, an embodiment of the present application provides a display method. The execution subject of the display method can be an electronic device, which can include but is not limited to: VR display devices, smartphones, and other electronic devices capable of synthesizing VR composite images based on the VST technology. The electronic device can be built with a VR display module or externally connected to a wearable VR display device to output the above VR composite image through the externally connected wearable VR display device. Among them, the VR display device includes a first chip, which processes the synthesis of real and virtual images and the output adaptation of the display module for image processing, so as to improve the smoothness of the output picture, reduce the dizziness of the user during viewing, and enhance the user's viewing experience.

[0151] Figure 4 FIG. shows a schematic structural diagram of a first electronic device provided by an embodiment of the present application. Refer to Figure 4 As shown, the electronic device may include a first chip 40. The first chip may include a first interface 41, a second interface 42, and at least one seventh interface 43. Among them, the first chip 40 can obtain the first image data of the first image (i.e., the real scene image) through the first interface 41, obtain the second image (i.e., the virtual image) through the second interface 42, and output the VR composite image to the display module through the seventh interface 43. It should be noted that if the electronic device is a VR display device, in addition to the first chip 40, the electronic device may further include other chips, a camera module, and a display module other than the first chip 40. The other chips can be used to process the exposure stage of the real scene image and the synthesis stage of the virtual image. The number of other chips can be one or more, which is not limited here.

[0152] Exemplarily, the first chip can be, for example, Figure 1 one of the chips in the chip system 11 in (a) as shown, and the chip system 11 may further include other chips in addition to the above first chip, and cooperate with other chips to achieve the purpose of displaying VR composite images.

[0153] In some implementation manners, the above other chips can be chips with image processing functions used by other electronic devices, such as chips applied to existing smartphones.

[0154] Exemplarily, the first chip can also be, for example, Figure 1The chip in the smart glasses 14 shown in (b) thereof. Other chips may be configured in the smart phone 13 or the smart glasses 14. When other chips are configured in the smart phone 13, the first interface 41 corresponding to the first chip and the second interface 42 may be wireless communication interfaces or wired interfaces, so as to be able to receive the real scene image and / or virtual image sent by the smart phone 13. Of course, the number of other chips may also be multiple. For example, the second chip for acquiring the real scene image may be disposed in the smart glasses 14, and the third chip for generating the synthetic image may be disposed in the smart phone 13. Specifically, it may be set according to the actual situation and is not limited herein. As described above, the second chip and the third chip may be chips in other electronic devices, thereby improving the utilization rate of the chips and reducing the production cost of the VR display device.

[0155] Exemplarily, the first chip may also be a chip in the smart phone 13 shown in (b) of Figure 1 thereof. Other chips may be configured in the smart phone 13 or the smart glasses 14. In this case, the output image finally generated by the first chip may be transmitted to the wearable glasses 14 wirelessly or by wire, so as to display the corresponding VR synthetic picture through the smart glasses 14. It should be noted that the second chip for generating the real scene image may be disposed on the smart glasses 14. In this case, the first interface may be a wireless communication interface or a wired interface, so as to be able to receive the first image data of the real scene image acquired by the smart glasses 14. Specifically, the setting manner of other chips may be set according to the actual situation and is not limited herein.

[0156] Specifically, Figure 5 shows a schematic flowchart of a display method provided by an embodiment of the present application. Combining Figure 4 the electronic device and Figure 5 the flowchart, the display method provided by the embodiment of the present application includes:

[0157] In S501, the first chip acquires the first image data of the first image through the first interface; the first image may be a real scene image acquired through the imaging module; the first image data may be partial data or all data of the first image.

[0158] In S502, the first chip acquires partial data or all data of the second image through the second interface.

[0159] In the embodiments of the present application, the first chip can obtain real and virtual images respectively through the above two interfaces, that is, the first image data corresponding to the real scene image captured by the camera module, and the virtual image that can be synthesized by software. This display method can be applied to the display field of VST technology, that is, there is a real scene image corresponding to the user's scene in the display screen, and this part of the image can be obtained by the camera module capturing the image of the user's scene; at the same time, some virtual objects can also be added to the above-mentioned scene, such as a synthesized video screen, or virtual objects can be added to the scene, and the virtual object can be an interactive virtual object, such as a virtual user, or a non-interactive virtual object, such as a virtual photo frame, etc. The first chip can perform image synthesis on the above two types of images, that is, complete the operation of combining reality and virtuality. For example Figure 3 the relevant steps of stage 3 in

[0160] In the embodiments of the present application, the first interface can be used to receive the first image data of the real scene image. Among them, the chip for obtaining the real scene image can be another chip different from the first chip. This chip can complete the exposure operation and preprocessing operation of the real scene image, and can also send the first image data after processing the real scene image to the first chip through the first interface, so as to realize that the exposure operation, the virtual-real fusion operation and the display operation of the real scene image can be executed in parallel.

[0161] In some implementation manners, the above first image data can be all the data of the first image, that is, another chip different from the first chip can transmit the first image data to the first chip after completing the exposure operation of the entire real scene image, so as to facilitate the subsequent operations of the first chip. Exemplarily, Figure 6 shows a partial timing schematic diagram of the VR composite image after the exposure of the entire real scene image is completed. Refer to Figure 6 As shown in (a) in , it is the timing flowchart of the existing VR display technology. The VR device needs to complete the exposure operation of a real scene image and the virtual-real synthesis operation of the real scene image before performing the exposure operation of the next real scene image, that is, the time delay between the composite images of each frame must be greater than the sum of the exposure duration T1, the data transmission duration T2 and the processing duration T3 required for the above virtual-real combination. Different from the existing VR display technology, refer to Figure 6As shown in (b), since the exposure operation of the real scene image and the virtual-real fusion operation can be completed by different chips. For example, another chip different from the first chip (such as the second chip) can perform the exposure operation of the entire real scene image and transmit it to the first chip, and then perform the exposure operation of the next real scene image. During the process of the second chip performing the exposure operation of the next real scene image, the first chip can perform the operations of subsequent steps. The two stages are carried out simultaneously, so that the time delay between each frame of VR composite images is reduced from T1 + T2 + T3 to T1 + T2, thereby increasing the smoothness of image display.

[0162] In some implementation manners, the above-mentioned first image data may be partial data of the real scene image. In this case, the first interface is an interface that supports the data transmission of fragmented images, such as an interface that supports pipeline processing. In this case, the first chip can receive the first image data sent row by row or multiple rows at a time in the real scene image, and perform subsequent virtual-real fusion and display operations based on the first image data sent row by row or multiple rows at a time. Exemplarily, Figure 7 shows a partial timing diagram of the VR composite image when the real scene image is fragmented. Refer to Figure 7 As shown in (a), as described above, the time delay between each frame of composite images must be greater than the sum of the exposure duration T1, the data transmission duration T2, and the processing duration T3 required for the above-mentioned virtual-real combination. Different from the existing VR display technology, refer to Figure 7 As shown in (b), the real scene image can be divided into two segments. That is, for the first frame of the real scene image, image 1 can be divided into segment 11 and segment 12. After segment 11 is exposed, segment 11 can be sent to the first chip for virtual-real fusion and display operations. At the same time, the first chip can continue to expose the part of segment 12, and after segment 12 is exposed, it can be transmitted to the first chip for subsequent processing. After the data is fragmented, the required transmission duration for each segment is reduced, so that the time delay between each frame of VR composite images is reduced from T1 + T2 + T3 to T1 + T2.

[0163] In some implementation manners, if the chip that performs the exposure of the real scene image can continue to capture the next frame of the real scene image after completing the exposure operation of the first segment in the first frame of the real scene image, that is, the exposure operation of the first segment of the subsequent frame of the real scene image can be synchronized with the exposure operation of the second segment of the previous frame of the real scene image, the time delay between each frame of VR composite images will be further shortened. For example, Figure 7As shown in (c) thereof. After the exposure of segment 11 of the real-scene image of the first frame is completed, the exposure operation of segment 12 is performed. At the same time, the exposure operation of segment 21 of the real-scene image of the subsequent frame (i.e., Image 2) can also be performed, so that the time delay between each frame of VR composite images is reduced to T1 / 2 + T2 / 2.

[0164] In the embodiment of the present application, image shard data transmission is supported between the chip for processing the exposure process of the real-scene image and the first chip, which can further reduce the time delay between different frames of VR composite images, thereby improving the smoothness of the picture, avoiding the user from feeling dizzy due to excessive picture time delay, and improving the user's viewing experience.

[0165] In the embodiment of the present application, the second interface can be specifically used to receive a second image, and the second image can be an image obtained by software synthesis, that is, a virtual image. The chip for generating the virtual image can be another chip different from the first chip (such as the third chip). If content that does not exist in the real-scene image needs to be added to the VR composite image, the corresponding virtual image (i.e., the second image) can be synthesized by the third chip, and then the virtual image can be sent to the first chip through the second interface, so that the first chip can perform image synthesis on the virtual image and the real-scene image.

[0166] In some implementation manners, the above virtual image also supports image shard transmission, that is, the first chip can obtain all the first image data of the second image through the second interface. For example, after the third chip generates the entire virtual image, it is sent to the first chip through the second interface; optionally, when the third chip receives a synchronization signal sent by the first chip or the second chip, it can send partial first image data of the virtual image to the first chip through the second interface. The first chip can perform image fusion and subsequent display operations on partial real-scene images and partial virtual images, thereby achieving the purpose of image shard processing to improve the smoothness of display.

[0167] In the embodiment of the present application, the chip for generating the real-scene image (i.e., the second chip) and the chip for generating the virtual image (i.e., the third chip) can be the same chip or different chips. It can be specifically set according to the actual situation.

[0168] Exemplarily, Figure 8 shows a schematic diagram of a VR display system provided by an embodiment of the present application. The VR display system may include a VR display device, and in some scenarios, it may further include other electronic devices, such as a smart phone. Refer to Figure 8As shown, the VR display device may include a first chip 81. In some scenarios, the VR display device may further include another chip 82. Of course, the chip 82 may also be disposed on other electronic devices, such as a smartphone. The chip 82 can be used to implement operations of generating real-scene images and generating virtual images. The first chip 81 may include an interface 811 and an interface 812. The interface 811 may be electrically connected to the interface 821 of the chip 82 and can be used to receive the first image data of the real-scene image sent by the chip 82. The interface 812 may be electrically connected to the interface 822 of the chip 82 and can be used to receive the virtual image sent by the chip 82.

[0169] Among them, the interface 811 and the interface 812 can transmit image shard data through electrical connection. Optionally, the interface 811 and the interface 821 may be SMIO interfaces. Since the SMIO interface provides a high-speed and low-latency communication channel and can support a bandwidth of up to 1TB / s, when transmitting real-scene data, the duration required for transmission can be reduced, thereby shortening the transmission delay between chips. It should be noted that the SMIO interface can support the transmission technology of image shards. Among them, the first chip and the second chip may be chips encapsulated based on the chiplet technology, and the interface between the two chips may be an SMIO interface, so as to enable high-speed data interconnection between the two chips to achieve the purpose of collaborative work. The interface between the two chips may be other interfaces that support high-speed data transmission.

[0170] Among them, the interface 812 and the interface 822 can transmit virtual images through electrical connection. Optionally, the interface 812 and the interface 822 may be MIPI interfaces.

[0171] In some implementation manners, the chip 81 may include a fusion module 814 and an image processing module 815 with the same number as the output channels. For example, if the VR display device outputs images through two screens, the number of the image processing modules 815 may be 2. The fusion module 814 can specifically be used to implement reprojection of virtual images, reprojection of real-scene images, and image fusion of the reprojected virtual and real images. The image processing module can be used to perform image processing on the synthesized image, such as preprocessing, anti-distortion processing, and anti-dispersion processing, so as to meet the display requirements of the screens of the corresponding output channels. Among them, the preprocessing may include image enhancement operations such as adjusting image contrast, color cast, and chromatic aberration.

[0172] In some implementations, the above chip 82 may include an ISP 823, a Graphics Processing Unit (GPU) 824, a Data Processing Unit (DPU) 825, and a Central Processing Unit (CPU) 826. Among them, the ISP 823 may be electrically connected to the interface 827, and the interface 827 may be connected to the imaging module 83, and may receive the image signal transmitted by the imaging module, perform exposure based on the image signal, convert it into corresponding first image data, and may send the first image data to the first chip 81 through the interface 821. At the same time, the chip 82 may generate a virtual image through the GPU 824, send the virtual image to the DPU 825 for primary rendering, and send the processed virtual image to the first chip through the interface 822.

[0173] In S503, the first chip generates output images corresponding to each output channel according to the first image data and the second image; each of the output channels corresponds to a display module; the output images can be displayed through the display module corresponding to the output channel.

[0174] In the embodiment of the present application, after obtaining the real scene image and the virtual image, the first chip may perform virtual-real fusion to obtain a VR composite image, that is, the above-mentioned output image, and may send the output image to the display module through the output channel.

[0175] In the embodiment of the present application, the first chip may receive a synchronization signal sent by other chips through the first interface or the second interface, generate an output image according to the synchronization signal, and send the output image to the corresponding display module for display.

[0176] Further, in the embodiment of the present application, the above S503 may further include S5031 and S5032:

[0177] In S5031, the first chip generates a composite image according to the first image data and the second image.

[0178] In the embodiment of the present application, after the first chip obtains the first image data of the real scene image and the virtual image, the above two images may be synthesized. Optionally, each virtual image may correspond to a display coordinate, and based on the above display coordinate, the virtual image may be added to the corresponding position in the real scene image, thereby generating a virtual-real combined VR image.

[0179] Further, in the embodiment of the present application, Figure 9 shows a schematic flow chart of generating a composite image provided by the embodiment of the present application. Refer to Figure 9 As shown, withFigure 5 Compared with the embodiments shown, in the embodiments of the present application, S901 may further be included before S5031, and the above S5031 may further include: S902 to S904, which are specifically described as follows:

[0180] In S901, the first chip obtains the first depth information of the first image and the second depth information of the second image through the first interface.

[0181] In the embodiments of the present application, when generating a VR composite image, the first chip can perform operations in two steps, namely a reprojection operation and a synthesis operation. Since the VR composite image is generally a three-dimensional image with a depth-of-field effect, after obtaining the first image data, three-dimensional projection can be performed to form a three-dimensional image with a stereoscopic viewing effect. Based on this, the first chip can obtain the first depth information of the real scene image and the second depth information of the virtual image through the first interface.

[0182] In S902, the first chip constructs a first depth image according to the first depth information and the first image data.

[0183] In the embodiments of the present application, the first chip can reproject the first image data of the real scene image according to the first depth information corresponding to the real scene image, that is, the two-dimensional real scene image can be projected into a three-dimensional image according to the first depth information, that is, the first depth image is obtained.

[0184] In some implementation manners, if the above real scene image sends the first image data in a manner of image sharding, that is, the above first image data is partial data of the entity image, the depth information corresponding to this partial image can be obtained from the first depth information, and the first depth image of this partial image can be reconstructed.

[0185] In S903, the first chip constructs a second depth image according to the second image and the second depth information.

[0186] In the embodiments of the present application, similarly, the first chip can also construct a three-dimensional image of the virtual image in the corresponding three-dimensional coordinate system according to the second depth information corresponding to the virtual image, that is, the above second depth image. For the specific process of constructing the second depth image, reference can be made to the description of S902, which will not be elaborated here.

[0187] It should be noted that the first chip can use the same three-dimensional coordinate system when reprojecting the first image data and the virtual image, so as to facilitate the subsequent synthesis operation of the real scene image and the virtual image.

[0188] In S904, the first chip generates a composite image according to the first depth image and the second depth image.

[0189] In an embodiment of the present application, the first chip may perform image fusion on the reprojected first depth image and the second depth image to form a three-dimensional image with a depth-of-field effect, that is, the above-mentioned synthetic image.

[0190] In an embodiment of the present application, the first chip may receive depth information about a real scene image and a virtual image to construct a depth image with a depth-of-field effect, and then may perform virtual-real fusion on the two depth images, which can improve the consistency between the synthetic image and the real scene, and then improve the immersion and authenticity when the user views the VR picture.

[0191] In S5032, the first chip may process the synthetic image through the image processing algorithms corresponding to each output channel to generate an output image corresponding to the output channel; each output channel may correspond to a display module; and the output image may be displayed through the display module corresponding to the output channel.

[0192] In an embodiment of the present application, the first chip may further include a seventh interface, and this seventh interface may be connected to the display module. Continuing to refer to Figure 8 As shown, the first chip 81 may further include an interface 813, and this interface 813 may be used to connect to the display module 84, and may send the generated synthetic image to the display module through the interface 813 to display the VR synthetic image through the display module.

[0193] Exemplarily, the VR display device may be a head-mounted display device, and each eye of the user corresponds to a display module. In this case, the number of the above-mentioned interfaces 813 may be two, which are used to output the synthetic image of the display module corresponding to the left eye and the synthetic image of the display module corresponding to the right eye.

[0194] In an embodiment of the present application, the first chip may perform image processing on the above-mentioned synthetic image through an image processing algorithm to generate an output image corresponding to a subsequent display module. It should be noted that if there are multiple display modules connected to the first chip, different display modules may correspond to different channels, such as a left-eye channel and a right-eye channel. The first chip may process the synthetic image through the image processing algorithm corresponding to the output channel according to the output requirements corresponding to different channels, so as to obtain an output image corresponding to the output channel, such as a right-eye output image and a left-eye output image, and display the corresponding output image through the display module corresponding to the output channel.

[0195] In some implementation manners, the above-mentioned image processing algorithm may include: an anti-distortion algorithm and an anti-dispersion algorithm.

[0196] As can be seen above, in a display method provided by an embodiment of the present application, when generating a VR composite image, content such as image synthesis and image processing can be completed by an independent first chip, so that the exposure operation of the real scene image can be executed synchronously with subsequent operations such as image synthesis and image processing. Therefore, when continuously outputting VR composite images is required, the waiting time required between each frame of composite images can be reduced, thereby improving the smoothness of the VR display screen. Compared with existing display technologies, since in the embodiment of the present application, when displaying a VR composite image, the acquisition of the real scene image and subsequent operations of image synthesis and image processing can be completed by different chips, and the logical operations of the two parts can be executed synchronously, thus shortening the waiting time required for VR image synthesis, reducing the time delay, improving the smoothness of the VR screen, reducing the dizziness of the user when viewing, and improving the user experience.

[0197] Figure 10 FIG. shows a schematic structural diagram of a second electronic device provided by an embodiment of the present application. Refer to Figure 10 As shown, compared with the Figure 4 embodiment shown, in the electronic device provided by the embodiment of the present application, in addition to including the first chip 10, it may further include a second chip 20 for processing the exposure operation of the real scene image. The first chip may include a first interface 101, a second interface 102, and at least one seventh interface 107; the second chip may include a third interface 103, a fourth interface 104, and an eighth interface 108. The functions implemented by each interface in the first chip 10 can refer to the relevant description of Embodiment 1, which will not be elaborated here.

[0198] In the embodiment of the present application, the first interface 101 of the first chip 10 is electrically connected to the third interface 103 of the second chip 20.

[0199] In some implementation manners, the first interface 101 and the third interface 103 may be interfaces supporting image fragment transmission.

[0200] In some implementation manners, the first interface 101 and the third interface 103 may be SMIO interfaces.

[0201] In the embodiment of the present application, the fourth interface 104 of the second chip 20 may be electrically connected to the interface of another chip for generating a virtual image. For example, it may be electrically connected to the sixth interface of the third chip 30, and data sent by the chip for generating a virtual image can be received through the sixth interface.

[0202] In some implementations, the above-mentioned chip 20 may include an ISP 201 and a storage unit. Specifically, the storage unit may include a register 202, and the amount of the first image data written can be recorded through the register 202 in the storage unit. Among them, the eighth interface 108 may be electrically connected to the input end of the ISP, the ISP 201 may be electrically connected to the register 202, and the register 202 may be electrically connected to the third interface 103. The ISP 201 may write the first image data regarding the real scene image obtained by exposure into the storage unit, and then may set the value of the register 203 according to the amount of data stored in the storage unit. For example, the value of the register may be set according to the number of written rows and columns. The second chip 20 may control the third interface 103 to obtain the first image data of the corresponding number of rows from the storage unit and send it to the first chip 10 according to the value of the register, so as to achieve the purpose of segmented transmission of the real scene image.

[0203] Exemplarily, the chip system composed of the first chip and the second chip may be the chip system 11 in (a) as shown in Figure 1 . In addition to the above-mentioned first chip and second chip in the chip system 11, other chips may also be included, so as to achieve the purpose of displaying the VR composite image.

[0204] Exemplarily, the chip system composed of the first chip and the second chip may also be the chip in the smart glasses 14 shown in (b) as shown in Figure 1 . Other chips may be configured in the smart phone 13 or the smart glasses 14. When other chips are configured in the smart phone 13, the second interface 102 in the first chip 10 and the fourth interface 104 of the second chip may be wireless communication interfaces or wired interfaces, so as to be able to receive the data related to the virtual image sent by the smart phone 13. Specifically, it can be set according to the actual situation and is not limited herein.

[0205] Exemplarily, the chip system composed of the first chip and the second chip may also be the chip in the smart phone 13 shown in (b) as shown in Figure 1 . Other chips may be configured in the smart phone 13 or the smart glasses 14. In this case, the output image finally generated by the first chip may be transmitted to the smart glasses 14 in a wireless or wired manner, so as to display the corresponding VR composite picture through the smart glasses 14.

[0206] Specifically, Figure 11 shows a schematic flowchart of a display method provided in another embodiment of the present application. Combining Figure 10 the electronic device and Figure 11 the flowchart, the display method provided in the embodiment of the present application includes:

[0207] In S1101, the second chip receives the first image data obtained by the exposure of the imaging module.

[0208] In S1102, the second chip sends the first image data to the first chip through the third interface.

[0209] In the embodiment of the present application, the second chip can be connected to the imaging module through the eighth interface, so as to receive the image signal of the real scene image transmitted by the imaging module. The second chip can convert the image signal of the real scene image into the first image data, write it into the storage unit, and when the preset transmission timing is met, the stored first image data can be sent to the first chip through the third interface, so as to complete the subsequent synthesis operation through the first chip.

[0210] In some implementation manners, the second chip can also receive the synchronization signal sent by the third chip through the sixth interface, and can send the corresponding first image data to the first chip according to the synchronization signal, so as to maintain the synchronization between multiple chips, improve the accuracy of the subsequent VR display image display, and avoid screen tearing.

[0211] In some implementation manners, after the exposure of the entire real scene image is completed, the second chip can send all the data of the real scene image to the first chip uniformly.

[0212] In some implementation manners, when the third interface and the first interface support image fragmentation transmission, the second chip can send the first image data of several rows of the real scene image to the first chip to implement the fragmentation transmission of the real scene image, so as to reduce the inter-frame delay of the VR composite image.

[0213] Correspondingly, when the third interface and the first interface support image fragmentation transmission, the second chip can also include an ISP and a read-write control unit. Specifically, S1101 and S1102 above can be: the graphics signal processor can receive the first image data obtained by the exposure of the imaging module, and can write the first image data into the storage unit, and can set the value of the register according to the data volume written into the storage unit; the second chip can send the first image data of the corresponding data volume to the first chip through the third interface according to the value of the register.

[0214] In the embodiment of the present application, the second chip may include an ISP and registers. The ISP may be connected to the eighth interface, receive the image signal sent by the camera module through the eighth interface, convert the image signal into first image data, and write the first image data into the corresponding storage unit, thus completing the operation in the above-mentioned stage 1. Among them, the process of the ISP writing the first image data may be to write the obtained first image data into the storage unit row by row. The ISP may set the value in the register according to the amount of data written into the storage unit. For example, the value in the register may be set according to the number of rows or columns of the written first image data.

[0215] In the embodiment of the present application, the second chip may determine the value of the above-mentioned register when meeting the preset read trigger condition (such as according to the locally generated synchronization signal or according to the synchronization signal sent by the third chip), so as to determine the amount of data that has been written, and obtain the first image data corresponding to the amount of data from the storage unit. For example, the value of the register may be 10, that is, obtain 10 rows of first image data in the storage unit, and may send it to the first chip through the third interface.

[0216] In some implementation manners, the first chip may generate a synchronization signal and send the synchronization signal to the second chip through the first interface. The second chip may determine the number of rows of the first image data recorded in the storage unit in response to the synchronization signal, and then may send the first image data corresponding to the amount of data to the first chip through the third interface.

[0217] In the embodiment of the present application, by setting a register in the second chip to determine the amount of the first image data that has been written, and then obtaining the first image data corresponding to the amount of data and sending it, the reading accuracy in the process of fragmentary transmission of the first image data is improved, and thus the delay in generating the entire VR composite image can be reduced.

[0218] Furthermore, in the embodiment of the present application, the fourth interface of the second chip may be electrically connected to the sixth interface of the third chip. The third chip may be a chip for generating a virtual image. The second chip may receive the depth image of the virtual image sent by the third chip:

[0219] In S1103, the first chip obtains a second image through the second interface; the second image may be a virtual image.

[0220] In S1104, the first chip generates a composite image according to the first image data and the second image.

[0221] In S1105, the first chip processes the synthesized image through the image processing algorithms corresponding to each output channel to generate an output image corresponding to the output channel; each output channel corresponds to a display module; the output image can be displayed through the display module corresponding to the output channel.

[0222] Specifically, the implementation manners of the steps of S1103 to S1105 can be exactly the same as those of the first embodiment. For the specific description of S1103, reference can be made to the relevant description of S502. For the specific description of S1104, reference can be made to the relevant description of S5031. For the description of S1105, reference can be made to the relevant description of S5032, and no further elaboration will be provided here.

[0223] Exemplarily, Figure 12 The schematic diagram of a VR display system provided by another embodiment of the present application is shown. The VR display system may include a VR display device. In some scenarios, the VR display system may further include other electronic devices in addition to the VR display device, such as a smart phone. Refer to Figure 12 As shown, the VR display device may include a first chip 121 and a second chip 122. In addition, the VR display device may further include a third chip 123. Of course, in some implementation scenarios, the third chip may be disposed on other electronic devices, such as a smart phone. Among them, for the functions of each interface and each module of the first chip 121, reference can be made to Figure 8 the relevant description of the chip 81 in the embodiment. For the functions of each interface and each module of the second chip 122, reference can be made to Figure 8 the relevant description of the chip 82 in the embodiment.

[0224] Different from the Figure 8 embodiment shown, in the second chip 122, it may include an ISP1221 and a register 1222. The implementation functions of the above two modules can refer to the relevant descriptions of S1101 and S1102, and no limitation will be made here.

[0225] In the embodiment of the present application, the third chip 123 can be used to synthesize a virtual image and a depth image corresponding to the virtual image. The third chip 123 can send the virtual image to the first chip 121 for virtual-real fusion, and can also send the depth image of the virtual image to the second chip 122 to extract the corresponding second depth information in the depth image through the second chip 122, so as to facilitate the reprojection by the fusion module 1211 in the subsequent first chip 121.

[0226] As can be seen from the above, in a display method provided by an embodiment of the present application, when generating a VR composite image, content such as image synthesis and image processing can be completed by an independent first chip, so that the exposure operation of the real scene image can be executed synchronously with subsequent operations such as image synthesis and image processing. Therefore, when continuous output of VR composite images is required, the waiting time required between each frame of composite images can be reduced, thereby improving the smoothness of the VR display screen. Compared with existing display technologies, since in the embodiment of the present application, when displaying a VR composite image, the acquisition of the real scene image and subsequent operations of image synthesis and image processing can be completed by different chips, and the logical operations of the two parts can be executed synchronously, thus shortening the waiting time required for VR image synthesis, reducing latency, improving the smoothness of the VR screen, reducing the dizziness of users when viewing, and improving the user experience.

[0227] Figure 13 FIG. shows a schematic structural diagram of a third electronic device provided by an embodiment of the present application. Refer to Figure 13 As shown, compared with Figure 4 and Figure 10 the embodiment shown, in the electronic device provided by the embodiment of the present application, in addition to including a first chip 131 and a second chip 132, it further includes a third chip 133 for generating virtual images.

[0228] The first chip may include a first interface 1301, a second interface 1302, and at least one seventh interface 1307; the second chip may include a third interface 1303, a fourth interface 1304, and an eighth interface 1308. The functions implemented by each interface in the first chip 131 and the second chip 132 can refer to the relevant descriptions in Embodiment 1 and Embodiment 2, which will not be elaborated here.

[0229] The third chip may include a fifth interface 1305 and a sixth interface 1306. Among them, the fifth interface 1305 may be electrically connected to the second interface 1302, and can be used to transmit virtual images. In some implementation scenarios, it can also be used to receive a synchronization signal sent by the first chip 131, and can also be used to send a synchronization signal to the first chip 131. The sixth interface 1306 may be electrically connected to the fourth interface 1304, and is used to transmit the second depth information of the virtual image. In some implementation scenarios, it can also be used to receive a synchronization signal sent by the second chip 132, and can also be used to send a synchronization signal to the second chip 132.

[0230] In some implementation manners, the fourth interface 1304 of the second chip 132 and the sixth interface 1306 of the third chip 133 may be interfaces based on the PCIE protocol. In particular, they may be interfaces of the PCIE4.0 protocol.

[0231] In some implementations, the second interface 1302 of the first chip 131 and the fifth interface 1305 of the third chip 133 can be interfaces based on the MIPI protocol.

[0232] In some implementations, the above-mentioned third chip 133 can include a GPU 1331 and a DPU 1332.

[0233] Among them, the DPU 1332 can be electrically connected to the fifth interface 1305. The DPU 1332 can be used to generate a virtual image and send the virtual image to the first chip 1331 through the fifth interface 1305. There can be an electrical connection between the DPU 1332 and the GPU 1331, and the virtual image can be transmitted to the GPU 1331.

[0234] Among them, the GPU 1331 can be electrically connected to the sixth interface 1306. The GPU 1331 can be used to generate a depth image corresponding to the virtual image and send the depth image to the second chip 132 through the sixth interface.

[0235] In some implementations, the above-mentioned third chip 133 can further include a CPU 1333. The CPU 1333 can be used to control the GPU 1331 and the DPU 1332 to perform corresponding operations, such as being used to generate a virtual image and being used to generate a depth image of the virtual image.

[0236] Specifically, Figure 14 shows a schematic flowchart of a display method provided by another embodiment of the present application. Combining Figure 13 the electronic device and Figure 14 the schematic flowchart, the display method provided by the embodiments of the present application includes:

[0237] In S1401, the second chip receives the first image data obtained by the exposure of the imaging module.

[0238] In S1406, the second chip sends the first image data, the first depth information, and the second depth information to the first chip through the third interface.

[0239] Specifically, the implementation manners of the steps of S1401 to S1402 are exactly the same as those of the second embodiment. For the specific description of S1401, reference can be made to the relevant description of S1101, and for the specific description of S1402, reference can be made to the relevant description of S1102, which will not be elaborated here.

[0240] In S1402, the third chip generates a second image.

[0241] In S1403, the third chip sends the second image to the first chip through the fifth interface.

[0242] In the embodiments of the present application, the third chip can generate corresponding virtual images according to the user's viewing needs. The virtual images can be, for example, Figure 2 the video picture in (b) of, or can also be virtual controls such as a virtual keyboard, etc., which can be specifically generated according to the actual situation. The number of virtual images to be generated for each frame can be one or multiple, which is not limited here.

[0243] In some implementation manners, the above first chip can send a synchronization signal to the third chip, and the third chip can obtain the synchronization signal sent by the first chip through the fifth interface. The synchronization signal can carry a frame number, and the third chip can send the corresponding virtual image to the first chip through the fifth interface according to the frame number, so that the first chip can complete operations such as virtual-real fusion and display.

[0244] In some implementation manners, the third chip can be a chip that controls the synchronization between chips. The third chip can generate a synchronization signal and send the synchronization signal to the first chip and the second chip through the fifth interface and the sixth interface, so as to ensure the synchronization of operations between each chip.

[0245] In some implementation manners, the second chip can be a signal that controls the synchronization between chips. In this case, the third chip can receive the synchronization signal sent by the second chip through the sixth interface, or can also receive the synchronization signal sent by the second chip forwarded by the first chip through the fifth interface, that is, the second chip can first send the synchronization signal to the first chip, and then the first chip can forward the synchronization signal to the third chip.

[0246] In S1404, the third chip generates the second depth information corresponding to the second image.

[0247] In S1405, the third chip sends the second depth information to the second chip through the sixth interface.

[0248] In the embodiments of the present application, the third chip can also generate the second depth information corresponding to the virtual image through the GPU, and can send the second depth information to the second chip, so that the second chip can send the second depth information to the first chip, and the first chip can re-project the virtual image in the three-dimensional coordinate system according to the second depth information to implement the synthesis of a VR composite image with a depth-of-field effect.

[0249] It should be noted that when the second chip sends the first image data to the first chip and does not receive the second depth information sent by the third chip, the second depth information corresponding to the virtual image of the previous frame can be used as the second depth information of the virtual image of the current frame, so as to avoid the output delay of the entire VR composite image caused by the delay in obtaining the second depth information, thereby improving the smoothness of the display.

[0250] In S1407, the first chip generates a composite image based on the first image data, the second image, the first depth information, and the second depth information.

[0251] In S1408, the first chip processes the composite image through the image processing algorithms corresponding to the respective output channels to generate output images corresponding to the output channels; each output channel corresponds to a display module; the output images can be displayed through the display modules corresponding to the output channels.

[0252] As can be seen from the above, in a display method provided by an embodiment of the present application, when generating a VR composite image, content such as image synthesis and image processing can be completed by an independent first chip, so that the exposure operation of the real scene image can be synchronized with subsequent operations such as image synthesis and image processing. Therefore, when continuous VR composite images need to be output, the waiting time required between each frame of composite images can be reduced, thereby improving the smoothness of the VR display screen. Compared with the existing display technology, since in the embodiment of the present application when displaying a VR composite image, the acquisition of the real scene image and the subsequent operations of image synthesis and image processing can be completed by different chips, the logical operations of the two parts can be synchronized, thereby shortening the waiting time required for VR image synthesis, reducing the time delay, improving the smoothness of the VR screen, reducing the dizziness of the user when viewing, and improving the user experience.

[0253] Figure 15 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 15 shown, the electronic device 15 in this embodiment includes: at least one processor 150 ( Figure 15 only one processor is shown in the figure, and the number of processors can match the number of chips actually included in the electronic device in the embodiment), a memory 151, and a program 152 stored in the memory 151 and executable on the at least one processor 150. When the processor 150 executes the program 152, the steps in any of the above-mentioned method embodiments for setting an electronic device are implemented.

[0254] The electronic device 15 may be a VR display device, a smart phone, etc. The electronic device may include, but is not limited to, a processor 150 and a memory 151. Those skilled in the art can understand that Figure 15 merely examples of the electronic device 15, which do not constitute a limitation on the electronic device 15, and may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output electronic devices, network access electronic devices, etc.

[0255] The so-called processor 150 may be a Central Processing Unit (CPU), and this processor 150 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0256] The memory 151 may be an internal storage unit of the electronic device 15 in some embodiments, such as the hard disk or memory of the electronic device 15. The memory 151 may also be an external storage electronic device of the electronic device 15 in some other embodiments, such as a plug-in hard disk equipped on the electronic device 15, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 151 may also include both the internal storage unit of the electronic device 15 and the external storage electronic device. The memory 151 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the program, etc. The memory 151 may also be used to temporarily store data that has been output or will be output.

[0257] It should be noted that, regarding the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of this application, for their specific functions and the technical effects brought, please refer to the method embodiment section for details and will not be elaborated here.

[0258] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0259] An embodiment of the present application further provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the foregoing method embodiments are implemented.

[0260] An embodiment of the present application further provides a readable storage medium, which stores a program that, when executed by a processor, can implement the steps in each of the foregoing method embodiments.

[0261] An embodiment of the present application provides a program product that, when running on an electronic device, enables the electronic device to execute and implement the steps in each of the foregoing method embodiments.

[0262] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0263] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0264] The above-mentioned embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A display method, characterized in that, Applied to an electronic device, the electronic device includes a first chip, and the display method includes: The first chip obtains first image data of a first image through a first interface; the first image is a real scene image obtained through a camera module; the first image data is partial data or all data of the first image; The first chip obtains second image data of a second image through a second interface; the second image data is partial data or all data of the second image; The first chip generates a composite image according to the first image data and the second image data; The first chip processes the composite image through image processing algorithms corresponding to respective output channels to generate an output image corresponding to the output channel; each output channel corresponds to a display module; the output image is displayed through the display module corresponding to the output channel.

2. The method according to claim 1, characterized in that It further includes: The first chip obtains first depth information of the first image and second depth information of the second image through the first interface; The first chip generating a composite image according to the first image data and the second image data includes: The first chip constructs a first depth image according to the first depth information and the first image data; The first chip constructs the second depth image according to the second image data and the second depth information; The first chip generates the composite image according to the first depth image and the second depth image.

3. The method according to claim 1, wherein The image processing algorithms include: an anti-distortion processing algorithm and an anti-color cast processing algorithm.

4. The method according to any one of claims 1-3, characterized in that The first chip obtaining first image data of a first image through a first interface includes: The first chip receives, through the first interface, the first image data of several rows in the first image sent by a second chip.

5. The method according to any one of claims 1-4, characterized in that, The electronic device further includes a second chip, and there is an electrical connection between a third interface of the second chip and the first interface of the first chip; the display method includes: The second chip receives the first image data obtained by the camera module through exposure; The second chip sends the first image data to the first chip through the third interface.

6. The method according to claim 5, characterized in that, The second chip includes a graphics signal processor and a storage unit; The graphics signal processor receives the first image data obtained by the camera module through exposure and sets the value of the storage unit according to the first image data; The value of the storage unit is used to determine the data volume of the first image data; The second chip sends the first image data corresponding to the data volume to the first chip through the third interface.

7. The method according to claim 5, characterized in that The first interface and the third interface perform data transmission of the first image data through electrical connection; the first image data is obtained by slicing the first image.

8. The method according to claim 5, wherein It further includes: The second chip receives second depth information of the second image sent by a third chip through a fourth interface; The second chip determines first depth information of the first image data; The first depth information and the second depth information are used to generate the composite image; The second chip sends the first image data to the first chip through a third interface, including: The second chip sends the first image data, the first depth information, and the second depth information to the first chip through the third interface.

9. The method according to any one of claims 1-8, characterized in that The electronic device further includes a third chip; there is an electrical connection between a fifth interface of the third chip and the second interface; the display method includes: The third chip generates the second image data; The third chip sends the second image data to the first chip through the fifth interface.

10. The method according to claim 9, wherein The third chip sends the second image data to the first chip through the fifth interface, including: The third chip receives a synchronization signal sent by the first chip through the fifth interface; In response to the synchronization signal, the third chip sends the second image data to the first chip through the fifth interface.

11. The method according to claim 9, wherein There is an electrical connection between a sixth interface of the third chip and a fourth interface of the second chip; the display method includes: The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the output image.

12. An electronic device, characterized in that, The electronic device includes a first chip; the first chip includes a first interface, a second interface, and at least one seventh interface; the first interface is electrically connected to a third interface of a second chip; the second interface is electrically connected to a fifth interface of a third chip; the seventh interface is electrically connected to a display module; the second chip and the third chip are the same chip or different chips; The first chip is configured to obtain the first image data of the first image sent by the second chip through the first interface; the first image is a real scene image obtained through an imaging module; the first image data is part or all of the data of the first image; The first chip is configured to obtain the second image data of the second image sent by the third chip through the second interface; the second image data is part or all of the data of the second image; The first chip is configured to generate a composite image based on the first image data and the second image data; The first chip is configured to process the composite image through an image processing algorithm corresponding to each output channel to generate an output image corresponding to the output channel; The first chip is configured to send the output image to the display module of the output channel corresponding to the seventh interface through the seventh interface, so as to display the output image through the display module.

13. The electronic device according to claim 12, characterized in that, The first chip is further configured to: obtain the first depth information of the first image and the second depth information of the second image data through the first interface; The first chip is configured to generate a composite image based on the first image data and the second image data, including: The first chip is configured to construct a first depth image based on the first depth information and the first image data; The first chip is configured to construct the second depth image based on the second image data and the second depth information; The first chip is configured to generate the composite image according to the first depth image and the second depth image.

14. The electronic device according to claim 12, wherein The image processing algorithm includes: an undistortion processing algorithm and an anti-color bias processing algorithm.

15. The electronic device according to any one of claims 12-14, characterized in that, The first chip is configured to receive, through the first interface, the first image data of several rows in the first image sent by the second chip.

16. The electronic device according to any one of claims 12-15, characterized in that, The electronic device further includes a second chip; an eighth interface of the second chip is electrically connected to the imaging module; The second chip is configured to receive the first image data obtained by the imaging module during exposure; The second chip is configured to send the first image data to the first chip through a third interface.

17. The electronic device according to claim 16, wherein The second chip includes a graphics signal processor and a storage unit; the graphics signal processor is electrically connected to the storage unit; the graphics signal processor is electrically connected to the eighth interface; the storage unit is electrically connected to the third interface; The graphics signal processor is configured to receive the first image data obtained by the imaging module during exposure, and set the value of the storage unit according to the first image data; The value of the storage unit is used to determine the data volume of the first image data; The second chip is configured to send the first image data corresponding to the data volume to the first chip through the third interface according to the value of the storage unit.

18. The electronic device according to claim 16, wherein The first interface and the third interface perform data transmission of the first image data through electrical connection; the first image data is obtained by fragmenting the first image.

19. The electronic device according to claim 16, wherein The second chip includes a fourth interface; the fourth interface is electrically connected to a sixth interface of the third chip; The second chip is configured to receive, through the fourth interface, the second depth information of the second image sent by the third chip; The second chip is configured to determine the first depth information of the first image data; The first depth information and the second depth information are used to generate the composite image; The second chip is configured to send the first image data, the first depth information, and the second depth information to the first chip through the third interface.

20. The electronic device according to any one of claims 12-19, characterized in that, The electronic device further includes a third chip; The third chip is configured to generate the second image data; The third chip is configured to send the second image data to the first chip through a fifth interface.

21. The electronic device according to claim 20, wherein The third chip is configured to send the second image data to the first chip through the fifth interface, including: The third chip is configured to receive, through the fifth interface, a synchronization signal sent by the first chip; The third chip is configured to respond to the synchronization signal and send the second image data to the first chip through the fifth interface.

22. The electronic device according to claim 20, wherein A sixth interface of the third chip is electrically connected to the fourth interface of the second chip; The third chip sends the second depth information of the second image data to the second chip through the sixth interface; the second depth information is used to generate the composite image.

23. A chip, characterized in that, The chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps performed by the first chip in the method according to any one of claims 1 to 4, or the steps performed by the second chip in the method according to any one of claims 5 to 8, or the steps performed by the third chip in the method according to any one of claims 9 to 11.

24. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 11.

Images