Image display method and electronic device

Abstract

Disclosed in the present application are an image display method and an electronic device. The method comprises: in response to a first user operation for instructing the three-dimensional display of a first image, an electronic device determining a foreground image and background image corresponding to the first image; displaying a second image on a first region, wherein the second image comprises a first foreground portion and a first background portion, and the pose of the electronic device when displaying the second image is a first pose; and when the pose of the electronic device is a second pose, displaying a third image on the first region, wherein the third image comprises a second foreground portion and a second background portion, the first region comprises a second region, and when the pose of the electronic device is different, picture content displayed in the second region is different. In the method, an electronic device can display different images corresponding to the same image during pose changes, and the image displayed by the electronic device comprises a foreground portion and a background portion, thereby presenting a two-dimensional image having a spatial three-dimensional effect.

Description

Image display method and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202411283657.3, filed on September 12, 2024, entitled "Image display method and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of terminal, and in particular to an image display method and an electronic device. BACKGROUND

[0003] With the development of augmented reality (AR) and virtual reality (VR) technologies, the display method of image and video gradually develops from 2D to 3D, aiming to present a more stereoscopic and three-dimensional effect to the user, and to provide a more immersive experience. The current 3D display usually uses a binocular device, which can play a photo or a video through two view angles at the same time, and present a spatial effect. However, the binocular device is expensive, and is not popularized in a large range at present, so the range of use is small.

[0004] How to present a stereoscopic effect with a sense of space to a two-dimensional image on an electronic device is a problem that needs to be solved in the industry. SUMMARY

[0005] The present application provides an image display method and an electronic device. In the method, the electronic device can display different images corresponding to the same image when the pose changes. The image displayed by the electronic device includes a foreground part and a background part, so as to present a stereoscopic effect with a sense of space to a two-dimensional image.

[0006] The present application will be described from different aspects below. It should be understood that the implementation and advantages of the different aspects below can be mutually referred to.

[0007] In a first aspect, the present application discloses an image display method, which can be executed by an electronic device or a module (e.g., a chip) of the electronic device. The method can include: in response to a first user operation for instructing to display a first image stereoscopically, determining a foreground image corresponding to the first image and a background image corresponding to the first image; displaying a second image on a first region, wherein the second image includes a first foreground part and a first background part, a pose of the electronic device when the electronic device displays the second image is a first pose, the first foreground part is obtained based on the foreground image and the first pose, and the first background part is obtained based on the background image and the first pose; and displaying a third image on the first region when the pose of the electronic device is a second pose, wherein the third image includes a second foreground part and a second background part, the second foreground part is obtained based on the foreground image and the second pose, and the second background part is obtained based on the background image and the second pose; and wherein the first region includes a second region, and the second region displays different content when the pose of the electronic device is different.

[0008] In the embodiments of the present application, the electronic device can generate a foreground image and a background image corresponding to a two-dimensional image (e.g., the first image described above); display an image corresponding to the two-dimensional image based on the pose of the electronic device and the foreground image and the background image, for example, the electronic device displays the second image when the pose of the electronic device is the first pose, and the content displayed by the electronic device is updated from the second image to the third image when the pose of the electronic device changes from the first pose to the second pose. In this method, the image corresponding to the two-dimensional image includes a foreground part and a background part, for example, the second image includes the first foreground part and the first background part, and the third image includes the second foreground part and the second background part. When the pose of the electronic device changes, the image corresponding to the two-dimensional image displayed by the electronic device changes, that is, the foreground part and the background part displayed by the electronic device also change, so that the two-dimensional image is presented with a stereoscopic effect with a sense of space.

[0009] In combination with the first aspect, in a possible implementation, the content displayed by the second region when the pose of the electronic device is the first pose belongs to the first background part, and the content displayed by the second region when the pose of the electronic device is the second pose belongs to the second foreground part; or, the content displayed by the second region when the pose of the electronic device is the first pose belongs to the first foreground part, and the content displayed by the second region when the electronic device displays the third image belongs to the second background part.

[0010] In the embodiments of the present application, when the pose of the electronic device changes, the foreground part and the background part of the displayed two-dimensional image of the electronic device change, for example, when the pose of the electronic device changes from the first pose to the second pose, the picture content displayed in the second area is updated from the picture content of the background image (i.e., the first background part) to the picture content of the foreground image (i.e., the second foreground part); for another example, when the pose of the electronic device changes from the first pose to the second pose, the picture content displayed in the second area is updated from the picture content of the foreground image (i.e., the first foreground part) to the picture content of the background image (i.e., the second background part), so as to present the two-dimensional image with a stereoscopic effect with a sense of space.

[0011] With reference to the first aspect, in a possible implementation, the second image further includes a first reality part, and the image captured by the electronic device at the first pose includes the first reality part; and / or, the third image further includes a second reality part, and the image captured by the electronic device at the second pose includes the second reality part.

[0012] In the embodiments of the present application, the second image and the third image can be AR images generated based on the background image, the foreground image and the captured image. In the method, the electronic device can present the two-dimensional image in a real scene, for example, the two-dimensional image can be a poster, and the real scene can be an exhibition hall, and the electronic device can display a picture of the poster in the exhibition hall, wherein the poster includes a foreground part and a background part, and when the user moves the electronic device (i.e., the pose of the electronic device changes), the user can see the change of the foreground part and the background part of the poster on the display screen of the electronic device, and the poster is presented with a stereoscopic effect with a sense of space. The method combines the two-dimensional image and the real scene, and can bring an immersive viewing effect to the user.

[0013] With reference to the first aspect, in a possible implementation, the method further includes: in response to a first user operation, determining a position of the observation plane in space; and determining a position of the foreground anchor plane in space and a position of the background anchor plane in space based on the position of the observation plane in space, wherein the foreground anchor plane and the background anchor plane are located in different planes, and the position of the foreground anchor plane in space and the position of the background anchor plane in space are used to determine the second image and / or the third image.

[0014] In the embodiments of the present application, the electronic device can determine the positions of the observation plane, the foreground anchor plane and the background anchor plane in space (such as a world coordinate system), and the positions of the observation plane, the foreground anchor plane and the background anchor plane are used to determine the display positions and display angles of the foreground image and the background image corresponding to the two-dimensional image, so as to support the electronic device to display the images corresponding to the two-dimensional image at different angles.

[0015] With reference to the first aspect, in a possible implementation manner, the plane where the observation surface is located, the plane where the foreground anchor surface is located, and the plane where the background anchor surface is located are parallel to each other; a distance between the plane where the observation surface is located and the plane where the foreground anchor surface is located is a first distance, a distance between the plane where the foreground anchor surface is located and the plane where the background anchor surface is located is a second distance, the first distance is greater than or equal to 0, and the second distance is greater than 0.

[0016] In the method, the second distance is greater than 0, the foreground anchor surface is used to display a foreground part (for example, the first foreground part) corresponding to the first image, and the background anchor surface is used to display a background part (for example, the first background part) corresponding to the first image, so as to achieve a display effect in which the foreground part and the background part are separated.

[0017] With reference to the first aspect, in a possible implementation manner, the method further includes: when the pose of the electronic device is a first pose, determining a first position of the foreground anchor surface in the image coordinate system and a second position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in the space, the position of the background anchor surface in the space, and the first pose; and rendering the foreground image and the background image based on the first position and the second position to obtain a second image.

[0018] With reference to the first aspect, in a possible implementation manner, the rendering, based on the first position and the second position, of the foreground image and the background image to obtain the second image includes: projecting the foreground image to the first position to obtain a first image texture; projecting the background image to the second position to obtain a second image texture; and rendering the first image texture and the second image texture to obtain the second image, a rendering result of the first image texture being a first foreground part and a rendering result of the second image texture including a first background part.

[0019] With reference to the first aspect, in a possible implementation manner, the method further includes: when the pose of the electronic device is a second pose, determining a third position of the foreground anchor surface in the image coordinate system and a fourth position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in the space, the position of the background anchor surface in the space, and the second pose; and rendering the foreground image and the background image based on the third position and the fourth position to obtain a third image.

[0020] With reference to the first aspect, in a possible implementation manner, the rendering, based on the third position and the fourth position, of the foreground image and the background image to obtain the third image includes: projecting the foreground image to the third position to obtain a third image texture; projecting the background image to the fourth position to obtain a fourth image texture; and rendering the third image texture and the fourth image texture to obtain the third image, a rendering result of the third image texture being a second foreground part and a rendering result of the fourth image texture including a second background part.

[0021] With reference to the first aspect, in a possible implementation, the method further includes: performing depth estimation on the first image to obtain a depth map; determining an average depth of the foreground image and an average depth of the background image based on the depth map; and determining the second distance based on the average depth of the foreground image and the average depth of the background image.

[0022] In the method, the second distance is determined through depth estimation, and a more reasonable display effect of separation of the foreground part and the background part can be presented.

[0023] With reference to the first aspect, in a possible implementation, the method further includes: performing semantic segmentation on the first image to obtain a foreground semantic template of the first image and a background semantic template of the first image; determining the background image based on the first image and the background semantic template of the first image; and determining the foreground image based on the first image and the foreground semantic template of the first image.

[0024] In the second aspect, the present application provides an electronic device. The electronic device can include a memory and a processor. The memory can be used to store a computer program. The processor can be used to invoke the computer program, so that the electronic device executes the first aspect or any possible implementation of the first aspect.

[0025] In the third aspect, the present application provides a computer program product including instructions, when the computer program product runs on an electronic device, the electronic device executes the first aspect or any possible implementation of the first aspect.

[0026] In the fourth aspect, the present application provides a computer readable storage medium including instructions, when the instructions run on an electronic device, the electronic device executes the first aspect or any possible implementation of the first aspect. The technical effects achieved by the above aspects can be mutually referred or referred to the beneficial effects of the method embodiments shown below, which will not be described here.

[0027] In the fifth aspect, the present application provides a chip including a processor and a communication interface. The communication interface is used to output and / or output data (including instructions), and / or the communication interface is used to receive and / or send data. When the processor executes the program instructions in the memory, the method described in the first aspect or any possible implementation of the first aspect is realized. BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1A is a schematic diagram of the hardware structure of an electronic device 100 according to an embodiment of the present application;

[0029] FIG. 1B is a software structure block diagram of the electronic device 100 according to an embodiment of the present application;

[0030] FIG. 2 is a schematic diagram of a calculation module according to an embodiment of the present application;

[0031] FIG. 3 is a schematic diagram of the positional relationship between an observation surface, a foreground anchor surface and a background anchor surface according to an embodiment of the present application;

[0032] FIG. 4 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0033] FIGS. 5A to 5D are some user interfaces provided by way of example according to an embodiment of the present application;

[0034] FIG. 5E is a schematic diagram of the stereoscopic effect of a two-dimensional image presented at different viewing angles according to an embodiment of the present application;

[0035] FIG. 6 is a flowchart of an image display method according to an embodiment of the present application;

[0036] FIG. 7A is a flowchart of another image display method according to an embodiment of the present application;

[0037] FIG. 7B is a flowchart of a method of obtaining a foreground image and a background image according to an embodiment of the present application;

[0038] FIG. 8A is a schematic diagram of semantic segmentation according to an embodiment of the present application;

[0039] FIG. 8B is a schematic diagram of obtaining a background image according to an embodiment of the present application;

[0040] FIG. 8C is a schematic diagram of obtaining a foreground image according to an embodiment of the present application;

[0041] FIG. 9A is a schematic diagram of initializing an observation surface according to an embodiment of the present application;

[0042] FIG. 9B is a schematic diagram of initializing a foreground anchor surface and a background anchor surface according to an embodiment of the present application;

[0043] FIG. 10 is a schematic diagram of the effect of different depth differences according to an embodiment of the present application;

[0044] FIG. 11A is a schematic diagram of an updated foreground anchor surface and a background anchor surface according to an embodiment of the present application;

[0045] FIG. 11B is a schematic diagram of a third image according to an embodiment of the present application;

[0046] FIG. 12 is a schematic diagram of the structure of an electronic device 200 according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0048] In the description of the present application, "first", "second", and the like are used only to distinguish different objects, and do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device, etc. comprising a series of steps or units is not limited to the listed steps or units, but can optionally also include steps or units not listed, or can optionally also include other steps or units inherent to these processes, methods, products, or devices, etc.

[0049] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this paper is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean: A alone, A and B exist at the same time, and B alone. In addition, "at least one item", "one or more items" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c; a and b; a and c; b and c; or a and b and c. Where a, b, and c can be single or multiple.

[0050] In the description of the present application, "exemplary" or "for example" and the like are used to mean as an example, illustration, or description. Any embodiment or design scheme described as "exemplary", "for example" or "for example" in this application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary", "for example" or "for example" and the like is intended to present the relevant concept in a specific manner.

[0051] It can be understood that in the description of the present application, "when", "if" and "if" all refer to the corresponding processing of the electronic device under certain objective circumstances, not the time limit, and it is not required that the electronic device has a judgment action when it is implemented. It also does not mean that there are other limitations. Among them, the electronic device makes corresponding processing under certain objective circumstances, including: meeting the objective circumstances, that is, being able to make the corresponding processing; or meeting the objective circumstances and other circumstances to make the corresponding processing.

[0052] "Simultaneously" in the present application can be understood as at the same time point, can also be understood as within a period of time, and can also be understood as within the same cycle. It can be understood in combination with the context.

[0053] In this application, the element using the singular is intended to represent "one or more", rather than "one and only one", unless otherwise specified.

[0054] In addition, the terms "system" and "network" are often used interchangeably herein.

[0055] It can be understood that in various embodiments of the present application, "A corresponds to B", "A and B correspond", "A corresponds to B" or the like, means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined based on A only, but B can also be determined based on A and / or other information.

[0056] At present, the schemes of image stereoscopic display mainly include the following:

[0057] Scheme 1: three-dimensional reconstruction of two-dimensional images (i.e. 2D images), such as three-dimensional modeling of original RGB images and other two-dimensional images through neural networks, neural radiance field (NeRF) rendering, Mesh reconstruction, etc., and then stereoscopic display based on the 3D model to obtain immersive photos with depth information. Scheme 2: using binocular devices to display the reconstructed scene, such as depth estimation of two-dimensional images, generating left and right binocular images with parallax, and displaying on binocular devices to present the effect of stereoscopic images.

[0058] Among them, the three-dimensional reconstruction process in scheme 1 has high computational complexity, large computational overhead, high latency and low precision; scheme 2 cannot be run on non-binocular devices (such as mobile phones, tablets, PCs and other flat terminals), and the cost of binocular devices is high, so the above methods are difficult to be large-scale commercialized.

[0059] The present application provides an image display method and an electronic device. In the method, the electronic device can generate a stereoscopic image with front and background depth information based on a two-dimensional image, realize three-dimensional stereoscopic display of a two-dimensional image (such as a monocular photo / video taken by a common camera), and solve the problem of three-dimensional reconstruction required by the existing method.

[0060] For example, the electronic device processes a two-dimensional image to obtain a foreground image and a background image corresponding to the two-dimensional image; then renders the foreground image on a foreground anchor surface and the background image on a background anchor surface; and displays the rendering result. Then, the observer can see different image contents of the foreground image and the background image from different perspectives, and have depth and occlusion effects.

[0061] The embodiments of the present application can also combine different postures of the electronic device (or the camera of the electronic device), and perform real-time stereoscopic display and interaction on the electronic device (such as a mobile phone), thereby solving the problem of the need for a binocular special-purpose device and greatly improving the landing application range of the technology.

[0062] For example, when the user moves, the electronic device can calculate the pose of the electronic device in real time through device information of an inertial measurement unit (IMU), perform perspective projection on images on the double-anchored surface (i.e., the foreground anchoring surface and the background anchoring surface) through the pose change, perform real-time rendering, present the effect of a real scene, and complete interaction with the user.

[0063] To more clearly and specifically introduce the image display method provided by the embodiments of the present application, the electronic device provided by the embodiments of the present application is introduced first.

[0064] The electronic device is a device with a display function.

[0065] For example, the electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and the like. Exemplary embodiments of the electronic device include, but are not limited to, a portable electronic device loaded with a Linux or other operating system. The electronic device can also be another portable electronic device, such as a laptop computer.

[0066] Optionally, the structure of the electronic device can refer to the structure of the electronic device 100 in FIGS. 1A and 1B, or can refer to the structure of the electronic device 200 in FIG. 12, and the structure of the electronic device is not limited in the present application.

[0067] Next, the electronic device provided by the embodiments of the present application is exemplarily introduced.

[0068] FIG. 1A shows a schematic diagram of the hardware structure of the electronic device 100.

[0069] Next, the embodiments are specifically described by taking the electronic device 100 as an example. It should be understood that the electronic device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0070] ​The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0071] Optionally, the sensor module 180 can include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0072] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0073] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0074] Among them, the controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0075] The processor 110 can also include a memory that stores instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The cache memory can hold instructions or data that the processor 110 has recently used or that the processor 110 uses on a regular basis. If the processor 110 needs to use the instructions or data again, it can be retrieved directly from the cache memory. This avoids repeated accesses and reduces the latency of the processor 110, thus improving the efficiency of the system.

[0076] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0077] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can include multiple sets of I2C buses. The processor 110 can be coupled to a touch sensor, a charger, a flash, a camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to a touch sensor through an I2C interface, so that the processor 110 and the touch sensor communicate through the I2C bus interface to realize the touch function of the electronic device 100.

[0078] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple sets of I2S buses. The processor 110 can be coupled to the audio module 170 through the I2S bus to realize communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can deliver an audio signal to the wireless communication module 160 through the I2S interface to realize the function of answering a phone through a Bluetooth headset.

[0079] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 can be coupled with the wireless communication module 160 through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, enabling the function of answering a phone call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0080] The UART interface is a universal serial bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, enabling Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, enabling the function of playing music through a Bluetooth headset.

[0081] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes the camera serial interface (CSI), the display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface, enabling the camera function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface, enabling the display function of the electronic device 100.

[0082] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0083] The SIM interface can be used to communicate with the SIM card interface 195, enabling the function of transmitting data to the SIM card or reading data in the SIM card.

[0084] The USB interface 130 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0085] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments. The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160, etc.

[0086] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device 100. The modem processor can include a modulator and a demodulator.

[0087] The wireless communication module 160 can provide a solution for wireless communication, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc., which are applied on the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0088] The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0089] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 can include 1 or N display screens 194, N being a positive integer greater than 1.

[0090] The electronic device 100 can implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display 194, and an application processor, etc. The camera 193 is used to capture a still image or a video. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts an optical signal into an electrical signal, and then transmits the electrical signal to an ISP to convert into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB, YUV, etc. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1. A digital signal processor is used to process a digital signal, in addition to processing a digital image signal, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0091] A video codec is used to compress or decompress a digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0092] An NPU is a neural-network (NN) computing processor, which quickly processes input information by drawing on the structure of a biological neural network, such as the transmission mode between human brain neurons, and can also continuously self-learn. Through the NPU, the electronic device 100 can implement intelligent cognitive applications such as image recognition, face recognition, voice recognition, text understanding, etc.

[0093] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, etc. files are saved in the external memory card.

[0094] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a face recognition function, a fingerprint recognition function, a mobile payment function, etc.), and the like. The data storage area can store data created during use of the electronic device 100 (such as face information template data, fingerprint information template, etc.). In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one of a magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0095] The electronic device 100 can implement an audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, and the like. For example, music playback, recording, and the like.

[0096] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or part of the functions of the audio module 170 can be disposed in the processor 110.

[0097] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into an acoustic signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0098] The receiver 170B, also known as a "earpiece", is used to convert an audio electrical signal into an acoustic signal. When the electronic device 100 answers a call or a voice message, the receiver 170B can be used to listen to the voice by being close to the ear of a person.

[0099] Microphone 170C, also called "microphone", "sounder", is used to convert sound signal into electrical signal. When making a call or sending voice message, the user can speak into the microphone 170C by putting the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, in addition to collecting sound signals, noise reduction function can also be realized. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, to realize the collection of sound signals, noise reduction, and also to identify the source of sound, to realize the function of directional recording, etc.

[0100] The earphone interface 170D is used to connect the wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0101] The key 190 includes the power-on key, the volume key, etc. The key 190 can be a mechanical key. It can also be a touch key. The electronic device 100 can receive the key input, and generate key signal input related to the user settings and function control of the electronic device 100.

[0102] The motor 191 can generate vibration prompt. The motor 191 can be used for incoming call vibration prompt, and also can be used for touch vibration feedback. For example, the touch operation acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The touch operation acting on different regions of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving information, alarm, game, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0103] The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and also can be used to indicate the message, the missed call, the notification, etc.

[0104] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to realize contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The electronic device 100 interacts with a network through the SIM card to realize functions such as call and data communication.

[0105] In the embodiments of the present application, the electronic device 100 can execute the image display method through the processor 110, and display the image rendered by the present application through the display screen 194. For example, the electronic device 100 can be a mobile device such as a mobile phone, a tablet, a PC, and the like, which has a processor 110, a camera 193, and the like.

[0106] FIG. 1B is a software structure block diagram of an electronic device 100 according to an embodiment of the present application.

[0107] The layered architecture divides software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom, application layer, application framework layer, runtime and system library, and kernel layer.

[0108] The application layer can include a series of application packages.

[0109] As shown in FIG. 1B, the application packages can include gallery, camera, calendar, call, WLAN, Bluetooth, music, video, and the like.

[0110] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0111] As shown in FIG. 1B, the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0112] The window manager is configured to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and the like.

[0113] The content provider stores and retrieves data and makes the data accessible to the application program. The data can include videos, images, audios, dialed and received phone numbers, browsing history and bookmarks, phone book, etc.

[0114] The view system includes visual controls, such as a control that displays text, a control that displays an image, etc. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon can include a view that displays text and a view that displays an image.

[0115] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call state (including call connection, call hang-up, etc.).

[0116] The resource manager provides various resources for the application program, such as localized strings, icons, images, layout files, video files, etc.

[0117] The notification manager enables the application program to display notification information in the status bar, which can be used to convey a type of message that can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of a download, a message reminder, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of a chart or a scrolling text, such as a notification of an application program running in the background, and can also be a notification that appears on the screen in the form of a dialog interface. For example, a text message is prompted in the status bar, a prompt sound is emitted, the electronic device vibrates, the indicator light flashes, etc.

[0118] The runtime includes a core library and a virtual machine. The runtime is responsible for the scheduling and management of the system.

[0119] The core library includes two parts: one part is the function function that the programming language (for example, the jave language) needs to call, and the other part is the core library of the system.

[0120] The application program layer and the application program framework layer run in the virtual machine. The virtual machine executes the programming files (for example, jave files) of the application program layer and the application program framework layer into binary files. The virtual machine is used to perform functions such as management of the object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0121] The system library can include multiple functional modules. For example: a calculation module, a surface manager, media libraries, a three-dimensional graphics processing library (for example: OpenGL ES, Vulkan), a two-dimensional graphics engine (for example: SGL), etc.

[0122] The computing module is configured to perform the image display method provided in the present application to process the image, such as image segmentation, image rendering, and image display, and the like. For details, refer to the relevant description below, which will not be described here.

[0123] It should be noted that the computing module can also be located in other layers in other layered software structures, or be split into multiple modules and arranged in different layers, which is not limited in the present application.

[0124] The surface manager is configured to manage the display subsystem and provide a fusion of 2-Dimensional (2D) and 3-Dimensional (3D) layers for multiple applications.

[0125] The media library supports playback and recording of a plurality of commonly used audio, video formats, and static image files, and the like. The media library can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0126] The 3D graphics processing library is configured to implement 3D graphics drawing, image rendering, synthesis, and layer processing, and the like.

[0127] The 2D graphics engine is a drawing engine for 2D drawing.

[0128] The kernel layer is a layer between hardware and software. The kernel layer can include display drivers, camera drivers, audio drivers, sensor drivers, virtual card drivers, and the like.

[0129] The following will illustrate the working flow of the software and hardware of the electronic device 100 in a capture and photographing scenario.

[0130] When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, a timestamp of the touch operation, and the like). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer, and identifies the control corresponding to the input event. Taking an example of the touch operation being a touch single-click operation and the control corresponding to the single-click operation being a control of a camera application icon, the camera application calls an interface of the application framework layer, starts the camera application, and then starts the camera driver through the kernel layer, and captures a still image or a video through the camera 193.

[0131] FIG. 2 is a schematic diagram of a computing module according to an embodiment of the present application.

[0132] As shown in FIG. 2, the computing module can include an offline computing module and an online computing module.

[0133] In some embodiments, a user can select a two-dimensional image in an electronic device for stereoscopic display, and accordingly, the electronic device can run an offline computing module and an online computing module for the two-dimensional image in response to the user operation, to realize three-dimensional stereoscopic display of the two-dimensional image. Optionally, the electronic device can run the offline computing module and the online computing module simultaneously, or sequentially, which is not limited in the present application.

[0134] The offline computing module is configured to provide data preparation for the online computing module. For example, the input of the offline computing module is the two-dimensional image (referred to as the first image for convenience of description) selected by the user, and the output of the offline computing module can be a foreground image corresponding to the first image and a background image corresponding to the first image. Optionally, the output of the offline computing module can also include a depth difference between the foreground image and the background image. It can be understood that the output of the offline computing module is the data preparation provided for the online computing module.

[0135] The online computing module is configured to display a stereoscopic image corresponding to the first image on the display screen of the electronic device, and to interact with the user according to the pose of the electronic device. It should be noted that the offline computing module can be run only once for an image, and the online computing module can be continuously and cyclically calculated to maintain continuous interaction with the user.

[0136] The offline computing module and the online computing module are described in detail below.

[0137] As shown in FIG. 2, the running process of the offline computing module can include the following steps: 1) semantic segmentation; 2) background completion; 3) obtaining a foreground image and a background image; and 4) calculating a depth difference.

[0138] For example, the offline computing module can perform semantic segmentation (or foreground semantic segmentation) and background completion on the first image to obtain a foreground image corresponding to the first image and a background image corresponding to the first image, and obtain a depth difference d between the background image and the foreground image, where d is a scalar value. Through the above steps, the complete background image and the foreground image corresponding to the first image can be obtained. The specific process can be referred to in the embodiments below, which is not expanded here.

[0139] It should be noted that the background completion algorithm is not limited in the present application, and the background completion algorithm can be a neural network-based image inpainting method. The above-mentioned depth difference can be pre-set or calculated by the offline computing module, and the method of obtaining the depth difference is not limited in the present application.

[0140] As shown in FIG. 2, the running process of the online computing module can include the following steps: 1) initializing a world coordinate system; 2) initializing an observation plane (which can also be referred to as a frame); 3) initializing a foreground anchor plane and a background anchor plane; 4) calculating a camera pose; 5) performing texture projection of a foreground image and a background image based on the foreground anchor plane, the background anchor plane and the camera pose; and 6) performing rendering display based on the result of the texture projection.

[0141] In some embodiments, the online computing module can acquire camera data, such as a video stream captured by a camera of the electronic device, and can also display (such as in real time) the video stream on the screen of the electronic device; initialize a world coordinate system; initialize the coordinates of the observation plane in the world coordinate system (which can also be referred to as placing the observation plane in space), such as using the center of the image captured by the camera as the center of the observation plane, determining the position of the observation plane in the world coordinate system based on a preset observation plane size; based on the coordinates of the observation plane in the world coordinate system; initialize the foreground anchor plane and the background anchor plane, i.e., determine the coordinates of the foreground anchor plane and the background anchor plane in the world coordinate system; calculate the pose (such as a 6DoF pose) of the electronic device, which can specifically be the pose of the camera of the electronic device, wherein the method of calculating the pose of the electronic device can be through a SLAM method or other methods, which are not limited in the present application; apply the pose information of the electronic device to the above two anchor planes, i.e., based on the position information of the electronic device, determine the coordinate positions of the two anchor planes relative to the observation plane; perform perspective projection on the foreground image and the background image respectively to obtain the image texture of the foreground anchor plane and the image texture of the background anchor plane, and then render the above image textures to the foreground anchor plane and the background anchor plane respectively; based on the observation plane, perform view frustum culling to obtain the texture within the imaging range of the observation plane, and complete rendering and display. Optionally, when the user moves the electronic device in any direction, the pose of the electronic device changes (such as the SLAM module can update the 6DoF pose of the device in real time), and the electronic device can cyclically perform the texture rendering process to complete the display and interaction of the stereoscopic image.

[0142] It should be understood that the world coordinate system is used to describe the position of an object in the real world, and the two-dimensional image coordinate system (referred to as image coordinate system) is used to describe the position of the displayed object on the screen; the coordinates of the world coordinate system and the coordinates of the image coordinate system can be mapped, i.e., any object (X, Y, Z) in the world coordinate system can be one-to-one mapped to the corresponding image coordinate system (u, v) through perspective projection.

[0143] The following exemplary introduces the process of initializing the foreground anchor plane and the background anchor plane.

[0144] It is assumed that the position of the observation plane in the world coordinate system is represented by formula (1). P_rec = (X_rec, Y_rec, Z_rec) formula (1)

[0145] It should be understood that the observation plane P_rec is determined and fixed in space, and remains relatively static with the surrounding objects and does not move with the movement of the electronic device.

[0146] Then, based on formula (1), the position of the foreground anchor plane in the world coordinate system can be represented by the following formula (2), and the position of the background anchor plane coordinate in the world coordinate system can be represented by formula (3).

[0147] Foreground anchor plane coordinate P_f = (X_rec, Y_rec, Z_rec-d_rec) formula (2)

[0148] Background anchor plane coordinate P_b = (X_rec, Y_rec, Z_rec-d_rec-d) formula (3)

[0149] Wherein, d_rec≥0, d>0.

[0150] It should be understood that, like the observation plane, the foreground anchor plane and the background anchor plane are determined and fixed in space, and remain relatively static with the surrounding objects and do not change with the change of the pose of the electronic device, and the coordinate positions are anchored in space.

[0151] FIG. 3 is a schematic diagram of the positional relationship between the observation plane, the foreground anchor plane and the background anchor plane according to an embodiment of the present application. FIG. 3 exemplarily shows the positional relationship between the observation point (i.e. the position of the electronic device), the observation plane, the foreground anchor plane and the background anchor plane in the world coordinate system (i.e. space). It is assumed that the user can hold the electronic device at the observation point, the observation plane can be located in front of the camera of the electronic device, the planes where the observation plane, the foreground anchor plane and the background anchor plane are located are parallel to each other, the distance between the plane where the observation plane is located and the plane where the foreground anchor plane is located is d_rec, and the distance between the plane where the foreground anchor plane is located and the plane where the background anchor plane is located is d. The dashed line in FIG. 3 is the visual angle range formed based on the observation plane, and the textures on the foreground anchor plane and the background anchor plane within the visual angle range can be displayed in the electronic device, that is, the observation plane is associated with the display of the foreground image and the background image. The size of the observation plane, the foreground anchor plane and the background anchor plane will be described below, and will not be expanded here.

[0152] First, the application scenarios applicable to the present application will be introduced below in combination with the schematic diagram of the application scenario of FIG. 4 and the user interfaces shown in FIGS. 5A to 5D.

[0153] The electronic device is an electronic device with display service, for example, a smart phone, a tablet, etc. The image processing method provided in the embodiments of the present application can be embodied as a "3D display" function on the electronic device. It should be noted that in some other embodiments of the present application, the function can also be referred to as "AR display" and the like, which is not limited in the present application.

[0154] FIG. 4 is a schematic diagram of an application scenario provided in the embodiments of the present application. In the scenario, user A can open the gallery in the electronic device, click the thumbnail of the first image in the gallery interface, and accordingly, the electronic device displays the browsing interface of the first image, for example, the user interface 51 shown in FIG. 5A. The user interface 51 includes a preview area 510 and a 3D display control 511, wherein the preview area 510 is used to display the first image, and the image content of the first image is exemplarily shown in FIG. 5A to include a cat, a stone beside the cat, and a tree behind the cat; the 3D display control 511 is used to start the "3D display" function described above. It should be understood that the user interface 51 also includes a sharing control and a collection control as shown in FIG. 5A, which is not limited in the present application.

[0155] In some embodiments, as shown in FIG. 4, assuming that user A holds the electronic device at an initial position (at this time, the pose of the electronic device is an initial pose), user A starts the "3D display" function of the electronic device for the first image, for example, user A clicks the 3D display control 511 in the user interface 51 shown in FIG. 5A, and accordingly, the electronic device can determine the observation surface in response to the user operation when detecting the user operation, determine the foreground anchor surface and the background anchor surface based on the observation surface, and then display the user interface 52 as shown in FIG. 5B based on the foreground anchor surface, the background anchor surface, the initial pose, and the first image. It should be understood that the observation surface is fixed in space after being determined, as shown in FIG. 4, when user A moves from the initial position to position 1 or position 2, the position of the observation surface in space is always unchanged.

[0156] The user interface 52 includes a preview area 520 and an adjustment area 521, the preview area 520 is configured to display the stereoscopic image of the first image at the initial viewing angle, and the adjustment area 521 is configured to adjust the currently displayed stereoscopic image. As shown in FIG. 5B, the adjustment area 521 includes a first control 521a, a second control 521b and a third control 521c. The first control 521a is configured to adjust the distance between the observation plane and the electronic device (or the camera of the electronic device), the second control 521b is configured to adjust the distance between the foreground anchor plane and the electronic device (or the camera of the electronic device), and the third control 521c is configured to adjust the distance between the background anchor plane and the electronic device (or the camera of the electronic device). It can be understood that, when the pose of the electronic device does not change, if the first control 521a is adjusted, the content of the image displayed in the picture changes, for example, as the distance between the observation plane and the electronic device increases, more picture content of the stereoscopic image corresponding to the first image is displayed, and vice versa; if the second control 521b or the third control 521c is adjusted, the distance between the foreground image and the background image corresponding to the first image changes in vision.

[0157] It should be noted that the user interface 52 is only an example, and the user interface 52 can also include a 3D mark or a display effect of imaging color at the edge of the image area, which is not limited in the present application.

[0158] As shown in FIG. 4, when the user A moves to position 1, it is assumed that the current pose of the electronic device is pose 1, and the viewing angle of the camera of the electronic device towards the observation plane is viewing angle 1. As an example, the display user interface of the electronic device can be the user interface 53 shown in FIG. 5C. The user interface 53 can include a preview area 530 and an adjustment area 521, the preview area 530 is configured to display the stereoscopic image of the first image at the viewing angle 1, and the adjustment area 521 is configured to adjust the currently displayed stereoscopic image. The details of the adjustment area 521 can be referred to the above, and will not be described here.

[0159] As can be seen from the comparison between FIG. 5B and FIG. 5C, when the viewing angle of the user changes (i.e., the pose of the electronic device changes), the picture content of the stereoscopic image presented in the electronic device is different. For example, the occlusion of the foreground part (such as the cat) to the background part (such as the tree) changes, in FIG. 5B, the cat does not occlude the left branch of the tree, while in FIG. 5C, the cat occludes the left branch of the tree. For another example, the size of the picture content of the foreground part and the background part changes, compared with FIG. 5B, the picture content on the left side in FIG. 5C is reduced.

[0160] As shown in FIG. 4, when the user A moves to position 2, assuming that the current pose of the electronic device is pose 2 and the camera of the electronic device faces the viewing surface at angle of view 2, the displayed user interface of the electronic device can be user interface 54 shown in FIG. 5D, for example. The user interface 54 includes a preview area 540 and an adjustment area 521, where the preview area 540 is used to display the stereoscopic image presented by the first image at angle of view 2, and the adjustment area 521 is used to adjust the currently displayed stereoscopic image. The details of the adjustment area 521 can be found in the foregoing, and will not be described here again.

[0161] As can be seen by comparing FIG. 5C with FIG. 5D, when the viewing angle of the user changes (i.e., the pose of the electronic device changes), the picture content of the stereoscopic image presented in the electronic device changes. In order to facilitate the comparison of the image in FIG. 5C with the image in FIG. 5D, refer to FIG. 5E.

[0162] FIG. 5E is a schematic diagram of the stereoscopic effect of a two-dimensional image presented at different viewing angles according to an embodiment of the present application.

[0163] The image on the left side in FIG. 5E is the image displayed in FIG. 5C, i.e., the image picture displayed by the electronic device at angle of view 1 (for convenience of description, referred to as image 1); the image on the right side in FIG. 5E is the image displayed in FIG. 5D, i.e., the image picture displayed by the electronic device at angle of view 2 (for convenience of description, referred to as image 2). FIG. 5E exemplarily shows region 1 and region 2 in the image coordinate system. For region 1, in image 1, region 1 displays the picture content of the background image, and in image 2, region 1 displays the picture content of the foreground image. As can be seen, when the viewing angle changes, the region originally displaying the background image is occluded by the foreground image, which can also be understood as the content in region 1 changes from the background “appearing” to being “occluded” by the foreground. For region 2, in image 1, region 1 displays the picture content of the foreground image, and in image 2, region 2 displays the picture content of the background image. As can be seen, when the viewing angle changes, the picture of the background image originally occluded by the foreground image becomes displayed, which can also be understood as the content in region 2 changes from being “occluded” by the foreground to the background “appearing”. In addition, the size of the picture content of the foreground part and the background part also changes. Compared with image 1, the left side picture content in image 2 becomes larger, and the right side picture content becomes smaller.

[0164] As can be seen from FIG. 5E, when the viewing angle of the user changes, the background content in region 2 changes from being “occluded” by the foreground to “appearing”, and the content in region 1 changes from the background “appearing” to being “occluded” by the foreground figure.

[0165] It should be noted that FIGS. 5B-5D only illustrate the changes of the first image, and in some embodiments of the present application, the "3D display" function described above can be an "AR" display, that is, the display interface can simultaneously display a stereoscopic image corresponding to the first image and an image obtained by superimposing and rendering the actual image captured by the camera, as shown in FIG. 11B.

[0166] It should be noted that in the embodiments of the present application, any one of the user operations can be a touch operation of the user (for example, a click operation, a long press operation, an up-slip operation, a down-slip operation, or a side-slip operation), can be a non-contact operation (for example, a gesture in the air), or can be a voice instruction of the user, and the embodiments of the present application do not make specific limitations.

[0167] The user can also open the "3D display" function in other ways, for example, by long pressing the two-dimensional image to open the "3D display" function, and the present application does not make limitations in this regard.

[0168] Next, the image display method provided by the present application is introduced.

[0169] FIG. 6 is a flowchart of an exemplary image display method provided by the embodiments of the present application.

[0170] The embodiments of the present application take the electronic device as an example, and the hardware and software architecture of the electronic device can refer to the related descriptions of the electronic device 100 in FIGS. 1A-2 or the related descriptions of the electronic device 200 in FIG. 12, and the present application does not make limitations in this regard. The method can include some or all of the following steps:

[0171] S601: In response to a first user operation for instructing to display a first image in a stereoscopic manner, the electronic device determines a foreground image corresponding to the first image and a background image corresponding to the first image.

[0172] In some embodiments, the electronic device can perform semantic segmentation on the first image to obtain a foreground semantic template of the first image and a background semantic template of the first image; then, based on the first image and the background semantic template of the first image, the background image is determined; and based on the first image and the foreground semantic template of the first image, the foreground image is determined. For example, the process can refer to the related content of step S701 below.

[0173] In an implementation, the electronic device can further determine, in response to the first user operation, a position of the observation plane in the space; determine, based on the position of the observation plane in the space, a position of the foreground anchor plane in the space and a position of the background anchor plane in the space, wherein the foreground anchor plane and the background anchor plane are located in different planes, and the position of the foreground anchor plane in the space and the position of the background anchor plane in the space are used to determine the second image and / or the third image. For example, the process can be referred to in the following description of step S704.

[0174] Optionally, the plane in which the observation plane is located, the plane in which the foreground anchor plane is located, and the plane in which the background anchor plane is located are parallel to each other in pairs; the distance between the plane in which the observation plane is located and the plane in which the foreground anchor plane is located is a first distance, the distance between the plane in which the foreground anchor plane is located and the plane in which the background anchor plane is located is a second distance, the first distance is greater than or equal to 0, and the second distance is greater than 0. For convenience of description, the first distance is denoted as d rec, and the second distance is denoted as a depth difference d.

[0175] For example, the second distance can be determined by depth estimation of the first image by the electronic device to obtain a depth map, determining an average depth of the foreground image (also referred to as an average foreground depth) and an average depth of the background image (also referred to as an average background depth) based on the depth map, and determining the second distance based on the average depth of the foreground image and the average depth of the background image.

[0176] S602: The electronic device displays the second image on the first region, wherein the second image includes a first foreground part and a first background part, the pose of the electronic device when displaying the second image is the first pose, the first foreground part is obtained based on the foreground image and the first pose, and the first background part is obtained based on the background image and the first pose.

[0177] In some embodiments, when the pose of the electronic device is the first pose, the electronic device can determine a first position of the foreground anchor plane in the image coordinate system and a second position of the background anchor plane in the image coordinate system based on the position of the foreground anchor plane in the space, the position of the background anchor plane in the space, and the first pose; and then render the foreground image and the background image based on the first position and the second position to obtain the second image. For example, the process can be referred to in the following description of steps S705 to S707.

[0178] For example, the electronic device can project the foreground image to the first position to obtain a first image texture, project the background image to the second position to obtain a second image texture, and then render the first image texture and the second image texture to obtain the second image, wherein the rendering result of the first image texture is the first foreground part, and the rendering result of the second image texture includes the first background part.

[0179] S603: In a case where the pose of the electronic device is the second pose, the electronic device displays a third image on the first region, where the third image includes a second foreground part and a second background part, the second foreground part is obtained based on the foreground image and the second pose, and the second background part is obtained based on the background image and the second pose; and the first region includes the second region, and the content of the image displayed on the second region is different in a case where the pose of the electronic device is different.

[0180] The image (e.g., the second image and the third image) displayed on the first region can be a normal image without superimposed rendered reality image, or can be an AR image with superimposed rendered reality image. For example, FIGS. 5B-5E exemplarily show that the image displayed on the first region is a normal image without superimposed rendered reality image. For another example, the first region can also display an AR image as shown in FIG. 11B.

[0181] The second region is a partial region in the first region. The second region can be a rectangle or a circle, and can be a regular shape or an irregular shape, which is not limited in the present application. For example, the first region can be the preview region 520 in the user interface 52 shown in FIG. 5B, or the preview region 530 in the user interface 53 shown in FIG. 5C, or the preview region 540 in the user interface 54 shown in FIG. 5D, and FIG. 5E exemplarily shows that the second region is a rectangle.

[0182] For example, the second region can be the region 1 or the region 2 in FIG. 5E, and the content of the image displayed on the region 1 is different in a case where the pose of the electronic device is different (i.e., the viewing angle is different), and the content of the image displayed on the region 2 is different in a case where the pose of the electronic device is different (i.e., the viewing angle is different).

[0183] For another example, the second region can also be a region in the first region when the electronic device displays the AR image as shown in FIG. 11B on the first region.

[0184] In some embodiments, when the pose of the electronic device is the second pose, the electronic device can determine a third position of the foreground anchor face in the image coordinate system and a fourth position of the background anchor face in the image coordinate system based on the position of the foreground anchor face in the space, the position of the background anchor face in the space, and the second pose; and render the foreground image and the background image based on the third position and the fourth position to obtain the third image. For example, the process can refer to the related content of steps S709-S711 below.

[0185] It can be understood that when the pose of the electronic device is changed from the first pose to the second pose, the content displayed by the electronic device on the first area is updated from the third image to the second image. Taking the normal image as an example, the second image can be the image displayed in the preview area 520 in the user interface 52 shown in FIG. 5B, and the third image can be the preview area 530 in the user interface 53 shown in FIG. 5C or the preview area 540 in the user interface 54 shown in FIG. 5D; or alternatively, the second image can be the preview area 530 in the user interface 53 shown in FIG. 5C, and the third image can be the preview area 540 in the user interface 54 shown in FIG. 5D.

[0186] For example, the process of rendering the third image by the electronic device can be: the electronic device projects the foreground image to the third position to obtain a third image texture; projects the background image to the fourth position to obtain a fourth image texture; and renders the third image texture and the fourth image texture to obtain the third image, wherein the rendering result of the third image texture is the second foreground part, and the rendering result of the fourth image texture includes the second background part.

[0187] For example, the process of rendering the third image by the electronic device can be: the electronic device projects the foreground image to the third position to obtain a third image texture; projects the background image to the fourth position to obtain a fourth image texture; and renders the third image texture and the fourth image texture to obtain the third image, wherein the rendering result of the third image texture is the second foreground part, and the rendering result of the fourth image texture includes the second background part.

[0188] Optionally, the second image further includes a first real part, and the image captured by the electronic device at the first pose includes the first real part; and / or the third image further includes a second real part, and the image captured by the electronic device at the second pose includes the second real part.

[0189] The method embodiment shown in FIG. 6 includes many possible implementation schemes. Some implementation schemes will be described below with reference to FIG. 7A. It should be noted that the related concepts, operations or logical relationships not explained in FIG. 7A can refer to the corresponding descriptions in the embodiment shown in FIG. 6.

[0190] In this application, the embodiment shown in FIG. 7A can be regarded as a separate embodiment, and the embodiment shown in FIG. 6 can not depend on the technical solution of FIG. 6; and some steps in the embodiment shown in FIG. 7A can also be regarded as a separate embodiment.

[0191] FIG. 7A is a flow diagram of another image display method according to an embodiment of the present application.

[0192] In the embodiments of the present application, the second image and the third image are taken as examples for AR images, and the process of displaying the AR images by the electronic device is related to the interfaces shown in FIGS. 5A to 5E. The content of the screens in FIGS. 5A to 5E can be changed to AR images.

[0193] As shown in FIG. 7A, the method includes the following steps:

[0194] S701: In response to a first user operation for indicating stereoscopic display of a first image, the electronic device determines a foreground image corresponding to the first image and a background image corresponding to the first image.

[0195] The first image is a two-dimensional image.

[0196] In some embodiments, the electronic device has a first application (such as a gallery), and the electronic device can display an application interface of the first application, which includes the first image. Then, when detecting a user operation for indicating stereoscopic display of the first image, the electronic device determines a foreground image corresponding to the first image and a background image corresponding to the first image.

[0197] Optionally, the first image can be a photo selected by the user from the gallery, or a photo taken by the user using the electronic device. The first image can be an RGB image taken by the camera of the electronic device, or a color electronic drawing work, an electronic poster, etc. made by rendering, synthesis, PS, electronic drawing, etc. If the first image is a photo taken by a binocular device (having a left view and a right view), the electronic device can select any one of the views as the first image. It should be understood that the electronic device is not a binocular device in the present application. After the user selects the photo, the offline calculation module is entered. The offline module only performs one calculation in the entire method running process, and the calculation result is used for subsequent processing of the online module.

[0198] For example, the step S701 can include the following steps shown in FIG. 7B:

[0199] S7011: The electronic device performs semantic segmentation on the first image to obtain a foreground semantic template of the first image and a background semantic template of the first image.

[0200] For example, the electronic device can use a neural network-based method for semantic segmentation (which can be referred to as foreground and background semantic segmentation), such as a U-Net network structure of the neural network. It should be noted that there can be multiple semantic segmentation methods, including but not limited to traditional methods and neural network-based methods, which are not limited in the present application.

[0201] FIG. 8A is a schematic diagram of semantic segmentation according to an embodiment of the present application. As shown in FIG. 8A, the electronic device can perform semantic segmentation on the first image to obtain a foreground semantic template (may be referred to as a semantic template or mask_f or foreground mask of the foreground part) of the first image and a background semantic template (may be referred to as a semantic template or mask_b or background mask of the background part) of the first image. In the example of FIG. 8A, black represents pixel 0 and white represents pixel 1. In the foreground semantic template, the pixels in the background region are black (i.e., pixel 0) and the pixels in the foreground region are white (i.e., pixel 1). In the background semantic template, the pixels in the background region are white (i.e., pixel 1) and the pixels in the foreground region are black (i.e., pixel 0).

[0202] S7012: The electronic device determines a background image corresponding to the first image based on the first image and the background semantic template of the first image.

[0203] FIG. 8B is a schematic diagram of obtaining a background image according to an embodiment of the present application. As shown in FIG. 8B, the electronic device can perform a point multiplication operation on the first image and the background semantic template of the first image to obtain a background image 1. Then, the electronic device can perform pixel filling on the region in which the pixels are missing in the background image 1 (i.e., the position corresponding to the foreground) to obtain a background image corresponding to the first image. In an example, the method of pixel filling can be image inpainting or a network structure based on Unet or other background completion algorithm, which is not limited in the present application. It should be noted that the object originally occluded by the foreground in the background image corresponding to the first image is filled completely. As shown in the background image corresponding to the first image in FIG. 8B, the tree originally occluded by the foreground cat is filled completely.

[0204] S7013: The electronic device determines a foreground image corresponding to the first image based on the first image and the foreground semantic template of the first image.

[0205] In an example, the foreground image img_f can be obtained by the following formula: img_f = img * mask_f, where img is the first image (or the original image), mask_f is the foreground semantic template, and * is a point multiplication operation.

[0206] FIG. 8C is a schematic diagram of obtaining a foreground image according to an embodiment of the present application. As shown in FIG. 8C, the electronic device can perform a point multiplication operation on the first image and the background semantic template of the first image to obtain a foreground image corresponding to the first image.

[0207] Optionally, the steps S7011 to S7013 can be calculated by the offline calculation module shown in FIG. 2. After calculating the foreground image and the background image, the offline calculation module can send the foreground image and the background image to the online calculation module.

[0208] Optionally, the offline computing module can further calculate a foreground-background depth difference (referred to as depth difference) based on the foreground image and the background image, and the data sent to the online computing module can further include the depth difference. For example, the offline computing module can further perform depth estimation on the first image to obtain a depth map depth; then, based on the depth map depth, calculate an average background depth depth_b = avg (depth * mask_b) and an average foreground depth depth_f = avg (depth * mask_f); finally, calculate the absolute value of the difference between the two to obtain the depth difference d = |depth_b - depth_f|. It should be noted that the method of calculating the depth difference is only an example, and the method of calculating the depth difference is not limited in the present application.

[0209] S702: The electronic device captures an image through the camera in response to the first user operation.

[0210] In some embodiments, the electronic device can call a camera interface, open the camera, obtain image data (such as an RGB image) collected by the camera, obtain a camera preview stream, and display the content of the camera preview stream on the display screen of the electronic device.

[0211] S703: The electronic device determines the position of the observation surface in the world coordinate system.

[0212] In some embodiments, the electronic device constructs a world coordinate system based on camera data, and determines an observation surface in the world coordinate system based on a preset rule. The position of the observation surface in the world coordinate system is fixed and does not change. For example, the camera data includes data of a gravity sensor and an acceleration sensor, which can be used to make the vertical axis of the world coordinate system perpendicular to the ground. The preset rule can take the center of the camera view angle (i.e., the center of the image of the camera view angle) as the center of the observation surface, and the size of the observation surface can be a preset size, such as a size of 1880mm in length, 1040mm in width, and 2000mm in depth (equivalent to an 85-inch television).

[0213] It should be understood that the world coordinate system is used to describe the environment around the camera; after the world coordinate system is initialized, the world coordinate system is fixed and does not change with the movement of the electronic device (i.e., the camera) and the person.

[0214] For example, the position of the observation surface in the world coordinate system can be a rectangular region formed by the upper left corner coordinate of the observation surface and the lower right corner coordinate of the observation surface. That is, the position of the observation surface in the world coordinate system can be represented by the upper left corner coordinate of the observation surface and the lower right corner coordinate of the observation surface.

[0215] FIG. 9A is a schematic diagram of initializing an observation surface according to an embodiment of the present application. The coordinate system in FIG. 9A is a world coordinate system, and the position of the observation surface in the world coordinate system is shown by way of example. The initialized position of the observation surface is POS rec = [(X1 rec, Y1 rec, Z1 rec), (X2 rec, Y2 rec, Z2 rec)], where (X1 rec, Y1 rec, Z1 rec) is the coordinate of the upper left corner of the observation surface, and (X2 rec, Y2 rec, Z2 rec) is the coordinate of the lower right corner of the observation surface. By way of example, the size of the observation surface can be 1880 mm in length, 1040 mm in width, and 2000 mm in depth. The above coordinates have the following constraint relationships: X1 rec - X2 rec = 1040, Y2 rec - Y1 rec = 1880, and Z1 rec = Z2 rec = 2000.

[0216] S704: The electronic device determines the position of the foreground anchor surface and the position of the background anchor surface in the world coordinate system based on the position of the observation surface in the world coordinate system.

[0217] In some embodiments, the electronic device determines the position of the foreground anchor surface and the position of the background anchor surface in the world coordinate system based on the observation surface in the world coordinate system and a first correspondence relationship, the first correspondence relationship including a correspondence relationship between the observation surface and the foreground anchor surface and a correspondence relationship between the observation surface and the background anchor surface. For example, the correspondence relationship between the observation surface and the foreground anchor surface is that the plane on which the observation surface is located is parallel to the plane on which the foreground anchor surface is located, and the distance between the two planes is d rec; and the plane on which the observation surface is located is parallel to the plane on which the background anchor surface is located, and the distance between the two planes is the depth difference d.

[0218] FIG. 9B is a schematic diagram of initializing the foreground anchor surface and the background anchor surface according to an embodiment of the present application. The coordinate system in FIG. 9B is a world coordinate system, the initialized position of the foreground anchor surface is POS f = [(X1 f, Y1 f, Z1 f), (X2 f, Y2 f, Z2 f)], where X1 f = X1 b, X2 f = X2 b, Y1 f = Y1 b, Y2 f = Y2 b, Z1 f = Z1 rec - d rec, and Z2 f = Z1 f; and the initialized position of the background anchor surface is POS b = [(X1 b, Y1 b, Z1 b), (X2 b, Y2 b, Z2 b)], where (X1 b, Y1 b, Z1 b) is the coordinate of the upper left corner of the anchor surface in the world coordinate system, and (X2 b, Y2 b, Z2 b) is the coordinate of the lower right corner, and X1 b = X1 rec, Y1 b = Y1 rec, X2 b = X2 rec, Y2 b = Y2 rec, Z1 b = Z1 rec - d rec - d, and Z2 b = Z1 b.

[0219] For example, two methods of obtaining the depth difference d are introduced as follows. Method 1: For electronic devices with weak computing capability, an empirical value can be preset as the depth difference d, such as 500 mm. Method 2: For electronic devices with strong computing capability, depth estimation can be performed on the original image (such as the first image described above) through an algorithm, and then the average depths df and db of the foreground and background regions are obtained in combination with the foreground and background regions, and then the depth difference d is calculated through d = |df-db|. The method 2 has higher accuracy, but the calculation amount is large, and in actual situations, the two methods need to be flexibly switched according to the scene.

[0220] For example, the depth difference d can be set as 30 cm by default. It should be understood that when the depth difference increases, the distance between the foreground anchor surface and the background anchor surface becomes farther, and vice versa, when the depth difference decreases, the distance between the foreground anchor surface and the background anchor surface becomes closer. The distance difference of different distances will affect the final presentation of the picture content. For example, the effects brought by different depth differences are shown in FIG. 10. The perspective of FIG. 10 can be the perspective 2 shown in FIG. 4. When the depth difference d is larger, the foreground image under this perspective is smaller and the distance between the foreground image and the background image is larger. It should be understood that when the depth difference d is larger, the foreground image can also be larger at other perspectives (such as the initial perspective shown in FIG. 4). The presentation effect of the picture content by the depth difference is related to the perspective.

[0221] S705: The electronic device determines a first position of the foreground anchor surface in the image coordinate system and a second position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in the world coordinate system, the position of the background anchor surface in the world coordinate system, and the first pose of the electronic device.

[0222] In some embodiments, the pose of the electronic device is the first pose, and the electronic device can determine the positional relationship between the foreground anchor surface and the background anchor surface relative to the electronic device based on the first pose, the position of the foreground anchor surface in the world coordinate system, and the position of the background anchor surface in the world coordinate system; and determine the first position of the foreground anchor surface in the image coordinate system and the second position of the background anchor surface in the image coordinate system based on the positional relationship.

[0223] S706: The electronic device projects the foreground image to the first position to obtain a first image texture of the foreground image, and projects the background image to the second position to obtain a second image texture corresponding to the background image.

[0224] For example, the electronic device projects pixels of the foreground image to the first position based on the position of the observation plane in the image coordinate system, generates a first image texture of the foreground image, and projects pixels of the background image to the second position, and generates a second image texture corresponding to the background image. The observation plane is used to determine the projection area on the foreground anchor plane and the background anchor plane, as shown in FIG. 4, and the anchor plane position within the dashed line range of the initial view angle is the projection area. It can be understood that the observation plane affects the projection results (i.e., the first image texture and the second image texture) of the foreground image and the background image, and the projection method is not limited in the present application.

[0225] S707: The electronic device generates a second image based on the first image texture, the second image texture, and an image captured by the current camera.

[0226] In some embodiments, the electronic device can render the first image texture and the second image texture on the foreground anchor plane and the background anchor plane respectively to obtain a first rendering result; and superimpose and render the first rendering result and an image captured by the current camera (e.g., a first real image) to obtain a second image.

[0227] S708: The electronic device displays the second image in the first area.

[0228] For example, the interface currently displayed by the electronic device is an application interface of a gallery application, and the first area can be a preview area in the application interface.

[0229] For example, the second image can include a first foreground part (e.g., a cat), a first background part (e.g., a tree and a rock), and a first real part (e.g., a home scene), wherein the first foreground part is the first image texture described above; the first background part is part of the picture in the second image texture, and the picture not displayed in the second image texture is blocked by the first image texture; and the first real part is part of the picture in the first real image, and the picture not displayed in the first real image is blocked by the first image texture and the second image texture.

[0230] S709: When the current pose of the electronic device is a second pose, determining a third position of the foreground anchor plane in the image coordinate system and a fourth position of the background anchor plane in the image coordinate system based on the position of the foreground anchor plane in the world coordinate system, the position of the background anchor plane in the world coordinate system, and the second pose of the electronic device.

[0231] In some embodiments, the electronic device can update the coordinates of the observation plane, the foreground anchor plane and the background anchor plane projected in the image coordinate system through the camera extrinsic parameters, and the projection process is perspective projection. For example, the electronic device can obtain the IMU data, calculate the real-time pose of the electronic device through the SLAM method, obtain the relative motion of the camera coordinates based on the pose change of the electronic device, that is, the camera extrinsic parameters, wherein the camera extrinsic parameters can include a rotation matrix R and a translation matrix T, wherein R is a 3x3 matrix, and T is a 3x1 matrix; based on R and T, the coordinates pos_rec of the observation plane, the coordinates pos_b of the background anchor plane and the coordinates pos_f of the foreground anchor plane in the image coordinate system are determined.

[0232] Taking point A as an example, how to update the position of point A in the image coordinate system when the pose of the electronic device changes is introduced. The updating process can include: converting the coordinates of point A in the world coordinate system into the coordinates of point A in the camera coordinate system 1; when the camera coordinate system moves, the coordinates of point A in the camera coordinate system change, and based on the camera extrinsic parameters, the coordinates of point A in the camera coordinate system are updated to coordinates 2; based on the camera intrinsic parameters, the coordinates 2 are projected into the image coordinate system to obtain the position of point A in the image coordinate system.

[0233] For example, for any point on the observation plane, the foreground anchor plane and the background anchor plane, the coordinates of the point in the world coordinate system are (Xw, Yw, Zw), and the coordinates of the point in the image coordinate system after projection are (u, v), and the updating process can be shown in the following formula:

[0234] Wherein, f, (u0, v0), d x and d y are camera intrinsic parameters, f is the focal length of the camera, (u0, v0) is the image center point coordinates, d x and d y are the pixel sizes in horizontal and vertical directions respectively; Zc is the normalization coefficient.

[0235] FIG. 11A is a schematic diagram of an updated foreground anchor plane and background anchor plane according to an embodiment of the present application. FIG. 11A exemplarily shows that in a home scene, assuming that the initial position of the user is in front of the sofa, the positions of the background anchor plane and the foreground anchor plane in the image coordinate system are the image center positions; when the user moves the electronic device to the side of the sofa, after the electronic device updates the positions of the background anchor plane and the foreground anchor plane in the image coordinate system, the background anchor plane and the foreground anchor plane are located at the left positions in the image coordinate system.

[0236] S710: The electronic device projects the foreground image to the third position to obtain the third image texture of the foreground image, and projects the background image to the fourth position to obtain the fourth image texture corresponding to the background image.

[0237] For example, the electronic device can project pixels of the foreground image to the third position based on the current observation plane position in the image coordinate system, generate a third image texture of the foreground image, and project pixels of the background image to the fourth position, and generate a fourth image texture corresponding to the background image.

[0238] S711: The electronic device generates a third image based on the third image texture, the fourth image texture, and the image captured by the current camera.

[0239] In some embodiments, the electronic device can render the third image texture and the fourth image texture on the foreground anchor plane and the background anchor plane respectively to obtain a second rendering result; and superimpose the second rendering result and the image captured by the current camera (e.g., the second real image) to obtain the third image.

[0240] FIG. 11B is a schematic diagram of a third image according to an embodiment of the present application. The third image shown in FIG. 11B can be an image generated by projecting the third image texture and the fourth image texture onto the foreground anchor plane and the background anchor plane shown in FIG. 11A, respectively. As shown in FIG. 11B, the third image includes a second foreground part (i.e., the cat), a second background part (i.e., the tree and the rock), and a second real part (i.e., the home scene), wherein the second foreground part is the third image texture, the second background part is part of the fourth image texture, and the second real part is part of the second real image. It should be noted that the first real image and the second real image are different because the pose of the electronic device changes when the first real image and the second real image are captured, and the content of the first real part and the second real part is also different.

[0241] S712: The electronic device displays the third image in the first area.

[0242] In some embodiments, when the pose of the electronic device changes from the first pose to the second pose, the electronic device can update the second image displayed in the first area to the third image.

[0243] Referring to FIG. 12, FIG. 12 is a schematic diagram of the structure of the electronic device 200 according to an embodiment of the present application.

[0244] As shown in FIG. 12, the electronic device 200 provided by the embodiments of the present application can be used to implement the method described in the method embodiments, and the structure of the electronic device implementing the method embodiments shown in FIG. 6 or FIG. 7A can be as shown in FIG. 12. The specific implementation can be referred to the description in the method embodiments. The electronic device 200 can be a terminal device (such as a mobile phone), or a chip therein. For example, the electronic device 200 includes one or more processors 801 and a transceiver 802. The electronic device 200 can further include a memory 803. In an implementation, the electronic device 200 further includes input and output devices (not shown in FIG. 12), such as a display screen and a keyboard.

[0245] The processor 801 is configured to control the entire electronic device 200, execute software programs, and process data of the software programs. The memory 803 is mainly configured to store the software programs and the data. The transceiver 802 can include a control circuit and an antenna. The control circuit is mainly configured to convert a baseband signal and a radio frequency signal and process the radio frequency signal. The antenna is mainly configured to transceive a radio frequency signal in the form of an electromagnetic wave. The input and output devices, such as a touch screen, a display screen, and a keyboard, are mainly configured to receive data input by a user and output data to the user.

[0246] When the electronic device 200 is powered on, the processor 801 can read the software programs in the memory 803, interpret and execute instructions of the software programs, and process data of the software programs. The processor 801, the transceiver 802, and the memory 803 can be connected through a communication bus.

[0247] In an implementation, the processor 801 can store instructions, which can be a computer program. The computer program can run on the processor 801, and can cause the electronic device 200 to execute the method described in the method embodiments. The computer program can be fixed in the processor 801, and in this case, the processor 801 can be implemented by hardware.

[0248] In an implementation, the electronic device 200 can include circuitry that can implement the functions of sending or receiving or communicating in the foregoing method embodiments. The processor and transceiver described in the present application can be implemented on an integrated circuit (IC), an analog IC, a radio frequency integrated circuit (RFIC), a mixed-signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device 200, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), positive channel metal oxide semiconductor (PMOS), bipolar junction transistor (BJT), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0249] It can be understood that the electronic device 200 shown in the embodiments of the present application can also have more components than those shown in FIG. 12, and the embodiments of the present application do not limit this. For the specific steps performed by the processor, reference can be made to the description of the method embodiments above.

[0250] In the embodiments of the present application, the memory 803 stores executable instructions, and the processor 801 executes the executable instructions to implement the foregoing image display method, such as the image display method in the embodiments shown in FIGS. 6 and 7A. That is, the memory 803 has instructions for executing the image display method.

[0251] The present application also provides a computer program for implementing the operations and / or processes performed by the electronic device in the method provided by the present application.

[0252] The present application also provides a computer readable storage medium having computer code stored therein, which, when executed on a computer, causes the computer to perform the operations and / or processes performed by the electronic device in the method provided by the present application.

[0253] The application further provides a computer program product comprising computer code or a computer program which, when run on a computer, causes the operations and / or processes performed by the electronic device in the method provided by the application to be performed.

[0254] The application further provides a chip comprising a processor and a communication interface. The communication interface is configured to output and / or output data (including instructions), and / or the communication interface is configured to receive and / or send data. When the processor executes program instructions in the memory, the aforementioned image display method, for example, the image display method in the embodiment shown in FIG. 6 and FIG. 7A.

[0255] In several embodiments provided in the application, it should be understood that the disclosed system, electronic device and method can be implemented in other manners. For example, the embodiments of the electronic device described above are merely schematic; for example, the division of the units is only a logical function division; there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, electronic devices or units, and can also be electrically, mechanically or in other forms of connection.

[0256] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected based on actual requirements to achieve the technical effects of the solutions provided in the embodiments of the application.

[0257] In addition, each functional unit in the embodiments of the application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0258] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned readable storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0259] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image display method characterized by, The method is applied to an electronic device, and the method comprises: in response to a first user operation for indicating a stereoscopic display of a first image, determining a foreground image corresponding to the first image and a background image corresponding to the first image; displaying a second image on a first region, wherein the second image comprises a first foreground part and a first background part, a pose of the electronic device when displaying the second image is a first pose, the first foreground part is obtained based on the foreground image and the first pose, and the first background part is obtained based on the background image and the first pose; in a case where the pose of the electronic device is a second pose, displaying a third image on the first region, wherein the third image comprises a second foreground part and a second background part, the second foreground part is obtained based on the foreground image and the second pose, and the second background part is obtained based on the background image and the second pose; wherein the first region comprises a second region, and a content of a picture displayed by the second region is different when the pose of the electronic device is different.

2. The method of claim 1, wherein: a content of a picture displayed by the second region when the pose of the electronic device is the first pose belongs to the first background part, and a content of a picture displayed by the second region when the pose of the electronic device is the second pose belongs to the second foreground part; or a content of a picture displayed by the second region when the pose of the electronic device is the first pose belongs to the first foreground part, and a content of a picture displayed by the second region when the electronic device displays the third image belongs to the second background part.

3. The method of claim 1 or 2, wherein: the second image further comprises a first real part, and an image captured by the electronic device when the pose of the electronic device is the first pose comprises the first real part; and / or the third image further comprises a second real part, and an image captured by the electronic device when the pose of the electronic device is the second pose comprises the second real part. The method further comprises: in response to the first user operation, determining a position of an observation plane in space; 4. The method according to any one of claims 1 to 3, characterized in that, based on the position of the observation plane in space, determining a position of a foreground anchor plane in space and a position of a background anchor plane in space, wherein the foreground anchor plane and the background anchor plane are located in different planes, and the position of the foreground anchor plane in space and the position of the background anchor plane in space are used to determine the second image and / or the third image. a plane in which the observation plane is located, a plane in which the foreground anchor plane is located, and a plane in which the background anchor plane is located are parallel to each other, a distance between the plane in which the observation plane is located and the plane in which the foreground anchor plane is located is a first distance, a distance between the plane in which the foreground anchor plane is located and the plane in which the background anchor plane is located is a second distance, the first distance is greater than or equal to 0, and the second distance is greater than 0. The method further comprises:

5. The method of claim 4, wherein, ​ 6. The method according to claim 4 or 5, characterized in that, ​ determining a first position of the foreground anchor face in an image coordinate system and a second position of the background anchor face in the image coordinate system based on the position of the foreground anchor face in space, the position of the background anchor face in space, and the first pose of the electronic device when the pose of the electronic device is the first pose; rendering the foreground image and the background image based on the first position and the second position to obtain the second image.

7. The method of claim 6, wherein, The rendering of the foreground image and the background image based on the first position and the second position to obtain the second image comprises: projecting the foreground image to the first position to obtain a first image texture; projecting the background image to the second position to obtain a second image texture; rendering the first image texture and the second image texture to obtain the second image, the rendering result of the first image texture being the first foreground part, and the rendering result of the second image texture comprising the first background part.

8. The method according to any one of claims 4-7, characterized in that, The method further comprises: determining a third position of the foreground anchor face in the image coordinate system and a fourth position of the background anchor face in the image coordinate system based on the position of the foreground anchor face in space, the position of the background anchor face in space, and the second pose of the electronic device when the pose of the electronic device is the second pose; rendering the foreground image and the background image based on the third position and the fourth position to obtain the third image.

9. The method of claim 8, wherein, The rendering of the foreground image and the background image based on the third position and the fourth position to obtain the third image comprises: projecting the foreground image to the third position to obtain a third image texture; projecting the background image to the fourth position to obtain a fourth image texture; rendering the third image texture and the fourth image texture to obtain the third image, the rendering result of the third image texture being the second foreground part, and the rendering result of the fourth image texture comprising the second background part.

10. The method according to any one of claims 5-8, characterized in that, The method further comprises: performing depth estimation on the first image to obtain a depth map; determining an average depth of the foreground image and an average depth of the background image based on the depth map; determining the second distance based on the average depth of the foreground image and the average depth of the background image.

11. The method according to any one of claims 1-10, characterized in that, The method further comprises: performing semantic segmentation on the first image to obtain a foreground semantic template of the first image and a background semantic template of the first image; determining the background image based on the first image and the background semantic template of the first image; determining the foreground image based on the first image and the foreground semantic template of the first image.

12. An electronic device, comprising: The electronic device comprises one or more processors and one or more memories; wherein the one or more memories are coupled with the one or more processors, and the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions which, when executed by the one or more processors, cause the electronic device to perform the method according to any one of claims 1-11.

13. A computer program product comprising instructions, characterized in that, The computer program product, when running on an electronic device, causes the electronic device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium comprising instructions, wherein: The instructions, when running on an electronic device, cause the electronic device to perform the method according to any one of claims 1-11.

15. A chip, characterized by The electronic device comprises a processor and a communication interface, the communication interface being configured to output and / or output data, and / or the communication interface being configured to receive and / or send data, and the processor, when executing program instructions in a memory, implements the method according to any one of claims 1-11.

Images