Image display method and electronic equipment

Abstract

The invention discloses an image display method and electronic equipment, and the method comprises the steps: the electronic equipment determines a foreground image and a background image corresponding to a first image in response to a first user operation used for indicating the stereoscopic display of the first image; a second image is displayed on the first area, the second image comprises a first foreground part and a first background part, and the pose when the electronic equipment displays the second image is the first pose; under the condition that the pose of the electronic equipment is the second pose, a third image is displayed on the first area, and the third image comprises a second foreground part and a second background part; wherein the first area comprises a second area, and when the poses of the electronic equipment are different, the picture contents displayed in the second area are different. In the method, the electronic equipment can display different images corresponding to the same image when the pose is changed, and the images displayed by the electronic equipment comprise a foreground part and a background part, so that the two-dimensional image presents a three-dimensional effect with a sense of space.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image display method and an electronic device. Background Technology

[0002] With the development of augmented reality (AR) and virtual reality (VR) technologies, the way images and videos are displayed is gradually evolving from 2D to 3D, aiming to present users with a more three-dimensional and immersive experience. Current 3D displays typically use binocular devices, which can simultaneously play photos or videos from two perspectives to create spatial effects. However, binocular devices are expensive and not yet widely adopted, serving only a niche audience.

[0003] How to present a three-dimensional effect with a sense of space in two-dimensional images on electronic devices is a problem that the industry urgently needs to solve. Summary of the Invention

[0004] This application provides an image display method and an electronic device. In this method, the electronic device can display different images corresponding to the same image when the pose changes. The image displayed by the electronic device includes a foreground part and a background part, thereby presenting a two-dimensional image with a sense of space and a three-dimensional effect.

[0005] The present application is described below from different aspects. It should be understood that the different implementation methods and beneficial effects described below can be referenced from each other.

[0006] In a first aspect, this application discloses an image display method, which can be executed by an electronic device or a module (e.g., a chip) of an electronic device. The method may include: in response to a first user operation for instructing a stereoscopic display of a first image, determining a foreground image corresponding to the first image and a background image corresponding to the first image; displaying a second image in a first region, wherein the second image includes a first foreground portion and a first background portion, the pose of the electronic device when displaying the second image is a first pose, the first foreground portion is obtained based on the foreground image and the first pose, and the first background portion is obtained based on the background image and the first pose; when the pose of the electronic device is a second pose, displaying a third image in the first region, wherein the third image includes a second foreground portion and a second background portion, the second foreground portion is obtained based on the foreground image and the second pose, and the second background portion is obtained based on the background image and the second pose; wherein the first region includes a second region, and the content displayed in the second region is different when the pose of the electronic device is different.

[0007] In this embodiment, the electronic device can generate a foreground image and a background image corresponding to a two-dimensional image (such as the first image described above). Based on the pose of the electronic device, the foreground image, and the background image, the device displays the image corresponding to the two-dimensional image. For example, when the electronic device is in the first pose, it displays the second image; when the pose of the electronic device changes from the first pose to the second pose, the content displayed by the electronic device is updated from the second image to the third image. In this method, the image corresponding to the two-dimensional image includes a foreground portion and a background portion. For example, the second image includes a first foreground portion and a first background portion, and the third image includes a second foreground portion and a second background portion. When the pose of the electronic device changes, the image corresponding to the two-dimensional image displayed by the electronic device changes, that is, the foreground portion and background portion displayed by the electronic device also change, thereby presenting a three-dimensional effect with a sense of space in the two-dimensional image.

[0008] In conjunction with the first aspect, in one possible implementation, when the electronic device is in a first pose, the content displayed in the second area belongs to the first background portion, and when the electronic device is in a second pose, the content displayed in the second area belongs to the second foreground portion; or, when the electronic device is in a first pose, the content displayed in the second area belongs to the first foreground portion, and when the electronic device displays a third image, the content displayed in the second area belongs to the second background portion.

[0009] In this embodiment, when the pose of the electronic device changes, the foreground and background portions of the two-dimensional image displayed by the electronic device change. For example, when the pose of the electronic device changes from a first pose to a second pose, the content of the image displayed in the second area is updated from the content of the background image (i.e., the first background portion mentioned above) to the content of the foreground image (i.e., the second foreground portion); or, for example, when the pose of the electronic device changes from a first pose to a second pose, the content of the image displayed in the second area is updated from the content of the foreground image (i.e., the first foreground portion mentioned above) to the content of the background image (i.e., the second background portion), thereby presenting a three-dimensional effect with a sense of space in the two-dimensional image.

[0010] In conjunction with the first aspect, in one possible implementation, the second image further includes a first real portion, which is included in the image captured by the electronic device in the first pose; and / or, the third image further includes a second real portion, which is included in the image captured by the electronic device in the second pose.

[0011] In this embodiment, the second and third images can be AR images generated based on a background image, a foreground image, and a captured image. In this method, an electronic device can present a two-dimensional image in a real-world scene. For example, the two-dimensional image can be a poster, and the real-world scene can be an exhibition hall. The electronic device can display the poster's image within the exhibition hall. The poster includes a foreground and a background. When the user moves the electronic device (i.e., the device's position changes), the foreground and background of the poster change on the device's display screen, presenting the poster with a three-dimensional effect. This method combines two-dimensional images with a real-world scene, providing users with an immersive viewing experience.

[0012] In conjunction with the first aspect, in one possible implementation, the method further includes: in response to a first user operation, determining the position of the observation surface in space; and based on the position of the observation surface in space, determining the position of the foreground anchor surface and the position of the background anchor surface in space, wherein the foreground anchor surface and the background anchor surface are located on different planes, and the positions of the foreground anchor surface and the background anchor surface in space are used to determine a second image and / or a third image.

[0013] In this embodiment, the electronic device can determine the positions of the observation plane, the foreground anchor plane, and the background anchor plane in space (such as the world coordinate system). The positions of the observation plane, the foreground anchor plane, and the background anchor plane are used to determine the display positions and display perspectives of the foreground image and the background image corresponding to the two-dimensional image, thereby supporting the electronic device to display the images corresponding to the two-dimensional image under different perspectives.

[0014] In conjunction with the first aspect, in one possible implementation, the plane containing the observation surface, the plane containing the foreground anchoring surface, and the plane containing the background anchoring surface are parallel to each other; the distance between the plane containing the observation surface and the plane containing the foreground anchoring surface is a first distance, and the distance between the plane containing the foreground anchoring surface and the plane containing the background anchoring surface is a second distance, wherein the first distance is greater than or equal to 0, and the second distance is greater than 0.

[0015] In this method, the second distance is greater than 0. The foreground anchoring surface is used to display the foreground part corresponding to the first image (such as the first foreground part mentioned above), and the background anchoring surface is used to display the background part corresponding to the first image (such as the first background part), so as to achieve the display effect of separating the foreground part and the background part.

[0016] In conjunction with the first aspect, in one possible implementation, the method further includes: when the pose of the electronic device is a first pose, determining a first position of the foreground anchor surface in the image coordinate system and a second position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in space, the position of the background anchor surface in space, and the first pose; and rendering the foreground image and the background image based on the first position and the second position to obtain a second image.

[0017] In conjunction with the first aspect, in one possible implementation, rendering a foreground image and a background image based on a first position and a second position to obtain a second image includes: projecting the foreground image onto the first position to obtain a first image texture; projecting the background image onto the second position to obtain a second image texture; and rendering the first image texture and the second image texture to obtain a second image, wherein the rendering result of the first image texture is a first foreground portion, and the rendering result of the second image texture includes a first background portion.

[0018] In conjunction with the first aspect, in one possible implementation, the method further includes: when the electronic device is in a second pose, determining a third position of the foreground anchor surface in the image coordinate system and a fourth position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in space, the position of the background anchor surface in space, and the second pose; and rendering the foreground image and the background image based on the third position and the fourth position to obtain a third image.

[0019] In conjunction with the first aspect, in one possible implementation, rendering the foreground image and the background image based on the third position and the fourth position to obtain the third image includes: projecting the foreground image onto the third position to obtain a third image texture; projecting the background image onto the fourth position to obtain a fourth image texture; and rendering the third image texture and the fourth image texture to obtain the third image, wherein the rendering result of the third image texture is a second foreground portion, and the rendering result of the fourth image texture includes a second background portion.

[0020] In conjunction with the first aspect, in one possible implementation, the method further includes: performing depth estimation on the first image to obtain a depth map; determining the average depth of the foreground image and the average depth of the background image based on the depth map; and determining a second distance based on the average depth of the foreground image and the average depth of the background image.

[0021] This method determines the second distance through depth estimation, which can present a more reasonable display effect that separates the foreground and background parts.

[0022] In conjunction with the first aspect, in one possible implementation, the method further includes: performing semantic segmentation on the first image to obtain a foreground semantic template and a background semantic template of the first image; determining a background image based on the first image and the background semantic template of the first image; and determining a foreground image based on the first image and the foreground semantic template of the first image.

[0023] Secondly, this application provides an electronic device. The electronic device may include a memory and a processor. The memory may be used to store computer programs. The processor may be used to invoke the computer programs, causing the electronic device to execute as described in the first aspect or any possible implementation thereof.

[0024] Thirdly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform as described in the first aspect or any possible implementation thereof.

[0025] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the first aspect or any possible implementation thereof. The technical effects achieved by the above aspects can be referred to each other or to the beneficial effects in the method embodiments shown below, and will not be repeated here.

[0026] Fifthly, this application provides a chip including a processor and a communication interface. The communication interface is used for outputting and / or outputting data (including instructions), and / or for receiving and / or sending data. When the processor executes program instructions in memory, it implements the method described in the first aspect or any possible embodiment of the first aspect. Attached Figure Description

[0027] Figure 1A This is a schematic diagram of the hardware structure of the electronic device 100 provided in the embodiments of this application;

[0028] Figure 1B This is a software structure block diagram of the electronic device 100 provided in the embodiments of this application;

[0029] Figure 2 This is a schematic diagram of a computing module provided in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating the positional relationship between an observation surface, a foreground anchoring surface, and a background anchoring surface, as provided in an embodiment of this application.

[0031] Figure 4 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0032] Figures 5A to 5D These are some user interfaces provided by example in the embodiments of this application;

[0033] Figure 5E This is a schematic diagram showing the stereoscopic effect of a two-dimensional image presented from different viewpoints in the embodiments of this application;

[0034] Figure 6 This is a flowchart illustrating an image display method provided in an embodiment of this application;

[0035] Figure 7A This is a schematic flowchart of another image display method provided in an embodiment of this application;

[0036] Figure 7B This is a schematic flowchart of a method for obtaining a foreground image and a background image provided in an embodiment of this application;

[0037] Figure 8A This is a schematic diagram illustrating semantic segmentation provided in an embodiment of this application;

[0038] Figure 8B This is a schematic diagram illustrating an embodiment of the present application for obtaining a background image;

[0039] Figure 8C This is a schematic diagram illustrating an embodiment of the present application for acquiring a foreground image;

[0040] Figure 9A This is a schematic diagram of an embodiment of the present application providing an initial observation surface;

[0041] Figure 9B This is a schematic diagram illustrating the initialization of a foreground anchoring surface and a background anchoring surface according to an embodiment of this application;

[0042] Figure 10 This is a schematic diagram illustrating the effect of different depth differences provided by an embodiment of this application;

[0043] Figure 11A This is a schematic diagram of an updated foreground anchoring surface and background anchoring surface provided in an embodiment of this application;

[0044] Figure 11B This is a schematic diagram of a third image provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the structure of the electronic device 200 provided in the embodiments of this application. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0047] In the description of this application, the terms "first," "second," etc., are used only to distinguish different objects and do not limit the quantity or order of execution, nor do they imply that they are necessarily different. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0048] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one item", "one or more of the following", or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c; a and b; a and c; b and c; or a and b and c. Here, a, b, and c can be single or multiple.

[0049] In the description of this application, the words "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design that is described as "exemplary," "for example," or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplary," "for example," or "for example" is intended to present the relevant concepts in a specific manner.

[0050] It is understood that in the description of this application, "when," "if," and "if" all refer to the electronic device performing a corresponding action under certain objective circumstances, and are not time-limited, nor do they require the electronic device to perform a judgment action, nor do they imply any other limitations. The electronic device performing a corresponding action under certain objective circumstances includes: satisfying the objective circumstances, i.e., being able to perform the corresponding action; or satisfying both the objective circumstances and other circumstances, in order to perform the corresponding action.

[0051] In this application, "simultaneously" can be understood as at the same point in time, within a period of time, or within the same cycle; the specific meaning can be determined by considering the context.

[0052] In this application, the use of singular designations for elements is intended to represent "one or more" rather than "one and only one," unless otherwise specified.

[0053] In addition, the terms “system” and “network” are often used interchangeably in this article.

[0054] It is understood that in the various embodiments of this application, expressions such as "A corresponds to B", "A and B correspond", "A corresponds to B", or similar expressions indicate that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0055] Currently, the main solutions for stereoscopic image display include the following:

[0056] Option 1: Reconstruct 3D images from 2D images. This can be done by using neural networks, neural radiance field (NeRF) rendering, mesh reconstruction, etc., to create a 3D model of the original RGB image or other 2D images. The resulting 3D model is then used for stereoscopic display to create an immersive photograph with depth information. Option 2: Display the reconstructed scene using a binocular device. This involves depth estimation of the 2D image to generate left and right binocular images with parallax, which are then displayed on a binocular device to achieve a stereoscopic effect.

[0057] Among them, the 3D reconstruction process in Scheme 1 has high computational complexity, large computational overhead, high latency and low accuracy; Scheme 2 cannot run on non-binocular devices (such as mobile phones, tablets, PCs and other planar terminals), and binocular devices are expensive. Therefore, the above methods are difficult to commercialize on a large scale.

[0058] This application provides an image display method and an electronic device. In this method, the electronic device can generate a stereoscopic image with front and back background depth information based on a two-dimensional image, thereby realizing the three-dimensional stereoscopic display of a two-dimensional image (such as a monocular photo / video taken by a regular camera) and solving the problem of the need for three-dimensional reconstruction in existing methods.

[0059] For example, an electronic device processes a two-dimensional image to obtain a foreground image and a background image corresponding to the two-dimensional image; then, it renders the foreground image on a foreground anchor surface and the background image on a background anchor surface; and displays the rendering result. Thus, an observer can see different image content from the foreground and background images from different perspectives, with depth and occlusion effects.

[0060] The embodiments of this application can also combine different postures of electronic devices (or cameras of electronic devices) to perform stereoscopic display and interaction on electronic devices (such as mobile phones) in real time, solving the problem of needing dedicated binocular equipment and greatly improving the scope of application of the technology.

[0061] For example, when a user moves, the electronic device can calculate the pose of the electronic device in real time using the device information of the inertial measurement unit (IMU). By changing the pose, the device can perform perspective projection on the image on the dual anchoring surfaces (i.e., the foreground anchoring surface and the background anchoring surface mentioned above), render it in real time, present the effect of a realistic scene, and complete the interaction with the user.

[0062] To provide a clearer and more detailed description of the image display method provided in the embodiments of this application, the electronic device provided in the embodiments of this application will be described first below.

[0063] Electronic devices are devices with display functions.

[0064] For example, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, etc. Exemplary embodiments of the electronic device include, but are not limited to, those equipped with... Portable electronic devices running Linux or other operating systems. These devices can also be other portable electronic devices, such as laptops.

[0065] Alternatively, the structure of the electronic device can be found in [reference needed]. Figures 1A to 1B The structure of the electronic device 100 in the text, or see the following: Figure 12 The structure of the electronic device 200 is not limited in this application.

[0066] The following describes an electronic device provided by an embodiment of this application.

[0067] Figure 1A A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0068] The following detailed description uses electronic device 100 as an example. It should be understood that electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0069] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc.

[0070] Optionally, the sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0071] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0072] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, GPU, image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0073] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0074] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0075] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0076] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to a touch sensor, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to a touch sensor through the I2C interface, enabling the processor 110 and the touch sensor to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0077] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0078] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0079] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0080] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0081] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0082] The SIM interface can be used to communicate with the SIM card interface 195 to transmit data to or read data from the SIM card.

[0083] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0084] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments. The charging management module 140 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc.

[0085] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on electronic device 100. Modem processor can include modulator and demodulator.

[0086] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0087] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0088] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0089] Electronic device 100 can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194, and application processor. Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. The digital signal processor is used to process digital signals, including digital image signals and other digital signals. For example, when electronic device 100 selects a frequency, the digital signal processor performs Fourier transforms on the frequency energy.

[0090] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0091] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0092] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0093] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as facial recognition, fingerprint recognition, mobile payment, etc.). The data storage area may store data created during the use of electronic device 100 (such as facial information template data, fingerprint information templates, etc.). Furthermore, internal memory 121 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0094] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0095] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0096] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0097] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0098] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0099] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0100] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0101] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0102] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0103] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and detach from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication.

[0104] In this embodiment, the electronic device 100 can execute the image display method through the processor 110 and display the image rendered by this application through the display screen 194. For example, the electronic device 100 can be a commonly used mobile device such as a mobile phone, tablet, or PC that has hardware devices such as a processor 110 and a camera 193.

[0105] Figure 1B This is a software structure block diagram of an electronic device 100 disclosed in an embodiment of this application.

[0106] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: the application layer, the application framework layer, the runtime and system libraries, and the kernel layer.

[0107] The application layer can include a series of application packages.

[0108] like Figure 1B As shown, the application package can include applications (also known as apps) such as gallery, camera, calendar, call, WLAN, Bluetooth, music, and video.

[0109] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0110] like Figure 1B As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0111] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0112] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0113] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0114] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0115] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0116] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog-style notifications on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0117] The runtime consists of the core libraries and the virtual machine. The runtime is responsible for system scheduling and management.

[0118] The core library consists of two parts: one part is the functionalities that the programming language (e.g., Java) needs to call, and the other part is the system's core library.

[0119] The application layer and application framework layer run in a virtual machine. The virtual machine executes the programming files (e.g., .jave files) of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0120] The system library can include multiple functional modules. For example: a calculation module, a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES, Vulkan), and 2D graphics engines (e.g., SGL).

[0121] The calculation module is used to execute the image display method provided in this application to process the image, such as image segmentation, image rendering and image display, etc. For details, please refer to the relevant description below, which will not be elaborated here.

[0122] It should be noted that the computing module may also be located in other layers in other layered software architectures, or may be split into multiple modules and set in different layers. This application does not limit this.

[0123] The Surface Manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0124] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0125] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0126] A 2D graphics engine is a graphics engine for 2D drawing.

[0127] The kernel layer is the layer between hardware and software. The kernel layer can contain display drivers, camera drivers, audio drivers, sensor drivers, virtual card drivers, etc.

[0128] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 100.

[0129] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0130] Figure 2 This is a schematic diagram of a computing module provided in an embodiment of this application.

[0131] like Figure 2 As shown, the computing module may include an offline computing module and an online computing module.

[0132] In some embodiments, a user can select a two-dimensional image in an electronic device for stereoscopic display. Correspondingly, the electronic device can respond to the user's operation by running an offline computing module and an online computing module for the two-dimensional image, thereby achieving a three-dimensional stereoscopic display of the two-dimensional image. Optionally, the electronic device can start running the offline computing module and the online computing module simultaneously, or sequentially; this application does not limit this.

[0133] The offline computing module prepares data for the online computing module. For example, the input to the offline computing module is a user-selected two-dimensional image (referred to as the first image for convenience), and the output can be a foreground image and a background image corresponding to the first image. Optionally, the output of the offline computing module may also include the depth difference between the foreground and background images. In essence, the output of the offline computing module serves as the data preparation for the online computing module.

[0134] The online calculation module is used to display the stereoscopic image corresponding to the first image on the screen of the electronic device, and to interact and display with the user based on the pose of the electronic device. It should be noted that the offline calculation module may run only once for a single image, while the online calculation module can continuously perform calculations in a loop to maintain continuous interaction with the user.

[0135] The offline computing module and the online computing module are described in detail below, with examples.

[0136] like Figure 2 As shown, the offline computing module can include the following steps: 1) semantic segmentation; 2) background completion; 3) obtaining foreground and background images; 4) calculating depth difference.

[0137] For example, the offline computing module can perform semantic segmentation (or foreground semantic segmentation) and background completion on the first image to obtain the foreground image and the background image corresponding to the first image; and obtain the depth difference d between the background image and the foreground image, where d is a scalar value. Through the above steps, for the first image, its corresponding complete background image and foreground image can be obtained. For specific processes, please refer to the embodiments below, which will not be elaborated here.

[0138] It should be noted that this application does not limit the background completion algorithm. For example, the background completion algorithm can be a pixel-filling (image inpainting) method based on neural networks. The depth difference mentioned above can also be preset or calculated by an offline computing module. This application does not limit the method for obtaining the depth difference.

[0139] like Figure 2 As shown, the operation of the online calculation module can include the following steps: 1) Initialize the world coordinate system; 2) Initialize the observation surface (also called the frame); 3) Initialize the foreground anchoring surface and the background anchoring surface; 4) Calculate the camera pose; 5) Perform texture projection of the background image and the foreground image based on the foreground anchoring surface, the background anchoring surface and the camera pose; 6) Render and display the results based on the texture projection.

[0140] In some embodiments, the online calculation module can acquire camera data, such as video streams captured by the camera of an electronic device, and display it on the screen of the electronic device (e.g., real-time display); initialize the world coordinate system; initialize the coordinates of the observation surface in the world coordinate system (also known as placing the observation surface in space), for example, using the center of the image captured by the camera as the center of the observation surface, and determining the position of the observation surface in the world coordinate system based on a preset observation surface size; initialize the foreground anchor surface and the background anchor surface based on the coordinates of the observation surface in the world coordinate system, i.e., determine the coordinates of the foreground anchor surface and the background anchor surface in the world coordinate system; and calculate the pose of the electronic device (e.g., 6DoF pose). Specifically, this can be the pose of the camera on the electronic device. The pose of the electronic device can be calculated using SLAM or other methods, which are not limited in this application. The pose information of the electronic device is applied to the two anchoring surfaces mentioned above; that is, based on the position information of the electronic device, the coordinate positions of the two anchoring surfaces and the observation surface relative to the electronic device are determined. The foreground image and background image are respectively subjected to perspective projection to obtain the image texture of the foreground anchoring surface and the image texture of the background anchoring surface. These image textures are then rendered onto the foreground anchoring surface and the background anchoring surface, respectively. Based on the observation surface, frustum culling is performed to obtain the texture within the imaging range of the observation surface, and rendering and display are completed. Optionally, when the user moves the electronic device in any direction, the pose of the electronic device changes (e.g., the SLAM module can update the 6DoF pose of the device in real time). The electronic device can then repeatedly execute the texture rendering process to complete the display and interaction of the stereoscopic image.

[0141] It should be understood that the world coordinate system is used to describe the position of objects in the real world, while the two-dimensional image coordinate system (or image coordinate system for short) is used to describe the position of objects displayed on the screen. The coordinates of the world coordinate system and the coordinates of the image coordinate system can be mapped, that is, any object (X,Y,Z) in the world coordinate system can be mapped one by one to the corresponding image coordinate system (u,v) through perspective projection.

[0142] The following example illustrates the process of initializing the foreground anchor plane and the background anchor plane.

[0143] Assume that the position of the observation surface in the world coordinate system is represented by formula (1).

[0144] P_rec=(X_rec,Y_rec,Z_rec) Formula (1)

[0145] It should be understood that once the observation surface P_rec is determined, it is fixed in space, remaining relatively stationary with respect to surrounding objects, and does not move with the movement of electronic devices.

[0146] Therefore, based on formula (1), the position of the foreground anchoring surface in the world coordinate system can be represented by the following formula (2); and the position of the background anchoring surface coordinates in the world coordinate system can be represented by formula (3).

[0147] Foreground anchoring plane coordinates P_f=(X_rec,Y_rec,Z_rec-d_rec) Formula (2)

[0148] Background anchoring plane coordinates P_b=(X_rec,Y_rec,Z_rec-drec-d) Formula (3)

[0149] Where d_rec≥0, d>0.

[0150] It should be understood that, similar to the observation surface, once the foreground anchoring surface and the background anchoring surface are determined, they remain relatively stationary with respect to surrounding objects and do not change with the pose of the electronic device; their coordinate positions will be anchored in space.

[0151] Figure 3 This is a schematic diagram illustrating the positional relationship between an observation surface, a foreground anchoring surface, and a background anchoring surface, as provided in an embodiment of this application. Figure 3 This example illustrates the positional relationships of the observation point (i.e., the location of the electronic device), the observation plane, the foreground anchor plane, and the background anchor plane in the world coordinate system (i.e., space). Assume the user can hold the electronic device at the observation point, the observation plane is located in the space in front of the camera of the electronic device, the planes containing the observation plane, the foreground anchor plane, and the background anchor plane are parallel to each other, the distance between the plane containing the observation plane and the plane containing the foreground anchor plane is d_rec, and the distance between the plane containing the foreground anchor plane and the plane containing the background anchor plane is d. Figure 3 The dashed lines in the diagram represent the viewing angle range formed based on the observation plane. The textures on the foreground and background anchor planes within this viewing angle range can be displayed on electronic devices. In other words, the observation plane is associated with the display of the foreground and background images. For details regarding the dimensions of the observation plane, foreground anchor plane, and background anchor plane, please refer to the relevant descriptions below; they will not be elaborated upon here.

[0152] First, the following exemplary combination Figure 4 Schematic diagram of application scenarios and Figures 5A to 5D The user interface shown introduces the application scenarios to which this application applies.

[0153] Electronic devices are devices equipped with display services, such as smartphones and tablets. The image processing method provided in this application can be manifested as a "3D display" function on an electronic device. It should be noted that in some other embodiments of this application, this function may also be referred to as "AR display," etc., and this application does not limit it to that.

[0154] Figure 4This is a schematic diagram of an application scenario provided by an embodiment of this application. In this scenario, user A can open the gallery on an electronic device, click on the thumbnail of a first image in the gallery interface, and correspondingly, the electronic device displays a browsing interface for the first image. For example, this browsing interface can be... Figure 5A The user interface 51 shown is included. The user interface 51 includes a preview area 510 and 3D display controls 511, wherein the preview area 510 is used to display a first image. Figure 5A The example shown in the first image includes a cat, a rock next to the cat, and a tree behind the cat; the 3D display control 511 is used to enable the aforementioned "3D display" function. It should be understood that the user interface 51 also includes, for example,... Figure 5A The sharing and favorite controls shown are not limited in this application.

[0155] In some embodiments, such as Figure 4 As shown, assume user A holds the electronic device in the initial position (the pose of the electronic device is the initial pose). User A activates the "3D display" function of the electronic device for the first image. For example, user A clicks... Figure 5A The 3D display control 511 in the user interface 51 shown allows the electronic device to respond to user actions by determining the observation plane, determining the foreground anchor plane and the background anchor plane based on the observation plane, and then displaying, based on the foreground anchor plane, the background anchor plane, the initial pose, and the first image, as shown. Figure 5B The user interface 52 is shown. It should be understood that once the observation surface is determined, its position in space is fixed, such as... Figure 4 As shown, when user A moves from the initial position to position 1 or position 2, the position of the observation surface in space remains unchanged.

[0156] The user interface 52 includes a preview area 520 and an adjustment area 521. The preview area 520 displays a stereoscopic image of the first image from an initial viewing angle; the adjustment area 521 is used to adjust the currently displayed stereoscopic image. Figure 5BThe example illustrates that adjustment area 521 includes a first control 521a, a second control 521b, and a third control 521c. The first control 521a is used to adjust the distance between the observation surface and the electronic device (or the camera of the electronic device); the second control 521b is used to adjust the distance between the foreground anchoring surface and the electronic device (or the camera of the electronic device); and the third control 521c is used to adjust the distance between the background anchoring surface and the electronic device (or the camera of the electronic device). Understandably, when the pose of the electronic device remains unchanged, adjusting the first control 521a will change the content of the image displayed on the screen. For example, the greater the distance between the observation surface and the electronic device, the more content the stereoscopic image corresponding to the first image contains, and vice versa. Adjusting the second control 521b or the third control 521c will change the visual distance between the foreground image and the background image corresponding to the first image.

[0157] It should be noted that the user interface 52 is only an example. The user interface 52 may also include display effects such as 3D logos or color imaging at the edges of the image area. This application does not limit this.

[0158] like Figure 4 As shown, when user A moves to position 1, assuming the current pose of the electronic device is pose 1, and the viewing angle of the camera of the electronic device towards the observation plane is viewpoint 1, for example, the user interface displayed by the electronic device can be... Figure 5C The user interface 53 is shown. The user interface 53 may include a preview area 530 and an adjustment area 521, wherein the preview area 530 is used to display the stereoscopic image presented by the first image at the viewing angle 1; the adjustment area 521 is used to adjust the currently displayed stereoscopic image, and the details of the adjustment area 521 can be found above, and will not be repeated here.

[0159] contrast Figure 5B and Figure 5C As can be seen, the content of the stereoscopic image presented on the electronic device changes when the user's viewing angle changes (i.e., the pose of the electronic device changes). For example, the occlusion of the background (such as a tree) by the foreground (e.g., a cat) changes. Figure 5B The cat did not block the left branch of the tree, and Figure 5C The cat obscures the left branch of the tree; similarly, the size of the foreground and background elements in the image changes compared to... Figure 5B , Figure 5C The content on the left side of the screen is reduced in size.

[0160] like Figure 4 As shown, when user A moves to position 2, assuming the current pose of the electronic device is pose 2, and the viewing angle of the camera of the electronic device towards the observation plane is viewpoint 2, for example, the user interface displayed by the electronic device can be... Figure 5DThe user interface 54 is shown. The user interface 54 includes a preview area 540 and an adjustment area 521. The preview area 540 is used to display the stereoscopic image presented by the first image at the viewing angle 2. The adjustment area 521 is used to adjust the currently displayed stereoscopic image. For details of the adjustment area 521, please refer to the above text, which will not be repeated here.

[0161] contrast Figure 5C and Figure 5D As can be seen, when the user's viewing angle changes (i.e., the pose of the electronic device changes), the content of the stereoscopic image displayed on the electronic device changes. For easier comparison... Figure 5C and Figure 5D For image differences, please refer to [link / reference]. Figure 5E .

[0162] Figure 5E This is a schematic diagram showing the stereoscopic effect of a two-dimensional image presented from different perspectives in the embodiments of this application.

[0163] Figure 5E The image on the left is Figure 5C The image displayed in the image is the image shown by the electronic device at viewing angle 1 (for ease of description, it is referred to as image 1); Figure 5E The image on the right is Figure 5D The image displayed in the image is the image shown by the electronic device at viewpoint 2 (referred to as image 2 for ease of description). Figure 5E The image exemplifies regions 1 and 2 in the image coordinate system. Region 1 displays the background image content in Image 1, while Region 1 displays the foreground image content in Image 2. It can be seen that when the viewing angle changes, the area that originally displayed the background image is obscured by the foreground image; this can also be understood as the content in Region 1 changing from being "present" in the background to being "obscured" by the foreground. Similarly, Region 2 displays the foreground image content in Image 1, while Region 2 displays the background image content in Image 2. It can be seen that when the viewing angle changes, the background image that was originally obscured by the foreground image becomes visible; this can also be understood as the content in Region 2 changing from being "obscured" by the foreground to being "present" in the background. Furthermore, the size of the foreground and background content also changes. Compared to Image 1, the content on the left side of Image 2 is larger, while the content on the right side is smaller.

[0164] Depend on Figure 5E As can be seen, when the user's perspective changes, the background content in area 2 changes from being "obscured" by the foreground to "appearing," while the content in area 1 changes from "appearing" in the background to being "obscured" by the foreground figure.

[0165] It should be noted that, Figures 5B to 5DThe changes in the first image are only illustrated. In other embodiments of this application, the aforementioned "3D display" function can be "AR" display, that is, an image obtained by superimposing and rendering the stereoscopic image corresponding to the first image and the actual image captured by the camera on the display interface, such as... Figure 11B As shown.

[0166] It should be noted that, in the embodiments of this application, any of the user operations can be a user's touch operation (such as a click operation, a long press operation, an up swipe operation, a down swipe operation, or a side swipe operation), or a non-contact operation (such as an air gesture), or a user's voice command. The embodiments of this application do not impose specific limitations on this.

[0167] Not limited to Figures 5A to 5D The method shown for enabling the "3D display" function of an electronic device for a two-dimensional image (such as the first image) can also be used by the user in other ways, such as long-pressing the two-dimensional image to enable the "3D display" function. This application does not limit this method.

[0168] Next, we will introduce the image display method provided in this application.

[0169] Figure 6 This is a schematic flowchart of an image display method provided by an exemplary embodiment of this application.

[0170] This application uses an electronic device as an example for illustration, and the hardware and software architecture of the electronic device can be found in the above description. Figures 1A to 2 Related descriptions of electronic devices 100 or Figure 12 The description of the electronic device 200 is not limited in this application. The method may include some or all of the following steps:

[0171] S601: In response to a first user operation for instructing a stereoscopic display of a first image, the electronic device determines a foreground image corresponding to the first image and a background image corresponding to the first image.

[0172] In some embodiments, the electronic device may perform semantic segmentation on the first image to obtain a foreground semantic template and a background semantic template of the first image; then, based on the first image and the background semantic template of the first image, a background image is determined; and based on the first image and the foreground semantic template of the first image, a foreground image is determined. For example, this process can be referred to in the relevant content of step S701 below.

[0173] In one implementation, the electronic device may further respond to a first user operation by determining the position of the observation surface in space; based on the position of the observation surface in space, determining the positions of the foreground anchor surface and the background anchor surface in space, wherein the foreground anchor surface and the background anchor surface are located on different planes, and the positions of the foreground anchor surface and the background anchor surface in space are used to determine the second image and / or the third image described below. For example, this process can be seen in the relevant content of step S704 below.

[0174] Optionally, the plane containing the observation surface, the plane containing the foreground anchoring surface, and the plane containing the background anchoring surface are all parallel to each other; the distance between the plane containing the observation surface and the plane containing the foreground anchoring surface is the first distance, and the distance between the plane containing the foreground anchoring surface and the plane containing the background anchoring surface is the second distance, wherein the first distance is greater than or equal to 0, and the second distance is greater than 0. For ease of description, this application also represents the first distance as d_rec and the second distance as the depth difference d.

[0175] For example, the method for determining the second distance may be: the electronic device performs depth estimation on the first image to obtain a depth map; based on the depth map, the average depth of the foreground image (also known as the average foreground depth) and the average depth of the background image (also known as the average background depth) are determined; based on the average depth of the foreground image and the average depth of the background image, the second distance is determined.

[0176] S602: The electronic device displays a second image on a first region, wherein the second image includes a first foreground portion and a first background portion, the pose of the electronic device when displaying the second image is a first pose, the first foreground portion is obtained based on the foreground image and the first pose, and the first background portion is obtained based on the background image and the first pose.

[0177] In some embodiments, when the electronic device is in a first pose, the electronic device can determine a first position of the foreground anchor surface in the image coordinate system and a second position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in space, the position of the background anchor surface in space, and the first pose; then, based on the first and second positions, the foreground image and the background image are rendered to obtain a second image. For example, this process can be referred to in steps S705 to S707 below.

[0178] For example, the process of an electronic device rendering a second image may be as follows: the electronic device projects a foreground image to a first position to obtain a first image texture; projects a background image to a second position to obtain a second image texture; and then renders the first image texture and the second image texture to obtain a second image, wherein the rendering result of the first image texture is a first foreground portion, and the rendering result of the second image texture includes a first background portion.

[0179] S603: When the electronic device is in a second pose, the electronic device displays a third image in a first region, wherein the third image includes a second foreground portion and a second background portion, the second foreground portion is obtained based on the foreground image and the second pose, and the second background portion is obtained based on the background image and the second pose; wherein the first region includes the second region, and the content displayed in the second region is different when the pose of the electronic device is different.

[0180] The image displayed in the first area (such as the second and third images) can be a regular image without overlay rendering of the real-world image, or an AR image with overlay rendering of the real-world image. For example, Figures 5B to 5E The example shows that the image displayed in the first area is a normal image without overlaying or rendering the real image. Alternatively, the first area could also display something like... Figure 11B The AR image shown.

[0181] The second region is a portion of the first region. The second region can be rectangular or circular, and can have a regular or irregular shape; this application does not limit this. For example, the first region can be... Figure 5B The preview area 520 in the user interface 52 shown Figure 5C The preview area 530 in the user interface 53 shown Figure 5D The preview area 540 in the user interface 54 shown, Figure 5E An example is shown where the second region is a rectangle.

[0182] For example, the second region can be Figure 5E The content displayed in area 1 is different when the electronic device is in different positions (i.e., different viewing angles), and the content displayed in area 2 is different when the electronic device is in different positions (i.e., different viewing angles).

[0183] For example, the second area can also be displayed by an electronic device in the first area, such as... Figure 11B The AR image shown is a region within the first region.

[0184] In some embodiments, when the electronic device is in a second pose, the electronic device can determine a third position of the foreground anchor surface in the image coordinate system and a fourth position of the background anchor surface in the image coordinate system based on the spatial position of the foreground anchor surface, the spatial position of the background anchor surface, and the second pose; based on the third and fourth positions, the foreground image and the background image are rendered to obtain a third image. For example, this process can be referred to in steps S709 to S711 below.

[0185] Understandably, when the pose of the electronic device changes from a first pose to a second pose, the content displayed on the first area of ​​the electronic device is updated from the third image to the second image. Taking a normal image as an example, the second image can be... Figure 5B The preview area 520 in the user interface 52 shown displays an image, and the third image can be... Figure 5C The preview area 530 in the user interface 53 shown Figure 5D The preview area 540 in the user interface 54 shown; or, the second image can be... Figure 5C The preview area 530 in the user interface 53 shown can be the third image. Figure 5D The preview area 540 in the user interface 54 shown.

[0186] For example, the process of an electronic device rendering a third image can be as follows: the electronic device projects a foreground image to a third position to obtain a third image texture; projects a background image to a fourth position to obtain a fourth image texture; and renders the third image texture and the fourth image texture to obtain a third image, wherein the rendering result of the third image texture is the second foreground part, and the rendering result of the fourth image texture includes the second background part.

[0187] Wherein, when the electronic device is in the first pose, the content displayed in the second area belongs to the first background part, and when the electronic device is in the second pose, the content displayed in the second area belongs to the second foreground part; or, when the electronic device is in the first pose, the content displayed in the second area belongs to the first foreground part, and when the electronic device displays the third image, the content displayed in the second area belongs to the second background part.

[0188] Optionally, the second image also includes a first real portion, which is included in the image captured by the electronic device in the first pose; and / or, the third image also includes a second real portion, which is included in the image captured by the electronic device in the second pose.

[0189] above Figure 6 The method embodiments shown include many possible implementation schemes, which will be discussed below. Figure 7A Some of the implementation schemes will be illustrated with examples. It should be noted that... Figure 7A For any unexplained related concepts, operations, or logical relationships, please refer to [link / reference]. Figure 6 The corresponding description in the illustrated embodiment.

[0190] In this application, Figure 7A The illustrated embodiment can be considered as a standalone embodiment. Figure 6 The embodiments shown can all be implemented without relying on Figure 6 The technical solution; Figure 7ASome of the steps in the illustrated embodiments can also be used as separate embodiments.

[0191] Figure 7A This is a flowchart illustrating another image display method provided in an embodiment of this application.

[0192] In this embodiment, the second and third images are used as examples of AR images for illustration. The interface related to the process of the electronic device displaying the AR image can be exemplarily referred to [reference needed]. Figures 5A to 5E ,Will Figures 5A to 5E Simply change the content of the screen in the image to an AR image.

[0193] like Figure 7A As shown, the method includes the following steps:

[0194] S701: In response to a first user operation instructing the first image to be displayed in stereoscopic mode, the electronic device determines a foreground image corresponding to the first image and a background image corresponding to the first image.

[0195] The first image is a two-dimensional image.

[0196] In some embodiments, the electronic device includes a first application (as shown in the gallery), and the electronic device can display the application interface of the first application, the application interface including the first image mentioned above; furthermore, when the electronic device detects a user operation for instructing the first image to be displayed in stereoscopic form, it determines a foreground image corresponding to the first image and a background image corresponding to the first image.

[0197] Optionally, the first image can be a photo arbitrarily selected by the user from a gallery, or a photo taken by the user using an electronic device. The first image can be an RGB image captured by the electronic device using a camera, or a color electronic painting, electronic poster, etc., created through rendering, compositing, Photoshop, electronic painting, etc. If the first image is a photo taken by a binocular device (having a left view and a right view), the electronic device can select either view as the first image. It should be understood that in this application, the electronic device is not a binocular device. After the user selects a photo, the system enters the offline calculation module. The offline module performs only one calculation during the entire method execution, and the calculation result is used for subsequent processing in the online module.

[0198] For example, step S701 described above may include Figure 7B The following steps are shown:

[0199] S7011: The electronic device performs semantic segmentation on the first image to obtain the foreground semantic template and the background semantic template of the first image.

[0200] For example, electronic devices can use neural network-based methods for semantic segmentation (also known as foreground-background semantic segmentation), such as a U-Net network structure. It should be noted that there can be various semantic segmentation methods, including but not limited to traditional methods and neural network-based methods, and this application does not limit them.

[0201] Figure 8A This is a schematic diagram illustrating semantic segmentation provided by an embodiment of this application. For example... Figure 8A As shown, the electronic device can perform semantic segmentation on the first image to obtain a foreground semantic template (which may be called the semantic template of the foreground part or mask_f or foreground mask) and a background semantic template of the first image (which may be called the semantic template of the background part or mask_b or background mask). Figure 8A For example, black represents pixel 0 and white represents pixel 1. The background area pixels of the foreground semantic template are black (i.e., pixel 0) and the foreground area pixels are white (i.e., pixel 1); the background area pixels of the background semantic template are white (i.e., pixel 1) and the foreground area pixels are black (i.e., pixel 0).

[0202] S7012: The electronic device determines the background image corresponding to the first image based on the first image and the background semantic template of the first image.

[0203] Figure 8B This is a schematic diagram illustrating an embodiment of this application for acquiring a background image. Figure 8B As shown, the electronic device can perform a dot product operation on the first image and its background semantic template to obtain background image 1; then, it can fill in the missing pixel areas (i.e., the foreground corresponding positions) in background image 1 to obtain the background image corresponding to the first image. For example, the pixel filling method can be image inpainting or a Unet-based network structure or other background completion algorithm; this application does not limit this. It should be noted that objects originally occluded by the foreground in the background image corresponding to the first image will be filled in. For example... Figure 8B In the background image corresponding to the first image shown, the tree that was originally obscured by the foreground kitten has been fully added.

[0204] S7013: The electronic device determines the foreground image corresponding to the first image based on the first image and the foreground semantic template of the first image.

[0205] For example, the foreground image img_f can be calculated using the following formula: img_f = img * mask_f, where img is the first image (or the original image), mask_f is the foreground semantic template, and * is the dot product operation.

[0206] Figure 8CThis is a schematic diagram illustrating an embodiment of this application for acquiring a foreground image. For example... Figure 8C As shown, the electronic device can perform a dot product operation on the first image and the background semantic template of the first image to obtain the foreground image corresponding to the first image.

[0207] Optionally, the above steps S7011 to S7013 can be performed by... Figure 2 The offline computing module, as shown, calculates the foreground and background images. After obtaining the foreground and background images, the offline computing module can send them to the online computing module.

[0208] Optionally, the offline calculation module can also calculate the foreground-background depth difference (hereinafter referred to as depth difference) based on the foreground and background images. The data sent to the online calculation module may include the depth difference in addition to the foreground and background images. For example, the offline calculation module can also perform depth estimation on the first image to obtain a depth map; then, based on the depth map, calculate the average background depth depth_b = avg(depth * mask_b) and the average foreground depth depth_f = avg(depth * mask_f); finally, calculate the absolute value of the difference between the two to obtain the depth difference d = |depth_b – depth_f|. It should be noted that the above method for calculating the depth difference is merely an example, and this application does not limit the method for calculating the depth difference.

[0209] S702: The electronic device, in response to the first user operation described above, captures an image using a camera.

[0210] In some embodiments, the electronic device can call the camera interface, open the camera, acquire image data (such as RGB images) captured by the camera, obtain a camera preview stream, and display the content of the camera preview stream on the display screen of the electronic device.

[0211] S703: Electronic equipment determines the position of the observation surface in the world coordinate system.

[0212] In some embodiments, the electronic device constructs a world coordinate system based on camera data; and determines an observation surface in the world coordinate system based on preset rules, with the position of the observation surface in the world coordinate system remaining fixed. For example, the camera data includes data from a gravity sensor and an acceleration sensor, which can be used to make the vertical axis of the world coordinate system perpendicular to the ground; the preset rules can use the center of the camera's field of view (i.e., the center of the image captured by the camera) as the center of the observation surface, and the size of the observation surface can be a preset size, such as 1880mm long, 1040mm wide, and 2000mm deep (equivalent to an 85-inch television).

[0213] It should be understood that the world coordinate system is used to describe the environment around the camera; once the world coordinate system is initialized, it is fixed and does not change with the movement of electronic devices (i.e., cameras) or people.

[0214] For example, the position of the observation surface in the world coordinate system can be the rectangular area formed by the coordinates of the upper left corner and the lower right corner of the observation surface. That is to say, the position of the observation surface in the world coordinate system can be represented by the coordinates of the upper left corner and the lower right corner of the observation surface.

[0215] Figure 9A This is a schematic diagram of an embodiment of the present application providing an initial observation surface. Figure 9A The coordinate system in this example is the world coordinate system. The position of the observation surface within this world coordinate system is shown as an example. The initial position of the observation surface is POS_rec = [(X1_rec,Y1_rec,Z1_rec),(X2_rec,Y2_rec,Z2_rec)], where (X1_rec,Y1_rec,Z1_rec) are the coordinates of the upper left corner of the observation surface, and (X2_rec,Y2_rec,Z2_rec) are the coordinates of the lower right corner of the observation surface. For example, the dimensions of the observation surface can be 1880mm in length, 1040mm in width, and 2000mm in depth. Then, the above coordinates have the following constraints: X1_rec - X2_rec = 1040, Y2_rec - Y1_rec = 1880, Z1_rec = Z2_rec = 2000.

[0216] S704: The electronic equipment determines the positions of the foreground anchoring surface and the background anchoring surface in the world coordinate system based on the position of the observation surface in the world coordinate system.

[0217] In some embodiments, the electronic device determines the positions of the foreground anchor plane and the background anchor plane in the world coordinate system based on the observation plane in the world coordinate system and a first correspondence. The first correspondence includes the correspondence between the observation plane and the foreground anchor plane and the correspondence between the observation plane and the background anchor plane. For example, the correspondence between the observation plane and the foreground anchor plane is as follows: the plane containing the observation plane is parallel to the plane containing the foreground anchor plane, and the distance between the two planes is d_rec; the plane containing the observation plane is parallel to the plane containing the background anchor plane, and the distance between the two planes is the depth difference d.

[0218] Figure 9B This is a schematic diagram of an embodiment of the present application providing an initialization of a foreground anchoring surface and a background anchoring surface. Figure 9BThe coordinate system in the image is the world coordinate system. The initial position of the foreground anchoring surface is POS_f = [(X1_f,Y1_f,Z1_f),(X2_f,Y2_f,Z2_f)], where X1_f = X1_b, X2_f = X2_b, Y1_f = Y1_b, Y2_f = Y2_b, Z1_f = Z1_rec - d_rec, Z2_f = Z1_f; the initial position of the background anchoring surface is POS_b = [(X1_b,Y1_f,Z1_f),Z2_f = Z2_b, Z1_f = Z1_rec - d_rec, Z2_f = Z1_f; _b,Z1_b),(X2_b,Y2_b,Z2_b)], where (X1_b,Y1_b,Z1_b) are the coordinates of the upper left corner of the anchoring surface in the world coordinate system, (X2_b,Y2_b,Z2_b) are the coordinates of the lower right corner, and X1_b=X1_rec,Y1_b=Y1_rec,X2_b=X2_rec,Y2_b=Y2_rec,Z1_b=Z1_rec-d_rec-d,Z2_b=Z1_b.

[0219] For example, two methods for obtaining the depth difference d are described below. Method 1: For electronic devices with limited computing power, an empirical value can be preset as the depth difference d, such as 500mm. Method 2: For electronic devices with greater computing power, an algorithm can be used to estimate the depth of the original image (such as the first image mentioned above). Then, combining the foreground and background regions, the average depths df and db of the foreground and background regions can be obtained. The depth difference d can then be calculated using d = |df - db|. Method 2 has higher accuracy, but it involves more computation. In practice, it is necessary to flexibly switch between the two methods depending on the scenario.

[0220] For example, the depth difference d can be set to 30cm by default. It should be understood that when the depth difference increases, the distance between the foreground and background anchor surfaces increases; conversely, when the depth difference decreases, the distance between the foreground and background anchor surfaces decreases. Different distance differences will affect the final image content. For example, the effects of different depth differences are as follows: Figure 10 As shown, Figure 10 The perspective can be Figure 4 As shown in viewpoint 2, the larger the depth difference d, the smaller the foreground image and the greater the distance between the foreground and background images. It should be understood that in other viewpoints, a larger depth difference d may also result in a larger foreground image (e.g., ...). Figure 4 The effect of depth difference on the presentation of the image content (as shown in the initial viewpoint) is related to the viewpoint.

[0221] S705: The electronic device determines the first position of the foreground anchor surface in the image coordinate system and the second position of the background anchor surface in the image coordinate system based on the position of the foreground anchor surface in the world coordinate system, the position of the background anchor surface in the world coordinate system, and the current first pose of the electronic device.

[0222] In some embodiments, the current pose of the electronic device is a first pose. The electronic device can determine the positional relationship between the foreground anchor surface and the background anchor surface relative to the electronic device based on the first pose, the position of the foreground anchor surface in the world coordinate system, and the position of the background anchor surface in the world coordinate system. Based on this positional relationship, the first position of the foreground anchor surface in the image coordinate system and the second position of the background anchor surface in the image coordinate system are determined.

[0223] S706: The electronic device projects a foreground image to a first position to obtain a first image texture of the foreground image, and projects a background image to a second position to obtain a second image texture corresponding to the background image.

[0224] For example, the electronic device projects pixels of the foreground image to a first position based on the position of the observation surface in the image coordinate system to generate a first image texture of the foreground image, and projects pixels of the background image to a second position to generate a second image texture corresponding to the background image. The observation surface is used to determine the projection areas on the foreground anchoring surface and the background anchoring surface, such as... Figure 4 As shown, the anchoring surface within the dashed line range forming the initial viewpoint is the projection area. Understandably, the observation surface affects the projection results of the foreground and background images (i.e., the first image texture and the second image texture), and this application does not limit the projection method.

[0225] S707: The electronic device generates a second image based on the first image texture, the second image texture, and the image captured by the current camera.

[0226] In some embodiments, the electronic device can render a first image texture and a second image texture onto a foreground anchor surface and a background anchor surface, respectively, to obtain a first rendering result; and overlay the first rendering result with an image captured by the current camera (such as a first real image) to obtain a second image.

[0227] S708: The electronic device displays a second image in the first area.

[0228] For example, the interface currently displayed on the electronic device is the application interface of the gallery application, and the first area can be the preview area in the application interface.

[0229] For example, the second image may include a first foreground portion (such as a cat), a first background portion (such as a tree or a rock), and a first reality portion (such as a home scene), wherein the first foreground portion is the aforementioned first image texture; the first background portion is a portion of the aforementioned second image texture, wherein any part of the second image texture not displayed is occluded by the first image texture; and the first reality portion is a portion of the aforementioned first reality image, wherein any part of the first reality image not displayed is occluded by the first image texture and the second image texture.

[0230] S709: When the current pose of the electronic device is the second pose, based on the position of the foreground anchor surface in the world coordinate system, the position of the background anchor surface in the world coordinate system, and the second pose of the electronic device, determine the third position of the foreground anchor surface in the image coordinate system and the fourth position of the background anchor surface in the image coordinate system.

[0231] In some embodiments, the electronic device can update the coordinates of the observation plane, foreground anchor plane, and background anchor plane projected onto the image coordinate system using camera extrinsic parameters. The projection process is perspective projection. For example, the electronic device can acquire IMU data and calculate its real-time pose using the SLAM method. Based on the pose change of the electronic device, the relative motion of the camera coordinates, i.e., the camera extrinsic parameters, can be obtained. The camera extrinsic parameters may include a rotation matrix R and a translation matrix T, where R is a 3x3 matrix and T is a 3x1 matrix. Based on R and T, the coordinates of the observation plane pos_rec, the coordinates of the background anchor plane pos_b, and the coordinates of the foreground anchor plane pos_f in the image coordinate system are determined.

[0232] Taking point A as an example, this section describes how to update the position of point A in the image coordinate system when the pose of an electronic device changes. This update process can include: converting the coordinates of point A in the world coordinate system to coordinate 1 in the camera coordinate system; when the camera coordinate system moves, the coordinates of point A in the camera coordinate system change, and based on the camera extrinsic parameters, updating the coordinates of point A in the camera coordinate system to coordinate 2; and based on the camera intrinsic parameters, projecting coordinate 2 onto the image coordinate system to obtain the position of point A in the image coordinate system.

[0233] For example, for any point on the three anchoring surfaces—the observation surface, the foreground anchoring surface, and the background anchoring surface—its coordinates in the world coordinate system are (Xw, Yw, Zw), and its coordinates in the image coordinate system after projection are (u, v). The update process can be shown in the following formula:

[0234]

[0235] Where f, (u0, v0), d x and d y Here, f is the camera intrinsic parameter, (u0, v0) is the camera focal length, and d is the coordinate of the image center point. x and d y These are the pixel dimensions in the horizontal and vertical directions, respectively; Z c This is the normalization coefficient.

[0236] Figure 11A This is a schematic diagram of an updated foreground anchoring surface and background anchoring surface provided in an embodiment of this application. Figure 11AThe example illustrates a home scene where, assuming the user's initial position is in front of the sofa, the background and foreground anchor surfaces are positioned at the center of the image coordinate system. When the user moves their electronic device to the side of the sofa, the device updates the positions of the background and foreground anchor surfaces in the image coordinate system, placing them on the left side of the image coordinate system.

[0237] S710: The electronic device projects a foreground image to a third position to obtain a third image texture of the foreground image, and projects a background image to a fourth position to obtain a fourth image texture corresponding to the background image.

[0238] For example, an electronic device can project pixels of a foreground image to a third position to generate a third image texture of the foreground image based on the current position of the observation surface in the image coordinate system, and project pixels of a background image to a fourth position to generate a fourth image texture corresponding to the background image.

[0239] S711: The electronic device generates a third image based on the third image texture, the fourth image texture, and the image captured by the current camera.

[0240] In some embodiments, the electronic device can render the third image texture and the fourth image texture onto the foreground anchor surface and the background anchor surface, respectively, to obtain a second rendering result; and overlay the second rendering result with an image captured by the current camera (such as a second real image) to obtain a third image.

[0241] Figure 11B This is a schematic diagram of a third image provided in an embodiment of this application. Figure 11B The third image shown can be the result of projecting the aforementioned third image texture and fourth image texture onto the aforementioned image. Figure 11A The image generated after the foreground anchor plane and background anchor plane are shown. (Example) Figure 11B As shown, the third image includes a second foreground portion (i.e., a cat), a second background portion (i.e., trees and rocks), and a second reality portion (i.e., a home scene). The second foreground portion is the texture of the aforementioned third image; the second background portion is a portion of the aforementioned fourth image texture, with any undisplayed elements in the fourth image texture occluded by the third image texture; the second reality portion is a portion of the aforementioned second reality image, with any undisplayed elements in the second reality image occluded by both the third and fourth image textures. It should be noted that because the pose of the electronic device changes when acquiring the first and second reality images, the first and second reality images are different, and the content of the first and second reality portions is also different.

[0242] S712: The electronic device displays a third image in the first area.

[0243] In some embodiments, when the pose of the electronic device changes from a first pose to a second pose, the electronic device can update the second image displayed in the first area to a third image.

[0244] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of the electronic device 200 provided in the embodiments of this application.

[0245] like Figure 12 As shown, the electronic device 200 provided in this application embodiment can be used to implement the methods described in the above method embodiments, such as implementing the above... Figure 6 or Figure 7A The structure of the electronic device in the method embodiment shown can be as follows: Figure 12 As shown, specific implementation details can be found in the description of the above method embodiments. The electronic device 200 can be a terminal device (such as a mobile phone), or a chip therein. For example, the electronic device 200 includes one or more processors 801 and transceivers 802. The electronic device 200 may further include a memory 803. In one implementation, the above electronic device 200 also includes input / output devices, such as a display screen and a keyboard, etc. Figure 12 (Not indicated).

[0246] The processor 801 controls the entire electronic device 200, executes software programs, and processes the data from those programs. The memory 803 primarily stores software programs and data. The transceiver 802 may include control circuitry and an antenna. The control circuitry is mainly used for converting baseband signals to radio frequency signals and processing radio frequency signals. The antenna is mainly used for transmitting and receiving radio frequency signals in the form of electromagnetic waves. Input / output devices, such as touchscreens, displays, and keyboards, are mainly used for receiving user input data and outputting data to the user.

[0247] When the electronic device 200 is powered on, the processor 801 can read the software program in the memory 803, interpret and execute the instructions of the software program, and process the data of the software program. The processor 801, transceiver 802, and memory 803 can be connected via a communication bus.

[0248] In one implementation, processor 801 may store instructions, which may be a computer program. This computer program, running on processor 801, causes electronic device 200 to perform the methods described in the above method embodiments. The computer program may be embedded in processor 801; in this case, processor 801 may be implemented in hardware.

[0249] In one implementation, the electronic device 200 may include circuitry that performs the functions of transmitting, receiving, or communicating as described in the aforementioned method embodiments. The processor and transceiver described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits (RFICs), mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices 200, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductors (CMOS), n-metal-oxide-semiconductor (NMOS), positive channel metal oxide semiconductors (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0250] It is understood that the electronic device 200 shown in the embodiments of this application may also have more than Figure 12 This application does not limit the use of other components or other special devices. For details on the specific steps performed by the processor, please refer to the description of the method embodiments above.

[0251] In this embodiment, the memory 803 stores executable instructions, and the processor 801 executes these executable instructions to implement the aforementioned image display method, for example... Figure 6 The image display method in the embodiment shown in Figure 7. That is, the memory 803 stores instructions for executing the image display method.

[0252] This application also provides a computer program for implementing the operations and / or processes performed by an electronic device in the method provided in this application.

[0253] This application also provides a computer-readable storage medium storing computer code that, when executed on a computer, causes the computer to perform the operations and / or processes performed by an electronic device in the method provided in this application.

[0254] This application also provides a computer program product comprising computer code or a computer program that, when run on a computer, causes the operations and / or processes performed by an electronic device in the method provided in this application to be executed.

[0255] This application also provides a chip, including a processor and a communication interface. The communication interface is used for outputting and / or outputting data (including instructions), and / or for receiving and / or sending data. When the processor executes program instructions in memory, the aforementioned image display method, for example... Figure 6 The image display method in the embodiment shown in Figure 7.

[0256] In the embodiments provided in this application, it should be understood that the disclosed systems, electronic devices, and methods can be implemented in other ways. For example, the embodiments of the electronic devices described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, electronic devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0257] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected based on actual needs to achieve the technical effects of the solutions provided in the embodiments of this application.

[0258] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0259] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0260] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image display method, characterized in that, Applied to electronic devices, the method includes: In response to a first user operation for instructing a stereoscopic display of a first image, a foreground image corresponding to the first image and a background image corresponding to the first image are determined; A second image is displayed in a first region, wherein the second image includes a first foreground portion and a first background portion, the pose of the electronic device when displaying the second image is a first pose, the first foreground portion is obtained based on the foreground image and the first pose, and the first background portion is obtained based on the background image and the first pose; When the pose of the electronic device is a second pose, a third image is displayed on the first region, wherein the third image includes a second foreground portion and a second background portion, the second foreground portion being obtained based on the foreground image and the second pose, and the second background portion being obtained based on the background image and the second pose; The first region includes a second region, and the content displayed in the second region varies depending on the position of the electronic device.

2. The method according to claim 1, characterized in that, When the electronic device is in the first pose, the content displayed in the second area belongs to the first background part; when the electronic device is in the second pose, the content displayed in the second area belongs to the second foreground part. or, When the electronic device is in the first pose, the content displayed in the second area belongs to the first foreground portion; when the electronic device displays the third image, the content displayed in the second area belongs to the second background portion.

3. The method according to claim 1 or 2, characterized in that, The second image also includes a first real portion, which is the image captured by the electronic device in the first pose. And / or, The third image also includes a second reality portion, which is the image captured by the electronic device in the second pose.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: In response to the first user's operation, determine the position of the observation surface in space; Based on the position of the observation surface in space, the positions of the foreground anchor surface and the background anchor surface in space are determined, wherein the foreground anchor surface and the background anchor surface are located on different planes, and the positions of the foreground anchor surface and the background anchor surface in space are used to determine the second image and / or the third image.

5. The method according to claim 4, characterized in that, The plane containing the observation surface, the plane containing the foreground anchoring surface, and the plane containing the background anchoring surface are all parallel to each other; the distance between the plane containing the observation surface and the plane containing the foreground anchoring surface is a first distance, and the distance between the plane containing the foreground anchoring surface and the plane containing the background anchoring surface is a second distance, wherein the first distance is greater than or equal to 0, and the second distance is greater than 0.

6. The method according to claim 4 or 5, characterized in that, The method further includes: When the electronic device is in the first pose, based on the position of the foreground anchor surface in space, the position of the background anchor surface in space, and the first pose, the first position of the foreground anchor surface in the image coordinate system and the second position of the background anchor surface in the image coordinate system are determined. Based on the first position and the second position, the foreground image and the background image are rendered to obtain the second image.

7. The method according to claim 6, characterized in that, The step of rendering the foreground image and the background image based on the first position and the second position to obtain the second image includes: The foreground image is projected onto the first position to obtain the first image texture; The background image is projected onto the second position to obtain the second image texture; The first image texture and the second image texture are rendered to obtain the second image. The rendering result of the first image texture is the first foreground part, and the rendering result of the second image texture includes the first background part.

8. The method according to any one of claims 4-7, characterized in that, The method further includes: When the electronic device is in the second pose, based on the position of the foreground anchor surface in space, the position of the background anchor surface in space, and the second pose, the third position of the foreground anchor surface in the image coordinate system and the fourth position of the background anchor surface in the image coordinate system are determined. Based on the third position and the fourth position, the foreground image and the background image are rendered to obtain the third image.

9. The method according to claim 8, characterized in that, The process of rendering the foreground image and the background image based on the third position and the fourth position to obtain the third image includes: The foreground image is projected onto the third position to obtain the third image texture; The background image is projected onto the fourth position to obtain the fourth image texture; The third image texture and the fourth image texture are rendered to obtain the third image. The rendering result of the third image texture is the second foreground part, and the rendering result of the fourth image texture includes the second background part.

10. The method according to any one of claims 5-8, characterized in that, The method further includes: Depth estimation is performed on the first image to obtain a depth map; Based on the depth map, the average depth of the foreground image and the average depth of the background image are determined; The second distance is determined based on the average depth of the foreground image and the average depth of the background image.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Semantic segmentation is performed on the first image to obtain the foreground semantic template and the background semantic template of the first image; The background image is determined based on the first image and its background semantic template; The foreground image is determined based on the first image and the foreground semantic template of the first image.

12. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-11.

13. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-11.

15. A chip, characterized in that, The device includes a processor and a communication interface, the communication interface being used to output and / or output data, and / or to receive and / or send data, wherein the method as described in any one of claims 1-11 is implemented when the processor executes program instructions in memory.

Images