Image display method and electronic device

Abstract

The present application provides an image display method and electronic device, which relate to the field of terminal technology. In the image display method, the difference between the depth of the adjusted pixel points of the target object in the N-th frame image and the depth of the corresponding pixel points on the target object in the N-1-th frame image is less than or equal to a first threshold. In this way, the difference in depth between the pixel points on the target object of the N-th frame image and the N-1-th frame image is small. In addition, since there is occlusion between the virtual object and the target object in the N-1-th frame; and there is occlusion between the virtual object and the target object in the N-th frame image; when the difference in depth between the pixel points on the target object of the N-th frame image and the N-1-th frame image is small, the occlusion relationship between the virtual object and the target object of the N-1-th frame image will not jump relative to the occlusion relationship between the virtual object and the target object of the N-th frame.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image display method and electronic device. Background Art

[0002] Current augmented reality (AR) technology can superimpose virtual objects on real images captured by electronic devices, making the virtual objects and real images complement each other, thereby achieving "enhancement" of the real images displayed by electronic devices.

[0003] Typically, when a virtual object is superimposed on each captured real-world image, an AR image is generated. In an AR image, the virtual object may occlude the target object in the real-world image. Understandably, when the depth of the real-world image is greater than the depth of the virtual object, the virtual object occludes the target object; when the depth of the real-world image is less than the depth of the virtual object, the target object occludes the virtual object.

[0004] However, when displaying AR images based on virtual objects and continuously captured multiple frames of images, the occlusion relationship between the virtual object and the target object is prone to sudden changes (i.e., the occlusion relationship between the target object and the virtual object in the next frame of image is distorted relative to the change in the occlusion relationship between the target object and the virtual object in the previous frame of image). This, in turn, leads to poor AR image quality. Summary of the Invention

[0005] The present application provides an image display method and an electronic device to improve the above-mentioned problem of poor quality of AR images.

[0006] In a first aspect, the present application provides an image display method, comprising: an electronic device captures an N-th frame image. Wherein, N is an integer greater than 1. The electronic device determines the depth of each pixel point of the target object in the N-th frame image, and the target object is an object in the N-th frame image that is occluded by a preset virtual object. When the difference between the depth of any pixel point of the target object in the N-th frame image and the depth of the corresponding pixel point on the target object in the N-1-th frame image is greater than a first threshold, the electronic device adjusts the depth of any pixel point so that the difference is less than or equal to the first threshold. When the electronic device displays the virtual object and the adjusted target object, the object with lower depth in the virtual object and the adjusted target object is displayed to occlude the object with higher depth.

[0007] The image display method provided by the present application is that the difference between the depth of the adjusted pixel points of the target object in the N-th frame image and the depth of the corresponding pixel points on the target object in the N-1-th frame image is less than or equal to a first threshold. In this way, the difference in depth between the pixel points on the target object in the N-th frame image and the N-1-th frame image can be made smaller. In addition, since the virtual object is occluded with the target object in the N-1-th frame; and the virtual object is also occluded with the target object in the N-th frame image; when the depth of the virtual object remains unchanged and the difference in depth between the pixel points on the target object in the N-th frame image and the N-1-th frame image is small, the occlusion relationship between the virtual object and the target object in the N-1-th frame image will not jump relative to the occlusion relationship between the virtual object and the target object in the N-1-th frame. In this way, the quality of the displayed image is improved.

[0008] In one possible implementation, the electronic device adjusts the depth of any pixel so that a difference is less than or equal to a first threshold, including: the electronic device adjusts the depth of any pixel based on the depths of pixels of the target object in M ​​frames preceding the Nth frame so that the difference is less than or equal to the first threshold, where M is an integer greater than or equal to 1.

[0009] Since the reference basis for adjusting the depth of any pixel is the depth of the pixel of the target object in the M frames before the Nth frame, the reliability of the depth of any pixel after adjustment can be made higher.

[0010] Furthermore, when M=1, the preceding M frames of the Nth frame image are the N-1th frame image, and the depth of any pixel of the target object in the Nth frame image before adjustment and the depth of any pixel of the target object in the Nth frame image after adjustment satisfy the condition: D1=D0-(Da-Dt). Wherein, D0 is the depth of any pixel of the target object in the Nth frame image before adjustment, D1 is the depth of any pixel of the target object in the Nth frame image after adjustment, and Da is the difference between the depth of any pixel of the target object in the Nth frame image and the depth of the corresponding pixel of the target object in the N-1th frame image.

[0011] Since the subsequent focus is to avoid a sudden change in the occlusion relationship between the target object and the virtual object in frame N compared to the occlusion relationship between the target object and the virtual object in frame N-1, the depth of any pixel is adjusted based on the depth of the pixel of the target object in frame N-1. This ensures a higher degree of reliability in the adjusted depth of any pixel.

[0012] Alternatively, further, when M=2, the first M frames of the Nth frame image include the N-1th frame image and the N-2th frame image, and the depth of any pixel of the target object in the Nth frame image before adjustment and the depth of any pixel of the target object in the Nth frame image after adjustment satisfy the condition: D1=D0+{D0-{[d1+(d1-d2)]*(1-c)+D0*c}}. Wherein, D0 is the depth of any pixel of the target object in the Nth frame image before adjustment, D1 is the depth of any pixel of the target object in the Nth frame image after adjustment, d1 is the depth of the pixel of the target object in the N-1th frame image, d2 is the depth of the pixel of the target object in the N-2th frame image, and c is the Kalman gain obtained by performing Kalman filtering on the Nth frame image.

[0013] Since the subsequent focus is to avoid a sudden change in the occlusion relationship between the target object and the virtual object in frame N compared to the occlusion relationship between the target object and the virtual object in frame N-1, the depth of any pixel is adjusted based on the depth of the target object pixel in both the N-1 and N-2 frames. This comprehensive reference ensures greater reliability in the depth of any pixel after adjustment.

[0014] Furthermore, the electronic device adjusts the depth of any pixel point so that the difference is less than or equal to the first threshold value, including: the electronic device calculates the estimated depth value of each pixel point of the target object in the Nth frame image based on the formula De=d1+(d1-d2). Wherein, De is the estimated depth value. The electronic device determines the actual depth value of any pixel point of the target object in the Nth frame image based on the formula Dr=De*(1-c)+D0*c, where Dr is the actual depth value. The electronic device calculates the difference Df between the depth of any pixel point of the target object in the Nth frame image and the actual depth value. The electronic device adjusts the depth of each pixel point of the target object in the Nth frame image according to the formula D1=D0+Df.

[0015] In one possible embodiment, before the electronic device adjusts the depth of any pixel point so that the difference is less than or equal to a first threshold, the method provided in the present application also includes: the electronic device updates the depth of each pixel point in the target object in the Nth frame image, so that the difference in the depth of each pixel point within a preset area range on the target object is less than a preset second threshold.

[0016] In this way, when the electronic device displays the target object and the virtual object in the Nth frame image, the occlusion relationship between the target object and the virtual object will not be distorted.

[0017] Furthermore, the electronic device updates the depth of each pixel in the target object in the Nth frame image so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold, including: the electronic device uses the average value of the depth of each pixel of the target object in the Nth frame image as the depth of each pixel in the target object in the Nth frame image. Alternatively, the electronic device uses the maximum value of the depth of each pixel of the target object in the Nth frame image as the depth of each pixel in the target object in the Nth frame image. Alternatively, the electronic device uses the median value of the depth of each pixel of the target object in the Nth frame image as the depth of each pixel in the target object in the Nth frame image. Alternatively, the electronic device uses the minimum value of the depth of each pixel of the target object in the Nth frame image as the depth of each pixel in the target object in the Nth frame image.

[0018] Alternatively, further, the electronic device updates the depth of each pixel in the target object in the Nth frame image so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold, including: the electronic device determines a target plane having the smallest sum of distances from each pixel of the target object in the Nth frame image. The electronic device projects each pixel of the target object in the Nth frame image onto the target plane. The electronic device updates the depth of each pixel of the target object in the Nth frame image to the depth of each pixel of the target object in the Nth frame image projected onto the target plane.

[0019] In one possible implementation, before the electronic device captures the Nth image frame, the image display method provided in this application further includes: the electronic device displaying a shooting preview interface, the shooting preview interface including a first control. In response to a triggering operation on the first control, the electronic device displays an icon of a virtual object. In response to a triggering operation on the virtual object icon, the electronic device displays the virtual object in the shooting preview interface.

[0020] In one possible implementation, before the electronic device captures the Nth image frame, the image display method provided in the present application further includes: the electronic device determining a first threshold based on a preset speed threshold and a preset frame rate of the captured image; wherein the speed threshold, the frame rate, and the first threshold satisfy the following conditions: Dt is the first threshold, v is the speed threshold, and f is the frame rate.

[0021] In this way, the reliability of determining the first threshold can be increased.

[0022] In one possible embodiment, when an electronic device displays a virtual object and an adjusted target object, an object with a lower depth in the virtual object and the adjusted target object is displayed to block an object with a higher depth, including: the electronic device determines whether the depth of each pixel of the target object after adjustment in the N-th frame image is less than the depth of the corresponding pixel in the virtual object. If the depth of each pixel of the target object after adjustment in the N-th frame image is lower than the depth of the pixel corresponding to the virtual object, the electronic device will block the display of the virtual object with the target object in the N-th frame image. If the depth of the pixel of the target object after adjustment in the N-th frame image is higher than the depth of the pixel corresponding to the virtual object, the electronic device will block the display of the target object in the N-th frame image.

[0023] In one possible implementation, the electronic device determines whether the adjusted pixel depth of the target object in the Nth frame image is less than the depths of each pixel of the virtual object, including: the electronic device determines whether the adjusted pixel depths of the target object in n consecutive frames of images including the Nth frame image are less than the depths of corresponding pixels of the virtual object, where n is an integer greater than or equal to 2.

[0024] In this way, the occlusion relationship can be further avoided from jumping.

[0025] In the second aspect, the present application also provides an image display device, comprising: an image acquisition unit, for acquiring the Nth frame image. Wherein, N is an integer greater than 1. A processing unit, for determining the depth of each pixel point of the target object in the Nth frame image, where the target object is an object in the Nth frame image that is occluded by a preset virtual object. The processing unit is also used to adjust the depth of any pixel point of the target object in the Nth frame image so that the difference between the depth of the corresponding pixel point in the N-1th frame image is greater than a first threshold, so that the difference is less than or equal to the first threshold. A display unit, for displaying the virtual object and the adjusted target object, so that the object with lower depth in the virtual object and the adjusted target object occludes the object with higher depth.

[0026] In one possible implementation, the processing unit is specifically configured to adjust the depth of any pixel of the target object in the M frames preceding the Nth frame so that the difference is less than or equal to a first threshold, where M is an integer greater than or equal to 1.

[0027] Furthermore, when M=1, the preceding M frames of the Nth frame image are the N-1th frame image, and the depth of any pixel of the target object in the Nth frame image before adjustment and the depth of any pixel of the target object in the Nth frame image after adjustment satisfy the condition: D1=D0-(Da-Dt). Wherein, D0 is the depth of any pixel of the target object in the Nth frame image before adjustment, D1 is the depth of any pixel of the target object in the Nth frame image after adjustment, and Da is the difference between the depth of any pixel of the target object in the Nth frame image and the depth of the corresponding pixel of the target object in the N-1th frame image.

[0028] Alternatively, further, when M=2, the first M frames of the Nth frame image include the N-1th frame image and the N-2th frame image, and the depth of any pixel of the target object in the Nth frame image before adjustment and the depth of any pixel of the target object in the Nth frame image after adjustment satisfy the condition: D1=D0+{D0-{[d1+(d1-d2)]*(1-c)+D0*c}}. Wherein, D0 is the depth of any pixel of the target object in the Nth frame image before adjustment, D1 is the depth of any pixel of the target object in the Nth frame image after adjustment, d1 is the depth of the pixel of the target object in the N-1th frame image, d2 is the depth of the pixel of the target object in the N-2th frame image, and c is the Kalman gain obtained by performing Kalman filtering on the Nth frame image.

[0029] Furthermore, the processing unit is specifically used to calculate the estimated depth value of each pixel of the target object in the Nth frame image based on the formula De=d1+(d1-d2). Wherein, De is the estimated depth value. Based on the formula Dr=De*(1-c)+D0*c, the actual depth value of any pixel of the target object in the Nth frame image is determined, where Dr is the actual depth value. The difference Df between the depth of any pixel of the target object in the Nth frame image and the actual depth value is calculated. According to the formula D1=D0+Df, the depth of each pixel of the target object in the Nth frame image is adjusted.

[0030] In one possible embodiment, the processing unit is further used to update the depth of each pixel in the target object in the Nth frame image, so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold.

[0031] Furthermore, the processing unit is specifically used for the electronic device to use the average value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image. Alternatively, the electronic device uses the maximum value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image. Alternatively, the electronic device uses the median value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image. Alternatively, the electronic device uses the minimum value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image.

[0032] Alternatively, the processing unit is further configured to determine a target plane having the minimum sum of distances to each pixel of the target object in the Nth frame image. Project each pixel of the target object in the Nth frame image onto the target plane. Update the depth of each pixel of the target object in the Nth frame image to the depth of each pixel of the target object in the Nth frame image projected onto the target plane.

[0033] In one possible embodiment, the display unit is further configured to display a shooting preview interface, the shooting preview interface including a first control. The display unit is further configured to display an icon of a virtual object in response to a triggering operation on the first control, and to display the virtual object in the shooting preview interface in response to a triggering operation on the virtual object icon.

[0034] In one possible implementation, the processing unit is further configured to determine a first threshold value based on a preset speed threshold value and a preset frame rate of the captured image. The speed threshold value, the frame rate, and the first threshold value satisfy the following conditions: Dt is the first threshold, v is the speed threshold, and f is the frame rate.

[0035] In one possible embodiment, the processing unit is further configured to determine whether the adjusted depth of each pixel of the target object in the Nth frame image is less than the depth of the corresponding pixel of the virtual object. The display unit is further configured to, if the adjusted depth of each pixel of the target object in the Nth frame image is less than the depth of the pixel corresponding to the virtual object, cause the electronic device to display the virtual object with the target object in the Nth frame image obstructing the display; and if the adjusted depth of each pixel of the target object in the Nth frame image is greater than the depth of the pixel corresponding to the virtual object, cause the electronic device to display the target object in the Nth frame image obstructing the display.

[0036] In one possible implementation, the processing unit is specifically configured to determine whether the depth of the adjusted pixel of the target object in n consecutive image frames, including the Nth image frame, is less than the depth of the pixel corresponding to the virtual object, where n is an integer greater than or equal to 2.

[0037] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, the memory being used to store code instructions; the processor being used to run the code instructions so that the electronic device executes the image display method described in the first aspect or any one of the implementations of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the computer executes the image display method described in the first aspect or any implementation of the first aspect.

[0039] In a fifth aspect, a computer program product includes a computer program. When the computer program is executed, the computer executes the image display method as described in the first aspect or any one of the implementations of the first aspect.

[0040] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the hardware system architecture of an electronic device provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the software system architecture of an electronic device provided in an embodiment of the present application;

[0043] Figure 3 One of the flow charts of the image display method provided in an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the first interface provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the architecture of the depth estimation model provided in an embodiment of the present application for calculating the depth of each pixel of the Nth frame image;

[0046] Figure 6 for Figure 3 Specific flow chart of S406 in FIG;

[0047] Figure 7 A schematic diagram of an interface for displaying a second AR image in a first interface provided in an embodiment of the present application;

[0048] Figure 8 The second flowchart of the image display method provided in the embodiment of the present application;

[0049] Figure 9This is a schematic diagram of the distance from the camera to each pixel of person A provided in an embodiment of the present application;

[0050] Figure 10 Schematic diagram 2 of the distance from the camera to each pixel point of person A provided in an embodiment of the present application;

[0051] Figure 11 The third flowchart of the image display method provided in the embodiment of the present application;

[0052] Figure 12 A functional module block diagram of an image display device provided in an embodiment of the present application;

[0053] Figure 13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0054] Figure 14 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first value and the second value are merely used to distinguish different values ​​and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit different values.

[0056] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0058] Current augmented reality (AR) technology can superimpose virtual objects on real images captured by electronic devices, making the virtual objects and real images complement each other, thereby achieving "enhancement" of the real images displayed by electronic devices.

[0059] Exemplarily, the electronic device can capture the N-1 frame image and obtain the cartoon character B. The N-1 frame image includes character A. For example, the depth of each pixel of character A in the N-1 frame image is greater than 5m. The depth is used to indicate the distance between each pixel of character A in the image and the electronic device, and the depth of each pixel of character A is calculated by a pre-trained depth estimation model. In addition, it can be understood that the cartoon character B is a virtual object, and each pixel of the cartoon character B also has depth. For example, the depth of each pixel of the cartoon character B pre-recorded by the electronic device is less than 3m. It can be seen that the depth of each pixel of the cartoon character B is less than the depth of each pixel on the character A.

[0060] The electronic device displays a first AR image on the video playback interface based on the N-1 frame image and cartoon character B. In the first AR image, the position of cartoon character B partially overlaps with the position of character A. Because the depth of each pixel of character B is less than the depth of each pixel of character A, the overlapping portion of the positions of character A and cartoon character B is obscured by cartoon character B.

[0061] When the electronic device captures the Nth frame image, the Nth frame image also includes character A. In this way, the electronic device generates a second AR image based on the cartoon character B and the Nth frame image. Among them, since the position of the camera of the electronic device when capturing the Nth frame image changes relative to the position when the electronic device captures the N-1th frame image (such as the electronic device shakes), and / or the character A in the Nth frame image moves relative to the character A in the N-1th frame, the content of the Nth frame image will be different from the image content of the N-1th frame. In this way, the depth of each pixel on the character A in the Nth frame image output by the pre-trained depth estimation model may be significantly different from the depth of the corresponding pixel on the character A in the N-1th frame image. For example, the depth of each pixel on the character A in the Nth frame image is less than 2.8m (the difference from the depth of the corresponding pixel on the character A in the N-1th frame image is 2.2m). Since the depth of the cartoon character B recorded by the electronic device is 3m, it can be seen that the depth of the cartoon character B is greater than the depth of each pixel on the character A. In the second AR image, the portion where the positions of the cartoon character B and the character A overlap is blocked by the character A.

[0062] In everyday life, the speed of person A's movement usually does not exceed a preset speed threshold, and the shaking amplitude of the electronic device when shooting a video is usually less than a preset amplitude threshold. When the frame rate of the electronic device capturing images is constant, the difference between the depth of the pixel in the N-1 frame image and the depth of the corresponding pixel in the corresponding N frame image will not reach 2.2m. In fact, the depth of each pixel on person A in the N-1 frame image output by the pre-trained depth estimation model differs by 2.2m from the depth of the corresponding pixel on person A in the N-1 frame image. This will cause the part where person A in the N-1 frame image overlaps with the cartoon character B to be occluded by cartoon character B, and suddenly switch to the part where the cartoon character B overlaps with the position of person A in the N frame image being occluded by person A. As a result, the user's visual experience is that person A in the second AR image is displayed in a jumpy manner relative to person A in the first AR image, resulting in poor image quality.

[0063] In view of this, the present application provides an image display method, in which an electronic device can determine the depth of each pixel in a target object in the Nth frame image, where the target object is an object in the Nth frame image that is occluded by a preset virtual object. Furthermore, when the difference between the depth of any pixel in the target object and the depth of the corresponding pixel on the target object in the N-1th frame image is greater than a first threshold, the electronic device adjusts the depth of any pixel so that the difference is less than or equal to the first threshold. In this way, when the electronic device displays the virtual object and the adjusted target object, the object with lower depth in the virtual object and the adjusted target object is displayed to occlude the object with higher depth.

[0064] In this way, since the depth difference between the adjusted pixel points of the target object in the Nth frame image and the depth difference between the corresponding pixel points on the target object in the N-1th frame image is less than or equal to the first threshold, the depth difference between the pixel points on the target object in the Nth frame image and the N-1th frame image can be made less than the first threshold. In this way, the depth difference between the pixel points on the target object in the Nth frame image and the N-1th frame image will not be distorted (for example, the target object in the N-1th frame image will not be 10 meters behind the virtual object and suddenly jump to the target object in the Nth frame image being 1 meter in front of the virtual object). Furthermore, the situation where the portion of the target object in the N-1th frame image that overlaps with the virtual object is blocked by the virtual object and suddenly switches to the portion of the target object that overlaps with the target object in the Nth frame image being blocked by the target object will not occur; or the situation where the portion of the target object in the N-1th frame image that overlaps with the virtual object is blocked by the target object and suddenly switches to the portion of the target object that overlaps with the target object in the Nth frame image being blocked by the virtual object will not occur. That is, the occlusion relationship between the virtual object and the target object in the Nth frame image does not change suddenly relative to the occlusion relationship between the virtual object and the target object in the N-1th frame image. In this way, the quality of the displayed image is improved.

[0065] It is understandable that the above-mentioned electronic devices may be user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic devices may be mobile phones, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic devices.

[0066] In order to better understand the embodiments of the present application, the structure of the electronic device according to the embodiments of the present application is introduced below. Figure 1A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0067] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0068] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 may include one or more processing units. The different processing units may be independent devices or integrated into one or more processors. The processor 110 may also be provided with a memory for storing instructions and data.

[0070] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0071] The electronic device implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0072] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than one.

[0073] The electronic device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0074] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0075] The external memory interface 120 can be used to connect an external memory card to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0076] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 can include a program storage area and a data storage area. For example, the internal memory 121 can be used to store the depth of each pixel of each frame of image captured by the camera 193, as well as the depth of each pixel of a virtual object.

[0077] Keys 190 include a power button, a volume button, and the like. Keys 190 can be mechanical or touch-sensitive. The electronic device can receive key inputs and generate key signal inputs related to user settings and function control of the electronic device. Indicator 192 can be an indicator light that can be used to indicate charging status and battery level changes.

[0078] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a microservice architecture, or a cloud architecture, etc., which will not be described in detail here. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device. Figure 2 The following is a software structure diagram of an electronic device applicable to embodiments of the present application. The layered architecture divides the software system of an electronic device into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers: application layer (applications), application framework layer (application framework), Android runtime (Android runtime) and system library, hardware abstraction layer (HAL), and kernel layer (kernel).

[0079] The application layer can include a series of application packages, and the application layer runs applications by calling the application programming interface (API) provided by the application framework layer. Figure 2As shown, the application package may include applications such as camera, beauty photo, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0080] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager, etc.

[0081] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc. The content provider is used to store and obtain data and make this data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for electronic devices. The resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar and can be used to convey notification-type messages.

[0082] The Android runtime consists of a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.

[0083] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. It also supports a variety of audio and video encoding formats. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0084] The hardware abstraction layer (HAL) can contain multiple library modules, such as a camera library module and a motor library module. The Android system can load the corresponding library modules for the device hardware, enabling the application framework layer to access the device hardware. Device hardware can include displays and cameras in electronic devices.

[0085] The kernel layer is the layer between hardware and software. It drives the hardware, making it work. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, and motor drivers, though this embodiment of the application does not limit these.

[0086] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0087] Explanation of terms in this patent application:

[0088] Depth estimation model: This model is used to obtain a depth map corresponding to the RGB image based on the input raw RGB (red, green, blue) image, the depth value of the depth sensor, and the camera's pose. The depth map includes the depth of each pixel in the RGB image.

[0089] Mask: The mask is the outside of the selection box (the inside of the selection box is the selection area). The mask is a grayscale image that acts like a piece of cloth, covering a portion of the processing area. When the entire image within the processing area is subjected to depth adjustment, blurring, coloring, and other operations, the part covered by the mask will not be affected.

[0090] Frame rate: The frequency (rate) at which bitmap images, in units of frames, appear continuously on the display.

[0091] Kalman filtering: An algorithm that uses the linear system state equation to optimally estimate the system state using observation data from the system's input and output. Because the observation data includes the effects of noise and interference in the system, optimal estimation can also be viewed as a filtering process.

[0092] Below, the image display method provided in the embodiment of the present application is described using an electronic device as a mobile phone 200, a target object as a character A, and a virtual object as a cartoon character B as an example. This example does not constitute a limitation on the embodiment of the present application. The following embodiments may be combined with each other, and the same or similar concepts or processes will not be repeated here. Figure 3 This is a flow chart of an embodiment of the image display method provided in the present application. Figure 3 As shown, the image display method provided in the embodiment of the present application includes:

[0093] S401: The mobile phone 200 captures the N-1th frame image, wherein the N-1th frame image includes person A, and N is an integer greater than 2.

[0094] For example, the way in which the mobile phone 200 captures the N-1 frame image can be: when it is necessary to record a video image of person A, the mobile phone 200 displays the system desktop. The system desktop includes a "camera" icon (not shown in the accompanying drawings). The camera 193 of the mobile phone 200 can be aimed at person A, and then Figure 4 As shown in (a) of FIG, in response to the user triggering the "camera" icon, the mobile phone 200 displays the first interface 501. It can be understood that the first interface 501 is the shooting preview interface of the "camera". The first interface 501 displays the image captured by the mobile phone 200.

[0095] S402: The mobile phone 200 displays the N-1th frame image and the cartoon character B on the first interface 501. The cartoon character B and the character A are blocked.

[0096] Still Figure 4 As shown in (a) in the figure, the first interface 501 also includes an "AR mode" control (i.e., a first control). The mobile phone 200 can also display a floating window on the first interface 501 in response to the user triggering the "AR mode" control. The floating window includes icons of multiple virtual objects, and the icons of the multiple virtual objects include an icon of cartoon character B. In addition, after displaying the floating window, the mobile phone 200 can capture the N-1th frame image, and the N-1th frame image includes character A.

[0097] For example, mobile phone 200 inputs the N-1th frame image to the pre-trained depth estimation model, and the depth estimation model outputs the depth of each pixel in the N-1th frame image. Typically, the depth of each pixel in the N-1th frame image output by the depth estimation model deviates from the actual depth of each pixel in the N-1th frame image.

[0098] like Figure 4 As shown in (b) of FIG. 1 , the mobile phone 200 displays the N-1 frame image on the first interface 501 based on the depth of each pixel point on the N-1 frame image. In addition, the mobile phone 200 also stores the depth of each pixel point on the N-1 frame image in the internal memory 121 .

[0099] In addition, mobile phone 200 can also identify person A in the N-1th frame image. Furthermore, mobile phone 200 can obtain the depth of each pixel of person A in the N-1th frame image. After identifying person A in the N-1th frame image, mobile phone 200 can also obtain the area in the N-1th frame image excluding person A. It can be understood that the area in the N-1th frame image excluding person A is the mask of person A in the N-1th frame image.

[0100] Still Figure 4 As shown in (b) in the figure, the mobile phone 200 can respond to the user's dragging operation on the icon of the cartoon character B and drag the cartoon character B to the N-1 frame image, and the position of the cartoon character B partially overlaps with the position of the character A. Furthermore, the mobile phone 200 inputs the position of the cartoon character B dragged to the N-1 frame image and the posture of the camera 193 of the mobile phone 200 into the depth estimation model to output the depth of each pixel of the cartoon character B. In addition, the mobile phone 200 also stores the depth of each pixel of the cartoon character B in the internal memory 121. It can be understood that, as Figure 4 As shown, the N-1th frame image and the cartoon character B constitute the first AR image 502 .

[0101] It can be understood that for a certain object, the smaller the depth of each pixel point of the object, the closer the distance between the object and the camera 193 of the mobile phone 200 is; conversely, the greater the depth of each pixel point of the object, the farther the distance between the object and the camera 193 of the mobile phone 200 is.

[0102] For example, in Figure 4 In the image, the depth of each pixel of character B is less than 2.5 meters, while the depth of each pixel of character A is greater than 3 meters. This shows that the depth of each pixel of character B is less than that of character A (i.e., the distance between character B and phone 200 is less than the distance between character A and phone 200). Therefore, for the overlapping portion C between the positions of character B and character A, character B obscures character A.

[0103] Of course, in other embodiments, the depth of each pixel of the cartoon character B may be greater than the depth of each pixel of the character A (i.e., the distance between the cartoon character and the mobile phone 200 is greater than the distance between the character A and the mobile phone 200). In this way, for the overlapping portion C between the positions of the cartoon character B and the character A, the character A obscures the cartoon character B (not shown in the drawings).

[0104] S403: The mobile phone 200 captures the Nth frame image, wherein the Nth frame image also includes person A.

[0105] It can be understood that the Nth frame image is the next frame image that is continuous with the N-1th frame image when the mobile phone 200 is capturing the video image. The position of the character A in the Nth frame image also overlaps with the cartoon character B. In other words, the character A in the Nth frame image is an object that is occluded by the cartoon character B.

[0106] S404: The mobile phone 200 determines the depth of each pixel of the person A in the Nth frame image.

[0107] For example, Figure 5 As shown, mobile phone 200 can input the N-1th frame image and the Nth frame image into the depth estimation model, and the depth estimation model can output the depth of each pixel in the Nth frame image. Generally, the depth of each pixel in the Nth frame image output by the depth estimation model deviates from the actual depth of each pixel in the Nth frame image.

[0108] In one embodiment, in everyday life, the speed of a moving object generally does not exceed a preset speed threshold. Thus, the depth difference between any pixel of person A in the Nth frame and the depth of the corresponding pixel of person A in the N-1th frame will not be greater than the first threshold.

[0109] However, since the depth of each pixel on the N-1 frame image output by the depth estimation model deviates from the actual depth of each pixel on the N-1 frame image; the depth of each pixel on the N frame image output by the depth estimation model deviates from the actual depth of each pixel on the N frame image. This may cause the depth of any pixel of person A in the N frame to differ from the depth of the pixel corresponding to person A in the N-1 frame to be greater than the first threshold, that is, the depth of any pixel of person A in the N frame to differ from the depth of the pixel corresponding to person A in the N-1 frame to be distorted. In this way, the depth of the pixel in person A in the N frame image can be adjusted based on the following S405.

[0110] S405: When the difference between the depth of any pixel point of the person A in the Nth frame image and the corresponding pixel point on the target object in the N-1th frame image is greater than a first threshold, the mobile phone 200 adjusts the depth of any pixel point so that the difference is less than or equal to the first threshold.

[0111] Specifically, the mobile phone 200 adjusts the depth of any pixel based on the depth of the pixel of the target object in the M frames before the Nth frame so that the difference is less than or equal to a first threshold, where M is an integer greater than or equal to 1.

[0112] For example, the specific implementation of S405 includes but is not limited to the following two methods:

[0113] The first method: When M = 1, the mobile phone 200 can obtain the depth of each pixel in the person A in the N-1th frame image from the internal memory 121. The mobile phone 200 can use the depth of each pixel in the person A in the N-1th frame image as a reference depth. The mobile phone 200 calculates the difference Da between the depth of any pixel in the person A in the N-1th frame image and the depth of the corresponding pixel in the person A in the N-1th frame image. The mobile phone 200 determines whether the difference Da between the depth of any pixel in the person A in the N-1th frame image and the depth of the corresponding pixel in the person A in the N-1th frame image is greater than a first threshold Dt. If the difference Da between the depth of any pixel in the person A in the N-1th frame image and the depth of the corresponding pixel in the person A in the N-1th frame image is greater than the first threshold Dt, the mobile phone 200 adjusts the depth of the pixel whose depth difference between the corresponding pixel in the N-1th frame image and the person A in the N-1th frame image is greater than the first threshold Dt according to the formula D1 = D0 - (Da - Dt). Where D1 is the depth after adjustment, and D0 is the depth before adjustment.

[0114] Since the subsequent focus is to avoid a sudden change in the occlusion relationship between the target object and the virtual object in frame N compared to the occlusion relationship between the target object and the virtual object in frame N-1, the depth of any pixel is adjusted based on the depth of the pixel of the target object in frame N-1. This ensures a higher degree of reliability in the adjusted depth of any pixel.

[0115] Second: When M = 2, the internal memory 121 of the mobile phone 200 also stores the depth of each pixel of person A in the N-2th frame image captured by the camera 193 of the mobile phone 200. Where N is an integer greater than 2. The mobile phone 200 can also obtain the depth of each pixel of person A in the N-2th frame image and the depth of each pixel of person A in the N-1th frame image from the internal memory 121.

[0116] Mobile phone 200 calculates the depth difference Da between any pixel of person A in the Nth frame and the depth of the corresponding pixel of person A in the N-1th frame. Mobile phone 200 determines whether the depth difference Da between any pixel of person A in the Nth frame and the depth of the corresponding pixel of person A in the N-1th frame is greater than a first threshold Dt. If so, mobile phone 200 calculates an estimated depth value De for each pixel of person A in the Nth frame based on the formula De=d1+(d1-d2), where d1 is the depth of the pixel of person A in the N-1th frame, and d2 is the depth of the pixel of person A in the N-2th frame. Mobile phone 200 then performs a Kalman filter on the Nth frame to obtain a Kalman gain c. Mobile phone 200 can then determine the actual depth value Dr for any pixel of person A in the Nth frame based on the formula Dr=De*(1-c)+D0*c. Wherein, D0 is the depth of any pixel of person A in the Nth frame image.

[0117] It should be noted that in the formula Dr=De*(1-c)+D0*c, the Kalman gain c is associated with the difference between the depth D0 of any pixel point in the person A of the N-th frame image and the corresponding estimated depth value De. From the above formula Dr=De*(1-c)+D0*c, it can be seen that the smaller the difference between the depth D0 of any pixel point in the person A of the N-th frame image and the corresponding estimated depth value De, the larger the Kalman gain c, indicating that the reliability of the depth D0 of any pixel point in the person A of the N-th frame image is higher; conversely, the larger the difference between the depth D0 of any pixel point in the person A of the N-th frame image and the corresponding estimated depth value De, the smaller the Kalman gain c, indicating that the reliability of the depth estimation value De of any pixel point in the person A of the N-1-th frame image is higher.

[0118] The mobile phone 200 can use the actual depth value Dr of any pixel of the person A in the Nth frame image as a reference depth. The mobile phone 200 calculates the difference Df between the depth D0 of any pixel of the person A in the Nth frame image and the actual depth value Dr, that is, Df=D0-Dr. Furthermore, the mobile phone 200 can adjust the depth of each pixel of the person A in the Nth frame image according to the formula D1=D0+Df. Among them, D1 is the depth after adjustment, D0 is the depth before adjustment, and Df is the difference between the depth of any pixel of the person A in the Nth frame image and the actual depth value Dr. In the embodiment of the present application, since the Kalman gain is obtained by performing Kalman filtering on the Nth frame image, the depth estimation value De of any pixel of the person A in the N-1th frame image is obtained based on the difference between the depth d1 of any pixel of the person A in the N-1th frame image and the depth d2 of the corresponding pixel of the person A in the N-2th frame image. In this way, the reliability of the actual depth value Dr of any pixel of person A in the Nth frame image is higher. The mobile phone 200 can add the corresponding difference Df to the depth of each pixel of person A in the Nth frame image, which can improve the reliability of the depth of each pixel of person A in the Nth frame image.

[0119] Since the subsequent focus is to avoid a sudden change in the occlusion relationship between the target object and the virtual object in frame N compared to the occlusion relationship between the target object and the virtual object in frame N-1, the depth of any pixel is adjusted based on the depth of the target object pixel in both the N-1 and N-2 frames. This comprehensive reference ensures greater reliability in the depth of any pixel after adjustment.

[0120] Based on the above, the adjusted depth D1 and the unadjusted depth D0 satisfy the condition: D1 = D0 + {D0 - {[d1 + (d1 - d2)] * (1 - c) + D0 * c}}. Where D1 is the adjusted depth, D0 is the unadjusted depth, d1 is the depth of the pixel of person A in the N-1th frame image, d2 is the depth of the pixel of person A in the N-2th frame image, and c is the Kalman gain obtained by performing a Kalman filter on the Nth frame image.

[0121] It is understandable that in the first approach described above, the depth of each pixel of person A in the Nth frame is adjusted based on the depth of each pixel of person A in the two frames preceding the Nth frame. Alternatively, the depth of each pixel of person A in the Nth frame may be adjusted based on the depth of each pixel of person A in the three, four, and five frames preceding the Nth frame, etc., without limitation herein.

[0122] As can be seen, the first and second methods described above can be summarized as follows: mobile phone 200 adjusts the depth of the pixels in the Nth frame based on the depth of the pixels of person A in the M frames preceding the Nth frame, so that the difference is less than or equal to the first threshold, where M is an integer greater than or equal to 1.

[0123] In addition, in daily life, people's movement speed usually does not exceed the preset speed threshold. When the frame rate of the image captured by mobile phone 200 is constant, the difference between the depth of the pixel point in the N-1 frame image and the depth of the corresponding pixel point in the corresponding N frame image will not be greater than the first threshold Dt. If it is greater than the first threshold Dt, it means that the depth of the pixel point output by the depth estimation model of mobile phone 200 is not accurate enough and needs to be adjusted to a depth with a difference less than or equal to the first threshold Dt. In this way, the reliability of the depth of the pixel point can be improved.

[0124] In addition, in daily life, the speed of the moving body usually does not exceed the preset speed threshold. Thus, the mobile phone 200 can determine the first threshold Dt according to the preset speed threshold and the frame rate of the image captured by the mobile phone 200. For example, the mobile phone 200 can determine the first threshold Dt according to the formula Determine a first threshold. Wherein, Dt is the first threshold, v is the preset speed threshold, and f is the frame rate. For example, when the preset speed threshold v is 6 m / s, if the frame rate f is equal to 15 fps, the first threshold Dt is equal to 0.4 m; if the frame rate f is equal to 30 fps, the first threshold Dt is equal to 0.2 m. Of course, in the embodiment of the present application, the preset speed threshold v and frame rate f can also be other values, which are not limited here. In this way, the reliability of determining the first threshold can be high.

[0125] S406 : When the mobile phone 200 displays the cartoon character B and the adjusted character A, the object with lower depth in the cartoon character B and the adjusted character A is displayed to block the object with higher depth.

[0126] For example, Figure 6As shown, the process of the mobile phone 200 displaying the cartoon character B and the adjusted character A may include: S701: the mobile phone 200 compares the depth of each pixel of the cartoon character B with the depth of the corresponding pixel of the adjusted character A (comparing pixels with the same coordinate position). S702: the mobile phone 200 determines whether the depth of each pixel in the adjusted character A is less than the depth of each pixel of the cartoon character B. S703: when the depth of each pixel in the adjusted character A (e.g., greater than 2.9m) is less than the depth of each pixel of the cartoon character B (e.g., less than 2.5m), the mobile phone 200 displays a second AR image (not shown in the drawings) in the first interface 501 based on the mask of the character A, the Nth frame image, and the image of the cartoon character B. The second AR image includes the character A displayed at the depth of each pixel in the adjusted character A and the cartoon character B displayed at the depth of each pixel of the cartoon character B, and in the overlapping part C of the character A and the cartoon character B, the cartoon character B still blocks the character A. It can be seen that the occlusion relationship between the cartoon character B and the character A in the N-th frame image does not change significantly compared to the occlusion relationship between the cartoon character B and the character A in the N-1-th frame image.

[0127] In other embodiments, Figure 6 As shown, S704: when the depth of each pixel of the adjusted character A is greater than the depth of each pixel of the cartoon character B, the mobile phone 200 Figure 7 As shown, mobile phone 200 displays a second AR image 503 on first interface 501 based on the mask of character A, the Nth frame image, and the image of cartoon character B. Second AR image 503 includes character A displayed at the adjusted depth of each pixel in character A and cartoon character B displayed at the depth of each pixel in cartoon character B. In the overlapping portion C between character A and cartoon character B, character A occludes cartoon character B. It is understandable that in this embodiment of the present application, since the difference between the adjusted depth of any pixel of character A in frame N and the depth of the corresponding pixel of character A in frame N-1 is less than the first threshold, even if the overlapping portion of character A in frame N-1 is occluded by cartoon character B, the overlapping portion of character B with character A in frame N is occluded by character A. The user will visually perceive the switching of the occlusion relationship between cartoon character B and character A as smooth, i.e., the switching of the occlusion relationship between cartoon character B and character A is not distorted, and thus does not affect the image quality.

[0128] In some optional implementations, it should be noted that, in order to further avoid the occlusion relationship between the cartoon character B and the character A in the Nth frame image, the occlusion relationship between the cartoon character B and the character A in the N-1th frame image is changed. Figure 6As shown, in S703, it can be determined whether the depth of each pixel point in the adjusted character A in the image captured by the mobile phone 200 for n consecutive frames (n is greater than or equal to 2) is less than the depth of each pixel point of the cartoon character B. If so, when the mobile phone 200 displays the AR image, the character A is displayed at the depth of each pixel point in the adjusted character A.

[0129] For example, when n is equal to 2, the mobile phone 200 determines whether the depth of each pixel in the adjusted character A in the N-1 frame image is less than the depth of each pixel in the cartoon character B, and determines whether the depth of each pixel in the adjusted character A in the N frame image is less than the depth of each pixel in the cartoon character B. If so, when displaying the AR image, the mobile phone 200 displays the character A at the depth of each pixel in the adjusted character A. In this way, when the depth of each pixel in the adjusted character A in two consecutive frames is less than the depth of each pixel in the cartoon character B, as Figure 7 As shown, when the mobile phone 200 displays the second AR image 503, the cartoon character B in the N-1th frame blocks the character A, and then switches to the character A in the Nth frame blocking the cartoon character B. In this way, the occlusion relationship can be further avoided from jumping.

[0130] Understandably, if Figure 8 As shown, the process of the above-mentioned image display method is as follows: S901: The mobile phone 200 obtains the mask of the person A in the Nth frame image, and obtains the depth map of the Nth frame image calculated based on the Nth frame image (i.e., the depth of each pixel of the Nth frame image). S902: Based on the mask of the person A in the Nth frame image and the depth map of the Nth frame image, the mobile phone 200 obtains the second depth map of the person A in the Nth frame image (i.e., the depth of each pixel of the person A). In addition, S903: The mobile phone 200 obtains the first depth map of the person A in the N-1th frame image (i.e., the depth of each pixel of the person A in the N-1th frame image). S904: The mobile phone 200 adjusts the second depth map of the person A so that the difference between the second depth map of the person A and the first depth map of the person A is less than a first threshold. S905: The mobile phone 200 displays the AR image. The AR image includes the person A and the cartoon character B in the adjusted Nth frame image, and the object with lower depth in the cartoon character B and the adjusted person A occludes the object with higher depth.

[0131] In summary, since the speed of the moving body in daily life scenes usually does not exceed the preset speed threshold, the depth of any pixel of the character A in the Nth frame and the depth of the pixel corresponding to the character A in the N-1th frame image do not exceed the first threshold. In the image display method provided by the embodiment of the present application, the difference between the depth of the adjusted pixel of the character A in the Nth frame image and the depth of the pixel corresponding to the character A in the N-1th frame image is less than or equal to the first threshold. In this way, the difference in depth between the pixel on the character A in the Nth frame image and the N-1th frame image can be made less than the first threshold. That is, the difference in depth between any pixel of the character A in the Nth frame and the depth of the pixel corresponding to the character A in the N-1th frame will not be distorted. In this way, the part where the position of the character A in the N-1th frame image overlaps with the cartoon character B will not be blocked by the cartoon character B, and the part where the cartoon character B suddenly switches to the part where the position of the character A in the Nth frame image overlaps with the character A will not be blocked by the character A in the Nth frame image. Furthermore, the occlusion relationship between the cartoon character B and the character A in the Nth frame image does not change suddenly compared to the occlusion relationship between the cartoon character B and the character A in the N-1th frame image. In this way, the quality of the displayed image is improved.

[0132] In addition, the depth of each pixel of character A in the N+1th frame image can also be adjusted using the method in S405 described above. This prevents the situation where the overlapping portion of character A in the N-1th frame image is blocked by character B, and then suddenly switches to the overlapping portion of character B in the Nth frame image being blocked by character A; and then switches back to the overlapping portion of character A in the N-1th frame image being blocked by character B. This improves the flickering display of character B in the image, further enhancing the quality of the displayed image.

[0133] It should be noted that, in the above embodiment, when displaying the person A in the N-1th frame image and the person A in the Nth frame image, the person A is displayed based on the depth of each pixel in the person A.

[0134] In another embodiment, in the above S402, the mobile phone 200 obtains the depth of each pixel of the character A on the N-1 frame image output by the depth estimation model. Then, the mobile phone 200 updates the depth of the pixel of the character A in the N-1 frame image so that the difference in the depth of each pixel within the preset area of ​​the character A is less than the preset second threshold. In this way, when the mobile phone 200 displays the character A and the cartoon character B in the N-1 frame image with the updated depth of each pixel in the first interface 501, the occlusion relationship between the character A and the cartoon character B will not be distorted. For example, Figure 4 In the example, there will be no situation where the shoulders of character A are blocked by cartoon character B, while the upper arms of character A are displayed.

[0135] Similarly, based on the above S404, the mobile phone 200 obtains the depth of each pixel in the Nth frame image output by the depth estimation model. Then, the mobile phone 200 updates the depth of the pixel of the character A on the Nth frame image so that the difference in the depth of each pixel within the preset area of ​​the character A is less than the preset second threshold. In this way, when the mobile phone 200 displays the character A and the cartoon character B in the Nth frame image in the first interface 501, the occlusion relationship between the character A and the cartoon character B will not be distorted. For example, Figure 7 In the example, there will be no situation where the shoulders of character A are blocked by cartoon character B, while the upper arms of character A are displayed.

[0136] For example, below, combined with Figure 9-10 It shows that the mobile phone 200 updates the depth of each pixel of the person A in the N-1th frame image or the Nth frame image so that the difference in the depth of each pixel within the preset area range on the person A is less than the two ways of the preset second threshold.

[0137] The first method: the mobile phone 200 calculates the average value of the depth of each pixel of the person A. The mobile phone 200 uses the average value of the depth of each pixel of the person A as the depth of each pixel of the person A. In this way, the difference in the depth of each pixel within the preset area of ​​the person A is 0. Furthermore, the difference in the depth of each pixel within the preset area of ​​the person A is less than the preset second threshold. For example, Figure 9 The small black dots in the image are used to indicate the individual pixels of character A. Figure 9 The distribution of each pixel point in the figure shows that the depth of each pixel point of character A is different, and Figure 9 The line segment 1001 in the figure indicates the average value (e.g., 2.5 m) of the depth of each pixel of person A. When the mobile phone 200 uses the average value of the depth of each pixel of person A as the depth of each pixel of person A, it can be understood that the distance from the camera 193 of the mobile phone 200 to each position of person A is equal.

[0138] In addition, the average value of the depth of each pixel point of the above-mentioned character A can also be replaced by the minimum value, maximum value and median value of the depth of each pixel point of character A, etc., which is not limited here. The principle and effect are the same as above and will not be repeated here.

[0139] The second method: the mobile phone 200 converts each pixel of the person A into a 3D point cloud space, and performs filtering on each pixel after conversion to the 3D point cloud space to remove noise in each pixel after conversion to the 3D point cloud space. Figure 10 As shown, the mobile phone 200 projects each pixel point after filtering to the target plane 1101 based on the principal component analysis method. Among them, the target plane 1101 is the plane with the smallest sum of distances to each pixel point after filtering. Then, the mobile phone 200 determines the depth of each pixel point after being projected onto the target plane. It can be understood that the depth of each pixel point after being projected onto the target plane 1101 can characterize the depth distribution trend of the pixel points of person A. In real life, the depth distribution of the pixel points of a certain area of ​​a person (such as the eye area, mouth area, and facial area, etc.) is continuous. In this way, the difference in the depth of each pixel point within the preset area range on person A is less than the preset second threshold.

[0140] exist Figure 9 or Figure 10 Based on the corresponding embodiments, Figure 11 As shown, the process of the image display method provided by the embodiment of the present application may include: S1201: The mobile phone 200 obtains the cartoon character B, the Nth frame image, and the posture of the camera 193 of the mobile phone 200 when shooting the Nth frame image. S1202: The mobile phone 200 calculates the projection image of the cartoon character B in the Nth frame image and the depth of each pixel point in the cartoon character B based on the posture of the camera 193. S1203: The mobile phone 200 calculates the mask of the character A based on the Nth frame image, and calculates the depth of each pixel point of the Nth frame image. Furthermore, S1204: The mobile phone 200 updates the depth of the pixel points of the character A in the mask, so that the difference in the depth of each pixel point within the preset area range on the character A is less than the preset second threshold value. S1205: The mobile phone 200 adjusts the depth of the updated pixel points of the character A in the mask. Thus, S1206: The mobile phone 200 displays the second AR image 503 on the first interface 501 based on the Nth frame image, the depth of each pixel of the adjusted character A, the projection of the cartoon character B in the Nth frame image, and the depth of each pixel of the cartoon character B. The second AR image 503 includes the cartoon character B and the adjusted character A. When the mobile phone 200 displays the cartoon character B and the adjusted character A, the object with lower depth in the cartoon character B and the adjusted character A obscures the display of the object with higher depth.

[0141] In addition, in the above embodiment, the target object in the Nth frame image is a person as an example. When the target object in the Nth frame image includes multiple people, the mobile phone 200 can identify the multiple people in the Nth frame image, the area in the N-1th frame image other than the multiple people, that is, the mask of the multiple people in the N-1th frame image. The mobile phone 200 applies the connected domain analysis algorithm to calculate the multiple connected domains in the mask. It can be understood that each connected domain is the area where a person is located. Furthermore, the mobile phone 200 can adjust the pixel points in each connected domain based on the above-mentioned S405 method (that is, adjust the depth of each pixel point of each person). Similarly, the mobile phone 200 displays the above-mentioned cartoon character B and the multiple people after adjusting the depth based on the above-mentioned S406 method.

[0142] In the above embodiment, the target object is a person A and the virtual object is a cartoon character B. The target object can also be a plant, an animal, a work of art, etc., without limitation. The virtual object can also be a virtual landscape, a virtual work of art, etc., without limitation.

[0143] In addition, in the above embodiment, the depth of each pixel of the target object in the image preview interface of the "camera" is adjusted as an example. In other embodiments, it can also be the image preview interface of some AR applications. For example, the image preview interface of a game application with AR image synthesis function or a video playback application, etc., is not limited here.

[0144] See also Figure 12 The present application also provides an image display device 1200, which is applied to the above-mentioned electronic device. The image display device 1200 includes an image acquisition unit 1201, a processing unit 1202 and a display unit 1203. The image acquisition unit 1201 is used to acquire the Nth frame image. N is an integer greater than 1. The processing unit 1202 is used to determine the depth of each pixel point of the target object in the Nth frame image, and the target object is an object in the Nth frame image that is occluded by the preset virtual object. The processing unit 1202 is also used to adjust the depth of any pixel point of the target object in the Nth frame image so that the difference between the depth of the corresponding pixel point in the N-1th frame image is greater than a first threshold, so that the difference is less than or equal to the first threshold. The display unit 1203 is used to display the virtual object and the adjusted target object, and display the object with lower depth in the virtual object and the adjusted target object so that the object with higher depth is occluded.

[0145] In one possible implementation, the processing unit 1202 is specifically configured to adjust the depth of any pixel of the target object in the M frames preceding the Nth frame so that the difference is less than or equal to a first threshold, where M is an integer greater than or equal to 1.

[0146] Furthermore, when M=1, the preceding M frames of the Nth frame image are the N-1th frame image, and the depth of any pixel of the target object in the Nth frame image before adjustment and the depth of any pixel of the target object in the Nth frame image after adjustment satisfy the condition: D1=D0-(Da-Dt). Wherein, D0 is the depth of any pixel of the target object in the Nth frame image before adjustment, D1 is the depth of any pixel of the target object in the Nth frame image after adjustment, and Da is the difference between the depth of any pixel of the target object in the Nth frame image and the depth of the corresponding pixel of the target object in the N-1th frame image.

[0147] Alternatively, further, when M=2, the first M frames of the Nth frame image include the N-1th frame image and the N-2th frame image, and the depth of any pixel of the target object in the Nth frame image before adjustment and the depth of any pixel of the target object in the Nth frame image after adjustment satisfy the condition: D1=D0+{D0-{[d1+(d1-d2)]*(1-c)+D0*c}}. Wherein, D0 is the depth of any pixel of the target object in the Nth frame image before adjustment, D1 is the depth of any pixel of the target object in the Nth frame image after adjustment, d1 is the depth of the pixel of the target object in the N-1th frame image, d2 is the depth of the pixel of the target object in the N-2th frame image, and c is the Kalman gain obtained by performing Kalman filtering on the Nth frame image.

[0148] Furthermore, the processing unit 1202 is specifically configured to calculate an estimated depth value for each pixel of the target object in the Nth frame image based on the formula De=d1+(d1-d2). Wherein, De is the estimated depth value. Based on the formula Dr=De*(1-c)+D0*c, the actual depth value of any pixel of the target object in the Nth frame image is determined, where Dr is the actual depth value. The difference Df between the depth of any pixel of the target object in the Nth frame image and the actual depth value is calculated. According to the formula D1=D0+Df, the depth of each pixel of the target object in the Nth frame image is adjusted.

[0149] In one possible embodiment, the processing unit 1202 is also used to update the depth of each pixel in the target object in the Nth frame image, so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold.

[0150] Furthermore, the processing unit 1202 is specifically configured for the electronic device to use the average value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image. Alternatively, the electronic device uses the maximum value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image. Alternatively, the electronic device uses the median value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image. Alternatively, the electronic device uses the minimum value of the depths of each pixel of the target object in the N-th frame image as the depth of each pixel of the target object in the N-th frame image.

[0151] Alternatively, the processing unit 1202 is further configured to determine a target plane having the minimum sum of distances to each pixel of the target object in the Nth frame image. Project each pixel of the target object in the Nth frame image onto the target plane. Update the depth of each pixel of the target object in the Nth frame image to the depth of each pixel of the target object in the Nth frame image projected onto the target plane.

[0152] In one possible implementation, display unit 1203 is further configured to display a shooting preview interface, the shooting preview interface including a first control. Display unit 1203 is further configured to display an icon of a virtual object in response to a triggering operation on the first control, and to display the virtual object in the shooting preview interface in response to a triggering operation on the virtual object icon.

[0153] In a possible implementation, the processing unit 1202 is further configured to determine a first threshold value based on a preset speed threshold value and a preset frame rate of the captured image. The speed threshold value, the frame rate, and the first threshold value satisfy the following conditions: Dt is the first threshold, v is the speed threshold, and f is the frame rate.

[0154] In one possible embodiment, the processing unit 1202 is further configured to determine whether the adjusted depth of each pixel of the target object in the Nth frame image is less than the depth of the corresponding pixel of the virtual object. The display unit 1203 is further configured to, if the adjusted depth of each pixel of the target object in the Nth frame image is less than the depth of the pixel corresponding to the virtual object, cause the electronic device to display the virtual object with the target object in the Nth frame image obstructing the display; and if the adjusted depth of each pixel of the target object in the Nth frame image is greater than the depth of the pixel corresponding to the virtual object, cause the electronic device to display the target object in the Nth frame image obstructing the display.

[0155] In one possible implementation, the processing unit 1202 is specifically configured to determine whether the depth of the adjusted pixel of the target object in n consecutive image frames, including the Nth image frame, is less than the depth of the pixel corresponding to the virtual object, where n is an integer greater than or equal to 2.

[0156] For example, Figure 13 A hardware structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 13 As shown, the electronic device includes a processor 1301, a communication circuit 1304 and at least one communication interface ( Figure 13 The communication interface 1303 is used as an example for explanation).

[0157] The processor 1301 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0158] Communications link 1304 may include circuitry that transmits information between the aforementioned components.

[0159] The communication interface 1303 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0160] Possibly, the electronic device may further include a memory 1302 .

[0161] The memory 1302 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a communication line 1304. The memory may also be integrated with the processor.

[0162] The memory 1302 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1301. The processor 1301 is used to execute the computer-executable instructions stored in the memory 1302, thereby implementing the image display method provided by the embodiment of the present application.

[0163] Possibly, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, and the embodiments of the present application do not specifically limit this.

[0164] In a specific implementation, as an embodiment, the processor 1301 may include one or more CPUs, such as Figure 13 CPU0 and CPU1 in.

[0165] In a specific implementation, as an embodiment, the electronic device may include multiple processors, such as Figure 13 1 and 1305. Each of these processors can be a single-CPU processor or a multi-CPU processor. A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0166] For example, Figure 14 A schematic diagram of the structure of a chip provided in an embodiment of the present application: The chip 140 includes one or more (including two) processors 1410 and a communication interface 1430 .

[0167] In some embodiments, the memory 1440 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0168] In the embodiment of the present application, the memory 1440 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1410. A portion of the memory 1440 may also include a non-volatile random access memory (NVRAM).

[0169] In the embodiment of the present application, the memory 1440, the communication interface 1430 and the memory 1440 are coupled together through the bus system 1420. In addition to the data bus, the bus system 1420 may also include a power bus, a control bus and a status signal bus. Figure 14 Various buses are labeled as bus system 1420.

[0170] The method described in the above embodiment of the present application can be applied to the processor 1410, or implemented by the processor 1410. The processor 1410 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the steps of the above method can be completed by the hardware integrated logic circuit in the processor 1410 or the instructions in the form of software. The above-mentioned processor 1410 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1410 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0171] The steps of the method disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 1440, and the processor 1410 reads the information in the memory 1440 and completes the steps of the above method in combination with its hardware.

[0172] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0173] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state disk (SSD)).

[0174] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0175] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers.

[0176] The above combinations should also be included within the scope of computer-readable media. The above are only specific embodiments of this application, but the scope of protection of this application is not limited to them. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image display method, characterized in that: The method comprises: The electronic device captures an N-th frame of image, where N is an integer greater than 1; The electronic device determines the depth of each pixel of the target object in the N-th frame image, where the target object is an object in the N-th frame image that is occluded by a preset virtual object; When a difference between the depth of any pixel of the target object in the N-th frame image and the depth of the corresponding pixel of the target object in the N-1-th frame image is greater than a first threshold, the electronic device adjusts the depth of the any pixel so that the difference is less than or equal to the first threshold; When the electronic device displays the virtual object and the adjusted target object, the object with lower depth among the virtual object and the adjusted target object is displayed to block the object with higher depth; The electronic device adjusting the depth of any one pixel so that the difference is less than or equal to the first threshold includes: The electronic device adjusts the depth of any one pixel based on the depths of the pixel of the target object in the M frames before the Nth frame, so that the difference is less than or equal to the first threshold, where M is an integer greater than or equal to 1; When M=2, the first M frames of the Nth frame image include the N-1th frame image and the N-2th frame image, and the depth of any pixel point of the target object in the Nth frame image before adjustment and the depth of any pixel point of the target object in the Nth frame image after adjustment meet the conditions: , where D0 is the depth of any pixel of the target object in the N-th frame image before adjustment, D1 is the depth of any pixel of the target object in the N-th frame image after adjustment, d1 is the depth of the pixel of the target object in the N-1-th frame image, d2 is the depth of the pixel of the target object in the N-2-th frame image, and c is the Kalman gain obtained by performing Kalman filtering on the N-th frame image.

2. The method according to claim 1, characterized in that When M=1, the preceding M frames of the Nth frame are the N-1th frame, and the depth of any pixel of the target object in the Nth frame before adjustment and the depth of any pixel of the target object in the Nth frame after adjustment meet the following conditions: , where D0 is the depth of any pixel point of the target object in the N-th frame image before adjustment, D1 is the depth of any pixel point of the target object in the N-th frame image after adjustment, Da is the difference between the depth of any pixel point in the target object of the N-th frame image and the depth of the corresponding pixel point in the target object of the N-1-th frame image, and Dt is the first threshold.

3. The method according to claim 1, characterized in that When M=2, the electronic device adjusts the depth of any one pixel so that the difference is less than or equal to the first threshold, including: The electronic device is based on the formula , calculating the estimated depth value of each pixel of the target object in the Nth frame image, wherein De is the estimated depth value; The electronic device is based on the formula , determining the actual depth value of any pixel of the target object in the Nth frame image, wherein Dr is the actual depth value; The electronic device calculates a difference Df between the depth of any pixel of the target object in the Nth frame image and the actual depth value; The electronic device is based on the formula , adjusting the depth of each pixel of the target object in the Nth frame image.

4. The method according to claim 1, wherein Before the electronic device adjusts the depth of any one pixel so that the difference is less than or equal to the first threshold, the method further includes: The electronic device updates the depth of each pixel in the target object in the Nth frame image so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold.

5. The method according to claim 4, characterized in that The electronic device updates the depth of each pixel in the target object in the Nth frame image so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold, including: The electronic device uses an average value of the depths of the pixels of the target object in the N-th frame image as the depth of the pixels of the target object in the N-th frame image; Alternatively, the electronic device uses the maximum value of the depths of the pixels of the target object in the N-th frame image as the depth of the pixels of the target object in the N-th frame image; Alternatively, the electronic device uses the median of the depths of the pixels of the target object in the N-th frame image as the depth of the pixels of the target object in the N-th frame image; Alternatively, the electronic device uses the minimum value of the depths of the pixels of the target object in the N-th frame image as the depth of the pixels of the target object in the N-th frame image.

6. The method according to claim 4, characterized in that The electronic device updates the depth of each pixel in the target object in the Nth frame image so that the difference in the depth of each pixel within a preset area range on the target object is less than a preset second threshold, including: The electronic device converts each pixel of the target object in the Nth frame image into a 3D point cloud space; The electronic device determines a target plane having the smallest sum of distances to each pixel point after conversion of the target object in the Nth frame image; The electronic device projects each pixel point of the target object in the Nth frame image onto the target plane; The electronic device updates the depth of each pixel point of the target object in the Nth frame image to: the depth of each pixel point of the target object in the Nth frame image projected onto the target plane.

7. The method according to any one of claims 1 to 6, characterized in that: Before the electronic device captures the Nth image frame, the method further includes: The electronic device displays a shooting preview interface, wherein the shooting preview interface includes a first control; The electronic device displays the icon of the virtual object in response to a triggering operation on the first control; The electronic device displays the virtual object in the shooting preview interface in response to a triggering operation on the icon of the virtual object.

8. The method according to any one of claims 1 to 6, characterized in that: Before the electronic device captures the Nth image frame, the method further includes: The electronic device determines the first threshold according to a preset speed threshold and a preset frame rate of the captured image; wherein the speed threshold, the frame rate, and the first threshold satisfy a condition: , Dt is the first threshold, v is the speed threshold, and f is the frame rate.

9. The method according to any one of claims 1 to 6, characterized in that: When the electronic device displays the virtual object and the adjusted target object, the display of the object with a lower depth in the virtual object and the adjusted target object obscures the display of the object with a higher depth, including: The electronic device determines whether the adjusted depth of each pixel of the target object in the Nth frame image is less than the depth of the corresponding pixel in the virtual object; If the adjusted depth of each pixel of the target object in the N-th frame image is lower than the depth of the pixel corresponding to the virtual object, the electronic device displays the virtual object by blocking the target object in the N-th frame image; If the adjusted depth of each pixel of the target object in the Nth frame image is higher than the depth of the pixel corresponding to the virtual object, the electronic device will block the display of the target object in the Nth frame image with the virtual object.

10. The method according to claim 9, characterized in that The electronic device determines whether the adjusted depth of each pixel of the target object in the Nth frame image is less than the depth of each pixel of the virtual object, including: The electronic device determines whether the depth of each adjusted pixel of the target object in n consecutive frames including the Nth frame is less than the depth of the pixel corresponding to the virtual object, where n is an integer greater than or equal to 2.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device is caused to perform the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer is caused to perform the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 10.

Images