Interface display method and electronic device

Abstract

Provided in the embodiments of the present application are an interface display method and an electronic device, by means of which a depth-of-field relationship between UI elements such as a clock component and an application icon and objects in a wallpaper image can be constructed, so as to present a 3D visual effect where the UI elements are interleavedly displayed among different objects in the wallpaper image. Moreover, the embodiments of the present application enable a user to drag a UI element to move on a wallpaper image, such that the UI element can present a 3D visual effect of interleaved display at any position on the wallpaper image, thereby improving the appeal of displaying a UI element on the wallpaper image.

Description

Interface display method and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202411400171.3, filed on September 30, 2024, with the State Intellectual Property Office of China, and the Chinese patent application No. 202411400171.3 has the title of “Interface display method and electronic device”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of electronic technology, and in particular, to an interface display method and an electronic device. BACKGROUND

[0003] A wallpaper image is usually used as a display background of an application icon, a clock component, a weather component and other user interface (UI) elements, which can beautify the user interface and meet the user's individual needs for the appearance of the user interface. Generally, the rendering levels of these UI elements and the wallpaper image are fixed, the rendering level of the wallpaper image is at the bottom layer, and the rendering level of the UI elements is at the upper layer, so that the clock component, the application icon and other UI elements can be displayed above the wallpaper image, which is convenient for the user to view. However, to some extent, this display effect lacks interest. SUMMARY

[0004] Embodiments of the present application provide an interface display method and an electronic device, which can construct the depth-of-field relationship between a UI element such as a clock component and an application icon and an object in a wallpaper image, and present a 3D visual effect that the UI element is displayed between different objects in the wallpaper image. Moreover, embodiments of the present application support the user to drag the UI element to move on the wallpaper image, so that the 3D visual effect that the UI element is displayed between different objects in the wallpaper image can be presented at any position on the wallpaper image, which improves the interest of displaying the UI element on the wallpaper image.

[0005] In a first aspect, embodiments of the present application provide an interface display method, which can include: an electronic device obtains a wallpaper image, generates a foreground image of a first region of the wallpaper image based on an original image of the first region and a foreground edge of the first region, and then obtains a first user interface according to the generated foreground image of the first region, a first UI element and the wallpaper image. Finally, the electronic device can display the first user interface. Wherein, the first user interface can include the wallpaper image and the first UI element, and the first UI element is displayed in the first object in the foreground image generated by the first UI element occluded.

[0006] In the first aspect, the first object is a foreground object in the first region. The first UI element is occluded by the first object such that the first UI element appears to have a depth-of-field effect in the first region. The first region is a region in which the first UI element (such as a clock component, an application icon, or the like) needs to present a depth-of-field effect, and it is also the display position of the first UI element on the wallpaper image. The first region can be a region selected by the user, for example, the user places the clock component on which region of the wallpaper image, and the region is the first region. The first region can also not be selected by the user, but be a default region set by the system, for example, the first region is the default display region of the first UI element each time the first user interface is opened. The position of the default region can be fixed, for example, the first UI element is always displayed in the same position by default each time the first user interface is opened. The position of the default region can also be variable, for example, the default display position of the first UI element is different when the first user interface is opened at different times.

[0007] The interface display method provided in the first aspect can support the first UI element to present a depth-of-field effect of being occluded by a first object in the foreground image of the first region on the wallpaper image. The foreground image of the first region is not cut out from the original wallpaper image, but is generated based on the original image and the foreground edge, which can support the construction of a depth-of-field effect for a clock component or the like in a region without a significant subject, and the transition effect between the foreground and the background is more natural.

[0008] In combination with the first aspect, in some embodiments, before generating the foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region, the electronic device can also crop the original image of the first region from the wallpaper image and perform edge detection on the original image of the first region to obtain the foreground edge of the first region.

[0009] Further, after the edge of the foreground region is extracted by edge detection, the electronic device can perform an inflation operation on the edge to expand the foreground edge to obtain an inflated foreground edge. In addition to the foreground region, the foreground fill-in image can also cover the inflated foreground edge, that is, the foreground fill-in image is performed on the foreground region and the inflated foreground edge to regenerate the foreground image. In this way, the region in which the foreground fill-in image needs to be performed at the foreground edge can be selected, and the edge burr can be improved. The inflated foreground edge can have a certain transparency. The transparency of the entire inflated foreground edge can be gradual, and the closer to the original foreground edge, the higher the transparency. The original foreground edge is the foreground edge before inflation. In this way, the fill-in foreground and the original background can be better fused together to form a natural depth-of-field effect.

[0010] In some embodiments, the first region of the wallpaper image can be divided into a plurality of sub-regions, and the first region of the foreground image can be generated based on the original image of each sub-region of the first region and the foreground edge of each sub-region of the first region.

[0011] The input of the first artificial intelligence model can include the original image of the first region and the foreground edge of the first region, and the output of the first artificial intelligence model can be the foreground of the first region. The training samples in the training data set of the first artificial intelligence model can include an input image, a foreground edge of the input image, and an output image, wherein the output image can be a foreground image of the input image.

[0012] In some embodiments, the first artificial intelligence model can be further configured with a loss function, which can be used to calculate the difference between the output image and the input image in the training sample. The difference can be composed of two parts: the image difference of the foreground between the output image and the input image, and the image difference of the foreground edge region between the output image and the input image. The weight of the image difference of the foreground can be greater than the weight of the image difference of the foreground edge region. That is, the foreground generated by the foreground inpainting needs to maintain high consistency with the foreground in the original image, while the foreground edge generated by the foreground inpainting can have some differences with the foreground edge in the original image.

[0013] In some embodiments, the first user interface can be obtained by performing layered rendering on the generated foreground image of the first region, the first UI element, and the wallpaper image. The layered rendering can include rendering the first UI element on the original wallpaper image, and rendering the foreground image of the first region on the first UI element. The original wallpaper image is at the bottom layer as a background layer, the foreground image of the first region is at the upper layer as a foreground layer, and the first UI element is at the middle layer. In this way, the depth-of-field effect that the first UI element is blocked by the foreground image of the first region can be presented in the first user interface.

[0014] In some embodiments, the size of the foreground layer can be the same as the size of the first region, or the size of the original wallpaper image. When the size of the foreground layer is the same as the size of the original wallpaper image, the area outside the first region in the foreground layer can be set to be transparent.

[0015] In some embodiments of the first aspect, in the first user interface, the first UI element can further occlude a second object in the background image of the first region, the second object having a depth greater than the depth of the first object. The second object is a background object in the first region. In this way, the depth-of-field effect of the first UI element being interposed between the first object and the second object can be presented by occluding the second object.

[0016] In some embodiments of the first aspect, the first user interface can include one or more of the following: a wallpaper setting preview interface, a lock screen interface, a desktop, an off-screen interface. In the wallpaper setting scenario, the first user interface can be a wallpaper setting preview interface, or an interface in which the wallpaper is officially displayed after the wallpaper setting is completed, such as a lock screen interface, a desktop, or an off-screen interface. In the official wallpaper display scenario, the first user interface can be a lock screen interface, a desktop, or an off-screen interface.

[0017] In some embodiments of the first aspect, the first UI element can include one or more of the following: a clock component, a weather component, an application icon, a folder icon.

[0018] In some embodiments of the first aspect, the position of the first region in the wallpaper image is variable. The position of the first region in the wallpaper image can be changed based on a user operation, which can include a user operation of moving the first UI element on the wallpaper image. The change in the position of the first region in the wallpaper image can also be triggered by an internal event of the electronic device. For example, the position of the first region for displaying the clock component is changed according to a preset trajectory or a random trajectory when the clock reaches the hour. In this example, the internal event is the clock reaching the hour. For another example, the position of the first region for displaying the weather component is changed according to the visual motion trajectory of the sun (sunrise, sunset, inclination). In this example, the internal event is the change in the visual motion trajectory of the sun. The present embodiments do not limit what the internal event is. The former way of changing the position of the first region can be applied to the aforementioned wallpaper setting scenario, and the latter way of changing the position of the first region can be applied to the aforementioned official wallpaper display scenario.

[0019] In a second aspect, the present embodiments provide a method for displaying an interface, which can include the following steps:

[0020] Step 1. The electronic device displays a first user interface, which can include a wallpaper image and a first UI element, the first UI element being displayed in a second region of the wallpaper image, the first UI element being occluded by a third object in the wallpaper image, and the first UI element occluding a fourth object in the wallpaper image. The third object and the fourth object are located in the second region of the wallpaper image.

[0021] Step 2. The electronic device moves the first UI element to a third region of the wallpaper image.

[0022] In step 3, the electronic device updates the first user interface, in which the first UI element is displayed in the third region, the first UI element is occluded by the fifth object in the wallpaper image, and the first UI element occludes the sixth object in the wallpaper image. The fifth object and the sixth object are located in the third region of the wallpaper image. In this way, a depth-of-field relationship in which the first UI element is interposed between the fifth object and the sixth object can be constructed, so that the first UI element can present a depth-of-field effect when displayed at the third region.

[0023] In the second aspect, the second region can be the first region at a certain time (before the first UI element is moved), i.e., the display position of the first UI element in the wallpaper image at a certain time. The position of the second region in the wallpaper image is different from the position of the third region in the wallpaper image in the following text. The third region can be the first region at another time (after the first UI element is moved), i.e., the display region of the first UI element in the wallpaper image at another time.

[0024] The method provided in the second aspect can present a depth-of-field effect of the UI element on the wallpaper image, and the position at which the depth-of-field effect is presented is not fixed. The user can move the UI element on the wallpaper image to experience the depth-of-field effect of the UI element at different positions, so that a 3D dynamic effect in which the UI element moves in the wallpaper image can be presented through the movement, which is more vivid.

[0025] In combination with the second aspect, in some embodiments, in the wallpaper image, the depth of the fourth object is greater than the depth of the third object. In the second region, the third object can be referred to as a foreground object, and the fourth object can be referred to as a background object. The third object can not be a significant subject in the wallpaper image. The third object and the fourth object can be part of a piece of the same type of object, and the third object does not constitute a significant subject relative to the fourth object, such as a tree in a piece of forest, grass in a piece of lawn, or wheat in a piece of wheat field, etc.

[0026] In combination with the second aspect, in some embodiments, the second region can be a region selected by the user, such as a region where the user drags the first UI element to display it. The second region can also be a region at a default position, such as a default display region of the first UI element each time the user opens the first user interface. The position of the default region can be unchangeable, such as the first UI element being displayed at the same position by default each time the first user interface is opened. The position of the default region can also be changeable, such as the default display position of the first UI element being different when the first user interface is opened at different times.

[0027] With reference to the second aspect, in some embodiments, the step of moving the first UI element can be triggered by a user operation. For example, when detecting a user operation of dragging the first UI element from the second region to the third region, the electronic device can move the first UI element from the second region to the third region. The user operation can refer to the operations of dragging the clock component 23 to the region 222, the region 223 in FIGS. 8A-8C.

[0028] The step of moving the first UI element can also be triggered by an internal event of the electronic device. For example, when the clock reaches the hour time, the clock component moves on the wallpaper image according to a preset trajectory or a random trajectory. In this example, the internal event is that the clock reaches the hour time. For another example, the weather component moves on the wallpaper image according to the visual motion trajectory (sunrise, sunset, inclination) of the sun. In this example, the internal event is the change of the visual motion trajectory of the sun. Embodiments of the present application do not limit what the internal event is.

[0029] With reference to the second aspect, in some embodiments, in the wallpaper image, the depth of the sixth object is greater than the depth of the fifth object. In the third region, the fifth object can be referred to as a foreground object, and the sixth object can be referred to as a background object. The fifth object can not be a significant subject in the wallpaper image. The fifth object and the sixth object can be part of a piece of the same type of object, and the fifth object does not constitute a significant subject relative to the sixth object, such as a tree in a piece of forest, grass in a piece of lawn, or wheat in a piece of wheat field, etc.

[0030] With reference to the second aspect, in some embodiments, before displaying the first user interface, the electronic device can perform a second layered rendering on the foreground fill of the second region, the first UI element, and the original wallpaper image to obtain the first user interface before updating. The specific implementation of the second layered rendering can include rendering the first UI element on the original wallpaper image and rendering the foreground fill of the second region on the first UI element. In this way, the first UI element can present a depth-of-field effect of being displayed between the third object and the fourth object in the second region.

[0031] With reference to the second aspect, in some embodiments, the foreground fill of the second region can be implemented based on an artificial intelligence model, such as inputting the original image of the second region and the mask image of the second region into a first artificial intelligence model to obtain a regenerated foreground of the second region. The first aspect has introduced how to perform foreground fill based on the first artificial intelligence model, which will not be repeated here.

[0032] In combination with the second aspect, in some embodiments, to enable the first UI element to present the depth-of-field effect in the third region, before updating the first user interface, the electronic device can perform first layered rendering on the supplemental foreground of the third region, the first UI element, and the original wallpaper image, to obtain an updated first user interface. The specific implementation of the first layered rendering can include: rendering the first UI element on top of the original wallpaper image, and rendering the supplemental foreground of the third region on top of the first UI element.

[0033] Since the foreground of the region for which the depth-of-field effect needs to be presented during the foreground supplementation is not cut out from the wallpaper image, but is generated based on the original image of the region and the foreground edge of the region, the interface display method of the second aspect can support constructing the depth-of-field effect for a UI element such as a clock component in an arbitrarily selected region, even if the region has no significant subject; and since the foreground is supplemented with the foreground edge of the region as a constraint, the transition effect between the foreground and the background is more natural, avoiding problems such as edge burrs.

[0034] In a third aspect, an electronic device is provided, which can include a processor and a memory; the memory is coupled to the processor, and the memory is configured to store a computer program; when the processor executes the computer program, the electronic device performs the method described in the first aspect or any possible implementation manner of the first aspect, or performs the method described in the second aspect or any possible implementation manner of the second aspect.

[0035] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program; when the computer program is executed by a processor, the method described in the first aspect or any possible implementation manner of the first aspect is implemented, or the method described in the second aspect or any possible implementation manner of the second aspect is performed.

[0036] In a fifth aspect, a computer program product is provided, which includes a computer program; when the computer program is executed by a processor, the method described in the first aspect or any possible implementation manner of the first aspect is implemented, or the method described in the second aspect or any possible implementation manner of the second aspect is performed. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 shows a depth-of-field effect construction function;

[0038] FIG. 2 shows an image processing framework on which the interface display method provided in the embodiments of the present application is based;

[0039] FIG. 3 shows layered rendering in the embodiments of the present application;

[0040] FIG. 4 shows the overall flow of an interface display method provided in the embodiments of the present application;

[0041] FIG. 5 shows an application flow of the interface display method provided by the embodiment of the present application in a lock screen wallpaper setting scenario;

[0042] FIG. 6 shows an artificial intelligence model for foreground filling in the embodiment of the present application;

[0043] FIG. 7 shows a specific implementation of the interface display method provided by the embodiment of the present application when the position of the first region changes;

[0044] FIGS. 8A to 8C exemplarily show technical effects of the embodiment of the present application applied in a lock screen wallpaper setting scenario;

[0045] FIG. 9 shows an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0046] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting to the present application.

[0047] To increase the interest of displaying UI elements on a wallpaper image, some simple depth-of-field effect construction functions are proposed. For example, as shown in FIG. 1, a clock component is displayed below a prominent subject such as a figure in the lock screen wallpaper, and the clock component is partially blocked by the figure to show the level of the wallpaper, constructing the depth-of-field relationship (i.e., the front-back relationship) between the figure and the clock component, making the lock screen interface more lively and having a 3D effect. However, this depth-of-field effect construction function can only be applied to subject objects of specific categories (such as figures, cats, dogs, etc.), and the wallpaper image in which the subject object is clearly visible, and the position at which the depth-of-field effect can be presented is fixed on the wallpaper image.

[0048] The embodiment of the present application provides an interface display method, which can construct the depth-of-field effect of a clock component, an application icon, or the like on a wallpaper image, and the position at which the UI element can present the depth-of-field effect is not fixed on the wallpaper image, supporting the user to move the UI element to select the position at which the depth-of-field effect is presented.

[0049] The interface display method provided by the embodiment of the present application can be based on the image processing framework shown in FIG. 2. This framework can be called a framework based on filling-in simulation of depth-of-field. As shown in FIG. 2, the framework can include two main steps of foreground filling and depth-of-field presentation:

[0050] 1. Foreground filling.

[0051] First, the region selection determines the region in the original image in which the clock component, the application icon, or the like needs to present the depth-of-field effect; then, the selected region is filled in the foreground.

[0052] The foreground inpainting is used to regenerate the foreground of the region, and the regenerated foreground (which can be referred to as an inpainting foreground) is the output result of the foreground inpainting. The regenerated foreground is not directly separated from the original image of the region, but is regenerated based on the original image of the region and the foreground edge. The foreground edge is a constraint condition of the foreground inpainting, that is, a foreground constraint in the image, and can also be referred to as a foreground contour. The foreground inpainting can be implemented based on an artificial intelligence model. The specific implementation of the foreground inpainting will be described in detail below, and will not be expanded here.

[0053] The original image in FIG. 2 can be a wallpaper image, and the region that needs to present the depth-of-field effect can be the first region, the second region, or the third region mentioned below. The region selection can be manually implemented. For example, the user places the clock component on which region of the wallpaper image, and the region is the region that needs to present the depth-of-field effect. The region selection can also be implemented by the electronic device based on internal events. The internal event can refer to the internal event mentioned below that triggers the mobile UI element, and the region where the UI element is located is the region that needs to present the depth-of-field effect.

[0054] 2. Depth-of-field presentation.

[0055] For the region that needs to present the depth-of-field effect, the inpainting foreground, the background, and the UI element are used for layered rendering to realize that the UI element presents the depth-of-field effect in the region. As shown in FIG. 3, the hierarchical relationship of the layered rendering can be as follows: the background is at the bottom layer, the inpainting foreground is at the upper layer, and the UI element is between the inpainting foreground and the background. That is, the UI element is rendered above the background layer, and the inpainting foreground is rendered above the UI element layer. In the result of the layered rendering, the depth-of-field relationship of the background, the inpainting foreground, and the UI element is: the depth of the UI element is less than the depth of the background and greater than the depth of the inpainting foreground.

[0056] When the layered rendering is performed, the size of the foreground layer and the background layer can be the same as the size of the original wallpaper image, wherein the content outside the inpainting foreground in the foreground layer can be set to be transparent, and the background layer can use the original wallpaper image. That is, the UI element and the foreground layer are layered and superimposed on the original wallpaper image.

[0057] Based on the image processing architecture shown in FIG. 2, embodiments of the present application can generate a foreground for any selected region, and then simulate the depth-of-field effect of the UI element in the region based on the foreground, even if the region has no significant subject. However, the existing depth-of-field effect simulation is based on monocular depth estimation technology, which can only estimate the depth of a specific category (such as a person, a cat, a dog, etc.) of significant foreground. Because the monocular depth estimation technology is usually implemented based on deep learning, it depends on a large-scale training data set and is limited by the training data set and can only be applicable to a certain or certain specific category of significant foreground.

[0058] Moreover, the original image of the selected region is used to fill the foreground of the selected region with the foreground edge as a constraint, which can better synthesize the foreground and the background, control the transition effect of the foreground and the background, and avoid edge burrs, missing and other problems. However, the traditional filling technology is to fill the image content of the foreground region by simply copying and pasting, and it itself does not have the ability to generate new image content. The foreground image filled back to the foreground region is cut out from the original image, and there are problems such as edge burrs, missing and the like.

[0059] FIG. 4 shows the overall flow of an interface display method provided by an embodiment of the present application. As shown in FIG. 4, the method can include the following steps:

[0060] S11. The electronic device acquires a wallpaper image.

[0061] The electronic device can be a mobile phone, a tablet computer, a smart screen, or the like, which has a display. The composition of the electronic device will be described in detail in subsequent embodiments, which will not be expanded here.

[0062] The electronic device can acquire the wallpaper image from the local gallery or the local system wallpaper based on the user selection operation, or can download the wallpaper image from the cloud server. The way in which the electronic device acquires the wallpaper image is not limited by the embodiments of the present application. The wallpaper image can also be automatically selected by the electronic device based on a preset mechanism, without manual selection by the user.

[0063] S12. The electronic device generates a foreground image of a first region based on an original image of the first region of the wallpaper image and a foreground edge of the first region.

[0064] The first region is the region where the first UI element (such as a clock component, an application icon, etc.) needs to present the depth-of-field effect, and it is also the display position of the first UI element on the wallpaper image.

[0065] The first region can be a region selected by the user, for example, the region where the user places the clock component on the wallpaper image. The first region can also not be selected by the user, but a default region set by the system, for example, the first region is the default display region of the first UI element each time the first user interface is opened. The position of the default region can be fixed, for example, the first UI element is always displayed at the same position each time the first user interface is opened. The position of the default region can also be variable, for example, the default display position of the first UI element is different when the first user interface is opened at different times.

[0066] S12 is foreground filling in the image processing framework shown in FIG. 2. Before performing S12, the electronic device can further crop the original image of the first region from the wallpaper image, and perform edge detection on the original image of the first region to obtain the foreground edge of the first region. Further, after the edge of the foreground region is extracted through edge detection, the electronic device can perform an inflation operation on the edge to expand the foreground edge to obtain an inflated foreground edge. The inflated foreground edge has been expanded to a region. In addition to the foreground region, the foreground filling can also cover the inflated foreground edge, that is, the foreground filling is performed on the foreground region and the inflated foreground edge to regenerate the foreground image. In this way, the region where the foreground filling needs to be performed at the foreground edge can be selected, and the edge burr can be improved. In the regenerated foreground, the inflated foreground edge can have a certain transparency. The transparency of the entire inflated foreground edge can be gradual, and the closer to the original foreground edge, the higher the transparency. The original foreground edge is the foreground edge before inflation. For example, in FIG. 5 later, the part of the tree branches in the filling foreground is more transparent closer to the region of the original edge, and can be better integrated with the background near the edge. In this way, the filling foreground and the original image background can be better integrated together to form a natural depth of field effect.

[0067] One specific implementation of S12 can include that the electronic device can input the original image of the first region and the foreground edge of the first region into a first artificial intelligence model to generate the foreground image of the first region.

[0068] The input of the first artificial intelligence model can include the original image of the first region and the foreground edge of the first region, and the output of the first artificial intelligence model can be the filling foreground of the first region. The training samples in the training data set of the first artificial intelligence model can include an input image, a foreground edge of the input image, and an output image, where the output image can be a foreground image of the input image. In addition, the first artificial intelligence model can be further configured with a loss function, which can be used to calculate the difference between the output image and the input image in the training sample, which can be composed of two parts: the image difference between the output image and the input image in the foreground, and the image difference between the output image and the input image in the foreground edge region. Wherein the weight of the image difference in the foreground can be greater than the weight of the image difference in the foreground edge region. That is, the foreground generated by the foreground filling needs to maintain high consistency with the foreground in the original image, while the foreground edge generated by the foreground filling can have some differences with the foreground edge in the original image. Here, the image difference can be counted by comparing the color values of each pixel.

[0069] The first artificial intelligence model for foreground filling will be described in detail later in conjunction with FIG. 5, which will not be expanded here.

[0070] S13. The electronic device obtains a first user interface according to the generated foreground image of the first region, the first UI element, and the wallpaper image.

[0071] Specifically, the electronic device can perform layered rendering on the generated foreground image of the first region, the first UI element, and the original wallpaper image to obtain the first user interface. The layered rendering can include rendering the first UI element on the original wallpaper image and rendering the foreground image of the first region on the first UI element. The original wallpaper image is at the bottom layer as a background layer, the foreground image is at the upper layer as a foreground layer, and the first UI element is at the middle layer. In this way, the first UI element can be presented in the first user interface with a depth-of-field effect that the first UI element is blocked by the foreground image of the first region.

[0072] In performing the layered rendering, the size of the foreground layer can be the same as the size of the first region or the size of the original wallpaper image. When the latter is used, the area outside the first region in the foreground layer can be set to be transparent.

[0073] S14. The electronic device displays the first user interface. The first user interface can include the wallpaper image and the first UI element, and the first UI element is blocked by the generated foreground image in the first region of the wallpaper image.

[0074] The first object is a foreground object in the first region. The first UI element is blocked by the first object, so that the first UI element presents a depth-of-field effect in the first region. Taking FIG. 8A as an example, the first user interface can be a lock screen interface 21, the first UI element can be a clock component 23, the first region can be a region 221, and the first object can be a mountain peak A. In this way, the clock component 23 can present a depth-of-field effect in the region 221 by being blocked by the mountain peak A.

[0075] In the first user interface, the first UI element can also block a second object in the background image of the first region, and the depth of the second object is greater than the depth of the first object. The second object is a background object in the first region. Taking FIG. 8A as an example, the second object can be a sky W in the region 221. In this way, the clock component 23 can present a depth-of-field effect of being displayed between the mountain peak A and the sky W by being blocked.

[0076] The interface display method described in S11-S14 can be applied to a wallpaper setting scenario, such as a lock screen wallpaper, a desktop wallpaper, and the like. S14 can be implemented to display a wallpaper preview interface after setting a lock wallpaper, and the first user interface is the wallpaper preview interface. In the lock screen wallpaper, the desktop wallpaper, and the like, S14 can also be implemented to display a lock screen interface, a desktop, an off-screen interface, and the like, and the first user interface can include one or more of the following: a lock screen interface, a desktop, and an off-screen interface.

[0077] FIG. 5 shows an application flow of the interface display method provided by the embodiments of the present application in a lock screen wallpaper setting scenario. Details are as follows:

[0078] 1. The user selects an image 21 to be set as a wallpaper.

[0079] The image 21 is a wallpaper image. The user can select the image 21 in a gallery or from a wallpaper image provided by the system, and the embodiments of the present application do not limit this.

[0080] 2. The user selects a display area of a clock component (such as “08:08”) on the wallpaper image 21.

[0081] In FIG. 5, the area in the rectangular dashed box 22 is the display area of the clock component “08:08”, and is also the area in which the clock component presents the depth-of-field effect. The user can drag the clock component on the wallpaper image to select the display area thereof.

[0082] 3. The original image of the selected area in step 2 is cropped from the wallpaper image, and edge detection is performed on the original image of the area to obtain a foreground edge.

[0083] The edge detection method may, for example, be a Sobel operator edge detection method, and is not limited thereto. Edge detection can also be achieved through other implementation manners. The result of edge detection can be represented by a mask image 33 of the area. The mask image 33 can be used to identify the foreground and background of the area, and thus identify the foreground edge. The mask image is one way to identify the foreground edge.

[0084] 4. Foreground inpainting is performed based on the original image of the area and the foreground edge to obtain an inpainted foreground.

[0085] The foreground inpainting can be implemented based on an artificial intelligence model. Details of the specific implementation of the foreground inpainting will be described below, and will not be expanded here.

[0086] 5. The clock component, the inpainted foreground, and the background are rendered in layers.

[0087] After layer rendering, the clock component can present a depth-of-field effect of being displayed between the foreground and the background at the area 22 of the wallpaper image 21.

[0088] 6. A lock screen wallpaper preview interface is displayed. As shown in FIG. 5, the lock screen wallpaper preview interface presents a depth-of-field effect of the clock component “08:08” being displayed in the middle of the trees in the wallpaper image, and the wallpaper effect is lively and interesting.

[0089] The interface display method described in S11-S14 above can also be applied to lock screen display, desktop display, screen-off display, and other formal wallpaper display scenarios. In such scenarios, the first user interface can include one or more of the following: a lock screen interface, a desktop, and a screen-off interface.

[0090] FIG. 6 shows a first artificial intelligence model for foreground inpainting.

[0091] As shown in FIG. 6, the first artificial intelligence model can be a deep learning end-to-end model, and the model structure can be a Transformer model structure. The input of the first artificial intelligence model can include: an original image of a selected region, an edge detection result of the selected region; and the output of the first artificial intelligence model can include: a regenerated foreground. The first artificial intelligence model can generate a foreground of the selected region based on the original image of the selected region, with the edge detection result of the selected region as a constraint condition. Here, the selected region can be a region determined by the aforementioned region selection to need to present a depth-of-field effect for the first UI element.

[0092] The overall design scheme of the model can be as follows:

[0093] 1. The original image of the selected region is subjected to image encoding by an encoding (Encoder) module to generate image embedding features. The image embedding features can be image high-level semantic features. The encoding (Encoder) module can be, for example, a UNet encoding module.

[0094] 2. The edge detection result (mask image) of the selected region is subjected to image encoding by an encoding (Encoder) module to generate mask image embedding features.

[0095] 3. The image embedding features and the mask image embedding features of the selected region are fused, for example, added or spliced, to obtain encoding information of the selected region.

[0096] 4. The encoding information of the selected region is subjected to image decoding (Decoder) to generate a foreground of the selected region. The decoding (Decoder) module can be, for example, a UNet decoding module.

[0097] The training samples in the training data set of the first artificial intelligence model can include: a first input, a second input, and an output, wherein the first input is an image, the second input is a mask image of the first input, and the output is also an image, and the foreground in the first input is used as the output.

[0098] The training data set of the first artificial intelligence model can be constructed as follows:

[0099] 1. A mask image of a certain original image is constructed by edge detection;

[0100] 2. The original image and its mask image are used as the first input and the second input in the training sample respectively

[0101] 3. The foreground image in the original image is used as the output in the training sample.

[0102] The loss function of the first artificial intelligence model can consider the foreground region and the foreground edge region at the same time. The loss function of the first artificial intelligence model can be used to calculate the image difference between the output and the first input, which can be composed of two parts: the image difference of the output and the first input on the foreground, and the image difference of the output and the first input on the foreground edge. Among them, the weight of the image difference on the foreground can be greater than the weight of the image difference on the edge region. That is, the foreground generated by the foreground fill image needs to maintain high consistency with the foreground in the original image, and the foreground edge generated by the foreground fill image can have some differences with the foreground edge in the original image.

[0103] The position of the first region in the wallpaper image is variable. The position of the first region in the wallpaper image can be changed based on user operations, which can include user operations of moving the first UI element on the wallpaper image. The change of the position of the first region in the wallpaper image can also be triggered by an internal event of the electronic device. For example, the position of the first region for displaying the clock component is changed according to a preset trajectory or a random trajectory when the clock reaches the hour time. In this example, the internal event is that the clock reaches the hour time. For another example, the position of the first region for displaying the weather component is changed according to the visual motion trajectory (sunrise, sunset, inclination) of the sun. In this example, the internal event is the change of the visual motion trajectory of the sun. The embodiments of the present application do not limit what the internal event is. The former way of changing the position of the first region can be applied to the wallpaper setting scene described above, and the latter way of changing the position of the first region can be applied to the formal wallpaper display scene described above.

[0104] The specific implementation of the embodiments of the present application when the position of the first region changes will be further illustrated below in combination with the interface display method shown in FIG. 7:

[0105] S21. The electronic device displays a first user interface, which can include a wallpaper image and a first UI element, the first UI element is displayed in a second region of the wallpaper image, the first UI element is blocked by a third object in the wallpaper image, and the first UI element blocks a fourth object in the wallpaper image. The third object and the fourth object are located in the second region of the wallpaper image.

[0106] The second region can be the first region at a certain time, i.e., the display position of the first UI element in the wallpaper image at a certain time. The position of the second region in the wallpaper image is different from the position of the third region in the wallpaper image hereinafter. The third region can be the first region at another time, i.e., the display region of the first UI element in the wallpaper image at another time.

[0107] Taking FIG. 8A as an example, the first user interface can be a lock screen interface 21, the first UI element can be a clock component 23, the second region can be a region 221, the third object can be a mountain peak A, and the fourth object can be a sky W in FIG. 8A. In this way, the clock component 23 can be more realistically presented as penetratingly displayed between the mountain peak A and the sky W by occlusion and occlusion.

[0108] The first UI element can also be a weather component, an application icon, a folder icon, and the like, without being limited to the clock component, where the application icon can include one or more of the following: an icon of an application program, an icon of an application function, or an icon of a system function. The first user interface can also be a desktop, an off-screen interface, and the like, without being limited to the lock screen interface.

[0109] In the wallpaper image, the depth of the fourth object is greater than the depth of the third object. In the second region, the third object can be referred to as a foreground object, and the fourth object can be referred to as a background object. The third object can not be a significant subject in the wallpaper image. The third object and the fourth object can be part of a piece of the same type of object, and the third object does not constitute a significant subject relative to the fourth object, such as a tree in a piece of forest, grass in a piece of lawn, or wheat in a piece of wheat field, and the like.

[0110] The significant subject can generally be segmented from the original image by a significant subject segmentation algorithm, and generally occupies a large area in the picture and has a clear segmentation edge.

[0111] Here, the third object in the second region occluding the first UI element can mean that the third object partially occludes the first UI element, and the first UI element occluding the fourth object in the second region can include that the first UI element partially or completely occludes the fourth object.

[0112] The second region can be a region selected by the user, such as the user dragging the first UI element to select where to display it. The second region can also be a region at a default position, such as the second region being a default display region of the first UI element each time the first user interface is opened by the user. The position of the default region can be unchangeable, such as the first UI element being displayed at the same position by default each time the first user interface is opened. The position of the default region can also be changeable, such as the default display position of the first UI element being different when the first user interface is opened at different times.

[0113] S22. The electronic device moves the first UI element to a third region of the wallpaper image.

[0114] The movement can be triggered by a user operation. For example, when detecting a user operation of dragging the first UI element from the second region to the third region, the electronic device can move the first UI element from the second region to the third region. The user operation can refer to the operation of dragging the clock component 23 to the region 222, the region 223 in FIGS. 8A-8C.

[0115] The movement can also be triggered by an internal event of the electronic device. For example, when the clock reaches the hour time, the clock component moves on the wallpaper image according to a preset trajectory or a random trajectory. In this example, the internal event is that the clock reaches the hour time. For another example, the weather component moves on the wallpaper image according to the visual motion trajectory (sunrise, sunset, inclination) of the sun. In this example, the internal event is that the visual motion trajectory of the sun changes. Embodiments of the present application do not limit what the internal event is.

[0116] S23. The electronic device updates the first user interface, in which the first UI element is displayed in the third region, the first UI element is occluded by a fifth object in the wallpaper image, and the first UI element occludes a sixth object in the wallpaper image. The fifth object and the sixth object are located in the third region of the wallpaper image. In this way, a depth-of-field relationship in which the first UI element is interposed between the fifth object and the sixth object can be constructed, so that the first UI element can present a depth-of-field effect when displayed at the third region.

[0117] Taking FIGS. 8A-8B as an example, the first user interface before the first UI element is moved is the user interface 21 shown in FIG. 8A, and the first user interface after the first UI element is moved can be the user interface 21 shown in FIG. 8B. The third region can be the region 222, the fifth object can be the tree B, the tree C, and the tree D, and the sixth object can be the tree E. In this way, the clock component 23 can be more realistically presented to be interposed and displayed between the tree B, the tree C, the tree D, and the tree E through occlusion and occlusion.

[0118] For another example, taking FIGS. 8B-8C as an example, the first user interface before the first UI element is moved is the user interface 21 shown in FIG. 8B, and the first user interface after the first UI element is moved can be the user interface 21 shown in FIG. 8C. The third region can be the region 223 in FIG. 8C, the fifth object can be the tree B, the tree F, and the tree G, and the sixth object can be the tree E. In this way, the clock component 23 can be more realistically presented to be interposed and displayed between the tree B, the tree F, the tree G, and the tree E through occlusion and occlusion.

[0119] In the wallpaper image, the depth of the sixth object is greater than the depth of the fifth object. In the third region, the fifth object can be referred to as a foreground object, and the sixth object can be referred to as a background object. The fifth object can not be a significant subject in the wallpaper image. The fifth object and the sixth object can be parts in a piece of the same type of object, and the fifth object does not constitute a significant subject relative to the sixth object, such as a tree in a piece of forest, or grass in a piece of lawn, or wheat in a piece of wheat field, and the like.

[0120] As can be seen from the overall flow shown in FIG. 7, the interface display method provided by the embodiments of the present application can present the depth-of-field effect of the UI element on the wallpaper image, and the position at which the depth-of-field effect is presented is not fixed. The user can move the UI element on the wallpaper image to experience the depth-of-field effect of the UI element at different positions, so as to present the 3D dynamic effect of the UI element moving in the wallpaper image through the movement, which is more vivid.

[0121] FIGS. 8A to 8C exemplarily show the technical effect of the embodiments of the present application applied to the lock screen wallpaper setting scene. As shown in FIGS. 8A to 8C, the lock screen interface 21 can include a wallpaper image 21 and a clock component 23, wherein the wallpaper image has objects such as sky, mountains, forest, river, and sun.

[0122] Initially, as shown in FIG. 8A, the clock component 23 is displayed in a region 221 in the wallpaper image 21, and the clock component 23 is specifically displayed below a mountain peak A in the region 221, that is, the clock component 23 is displayed in a blocked manner by the mountain peak A, thereby constructing a depth-of-field relationship that the mountain peak A is in front and the clock component 23 is behind, so that the clock component 23 presents the depth-of-field effect at the region 221.

[0123] Subsequently, as shown in FIG. 8B, the user drags the clock component 23 to move on the wallpaper image 21, and the clock component 23 moves from the region 221 to a region 222. The clock component 23 is displayed below trees B, C, and D in the region 222 but is displayed above a tree E in the region 222, that is, the clock component 23 is blocked by the trees B, C, and D, and the clock component 23 blocks the tree E, thereby constructing a depth-of-field relationship that the trees B, C, and D are in front, the tree E is behind, and the clock component 23 is interposed between the front trees B, C, and D and the back tree E, so that the clock component 23 also presents the depth-of-field effect at the region 222.

[0124] Then, as shown in FIG. 8C, when the user continues to drag the clock component 23 to move on the wallpaper image 21, the clock component 23 moves from the area 222 to the area 223, the clock component 23 is displayed below the trees B, F and G in the area 222 but above the tree E in the area 222, that is, the clock component 23 is occluded by the trees B, F and G, and the clock component 23 occludes the tree E, thereby constructing a depth-of-field relationship that the trees B, F and G are in front and the tree E is behind, and the clock component 23 is interposed between the front trees B, F and G and the back tree E, so that the clock component 23 also presents a 3D effect at the area 223.

[0125] In summary, FIGS. 8A-8C, the user drags the clock component 23 to move to each new position, and the depth-of-field effect of the clock component 23 at the new position can be observed, and this dynamic process can present a 3D dynamic effect that the clock component 23 shuttles through the forest, which is more lively, and the spatial hierarchy of the wallpaper image can be more comprehensively felt from the depth-of-field effect of the clock component 23 at multiple positions, so that the 3D effect is more realistic.

[0126] In order to enable the first UI element to present a depth-of-field effect in the second area, before the foregoing S21, the electronic device can perform second layered rendering on the supplementary foreground of the second area, the first UI element and the original wallpaper image, to obtain the first user interface before updating. The specific implementation of the second layered rendering can include: rendering the first UI element on the original wallpaper image, and rendering the supplementary foreground of the second area on the first UI element.

[0127] Referring to the image processing framework shown in FIG. 2, the original image of the second area can be cropped from the wallpaper image, and the foreground edge of the second area can be obtained by performing edge detection on the original image of the second area. The foreground edge of the second area can be indicated by a mask image of the second area. The mask image of the second area can be used to identify the foreground and the background of the second area, and thus can serve as an indication of the foreground edge.

[0128] The foreground supplementary image of the second area can be implemented based on an artificial intelligence model, such as: inputting the original image of the second area and the mask image of the second area into a first artificial intelligence model to obtain a regenerated foreground of the second area. How to implement the foreground supplementary image based on the first artificial intelligence model has been described above, and will not be repeated here.

[0129] Similarly, in order to enable the first UI element to present a depth-of-field effect in the third area, before the foregoing S23, the electronic device can perform first layered rendering on the supplementary foreground of the third area, the first UI element and the original wallpaper image, to obtain the first user interface after updating. The specific implementation of the first layered rendering can include: rendering the first UI element on the original wallpaper image, and rendering the supplementary foreground of the third area on the first UI element.

[0130] Referring to the image processing framework shown in FIG. 2, the original image of the third region can be cropped from the wallpaper image, and the foreground edge of the third region can be obtained by performing edge detection on the original image of the third region. The foreground edge of the third region can be indicated by the mask image of the third region. The mask image of the third region can be used to identify the foreground and the background of the third region, and thus can serve as an indication of the foreground edge.

[0131] The foreground inpainting of the third region can also be implemented based on an artificial intelligence model. For example, the original image of the third region and the mask image of the third region can be input into a first artificial intelligence model to obtain a regenerated foreground of the third region. How to perform foreground inpainting based on the first artificial intelligence model has been described above, and thus will not be described again here.

[0132] As can be seen, since the foreground of the region that needs to present the depth-of-field effect during foreground inpainting is not cut out from the wallpaper image, but is generated based on the original image of the region and the foreground edge of the region, the embodiments of the present application can support constructing a depth-of-field effect for a UI element such as a clock component in an arbitrarily selected region, even if the region does not have a significant subject. Moreover, since the foreground edge of the region is used as a constraint condition during foreground inpainting, the transition effect between the foreground and the background is more natural, and problems such as edge burrs are avoided.

[0133] FIG. 9 exemplarily shows an electronic device 300 provided by an embodiment of the present application.

[0134] The electronic device 300 can have both human-computer interaction capability and computing capability. The device type of the electronic device 300 can be any one of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and a smart large-screen smart home device, a smart watch, a smart glass wearable device, an augmented reality (AR), a virtual reality (VR), a mixed reality (MR), an extended reality (XR) device, a vehicle-mounted device or a smart city device, and the like.

[0135] As shown in FIG. 9, the electronic device 300 can include a processor 110, a memory 120, a display 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 can include a gyroscope sensor 180B, an acceleration sensor 180E, a touch sensor 180K, and the like. The various parts in the electronic device 300 can be connected through a bus.

[0136] The processor 110 can be responsible for providing computing power and can be used as a computing module of the electronic device 300. The display 130, the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, the sensor module 180, the key 190, the motor 191, the indicator 192, the camera 193, and the like can be responsible for providing human-computer interaction capability and can be used as a human-computer interaction module of the electronic device 300. When the computing module in the electronic device 300 has strong computing power, the electronic device 300 can independently execute the interface display method provided in the embodiments of the present application. When the computing module in the electronic device 300 does not have strong computing power, the electronic device 300 can also only execute the human-computer interaction steps in the interface display method provided in the embodiments of the present application, such as displaying the first user interface, moving the first UI element on the wallpaper image, and the like, while the image processing steps such as foreground filling, layered rendering, and the like in the method can be executed by the cloud side server.

[0137] The processor 110 can be one or more, which can be integrated in an integrated circuit of a system on chip (SOC). The SOC is a system-level chip. The processor 110 can include a central processing unit (CPU), a graphic processing unit (GPU), a neural-network processing unit (NPU), etc. Among them, the CPU can include an application processor (AP), a baseband processor (BP), etc., where the AP can be responsible for running the operating system, user interface, and application program on the electronic device 300; the BP can be responsible for transmitting and receiving wireless signals and managing radio frequency services. The GPU can be responsible for graphics rendering, shading, material filling, rendering, output, etc. according to the rendering instructions and data from the CPU. The NPU can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between human brain neurons, and can also continuously self-learn. The NPU can be used to run artificial intelligence algorithms, such as editing instruction recommendation algorithms, image processing algorithms, image understanding algorithms, etc. The CPU and GPU can be used to render and synthesize the picture to be sent to the display 130.

[0138] The processor 110 can include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0139] The processor 110 can be provided with a cache memory, which can be used to save instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the efficiency of program running.

[0140] The memory 120 can include a program storage area and a user data storage area. The program storage area can store an operating system and one or more application programs (e.g., a game application), and the data storage area can store data (e.g., photos, contacts) created by a user during use of the electronic device 300. The memory 120 can be a high-speed random access memory, or a nonvolatile memory such as a magnetic disk, a flash memory, a universal flash storage (UFS), or the like. The memory 120 can also be an external memory card such as a Micro SD card.

[0141] The memory 120 can also store code instructions of the picture editing method provided in the embodiments of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the electronic device 300 can perform the steps performed by the human-computer interaction module and / or the computing module in the picture editing method provided in the embodiments of the present application.

[0142] The memory 120 can also be integrated with the processor 110 in an integrated circuit of a SOC.

[0143] As shown in FIG. 9, the electronic device 300 can implement a display function through a SOC, a DDIC 140, and a display 130.

[0144] The display 130 has a plurality of refresh rates. A refresh rate indicates the number of times a display screen refreshes a display picture in 1 second. For example, a 60-hertz (Hz) refresh rate indicates that the display screen refreshes a display picture 60 times in 1 second. The display 130 can use an LTPO display panel, which allows the refresh rate to be reduced to a low refresh rate such as 10 Hz or 1 Hz, thereby supporting reduction of power consumption of the display screen.

[0145] The display driving integrated circuit (DDIC) 140 can be used as a control core of the display 130, drive the display 130 to work, and receive data such as image data and some instructions from the SOC (processor 110). The DDIC 140 can send driving signals and data to the display panel of the display 130 in the form of electrical signals, thereby controlling the screen brightness and color, making image information such as letters and pictures appear on the screen, and completing screen refresh.

[0146] The image data of a picture to be displayed sent by the SOC to the DDIC 140 can be stored in a frame buffer to complete display (or image sending). Then, the DDIC 140 takes out the image data from the frame buffer and drives the display 130 to display.

[0147] The wireless communication function of the electronic device 300 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0148] The antenna 1 and the antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0149] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device 300. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert the signals into electromagnetic waves radiated by the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be arranged in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be arranged in the same device.

[0150] The modem processor can include a modulator and a demodulator. The modulator is used to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the loudspeaker 170A, the microphone 170B, etc.), or displays an image or a video through the display 130. In some embodiments, the modem processor can be an independent device. In some other embodiments, the modem processor can be independent of the processor 110, and arranged in the same device as the mobile communication module 150 or other functional modules.

[0151] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device 300. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, frequency-modulates and filters the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, frequency-modulate it, amplify it, and radiate it as an electromagnetic wave via the antenna 2.

[0152] In some embodiments, antenna 1 and mobile communication module 150 of electronic device 300 are coupled, and antenna 2 and wireless communication module 160 are coupled, so that electronic device 300 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0153] Electronic device 300 can implement a photographing function through an ISP, camera 193, a video codec, a GPU, display 130, and an application processor, etc.

[0154] The ISP is used to process data fed back by camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into a visible image. The ISP can also optimize algorithms for noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in camera 193.

[0155] The camera 193 is used to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in RGB, YUV, or the like. In some embodiments, the electronic device 300 can include one or N cameras 193, where N is a positive integer greater than one.

[0156] A video codec is used to compress or decompress digital videos. The electronic device 300 can support one or more video codecs. In this way, the electronic device 300 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0157] The electronic device 300 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, and the like. For example, music playing, voice recording, and the like.

[0158] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0159] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 300 can listen to music or listen to a hands-free call through the speaker 170A.

[0160] The receiver 170B, also known as a "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 300 answers a call or a voice message, the receiver 170B can be held close to the ear to listen to the voice.

[0161] Microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak into the microphone 170C by the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The electronic device 300 can be provided with at least one microphone 170C. In other embodiments, the electronic device 300 can be provided with two microphones 170C, in addition to collecting sound signals, it can also realize the noise reduction function. In other embodiments, the electronic device 300 can also be provided with three, four or more microphones 170C, to realize the collection of sound signals, noise reduction, and can also identify the sound source, realize the directional recording function, etc.

[0162] The earphone interface 170D is used to connect the wired earphone. The earphone interface 170D can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0163] The key 190 includes a power key, a volume key, etc. The key 190 can be a mechanical key, or a touch key. The electronic device 300 can receive a key input, and generate a key signal input related to the user settings and function control of the electronic device 300. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195, to realize contact and separation with the electronic device 300.

[0164] The structure shown in FIG. 9 does not constitute a specific limitation on the electronic device 300. The electronic device 300 can include more or fewer parts than shown, or combine certain parts, or split certain parts, or different part arrangements. The various parts shown can be implemented in hardware, software, or a combination of software and hardware.

[0165] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the steps executed by the human-computer interaction module in each method embodiment, or the steps executed by the human-computer interaction module and the calculation module.

[0166] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the steps executed by the human-computer interaction module in each method embodiment, or the steps executed by the human-computer interaction module and the calculation module.

[0167] The embodiments of the present application further provide a computer program product, when the computer program product is run on a terminal device, the terminal device can implement the steps performed by the man-machine interaction module, or the steps performed by the man-machine interaction module and the calculation module in each method embodiment.

[0168] The embodiments of the present application further provide a computer program product, when the computer program product is run on a server, the server can implement the steps performed by the calculation module in each method embodiment.

[0169] The embodiments of the present application further provide a chip system, the chip system includes a processor, the processor is coupled with a memory, the processor executes a computer program stored in the memory, to implement the steps performed by the man-machine interaction module, or the steps performed by the man-machine interaction module and the calculation module in any method embodiment of the present application. The chip system can be a single chip, or a chip module composed of multiple chips.

[0170] The embodiments of the present application further provide a chip system, the chip system includes a processor, the processor is coupled with a memory, the processor executes a computer program stored in the memory, to implement the steps performed by the calculation module in any method embodiment of the present application. The chip system can be a single chip, or a chip module composed of multiple chips.

[0171] The term "user interface (UI), simply interface" in the specification and drawings of the present application is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The user interface of the application program is the source code written by a specific computer language such as java and extensible markup language (XML), and the interface source code is parsed, rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, texts, buttons and other controls. The control (widget) is the basic element of the user interface, and typical controls include toolbar, menu bar, text box, button, scrollbar, picture and text. The properties and content of the controls in the interface are defined by tags or nodes, such as XML <textview> 、 <imgview> 、

[0172] <videoview>The interface is defined by nodes that specify the controls contained in the interface. One node corresponds to one control or property in the interface, and the nodes are parsed and rendered to present the content visible to the user. In addition, many applications, such as hybrid applications, also contain web pages in the interface. A web page, also referred to as a page, can be understood as a special control embedded in the interface of an application. The web page is a source code written in a specific computer language, such as hyper text markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed by a browser or a web page display component similar to the function of a browser to present content recognizable to the user. The specific content contained in the web page is also defined by tags or nodes in the web page source code, such as HTML defines the content of the web page by tags such as 、 、 <video> 、 <canvas>to define the elements and attributes of a web page.

[0173] A common form of user interface is a graphic user interface (GUI), which refers to a user interface that displays in a graphical manner. It can be an icon, window, control, etc. interface element displayed in the display screen of an electronic device, wherein the control can include an icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc. visual interface element.

[0174] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state disk), etc.

[0175] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware to complete, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various program code storage media.

[0176] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An interface display method characterized by, The method comprises: obtaining a wallpaper image; generating a foreground image of a first region of the wallpaper image based on an original image of the first region of the wallpaper image and a foreground edge of the first region; obtaining a first user interface according to the generated foreground image of the first region, a first UI element, and the wallpaper image; displaying the first user interface, the first user interface comprising the wallpaper image and the first UI element, the first UI element being displayed in the first region of the wallpaper image and being occluded by the generated foreground image.

2. The method of claim 1, wherein, Further comprising: performing edge detection on the original image of the first region to obtain the foreground edge of the first region.

3. The method of claim 1 or 2, wherein, The generating of the foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region comprises: inputting the original image of the first region and the foreground edge of the first region into a first artificial intelligence model to generate the foreground image of the first region.

4. The method of claim 3, wherein, The training sample in the training data set of the first artificial intelligence model comprises: a first input image, a foreground edge of the first input image, and an output image, wherein the output image is a foreground image of the first input image.

5. The method of claim 3 or 4, wherein, The loss function of the first artificial intelligence model is used to calculate the difference between the output image and the first input image, the difference comprising: image difference of the output image and the first input image on the foreground, and image difference of the output image and the first input image on the foreground edge region; wherein the weight of the image difference on the foreground is greater than the weight of the image difference on the foreground edge region.

6. The method of any one of claims 1-5, wherein, Before the generating of the foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region, further comprising: performing inflation processing on the foreground edge of the first region to obtain an inflated foreground edge.

7. The method of claim 6, wherein, The inflated foreground edge is transparent, and the transparency is gradual, the closer to the foreground edge before inflation, the higher the transparency.

8. The method of any one of claims 1-7, wherein, In the first user interface, the first UI element occludes a second object in the background image of the first region; in the wallpaper image, the depth of the second object is greater than the depth of a first object, the first object being an object in the foreground image of the first region.

9. The method of any one of claims 1-8, wherein, The obtaining of the first user interface according to the generated foreground image of the first region, the first UI element, and the wallpaper image comprises: performing layered rendering on the generated foreground image of the first region, the first UI element, and the wallpaper image to obtain the first user interface; the layered rendering comprises: rendering the foreground image of the first region above the first UI element, and rendering the first UI element above the wallpaper image.

10. The method of any one of claims 1-9, wherein, The position of the first region in the wallpaper image is variable.

11. The method of claim 10, wherein, The position of the first region in the wallpaper image is changed based on a user operation, the user operation comprising a user operation of moving the first UI element on the wallpaper image.

12. The method of any one of claims 1-11, wherein, The first user interface includes one or more of the following: a wallpaper setting preview interface, a lock screen interface, a desktop, an off-screen interface.

13. The method of any one of claims 1-12, wherein, The first UI element includes one or more of the following: a clock component, a weather component, an application icon, a folder icon.

14. An interface display method characterized by comprising: Comprise: Display a first user interface, the first user interface includes a wallpaper image and a first UI element, the first UI element is displayed in a second region of the wallpaper image, the first UI element is blocked by a third object in the wallpaper image, the first UI element blocks a fourth object in the wallpaper image, the third object, the fourth object is located in the second region of the wallpaper image; Move the first UI element to a third region of the wallpaper image; Update the first user interface, and in the updated first user interface, the first UI element is displayed in the third region, the first UI element is blocked by a fifth object in the wallpaper image, the first UI element blocks a sixth object in the wallpaper image, the fifth object, the sixth object is located in the third region of the wallpaper image.

15. The method of claim 14, wherein, In the wallpaper image, the depth of the third object is less than the depth of the fourth object, and the depth of the fifth object is less than the depth of the sixth object.

16. The method of any one of claims 14-15, wherein, Before updating the first user interface, further comprising: performing first layered rendering on the foreground image of the third region, the first UI element and the wallpaper image to obtain the updated first user interface; the first layered rendering comprises: rendering the first UI element on the wallpaper image, and rendering the foreground image of the third region on the first UI element; wherein the foreground image of the third region comprises the fifth object.

17. The method of any one of claims 14-16, wherein, The foreground image of the third region is generated based on the original image of the third region and the foreground edge of the third region, and the foreground image of the third region comprises the fifth object.

18. The method of claim 17, wherein, Further comprising: perform edge detection on the original image of the third region to obtain the foreground edge of the third region; input the original image of the third region and the foreground edge of the third region into a first artificial intelligence model to generate the foreground image of the third region.

19. The method of any one of claims 17-18, wherein, Further comprising: Before regenerating the foreground image of the third region based on the original image of the third region and the foreground edge of the third region, performing inflation processing on the foreground edge of the third region to obtain the inflated foreground edge of the third region.

20. The method of claim 19, wherein, The inflated foreground edge of the third region is transparent, and the transparency is gradual, that is, the closer to the foreground edge before inflation, the higher the transparency.

21. The method of any one of claims 14-20, wherein, Before displaying the first user interface, further comprising: performing second layered rendering on the foreground image of the second region, the first UI element and the wallpaper image to obtain the first user interface before updating; the second layered rendering comprises: rendering the first UI element on the wallpaper image, and rendering the foreground image of the second region on the first UI element; wherein the foreground image of the second region comprises the third object.

22. The method of any one of claims 14-21, wherein, The foreground image of the second region is generated based on the original image of the second region and the foreground edge of the second region, and the foreground image of the second region includes the third object.

23. The method of claim 22, wherein, Further comprising: performing edge detection on the original image of the second region to obtain the foreground edge of the second region; inputting the original image of the second region and the foreground edge of the second region into the first artificial intelligence model to generate the foreground image of the second region.

24. The method of any one of claims 22-23, wherein, Further comprising: performing dilation processing on the foreground edge of the second region before regenerating the foreground of the second region based on the original image of the second region and the foreground edge of the second region.

25. The method of claim 24, wherein, The dilated foreground edge of the second region is transparent, and the transparency is gradual, that is, the closer to the foreground edge before dilation, the higher the transparency.

26. The method of claim 18 or 23, wherein, The training sample in the training data set of the first artificial intelligence model includes: a first input image, a foreground edge of the first input image, and an output image, wherein the output image is a foreground image of the first input image.

27. The method of claim 26, wherein, The loss function of the first artificial intelligence model is used to calculate the difference between the output image and the first input image, and the difference includes: the image difference between the output image and the first input image in the foreground, and the image difference between the output image and the first input image in the foreground edge region; wherein the weight of the image difference in the foreground is greater than the weight of the image difference in the foreground edge region.

28. The method of any one of claims 14-27, wherein, The third object and the fifth object are not significant subjects in the wallpaper image.

29. The method of any one of claims 14-28, wherein, The first user interface includes one or more of the following: a wallpaper setting preview interface, a lock screen interface, a desktop, and an off-screen interface.

30. The method of any one of claims 14-29, wherein, The first UI element includes one or more of the following: a clock component, a weather component, an application icon, and a folder icon.

31. An electronic device, comprising: including: a processor, a memory, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method of any one of claims 1-30.

32. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-30.

33. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-30.

Images