Interface display method and electronic equipment

Abstract

The embodiment of the invention provides an interface display method and electronic equipment, which can construct a depth-of-field relationship between UI elements such as a clock component and an application icon and objects in a wallpaper image, and present a 3D visual effect that the UI elements are alternately displayed among different objects in the wallpaper image. Moreover, according to the embodiment of the invention, the user is supported to drag the UI element to move on the wallpaper image, so that the UI element can present a 3D visual effect of alternate display at any position on the wallpaper image, and the interestingness of displaying the UI element on the wallpaper image is improved.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to interface display methods and electronic devices. Background Technology

[0002] Wallpaper images are commonly used as backgrounds for app icons, clock widgets, weather widgets, and other user interface (UI) elements. They enhance the user interface and satisfy users' personalized needs for its appearance. Typically, the rendering hierarchy of these UI elements and the wallpaper image is fixed; the wallpaper image is rendered at the bottom layer, and the UI elements at the top layer, allowing UI elements like clock widgets and app icons to appear on top of the wallpaper image for easy viewing. However, to some extent, this display effect lacks visual appeal. Summary of the Invention

[0003] This application provides an interface display method and an electronic device that can construct a depth relationship between UI elements such as clock components and application icons and objects in a wallpaper image, presenting a 3D visual effect of UI elements interspersed among different objects in the wallpaper image. Furthermore, this application supports users dragging UI elements across the wallpaper image, enabling UI elements to be displayed in an interspersed 3D visual effect at any position on the wallpaper image, thus enhancing the enjoyment of displaying UI elements on the wallpaper image.

[0004] In a first aspect, embodiments of this application provide an interface display method, which may include: an electronic device acquiring a wallpaper image; generating a foreground image of the first region based on an original image of a first region of the wallpaper image and the foreground edge of the first region; and then obtaining a first user interface based on the generated foreground image of the first region, a first UI element, and the wallpaper image. Finally, the electronic device may display the first user interface. The first user interface may include the wallpaper image and the first UI element, wherein the first UI element is displayed in the first region of the wallpaper image and is occluded by a first object in the generated foreground image.

[0005] In the first aspect, the first object is the foreground object in the first area. The first UI element is occluded by the first object, causing the first UI element to exhibit a depth-of-field effect in the first area. The first area is the area where the first UI element (such as a clock component, application icon, etc.) needs to exhibit a depth-of-field effect; it is also the display position of the first UI element on the wallpaper image. The first area can be a user-selected area, for example, the area on the wallpaper image where the user places the clock component is the first area. The first area may also not be user-selected, but rather a system-set default area, for example, the first area is the default display area of ​​the first UI element every time the user opens the first user interface. The position of this default area can be constant, for example, the first UI element is always displayed in the same position every time the first user interface is opened. The position of this default area can also change, for example, the default display position of the first UI element is different when the first user interface is opened at different times.

[0006] The interface display method provided in the first aspect can support the first UI element to present a depth effect on the wallpaper image, which is occluded by the first object in the foreground image of the first area. The foreground image of the first area is not extracted from the original wallpaper image, but generated based on the original image and the foreground edge. It can support the construction of depth effect for UI elements such as clock components in areas without significant subjects, and the transition effect between the foreground and the background is more natural.

[0007] In conjunction with the first aspect, in some embodiments, before generating a foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region, the electronic device may also crop the original image of the first region from the wallpaper image and perform edge detection on the original image of the first region to obtain the foreground edge of the first region.

[0008] Furthermore, after extracting the edges of the foreground region through edge detection, the electronic device can perform a dilation operation on these edges to expand the foreground edge, resulting in a dilated foreground edge. In addition to the foreground region, the foreground overlay can also encompass the dilated foreground edge; that is, a foreground overlay is performed on both the foreground region and the dilated foreground edge to regenerate the foreground image. This allows for the selection of areas at the foreground edge that require foreground overlay, improving edge sharpness. The dilated foreground edge can have a certain degree of transparency. The transparency of the entire dilated foreground edge can be gradual, with higher transparency closer to the original foreground edge. The original foreground edge is the foreground edge before dilation. In this way, the overlaid foreground image and the original background can blend better, creating a natural depth-of-field effect.

[0009] In conjunction with the first aspect, in some embodiments, a specific implementation of generating a foreground image of a first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region may include: the electronic device may input the original image of the first region and the foreground edge of the first region into a first artificial intelligence model to generate a foreground image of the first region.

[0010] The input to the first artificial intelligence model may include the original image of the first region and the foreground edges of the first region, and the output of the first artificial intelligence model may be the foreground of the complement image of the first region. The training samples in the training dataset of the first artificial intelligence model may include: the input image, the foreground edges of the input image, and the output image, wherein the output image may be the foreground image of the input image.

[0011] In conjunction with the first aspect, in some embodiments, the first artificial intelligence model may also be configured with a loss function, which can be used to calculate the difference between the output image and the input image in the training samples. This difference may consist of two items: the image difference between the output image and the input image in the foreground, and the image difference between the output image and the input image in the foreground edge region. The weight of the image difference in the foreground may be greater than the weight of the image difference in the foreground edge region. That is, the foreground generated by the foreground complement needs to maintain a high degree of consistency with the foreground in the original image, while the foreground edges generated by the foreground complement may differ somewhat from the foreground edges in the original image.

[0012] In conjunction with the first aspect, in some embodiments, the specific implementation of obtaining the first user interface based on the generated foreground image of the first region, the first UI element, and the wallpaper image may include: the electronic device may perform layered rendering of the generated foreground image of the first region, the first UI element, and the original wallpaper image to obtain the first user interface. This layered rendering may include: rendering the first UI element on top of the original wallpaper image, and rendering the foreground image of the first region on top of the first UI element. The original wallpaper image serves as the background layer at the bottom, the foreground image serves as the foreground layer at the top, and the first UI element is in the middle layer. In this way, a depth-of-field effect can be presented in the first user interface, showing the first UI element being occluded by the foreground image of the first region.

[0013] When performing this layered rendering, the size of the foreground layer can be the same as the size of the first area, or the same as the size of the original wallpaper image. When using the latter, the area outside the first area in the foreground layer can be set to transparent.

[0014] In conjunction with the first aspect, in some embodiments, in the first user interface, the first UI element may also occlude a second object in the background image of the first area, the second object having a greater depth than the first object. The second object is a background object in the first area. Thus, through occlusion and being occluded, a depth-of-field effect can be achieved where the first UI element is interspersed between the first and second objects.

[0015] In conjunction with the first aspect, in some embodiments, the first user interface may include one or more of the following: a wallpaper setting preview interface, a lock screen interface, a desktop, and an always-on display interface. In the wallpaper setting scenario, the first user interface can be a wallpaper setting preview interface, or it can be the interface for officially displaying the wallpaper after the wallpaper setting is completed, such as a lock screen interface, desktop, or always-on display interface. In the official wallpaper display scenario, the first user interface can be a lock screen interface, desktop, always-on display interface, etc.

[0016] In conjunction with the first aspect, in some embodiments, the first UI element may include one or more of the following: a clock component, a weather component, an application icon, and a folder icon.

[0017] In conjunction with the first aspect, in some embodiments, the position of the first area within the wallpaper image is variable. The position of the first area within the wallpaper image can be changed based on user actions, which may include moving a first UI element on the wallpaper image. The change in the position of the first area within the wallpaper image can also be triggered by internal events of the electronic device. For example, when the clock reaches the hour, the position of the first area used to display the clock component is changed according to a preset or random trajectory. In this example, the internal event is the clock reaching the hour. As another example, the position of the first area used to display the weather component is changed according to the apparent motion trajectory of the sun (sunrise, sunset, tilt). In this example, the internal event is the change in the apparent motion trajectory of the sun. This application does not limit what the internal event is. The former method of changing the position of the first area can be applied to the aforementioned wallpaper setting scenario, and the latter method of changing the position of the first area can be applied to the aforementioned formal wallpaper display scenario.

[0018] Secondly, embodiments of this application provide an interface display method, which may include the following steps:

[0019] Step 1. The electronic device displays a first user interface, which may include a wallpaper image and a first UI element. The first UI element is displayed in a second area of ​​the wallpaper image. The first UI element is obscured by a third object in the wallpaper image, and the first UI element obscures a fourth object in the wallpaper image. The third and fourth objects are located in the second area of ​​the wallpaper image.

[0020] Step 2. The electronic device moves the first UI element to the third area of ​​the wallpaper image.

[0021] Step 3. The electronic device updates the first user interface. In the updated first user interface, the first UI element is displayed in the third area. The first UI element is obscured by the fifth object in the wallpaper image, and the first UI element also obscures the sixth object in the wallpaper image. The fifth and sixth objects are located in the third area of ​​the wallpaper image. In this way, a depth-of-field relationship can be constructed where the first UI element is interspersed between the fifth and sixth objects, so that the first UI element can present a depth-of-field effect when displayed in the third area.

[0022] Secondly, the second region can be the first region at a certain time (before the first UI element is moved), that is, the display position of the first UI element in the wallpaper image at that time. The position of the second region in the wallpaper image is different from the position of the third region in the wallpaper image as described later. The third region can be the first region at another time (after the first UI element is moved), that is, the display area of ​​the first UI element in the wallpaper image at another time.

[0023] The second method can present the depth-of-field effect of UI elements on the wallpaper image, and the position that can present the depth-of-field effect is not fixed. Users can move UI elements on the wallpaper image to experience the depth-of-field effect of UI elements at different positions. Thus, the movement can present a 3D dynamic effect of UI elements moving through the wallpaper image, which is more vivid.

[0024] In conjunction with the second aspect, in some embodiments, the depth of the fourth object in the wallpaper image is greater than the depth of the third object. In the second region, the third object can be referred to as a foreground object, and the fourth object as a background object. The third object may not be a prominent subject in the wallpaper image. The third and fourth objects can be parts of a group of objects of the same type, where the third object does not constitute a prominent subject relative to the fourth object, such as trees in a forest, grass in a lawn, or wheat in a wheat field, etc.

[0025] In conjunction with the second aspect, in some embodiments, the second region can be a user-selected region, such as where the user chooses to display the first UI element by dragging it. The second region can also be a region at a default location, for example, the default display area of ​​the first UI element each time the user opens the first user interface. The position of this default region can be constant, for example, the first UI element is always displayed in the same position each time the first user interface is opened. The position of this default region can also be variable, for example, the default display position of the first UI element is different when the first user interface is opened at different times.

[0026] In conjunction with the second aspect, in some embodiments, the step of moving the first UI element can be triggered by a user action. For example, when a user action of dragging the first UI element from the second area to the third area is detected, the electronic device can move the first UI element from the second area to the third area. This user action can be referred to... Figures 8A to 8C The operation of dragging the clock component 23 to area 222 and area 223.

[0027] The step of moving the first UI element can also be triggered by internal events of the electronic device. For example, when the clock reaches the hour, the clock component moves on the wallpaper image according to a preset or random trajectory. In this example, the internal event is the clock reaching the hour. As another example, the weather component moves on the wallpaper image based on the apparent motion trajectory of the sun (sunrise, sunset, tilt). In this example, the internal event is the change in the apparent motion trajectory of the sun. This application does not limit what the internal event is.

[0028] In conjunction with the second aspect, in some embodiments, the depth of the sixth object in the wallpaper image is greater than the depth of the fifth object. In the third region, the fifth object can be referred to as a foreground object, and the sixth object as a background object. The fifth object may not be a prominent subject in the wallpaper image. The fifth and sixth objects can be parts of a group of objects of the same type, where the fifth object does not constitute a prominent subject relative to the sixth object, such as trees in a forest, grass in a lawn, or wheat in a wheat field, etc.

[0029] In conjunction with the second aspect, in some embodiments, before displaying the first user interface, the electronic device can perform a second layered rendering of the foreground of the second region, the first UI element, and the original wallpaper image to obtain the first user interface before the update. Specific implementations of the second layered rendering may include: rendering the first UI element on top of the original wallpaper image, and rendering the foreground of the second region on top of the first UI element. In this way, the first UI element can present a depth-of-field effect in the second region, interspersed among the third and fourth objects.

[0030] In conjunction with the second aspect, in some embodiments, the foreground complementation of the second region can be implemented based on an artificial intelligence model. For example, the original image of the second region and the mask image of the second region are input into a first artificial intelligence model to obtain a regenerated foreground of the second region. The first aspect has already described how to perform foreground complementation based on the first artificial intelligence model, and will not be repeated here.

[0031] In conjunction with the second aspect, in some embodiments, in order to achieve a depth-of-field effect for the first UI element in the third region, the electronic device may perform a first layered rendering of the foreground of the supplementary image in the third region, the first UI element, and the original wallpaper image before updating the first user interface, thereby obtaining the updated first user interface. Specific implementations of the first layered rendering may include: rendering the first UI element on top of the original wallpaper image, and rendering the foreground of the supplementary image in the third region on top of the first UI element.

[0032] Since the foreground of the area that needs to present a depth-of-field effect during foreground overlay is not extracted from the wallpaper image, but generated based on the original image of the area and the foreground edge of the area, the second aspect of the interface display method can support the construction of depth-of-field effects for UI elements such as clock components in any selected area, even if the area does not have a prominent subject; moreover, since the foreground overlay is constrained by the foreground edge of the area, the transition effect between the foreground and the background is more natural, avoiding problems such as edge jaggedness.

[0033] Thirdly, embodiments of this application provide an electronic device that may include a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, causes the electronic device to perform a method as described in the first aspect or any possible implementation thereof, or to perform a method as described in the second aspect or any possible implementation thereof.

[0034] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof, or performs the method described in the second aspect or any possible implementation thereof.

[0035] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation of the first aspect, or performs the method described in the second aspect or any possible implementation of the second aspect. Attached Figure Description

[0036] Figure 1 This demonstrates a depth-of-field effect construction function;

[0037] Figure 2 This illustrates the image processing framework upon which the interface display method provided in the embodiments of this application is based;

[0038] Figure 3 Layered rendering is shown in an embodiment of this application;

[0039] Figure 4 The overall flow of an interface display method provided in an embodiment of this application is illustrated;

[0040] Figure 5 This illustrates the application flow of the interface display method provided in this application embodiment in the lock screen wallpaper setting scenario;

[0041] Figure 6 An artificial intelligence model for foreground complementation is shown in an embodiment of this application;

[0042] Figure 7 This illustrates a specific implementation of the interface display method provided in this application when the position of the first area changes:

[0043] Figures 8A to 8C This application demonstrates the technical effects of its embodiments in the context of setting lock screen wallpapers.

[0044] Figure 9 An electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0045] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0046] To enhance the visual appeal of displaying UI elements on wallpaper images, some simple depth-of-field effects have been proposed. For example, ... Figure 1 As shown, the clock component is displayed below prominent subjects such as people in the lock screen wallpaper, with the clock component partially obscured by the people. This creates a sense of depth in the wallpaper, constructing a depth-of-field relationship (i.e., a foreground / background relationship) between the prominent subjects and the clock component, making the lock screen interface more vivid and giving it a 3D effect. However, this depth-of-field effect construction function can only be applied to wallpaper images with specific types of subjects (such as people, cats, dogs, etc.) and where the subjects are clearly visible. Moreover, the position where the depth-of-field effect can be presented is fixed on the wallpaper image.

[0047] This application provides an interface display method that can construct a depth-of-field effect on a wallpaper image for UI elements such as clock components and application icons. Moreover, the position where the UI elements can present the depth-of-field effect is not fixed on the wallpaper image, and users can move the UI elements to select the position where the depth-of-field effect is presented.

[0048] The interface display method provided in this application embodiment can be based on Figure 2 The image processing framework shown can be termed a framework based on simulated depth of field using complementary images. Figure 2 As shown, the framework can include two main steps: foreground overlay and depth-of-field rendering.

[0049] 1. Foreground supplementary image.

[0050] First, the regions in the original image that need to be rendered with a depth-of-field effect for UI elements such as clock components and app icons are determined through region selection; then, foreground supplementation is performed on the selected regions.

[0051] Foreground complements are used to regenerate the foreground of the region. The regenerated foreground (also called complement foreground) is the output of the foreground complement. The regenerated foreground is not directly separated from the original image of the region, but rather regenerated based on the original image and foreground edges. Foreground edges are constraints on the foreground complement, i.e., foreground constraints in the image, also known as foreground contours. Foreground complements can be implemented using artificial intelligence models. The specific implementation of foreground complements will be explained in detail later; it will not be elaborated on here.

[0052] Figure 2 The original image in the image can be a wallpaper image, and the area where the depth-of-field effect needs to be displayed can be the first, second, or third area mentioned later. Area selection can be manual. For example, the area where the user places the clock component on the wallpaper image is the area where the depth-of-field effect needs to be displayed. Area selection can also be implemented by the electronic device based on an internal event. This internal event can be referred to later as the internal event that triggers moving UI elements; the area where the UI element is located is the area where the depth-of-field effect needs to be displayed.

[0053] 2. Depth of field rendering.

[0054] For areas requiring a depth-of-field effect, layered rendering is used with foreground, background, and UI elements to achieve a depth-of-field effect for the UI elements within that area. For example... Figure 3 As shown, the layered rendering hierarchy can be as follows: the background is at the bottom, the foreground supplementary image is at the top, and the UI elements are positioned between the foreground supplementary image and the background. That is, the UI elements are rendered above the background layer, and the foreground supplementary image is rendered above the UI element layer. In the result of layered rendering, the depth relationship between the background, foreground supplementary image, and UI elements is: the depth of the UI elements is less than the depth of the background, and greater than the depth of the foreground supplementary image.

[0055] When performing layered rendering, the foreground and background layers can be the same size as the original wallpaper image. In the foreground layer, the content outside the foreground element can be set to transparent, and the background layer can use the original wallpaper image. In other words, UI elements and the foreground layer are layered and overlaid on top of the original wallpaper image.

[0056] based on Figure 2The image processing architecture shown in this application can generate a foreground for any selected region and then simulate the depth effect of UI elements in that region based on the foreground, even if there is no salient subject in that region. However, existing depth effect simulations are based on monocular depth estimation techniques, which can only estimate the depth of salient foregrounds of specific categories (such as people, cats, dogs, etc.). This is because monocular depth estimation techniques are usually implemented based on deep learning, which relies on large-scale training datasets and is limited to one or more specific categories of salient foregrounds.

[0057] Furthermore, using the foreground edge of the selected region as a constraint and employing the original image of that region for foreground completion allows for better synthesis of the foreground and background, better control of the transition between them, and avoids issues such as edge jaggedness and missing parts. However, traditional completion techniques fill in the foreground area image content through simple copy-paste operations; they lack the ability to generate new image content. The foreground image filled back into the foreground area is extracted from the original image, which is prone to edge jaggedness and missing parts.

[0058] Figure 4 The overall flow of an interface display method provided in an embodiment of this application is illustrated. For example... Figure 4 As shown, the method may include the following steps:

[0059] S11. Electronic device acquires wallpaper image.

[0060] The electronic device can be a mobile phone, tablet computer, smart screen, or other device with a display. The composition of the electronic device will be described in detail in subsequent embodiments, and will not be elaborated here.

[0061] Electronic devices can obtain wallpaper images from a local gallery or local system wallpapers based on user selection, or they can download wallpaper images from a cloud server. This application embodiment does not limit the method by which the electronic device obtains wallpaper images. Wallpaper images can also be automatically selected by the electronic device based on a preset mechanism, without requiring manual selection by the user.

[0062] S12. The electronic device generates a foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region.

[0063] The first area is the area where the first UI element (such as a clock component, application icon, etc.) needs to present a depth effect; it is also the display position of the first UI element on the wallpaper image.

[0064] The first area can be a user-selected area, such as the area on the wallpaper image where the clock component is placed. Alternatively, the first area can be a system-set default area, such as the default display area of ​​the first UI element each time the user opens the first user interface. The position of this default area can be constant, for example, the first UI element always appears in the same position each time the first user interface is opened. Or, the position can change, for example, the default display position of the first UI element may differ depending on when the first user interface is opened.

[0065] S12 is... Figure 2 The foreground complement image is shown in the image processing framework. Before executing S12, the electronic device can also extract the original image of the first region from the wallpaper image and perform edge detection on the original image of the first region to obtain the foreground edge of the first region. Further, after extracting the edge of the foreground region through edge detection, the electronic device can perform a dilation operation on the edge to expand the foreground edge and obtain the dilated foreground edge. The dilated foreground edge has been expanded into a region. In addition to the foreground region, the foreground complement image can also cover the dilated foreground edge, that is: perform foreground complement image on the foreground region and the dilated foreground edge to regenerate the foreground image. In this way, the area where the foreground edge needs to be complemented can be selected to improve edge jaggedness. In the regenerated foreground, the dilated foreground edge can have a certain degree of transparency. The transparency of the entire dilated foreground edge can be gradual, with higher transparency closer to the original foreground edge. The original foreground edge is the foreground edge before dilation. For example, in the following Figure 5 In the foreground of the supplementary image, the area of ​​the treetops closer to the original edge is more transparent, allowing it to blend better with the background near the edge. This results in a more natural depth-of-field effect as the foreground of the supplementary image merges seamlessly with the background of the original image.

[0066] One specific implementation of S12 may include: the electronic device may input the original image of the first region and the foreground edge of the first region into the first artificial intelligence model to generate the foreground image of the first region.

[0067] The input to the first artificial intelligence model can include the original image of the first region and the foreground edges of the first region. The output of the first artificial intelligence model can be the foreground complement of the first region. The training samples in the training dataset of the first artificial intelligence model can include: the input image, the foreground edges of the input image, and the output image, where the output image can be the foreground image of the input image. Additionally, the first artificial intelligence model can be configured with a loss function, which can be used to calculate the difference between the output image and the input image in the training samples. This difference can consist of two items: the image difference between the output image and the input image in the foreground, and the image difference between the output image and the input image in the foreground edge region. The weight of the image difference in the foreground can be greater than the weight of the image difference in the foreground edge region. That is, the foreground generated by the foreground complement needs to maintain a high degree of consistency with the foreground in the original image, while the foreground edges generated by the foreground complement can have some differences from the foreground edges in the original image. Here, the image difference can be statistically analyzed by comparing color values ​​pixel by pixel.

[0068] The following text will combine Figure 5 The first artificial intelligence model used for foreground inpainting will be described in detail here, but will not be elaborated on.

[0069] S13. The electronic device obtains a first user interface based on the foreground image of the first region, the first UI element, and the wallpaper image generated.

[0070] Specifically, the electronic device can perform layered rendering of the foreground image of the generated first area, the first UI element, and the original wallpaper image to obtain a first user interface. This layered rendering can include: rendering the first UI element on top of the original wallpaper image, and then rendering the foreground image of the first area on top of the first UI element. The original wallpaper image serves as the background layer at the bottom, the foreground image serves as the foreground layer at the top, and the first UI element is in the middle layer. In this way, a depth-of-field effect can be presented in the first user interface, showing the first UI element being occluded by the foreground image of the first area.

[0071] When performing this layered rendering, the size of the foreground layer can be the same as the size of the first area, or the same as the size of the original wallpaper image. When using the latter, the area outside the first area in the foreground layer can be set to transparent.

[0072] S14. The electronic device displays a first user interface. The first user interface may include a wallpaper image and a first UI element, the first UI element being displayed in a first area of ​​the wallpaper image and the first UI element being occluded by a generated foreground image.

[0073] The first object is the foreground object in the first area. The first UI element is occluded by the first object, causing the first UI element to exhibit a depth-of-field effect in the first area. (Continued...) Figure 8A For example, the first user interface can be the lock screen interface 21, the first UI element can be the clock component 23, the first area can be area 221, and the first object can be mountain peak A. In this way, the clock component 23 can be presented in the area 221 by being occluded by mountain peak A.

[0074] In the first user interface, the first UI element can also obscure a second object within the background image of the first area, where the depth of the second object is greater than the depth of the first object. The second object is the background object within the first area. Similarly... Figure 8A For example, the second object can be the sky W in region 221. In this way, the clock component 23 can be interspersed between the mountain peak A and the sky W through occlusion and being occluded, creating a depth-of-field effect.

[0075] The interface display method described in S11-S14 above can be applied to wallpaper setting scenarios, such as lock screen wallpaper and desktop wallpaper settings. Specifically, S14 can be implemented as: displaying a wallpaper preview interface after setting a lock wallpaper, where the first user interface is the wallpaper preview interface. In lock screen wallpaper and desktop wallpaper setting scenarios, S14 can also be implemented as: displaying a lock screen interface, desktop, or always-on screen interface, where the first user interface can include one or more of the following: lock screen interface, desktop, and always-on screen interface.

[0076] Figure 5 This paper illustrates the application flow of the interface display method provided in this application embodiment in a lock screen wallpaper setting scenario. The details are as follows:

[0077] 1. The user selects the image 21 that they want to set as wallpaper.

[0078] Image 21 is the wallpaper image. Users can select image 21 from the gallery or from the wallpaper images provided by the system; this embodiment does not limit this selection.

[0079] 2. The user selects the display area for the clock component (such as "08:08") on the wallpaper image 21.

[0080] Figure 5 In the image, the area within the rectangular dashed box 22 is the display area for the clock component "08:08," and also the area where the clock component presents a depth-of-field effect. Users can drag the clock component on the wallpaper image to select its display area.

[0081] 3. Extract the original image of the selected area from the wallpaper image in step 2 above, and perform edge detection on the original image of the area to obtain the foreground edge.

[0082] Edge detection methods can include, for example, the Sobel operator edge detection method, but are not limited to this; edge detection can also be achieved through other implementations. The result of edge detection can be represented by a mask map 33 of the region. The mask map 33 can be used to identify the foreground and background of the region, and thus identify the foreground edges. The mask map is one way to identify foreground edges.

[0083] 4. Based on the original image of the region and the foreground edges, perform foreground supplementation to obtain the supplemented foreground image.

[0084] Foreground interpolation can be implemented using artificial intelligence models. The specific implementation of foreground interpolation will be explained in detail below, but will not be elaborated upon here.

[0085] 5. Perform layered rendering of the clock component, the foreground after the overlay image, and the background.

[0086] After layered rendering, in area 22 of wallpaper image 21, the clock component can present a depth-of-field effect that interweaves between the foreground and background.

[0087] 6. Display the lock screen wallpaper preview interface. For example... Figure 5 As shown, the lock screen wallpaper preview interface presents a depth-of-field effect where the clock component "08:08" is interspersed in the forest in the wallpaper image, making the wallpaper effect vivid and interesting.

[0088] The interface display method described in S11-S14 above can also be applied to formal wallpaper display scenarios such as lock screen display, desktop display, and always-on display. In such scenarios, the first user interface may include one or more of the following: lock screen interface, desktop, and always-on display.

[0089] Figure 6 The first artificial intelligence model for foreground complementation is shown.

[0090] like Figure 6 As shown, the first artificial intelligence model can be a deep learning end-to-end model, and the model structure can be a Transformer model structure. The input to the first artificial intelligence model can include: the original image of the selected region and the edge detection results of the selected region; the output of the first artificial intelligence model can include: the regenerated foreground. The first artificial intelligence model can generate the foreground of the selected region based on the original image of the selected region, using the edge detection results of the selected region as constraints. Here, the selected region can be the region determined by the aforementioned region selection that needs to present a depth-of-field effect for the first UI element.

[0091] The overall design scheme of the model can be as follows:

[0092] 1. The original image of the selected region is encoded using an encoder module to generate image embedding features. These image embedding features can be high-level semantic features of the image. The encoder module can be, for example, the UNet encoding module.

[0093] 2. The edge detection results (mask map) of the selected region are encoded by the Encoder module to generate mask map embedding features.

[0094] 3. The image embedding features of the selected region are fused with the mask image embedding features, for example, by addition or splicing, to obtain the encoded information of the selected region.

[0095] 4. The encoded information of the selected region is used to generate the foreground of the selected region through the image decoding module. The decoding module can be, for example, the UNet decoding module.

[0096] The training samples in the training dataset of the first artificial intelligence model may include: a first input, a second input, and an output, wherein the first input is an image, the second input is a mask image of the first input, and the output is also an image, with the foreground in the first input used as the output.

[0097] The training dataset for the first artificial intelligence model can be constructed as follows:

[0098] 1. Construct a mask image for a given original image using edge detection;

[0099] 2. Use the original image and its mask image as the first and second inputs in the training samples, respectively.

[0100] 3. The foreground image in the original image is used as the output in the training samples.

[0101] The loss function of the first AI model can simultaneously consider both the foreground region and the foreground edge region. This loss function can be used to calculate the image difference between the output and the first input, which can consist of two parts: the image difference between the output and the first input in the foreground, and the image difference between the output and the first input in the foreground edges. The weight of the image difference in the foreground can be greater than the weight of the image difference in the edge region. That is, the foreground generated by the foreground complement needs to maintain a high degree of consistency with the foreground in the original image, while the foreground edges generated by the foreground complement can have some differences from the foreground edges in the original image.

[0102] The position of the first area within the wallpaper image is variable. This position can be changed based on user actions, such as moving a first UI element on the wallpaper image. The change in the position of the first area can also be triggered by internal events of the electronic device. For example, when the clock reaches the hour, the position of the first area used to display the clock component changes according to a preset or random trajectory. In this example, the internal event is the clock reaching the hour. Another example is changing the position of the first area used to display the weather component based on the apparent motion trajectory of the sun (sunrise, sunset, tilt). In this example, the internal event is the change in the apparent motion trajectory of the sun. This application does not limit what the internal event is. The former method of changing the position of the first area can be applied to the aforementioned wallpaper setting scenario, while the latter method can be applied to the aforementioned formal wallpaper display scenario.

[0103] The following will combine Figure 7 The interface display method shown further clarifies the specific implementation of this application embodiment when the position of the first region changes:

[0104] S21. An electronic device displays a first user interface, which may include a wallpaper image and a first UI element. The first UI element is displayed in a second area of ​​the wallpaper image. The first UI element is obscured by a third object in the wallpaper image, and the first UI element obscures a fourth object in the wallpaper image. The third object and the fourth object are located in the second area of ​​the wallpaper image.

[0105] The second area can be the first area at a certain time, that is, the display position of the first UI element in the wallpaper image at that time. The position of the second area in the wallpaper image is different from the position of the third area in the wallpaper image as described later. The third area can be the first area at another time, that is, the display area of ​​the first UI element in the wallpaper image at another time.

[0106] by Figure 8A For example, the first user interface can be the lock screen interface 21, the first UI element can be the clock component 23, the second area can be area 221, the third object can be mountain peak A, and the fourth object can be... Figure 8A The sky W in the middle. In this way, the clock component 23 can be more realistically presented as it is interspersed between the mountain peak A and the sky W through the occlusion and being occluded.

[0107] The primary UI element is not limited to a clock widget; it can also be a weather widget, app icon, folder icon, or other UI elements. App icons can include one or more of the following: application icons, app function icons, or system function icons. The primary user interface is not limited to the lock screen; it can also be the desktop, the always-on display, etc.

[0108] In the wallpaper image, the depth of the fourth object is greater than the depth of the third object. In the second area, the third object can be called the foreground object, and the fourth object can be called the background object. The third object does not have to be a prominent subject in the wallpaper image. The third and fourth objects can be parts of a group of objects of the same type, where the third object does not constitute a prominent subject relative to the fourth object, such as trees in a forest, grass in a lawn, or wheat in a wheat field, etc.

[0109] A salient subject can usually be segmented from the original image using a salient subject segmentation algorithm. It is generally a subject that occupies a large area in the image and has a clear segmentation edge.

[0110] Here, the third object in the second area occluding the first UI element can mean that the third object partially occludes the first UI element, and the first UI element occluding the fourth object in the second area can mean that the first UI element partially or completely occludes the fourth object.

[0111] The second area can be a user-selected area, such as where the user chooses to display the first UI element by dragging it. The second area can also be the area at a default position, such as the default display area of ​​the first UI element each time the user opens the first user interface. The position of this default area can be constant, for example, the first UI element is always displayed in the same position each time the first user interface is opened. Alternatively, the position of this default area can change, for example, the default display position of the first UI element is different when the first user interface is opened at different times.

[0112] S22. The electronic device moves the first UI element to the third area of ​​the wallpaper image.

[0113] This movement can be triggered by a user action. For example, when a user action is detected dragging a first UI element from a second area to a third area, the electronic device can move the first UI element from the second area to the third area. This user action can be referenced... Figures 8A to 8C The operation of dragging the clock component 23 to area 222 and area 223.

[0114] This movement can also be triggered by internal events of the electronic device. For example, when the clock reaches the hour, the clock component moves on the wallpaper image along a preset or random trajectory. In this example, the internal event is the clock reaching the hour. As another example, the weather component moves on the wallpaper image according to the apparent motion trajectory of the sun (sunrise, sunset, tilt). In this example, the internal event is the change in the apparent motion trajectory of the sun. This application does not limit what the internal event is.

[0115] S23. The electronic device updates its first user interface. In the updated first user interface, the first UI element is displayed in the third area. The first UI element is obscured by a fifth object in the wallpaper image, and the first UI element also obscures a sixth object in the wallpaper image. The fifth and sixth objects are located in the third area of ​​the wallpaper image. In this way, a depth-of-field relationship can be constructed where the first UI element is interspersed between the fifth and sixth objects, so that the first UI element can present a depth-of-field effect when displayed in the third area.

[0116] by Figures 8A-8B For example, the first user interface before the first UI element is... Figure 8A The user interface 21 shown can be the first user interface after moving the first UI element. Figure 8B The user interface 21 shown can have a third area, which could be area 222; a fifth object, which could be trees B, C, and D; and a sixth object, which could be tree E. This allows for a more realistic depiction of the clock component 23 interspersed among trees B, C, D, and E through occlusion and being occluded, creating a depth-of-field effect.

[0117] And with Figures 8B-8C For example, the first user interface before the first UI element is... Figure 8B The user interface 21 shown can be the first user interface after moving the first UI element. Figure 8C The user interface 21 shown has a third area. Figure 8C In region 223, the fifth object is tree B, tree F, and tree G, and the sixth object is tree E. In this way, the clock component 23 can be more realistically presented as it is interspersed among trees B, tree F, tree G, and tree E through occlusion and being occluded.

[0118] In the wallpaper image, the depth of the sixth object is greater than the depth of the fifth object. Within the third area, the fifth object can be called the foreground object, and the sixth object can be called the background object. The fifth object does not necessarily have to be a prominent subject in the wallpaper image. The fifth and sixth objects can be parts of a group of objects of the same type, where the fifth object does not constitute a prominent subject relative to the sixth object, such as trees in a forest, grass in a lawn, or wheat in a wheat field, etc.

[0119] from Figure 7 As can be seen from the overall process shown, the interface display method provided in this application embodiment can present the depth effect of UI elements on the wallpaper image, and the position that can present the depth effect is not fixed. Users can move UI elements on the wallpaper image to experience the depth effect of UI elements at different positions, thereby presenting a 3D dynamic effect of UI elements moving through the wallpaper image through this movement, which is more vivid.

[0120] Figures 8A to 8CThis example illustrates the technical effects of applying the embodiments of this application to a lock screen wallpaper setting scenario. For instance... Figures 8A to 8C As shown, the lock screen interface 21 may include a wallpaper image 21 and a clock component 23, wherein the wallpaper image contains objects such as sky, mountains, forests, rivers and sun.

[0121] At first, such as Figure 8A As shown, the clock component 23 is displayed in area 221 of the wallpaper image 21. Specifically, the clock component 23 is displayed below the mountain peak A in area 221, that is, the clock component 23 is obscured by the mountain peak A, thus creating a depth-of-field relationship with the mountain peak A in front and the clock component 23 behind, so that the clock component 23 presents a depth-of-field effect in area 221.

[0122] Subsequently, as Figure 8B As shown, the user drags the clock component 23 on the wallpaper image 21. The clock component 23 moves from area 221 to area 222. The clock component 23 is displayed below trees B, C, and D in area 222 but above tree E in area 222. That is, the clock component 23 is occluded by trees B, C, and D, and the clock component 23 also occludes tree E, thus creating a depth-of-field relationship where trees B, C, and D are in front and tree E is behind, and the clock component 23 is interspersed between the trees B, C, and D in front and the tree E behind, so that the clock component 23 also presents a depth-of-field effect in area 222.

[0123] Next, as Figure 8C As shown, the user continues to drag the clock component 23 on the wallpaper image 21. The clock component 23 moves from region 222 to region 223. The clock component 23 is displayed below trees B, F, and G in region 222 but above tree E in region 222. That is, the clock component 23 is occluded by trees B, F, and G, and the clock component 23 also occludes tree E, thus creating a depth relationship where trees B, F, and G are in front and tree E is behind, and the clock component 23 is interspersed between the trees B, F, and G in front and the tree E behind, so that the clock component 23 also presents a 3D effect in region 223.

[0124] comprehensive Figures 8A to 8C As you can see, when the user drags the clock component 23 to a new position, they can observe the depth effect of the clock component 23 at that new position. This dynamic process can present a 3D dynamic effect of the clock component 23 moving through the forest, which is more vivid. Moreover, the user can more comprehensively feel the spatial layers of the wallpaper image from the depth effect of the clock component 23 at multiple positions, making the 3D effect more realistic.

[0125] To achieve a depth-of-field effect for the first UI element in the second area, prior to S21, the electronic device can perform a second layer rendering on the foreground of the second area, the first UI element, and the original wallpaper image to obtain the first user interface before the update. The specific implementation of the second layer rendering may include: rendering the first UI element on top of the original wallpaper image, and rendering the foreground of the second area on top of the first UI element.

[0126] Refer to the previous Figure 2 The image processing framework shown allows the original image of the second region to be extracted from the wallpaper image, and the foreground edges of the second region to be obtained by edge detection of the original image of the second region. Specifically, the foreground edges of the second region can be indicated by a mask map of the second region. This mask map can be used to identify the foreground and background of the second region, thus serving to indicate the foreground edges.

[0127] Foreground completion of the second region can be achieved using an artificial intelligence model. For example, the original image of the second region and the mask image of the second region can be input into a first artificial intelligence model to obtain a regenerated foreground of the second region. The method for foreground completion based on the first artificial intelligence model has already been described above and will not be repeated here.

[0128] Similarly, in order to achieve a depth-of-field effect for the first UI element in the third region, prior to S23, the electronic device can perform a first layered rendering of the foreground image of the third region, the first UI element, and the original wallpaper image to obtain an updated first user interface. The specific implementation of the first layered rendering may include: rendering the first UI element on top of the original wallpaper image, and rendering the foreground image of the third region on top of the first UI element.

[0129] Refer to the previous Figure 2 The image processing framework shown allows the original image of the third region to be extracted from the wallpaper image, and the foreground edges of the third region to be obtained by edge detection of the original image of the third region. Specifically, the foreground edges of the third region can be indicated by a mask map of the third region. This mask map can be used to identify the foreground and background of the third region, thus serving to indicate the foreground edges.

[0130] Foreground completion of the third region can also be achieved using an artificial intelligence model. For example, the original image of the third region and its mask image can be input into a first artificial intelligence model to obtain a regenerated foreground of the third region. The method for foreground completion based on the first artificial intelligence model has already been described above and will not be repeated here.

[0131] As can be seen, since the foreground of the area that needs to present a depth-of-field effect during foreground overlay is not extracted from the wallpaper image, but generated based on the original image of the area and the foreground edge of the area, this embodiment of the application can support the construction of a depth-of-field effect for UI elements such as clock components in any selected area, even if the area does not have a significant subject; moreover, since the foreground edge of the area is used as a constraint during foreground overlay, the transition effect between the foreground and the background is more natural, avoiding problems such as edge jaggedness.

[0132] Figure 9 An electronic device 300 provided in an embodiment of this application is illustrated by way of example.

[0133] Electronic device 300 can possess both human-computer interaction capabilities and computing capabilities. The device type of electronic device 300 can be any of the following: mobile phone, tablet computer, handheld computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), smart home devices such as smart screens, wearable devices such as smartwatches and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices, or smart city devices, etc.

[0134] like Figure 9 As shown, the electronic device 300 may include: a processor 110, a memory 120, a display 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180B, an accelerometer sensor 180E, and a touch sensor 180K, etc. The various components of the electronic device 300 can be connected via a bus.

[0135] The processor 110 provides computing power and can be used as the computing module of the electronic device 300. The display 130, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, and other input / output components provide human-computer interaction capabilities and can be used as the human-computer interaction module of the electronic device 300. When the computing module within the electronic device 300 has powerful computing capabilities, the electronic device 300 can independently execute the interface display method provided in this application embodiment. When the computing module within the electronic device 300 does not have powerful computing capabilities, the electronic device 300 can also only execute the human-computer interaction steps in the interface display method provided in this application embodiment, such as displaying the first user interface and moving the first UI element on the wallpaper image, while the image processing steps such as foreground supplementation and layered rendering in this method can be executed by a cloud server.

[0136] Processors 110 can be one or more, and they can be integrated into an integrated circuit of a system-on-a-chip (SOC). An SOC is a system-on-a-chip. Processors 110 may include a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU). The CPU may include an application processor (AP) and a baseband processor (BP). The AP is responsible for running the operating system, user interface, and applications on the electronic device 300; the BP is responsible for transmitting and receiving wireless signals and managing radio frequency services. The GPU is responsible for graphics rendering, performing tasks such as shading, material filling, rendering, and output based on rendering instructions and data from the CPU. The NPU, by referencing biological neural network structures, such as the transmission patterns between neurons in the human brain, can quickly process input information and continuously learn. The NPU can be used to run artificial intelligence algorithms, such as instruction recommendation algorithms, image processing algorithms, and image understanding algorithms. The CPU and GPU can be used to render and synthesize the image to be displayed on the monitor 130.

[0137] The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0138] The processor 110 may include a cache memory, which can be used to store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

[0139] The memory 120 may include a program storage area and a user data storage area. The program storage area may store the operating system and one or more applications (such as games), while the data storage area may store data created by the user during the use of the electronic device 300 (such as photos and contacts). The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a hard disk, flash memory, or universal flash storage (UFS). The memory 120 may also be an external memory card, such as a Micro SD card.

[0140] The memory 120 may also store code instructions for the image editing method provided in the embodiments of this application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the electronic device 300 may execute the steps performed by the human-computer interaction module and / or the calculation module in the image editing method provided in the embodiments of this application.

[0141] The memory 120 can also be integrated with the processor 110 into the integrated circuit of the SOC.

[0142] like Figure 9 As shown, the electronic device 300 can realize the display function through SOC, DDIC 140, and display 130.

[0143] The display 130 has multiple refresh rates. The refresh rate indicates the number of times the display refreshes the image per second. For example, a 60 Hz refresh rate means the display refreshes the image 60 times per second. The display 130 can use an LTPO display panel, allowing the refresh rate to be reduced to lower rates, such as 10Hz or 1Hz, thereby supporting reduced power consumption.

[0144] The display driver integrated circuit (DDIC) 140 serves as the control core of the display 130, driving the display 130 to operate and receiving data from the SOC (processor 110), such as image data and some instructions. The DDIC 140 can send drive signals and data to the display panel of the display 130 in the form of electrical signals, thereby controlling the screen brightness and color, enabling image information such as letters and pictures to be displayed on the screen and completing the screen refresh.

[0145] The image data to be displayed sent by the SOC to the DDIC 140 can be stored in the frame buffer to complete the display sending (or image sending). Then, the DDIC 140 retrieves the image data from the frame buffer and drives the display 130 to display it.

[0146] The wireless communication function of electronic device 300 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0147] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 300 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0148] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 300. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0149] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display 130. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0150] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 300, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0151] In some embodiments, antenna 1 of electronic device 300 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 300 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0152] Electronic device 300 can achieve shooting function through ISP, camera 193, video codec, GPU, display 130 and application processor.

[0153] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0154] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 300 may include one or N cameras 193, where N is a positive integer greater than 1.

[0155] Video codecs are used to compress or decompress digital video. Electronic device 300 may support one or more video codecs. Thus, electronic device 300 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0156] Electronic device 300 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0157] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0158] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 300 can listen to music or make hands-free calls through the speaker 170A.

[0159] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 300 answers a telephone call or voice message, the receiver 170B can be brought close to the listener's ear to receive the voice message.

[0160] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 300 may have at least one microphone 170C. In some embodiments, electronic device 300 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 300 may have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0161] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0162] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 300 can receive button input and generate key signal inputs related to user settings and function control of electronic device 300. Motor 191 can generate vibration prompts. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with electronic device 300.

[0163] Figure 9 The illustrated structure does not constitute a specific limitation on the electronic device 300. The electronic device 300 may include more or fewer parts than illustrated, or combine some parts, or split some parts, or arrange different parts. The various parts illustrated may be implemented in hardware, software, or a combination of software and hardware.

[0164] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps performed by the human-computer interaction module in the above-described method embodiments, or the steps performed by the human-computer interaction module and the computing module.

[0165] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps performed by the computing module in the above-described method embodiments.

[0166] This application also provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps executed by the human-computer interaction module in the above-described method embodiments, or the steps executed by the human-computer interaction module and the computing module.

[0167] This application also provides a computer program product that, when run on a server, enables the server to implement the steps executed by the computing module in the above-described method embodiments.

[0168] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps performed by the human-computer interaction module in any of the method embodiments of this application, or the steps performed by the human-computer interaction module and the computing module. The chip system can be a single chip or a chip module composed of multiple chips.

[0169] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps performed by the computing module in any of the method embodiments of this application. The chip system can be a single chip or a chip module composed of multiple chips.

[0170] The term "user interface (UI)," or simply "interface," used in the specification and accompanying drawings of this application, refers to the medium through which an application or operating system interacts and exchanges information with the user. It facilitates the conversion between the internal form of information and a form acceptable to the user. The user interface of an application is written in source code using specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content, such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags.

[0171] <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、

[0172] <canvas>To define the elements and attributes of a webpage.

[0173] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0174] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0176] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An interface display method characterized by, The method comprises: obtaining a wallpaper image; generating a foreground image of a first region of the wallpaper image based on an original image of the first region of the wallpaper image and a foreground edge of the first region; obtaining a first user interface according to the generated foreground image of the first region, a first UI element, and the wallpaper image; displaying the first user interface, the first user interface comprising the wallpaper image and the first UI element, the first UI element being displayed in the first region of the wallpaper image and being occluded by the generated foreground image.

2. The method of claim 1, wherein, Further comprising: performing edge detection on the original image of the first region to obtain the foreground edge of the first region.

3. The method of claim 1 or 2, wherein, The generating of the foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region comprises: inputting the original image of the first region and the foreground edge of the first region into a first artificial intelligence model to generate the foreground image of the first region.

4. The method of claim 3, wherein, The training sample in the training data set of the first artificial intelligence model comprises: a first input image, a foreground edge of the first input image, and an output image, wherein the output image is a foreground image of the first input image.

5. The method of claim 3 or 4, wherein, The loss function of the first artificial intelligence model is used to calculate the difference between the output image and the first input image, the difference comprising: image difference of the output image and the first input image on the foreground, and image difference of the output image and the first input image on the foreground edge region; wherein the weight of the image difference on the foreground is greater than the weight of the image difference on the foreground edge region.

6. The method of any one of claims 1-5, wherein, Before the generating of the foreground image of the first region based on the original image of the first region of the wallpaper image and the foreground edge of the first region, further comprising: performing inflation processing on the foreground edge of the first region to obtain an inflated foreground edge.

7. The method of claim 6, wherein, The inflated foreground edge is transparent, and the transparency is gradual, the closer to the foreground edge before inflation, the higher the transparency.

8. The method of any one of claims 1-7, wherein, In the first user interface, the first UI element occludes a second object in the background image of the first region; in the wallpaper image, the depth of the second object is greater than the depth of a first object, the first object being an object in the foreground image of the first region.

9. The method of any one of claims 1-8, wherein, The obtaining of the first user interface according to the generated foreground image of the first region, the first UI element, and the wallpaper image comprises: performing layered rendering on the generated foreground image of the first region, the first UI element, and the wallpaper image to obtain the first user interface; the layered rendering comprises: rendering the foreground image of the first region above the first UI element, and rendering the first UI element above the wallpaper image.

10. The method of any one of claims 1-9, wherein, The position of the first region in the wallpaper image is variable.

11. The method of claim 10, wherein, The position of the first region in the wallpaper image is changed based on a user operation, the user operation comprising a user operation of moving the first UI element on the wallpaper image.

12. The method of any one of claims 1-11, wherein, The first user interface includes one or more of the following: a wallpaper setting preview interface, a lock screen interface, a desktop, an off-screen interface.

13. The method of any one of claims 1-12, wherein, The first UI element includes one or more of the following: a clock component, a weather component, an application icon, a folder icon.

14. An interface display method characterized by comprising: Comprise: Display a first user interface, the first user interface includes a wallpaper image and a first UI element, the first UI element is displayed in a second region of the wallpaper image, the first UI element is blocked by a third object in the wallpaper image, the first UI element blocks a fourth object in the wallpaper image, the third object, the fourth object is located in the second region of the wallpaper image; Move the first UI element to a third region of the wallpaper image; Update the first user interface, and in the updated first user interface, the first UI element is displayed in the third region, the first UI element is blocked by a fifth object in the wallpaper image, the first UI element blocks a sixth object in the wallpaper image, the fifth object, the sixth object is located in the third region of the wallpaper image.

15. The method of claim 14, wherein, In the wallpaper image, the depth of the third object is less than the depth of the fourth object, and the depth of the fifth object is less than the depth of the sixth object.

16. The method of any one of claims 14-15, wherein, Before the update of the first user interface, further comprising: performing first layered rendering on the foreground image of the third region, the first UI element and the wallpaper image to obtain the updated first user interface; the first layered rendering comprises: rendering the first UI element on the wallpaper image, and rendering the foreground image of the third region on the first UI element; wherein the foreground image of the third region comprises the fifth object.

17. The method of any one of claims 14-16, wherein, The foreground image of the third region is generated based on the original image of the third region and the foreground edge of the third region, and the foreground image of the third region comprises the fifth object.

18. The method of claim 17, wherein, Further comprising: perform edge detection on the original image of the third region to obtain the foreground edge of the third region; inputting the original image of the third region and the foreground edge of the third region into a first artificial intelligence model to generate the foreground image of the third region.

19. The method of any one of claims 17-18, wherein, Further comprising: Before regenerating the foreground image of the third region based on the original image of the third region and the foreground edge of the third region, performing inflation processing on the foreground edge of the third region to obtain the inflated foreground edge of the third region.

20. The method of claim 19, wherein, The inflated foreground edge of the third region is transparent, and the transparency is gradual, that is, the closer to the foreground edge before inflation, the higher the transparency.

21. The method of any one of claims 14-20, wherein, Before the display of the first user interface, further comprising: performing second layered rendering on the foreground image of the second region, the first UI element and the wallpaper image to obtain the first user interface before the update; the second layered rendering comprises: rendering the first UI element on the wallpaper image, and rendering the foreground image of the second region on the first UI element; wherein the foreground image of the second region comprises the third object.

22. The method of any one of claims 14-21, wherein, The foreground image of the second region is generated based on the original image of the second region and the foreground edge of the second region, and the foreground image of the second region includes the third object.

23. The method of claim 22, wherein, Further comprising: performing edge detection on the original image of the second region to obtain the foreground edge of the second region; inputting the original image of the second region and the foreground edge of the second region into the first artificial intelligence model to generate the foreground image of the second region.

24. The method of any one of claims 22-23, wherein, Further comprising: performing dilation processing on the foreground edge of the second region before regenerating the foreground of the second region based on the original image of the second region and the foreground edge of the second region.

25. The method of claim 24, wherein, The dilated foreground edge of the second region is transparent, and the transparency is gradual, that is, the closer to the foreground edge before dilation, the higher the transparency.

26. The method of claim 18 or 23, wherein, The training sample in the training data set of the first artificial intelligence model includes: a first input image, a foreground edge of the first input image, and an output image, wherein the output image is a foreground image of the first input image.

27. The method of claim 26, wherein, The loss function of the first artificial intelligence model is used to calculate the difference between the output image and the first input image, and the difference includes: the image difference between the output image and the first input image in the foreground, and the image difference between the output image and the first input image in the foreground edge region; wherein the weight of the image difference in the foreground is greater than the weight of the image difference in the foreground edge region.

28. The method of any one of claims 14-27, wherein, The third object and the fifth object are not significant subjects in the wallpaper image.

29. The method of any one of claims 14-28, wherein, The first user interface includes one or more of the following: a wallpaper setting preview interface, a lock screen interface, a desktop, and an off-screen interface.

30. The method of any one of claims 14-29, wherein, The first UI element includes one or more of the following: a clock component, a weather component, an application icon, and a folder icon.

31. An electronic device, comprising: including: a processor, a memory, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method of any one of claims 1-30.

32. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-30.

33. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-30.

Images