Display method, electronic device and computer readable medium

Abstract

The present application relates to the technical field of computers, and in particular to a display method, an electronic device and a computer readable medium. By adjusting display parameters affecting foreground elements in a set target interface, comprising related parameters such as the display positions of the foreground elements in a corresponding background image area, the foreground elements on the set target interface are arranged at positions where the foreground elements have an interaction effect with a salient subject. In this way, on the basis of the display method provided in the present application, when a background image is set for a target interface, foreground elements such as a time component and a notification component are arranged at positions where the foreground elements have interactivity with features such as actions of salient subjects such as characters in an original background image, so that the interestingness of the target interface can be improved.

Description

Display method, electronic device, and computer-readable medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 30, 2023, with application number 202311654628.9 and application name “Display Method, Electronic Device and Computer-Readable Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of computer technology, and in particular to a display method, electronic equipment, and computer-readable medium. Background Art

[0003] As mobile phones, tablets, smartwatches, and other electronic devices play an increasingly important role in people's lives, the screen content displayed on these devices is also carrying more and more functions, and users' requirements for the aesthetics of screen content are becoming increasingly higher. Among them, using images stored locally on the electronic device as interface backgrounds, such as lock screen wallpapers or desktop wallpapers, has become a common choice for users. Such images can be, for example, images captured by a camera or images downloaded by the user.

[0004] Due to the limitations of the terminal device's screen size and card size, images need to be cropped to a specific aspect ratio. Furthermore, the phone's lock screen or desktop also displays clock components, lock screen notifications, or floating notifications, which are typically located in fixed locations on the lock screen desktop. This results in the cropped image serving as the interface background often being obscured by the clock component, lock screen notifications, or floating notifications. The resulting composition, combined with the cropped image and related components or floating windows, lacks aesthetic appeal, impacting the user experience.

[0005] Summary of the Invention

[0006] In view of this, the present application provides a display method, an electronic device and a computer-readable medium, which can control the layout of relevant components on the background image in some interface display scenarios so that an interactive effect is generated between the relevant components and the above-mentioned significant subjects.

[0007] In a first aspect, the present application provides a display method, applied to an electronic device, the method comprising: detecting an interface display instruction; displaying a first interface in response to the interface display instruction, wherein the first interface includes at least one first foreground element and a first background image area in a first background image, wherein the display parameters of the first foreground element in the first background image area include position parameters of the first foreground element, and the position parameters are related to semantic information of at least one display element in the first background image area.

[0008] For example, the first foreground element may be a time component, notification component, etc. displayed on the desktop or lock screen interface, and the first background image may be an image set by the user as wallpaper for the desktop or lock screen interface, which may be described as the original background image below. The corresponding first background image area may be a partial area cropped from the first background image to adapt to the screen size of an electronic device such as a mobile phone. The display parameters including the position parameters of the first foreground element may be, for example, parameters related to determining the display position of the time component, notification component, etc. on the desktop or lock screen interface. And at least one display element in the first background image area may be, for example, a significant subject in the original background image, including a significant subject in a person, animal, plant, food, building, or landscape.

[0009] It can be understood that the above-mentioned semantic information can be the result of image semantic understanding of the original background image. The semantic information can include descriptions of one or more significant subjects included in the image, the actions or forms of the significant subjects, and related descriptions such as the corresponding action objects or form extension directions.

[0010] In a possible implementation of the first aspect, the semantic information includes one or more of the following features: action-related features; and morphology-related features.

[0011] In a possible implementation of the first aspect above, the action-related features include one or more of the following features: features used to describe the action; features used to describe the object of the action.

[0012] For example, among the above-mentioned action-related features, the features used to describe the action can be "kick", "run", "jump", "hold", "support" and other features; the features used to describe the object of the action can be "football", "flower" and other action object features, which are not limited here.

[0013] In a possible implementation of the first aspect above, the morphology-related features include one or more of the following features: features for describing the morphology; features for describing the extension direction of the morphology.

[0014] For example, among the above-mentioned morphology-related features, the features used to describe the morphology may be features describing the morphology of a "big tree"; the above-mentioned features used to describe the extension direction of the morphology may be features describing the extension direction of the branches and leaves of a big tree, etc., and there is no limitation here.

[0015] It is understood that the aforementioned action-related and / or morphology-related features may be features of any of the at least one display element. If the original background image contains multiple salient subjects, the representativeness of each salient subject may be calculated using a saliency detection algorithm or other means, and the salient subject with the highest representativeness or one above a corresponding threshold may be determined as the display element to which the aforementioned action-related and / or morphology-related features belong.

[0016] The size of the first background image area is related to the screen size of the electronic device and the size parameters in the initial display parameters of the first foreground element. The screen size of the electronic device may include the width and height values ​​of the screen, based on which the aspect ratio of the screen can be determined, and then the aspect ratio of the first background image area can be determined, and then the aspect ratio can be used to determine an appropriate cropping area in the original background image as the first background image area. In addition, the initial display parameters of the first foreground element may be, for example, the size, color, font and other related parameters of the time component, notification component, etc., wherein the size of the time component, notification component, etc. can be understood as the size parameters in the initial display parameters.

[0017] In a possible implementation of the first aspect above, before displaying the first interface, the above method also includes: performing image semantic understanding on the first background image to obtain semantic information related to at least one display element and regional position information corresponding to the features contained in the semantic information; determining the first regional position information corresponding to the first feature among the features contained in the semantic information; and determining the position parameters of the first foreground element based on the first regional position information.

[0018] For example, the first feature may be a feature describing an action, such as "kick." In this case, the position parameter of the first foreground element is determined based on the first region position information corresponding to the first feature. The position parameter of the first foreground element may be the position information of the region where the virtual object corresponding to the action described by the first feature is located, such as the region where the virtual object performing the action "kick" (e.g., a soccer ball or other object) may be located.

[0019] In a possible implementation of the first aspect above, the first feature is a feature of an object describing an action, and determining the position parameter of the first foreground element based on the first area position information includes: determining the first area position information as the position parameter of the first foreground element.

[0020] For example, the first background image can be understood as the original background image. The semantic information related to the at least one display element obtained corresponding to the image semantic understanding can include information such as the salient subject located in the first background image area, the salient subject's action, and the action object. The action object is the object of the salient subject's action. For example, the object of the action "kick" can be "soccer ball." The first feature can be, for example, the salient active action object, such as "soccer ball." The first region position information corresponding to the first feature can be the location information of the soccer ball in the corresponding image.

[0021] The features contained in the above-mentioned semantic information can be, for example, fields that can express different regional position information in the semantic information extracted and obtained during the semantic segmentation process described in the following embodiments. Accordingly, the regional position information corresponding to the above-mentioned features can correspond to the regional position information corresponding to each field in the semantic information. Therefore, the process of determining the position parameters of the first foreground element described in the above implementation can be understood as a process of replacing the initial position parameters of the first foreground element with the first regional position information corresponding to the above-mentioned first feature. For example, the initial position parameters of the time component, notification component, etc. are replaced with the position parameters determined corresponding to the position information of "football" in the corresponding image.

[0022] In a possible implementation of the first aspect above, the display parameters also include color parameters of the first foreground element, and the color parameters are related to one or more of the following colors: the initial color of the first foreground element; the background color of the area where the features of the object used to describe the action are located; the background color of the area where the features of the object used to describe the extension direction of the form are located; the color of the first display element in at least one display element.

[0023] For example, in some embodiments, the color parameters included in the display parameters of the first foreground element may be color parameters in the initial display parameters of the first foreground element, such as preset color parameters corresponding to the time component, notification component, etc., i.e., color parameters corresponding to the color parameters indicated by the first color information. In other embodiments, the color parameters included in the display parameters of the first foreground element may also be color parameters corresponding to the background color of the area containing the action object of the salient subject in the original background image, such as the color parameters corresponding to the background color of the area containing the "football" or "flower." When the salient subject in the original background image is a landscape or building, such as a "big tree," the color of the first foreground element may also be the background color of the area indicating the direction of extension of the salient subject's form in the corresponding image. For example, the color of the time component may be the background color of the area in the direction of extension of the "big tree's" branches and leaves. The color of the first foreground element may also be a color that matches the color of the salient subject. For example, the color of the time component may be a color that is similar to or contrasts with the color of the person's clothing, without limitation.

[0024] In a possible implementation of the first aspect above, the method for determining the first background image area includes: determining the size parameters of a first cropping frame for cropping the first background image area on the first background image based on the screen size of the electronic device; determining at least one cropping area that meets the size parameters of the first cropping frame and contains the first display element based on the first semantic information of the first display element, the position information of the first display element in the first background image, and the size parameters of the first display element, wherein at least one display element includes the first display element; and determining the first cropping area among the at least one cropping area as the first background image area.

[0025] In a possible implementation of the first aspect above, the size parameters of the first cropping box are also related to the size parameters of the first foreground element, and the method for determining the first background image area includes: determining the size parameters of the first cropping box based on the screen size of the electronic device and the size parameters of the first foreground element.

[0026] Specifically, based on the screen size of the electronic device and the size of first foreground elements such as the time component and notification component, several cropping regions can be determined from the original background image. Furthermore, based on semantic information related to display elements such as a prominent subject in the original background image, such as "athlete kicking football," and the position information and size parameters of the prominent subject in the original background image, a cropping region, such as the first cropping region, can be selected from among several cropping regions that meet the aforementioned first cropping frame size parameters to serve as the first background image region.

[0027] In a possible implementation of the first aspect above, at least one cropping area is a plurality of cropping areas, and determining a first cropping area among the at least one cropping area as the first background image area includes: performing an aesthetic score on the at least one cropping area, and determining the first cropping area whose aesthetic score satisfies a third condition as the first background image area, wherein the third condition includes the aesthetic score being higher than a second threshold, or the highest value among the aesthetic scores.

[0028] That is, the first background image area can be a cropping area whose corresponding aesthetic score is higher than the corresponding threshold among several cropping areas that meet the first cropping frame size parameters, or it can be a cropping area with the highest corresponding aesthetic score, and there is no limitation here.

[0029] In a possible implementation of the first aspect above, the first interface includes a lock screen interface or a desktop, and the above-mentioned interface display instruction includes: a display instruction triggered by a first user operation indicating display of the lock screen interface; or a display instruction triggered by a second user operation indicating display of the desktop.

[0030] The first interface may be a target interface for the user to perform operations such as setting a background image. The target interface may be, for example, a desktop or lock screen interface displayed on an electronic device such as a mobile phone, and is not limited here.

[0031] It can be understood that when adjusting the display parameters of the foreground elements that can affect the layout of the set target interface, the screen size and the size and position of the salient subject in the original background image are used as layout constraints. The foreground elements on the set target interface can be integrated with some action-related features, morphology-related features, color features, and representative features of the salient subject, ultimately producing a highly interactive and aesthetically pleasing interface display effect.

[0032] In some embodiments, the above-mentioned process of determining the display parameters of the first foreground element, etc., or the above-mentioned process of fusion, can be implemented using a pre-trained machine learning model or a neural network model, for example, through saliency detection and a composition scoring model, etc., and is not limited here.

[0033] In a second aspect, the present application provides an electronic device comprising: one or more processors; one or more memories; one or more memories storing one or more programs, which, when executed by one or more processors, enables the electronic device to execute the display method provided in the above-mentioned first aspect and various possible implementations of the first aspect.

[0034] In a third aspect, the present application provides a computer-readable medium having instructions stored thereon. When the instructions are executed on a computer, the computer executes the display method provided in the first aspect and various possible implementations of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the display method provided in the above-mentioned first aspect and various possible implementations of the first aspect.

[0036] The beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions of the first aspect and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram showing an application scenario of a display method provided in an embodiment of the present application.

[0038] Figure 2a shows a schematic diagram of the lock screen interface displayed on the mobile phone in response to the user setting an image containing a human subject as the target interface background image provided by an embodiment of the present application.

[0039] Figure 2b shows another embodiment of the present application provided in response to the user setting an image containing a human subject as the target interface background image, and a schematic diagram of the lock screen interface displayed by the mobile phone.

[0040] FIG2c is a schematic diagram of a lock screen interface displayed on a mobile phone in response to a user setting a landscape image as the background image of a target interface provided by an embodiment of the present application.

[0041] FIG2 d is a schematic diagram of the desktop displayed on the mobile phone in response to the user setting an image containing a human subject as the background image of the target interface provided by an embodiment of the present application.

[0042] FIG3 shows a schematic diagram of a system framework and implementation principle of an embodiment of the present application that can implement the above-mentioned display method.

[0043] FIG4 is a schematic diagram showing an implementation flow of a display method provided in Example 1 of the present application.

[0044] FIG5 a is a schematic diagram of an image browsing interface supported by user operations displayed on a mobile phone provided in Example 1 of the present application.

[0045] FIG5 b is a schematic diagram of a setting interface supporting user operations displayed on a mobile phone provided in Example 1 of the present application.

[0046] FIG5c is a schematic diagram of another setting interface supporting user operations displayed on a mobile phone provided in Example 1 of the present application.

[0047] Figure 5d shows a schematic diagram of a lock screen interface displayed on a mobile phone provided in Example 1 of the present application.

[0048] FIG6 is a schematic diagram showing a process of determining recommended layout parameters according to the layout constraints of an input model by a composition scoring model provided in Example 1 of the present application.

[0049] FIG7 is a schematic diagram showing the technical implementation principles of image semantic understanding and natural language understanding used in a display method provided in Example 1 of the present application.

[0050] FIG8 is a schematic diagram showing an implementation flow of a display method provided in Example 2 of the present application.

[0051] FIG9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0052] FIG10 shows a block diagram of the operating system software structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0054] It should also be stated that the steps in the methods and processes in the embodiments of the present application are numbered for ease of reference, rather than to limit the order of precedence. If there is a sequence between the steps, the written description shall prevail.

[0055] FIG1 shows a schematic diagram of an application scenario of a display method.

[0056] As shown in Figure 1, the scene includes a mobile phone 100. In the lock screen state, the lock screen interface 101 displayed by the mobile phone 100 may include a lock screen wallpaper, a clock component 102, and a lock screen notification 103. The lock screen wallpaper can be set by the user using a photo taken with the mobile phone 100 or an image downloaded, such as the wallpaper set corresponding to the "athlete playing football" image shown in Figure 1. At present, the lock screen wallpaper set by the user is usually a cropped image of the original background image that adapts the screen aspect ratio. On this basis, the mobile phone 100 then draws, renders, and synthesizes the clock components, lock screen notifications, etc. to be laid out with the cropped image according to the preset layout rules, and displays them, which can be displayed as the lock screen interface 101 shown in Figure 1.

[0057] However, since the cropping rules adopted by the mobile phone 100 in response to the user's settings for cropping the image are usually fixed preset rules such as center cropping, and the layout rules adopted by the mobile phone 100 when synthesizing the lock screen interface are usually also arranged according to the preset positions and preset sizes of the corresponding components, the components displayed on the lock screen desktop of the mobile phone 100 may block the features of the prominent subjects such as the characters in the lock screen wallpaper, such as the characters' faces, limbs, movements, etc. In addition, when the mobile phone 100 receives a notification message (such as WeChat TM , missed calls, etc.), the position where the lock screen notification 103 is displayed may also cover the features of the prominent subject in the lock screen wallpaper, which will result in poor aesthetics of the lock screen interface and a decrease in the user's visual experience.

[0058] In order to solve the above problems, the present application provides a display method. Specifically, the method adjusts the display parameters affecting the foreground elements in the set target interface, including the size of the foreground elements, the position relative to the significant subject in the background image, and the color and other parameters, so as to achieve the layout of the foreground elements on the set target interface to a position that can produce an interactive effect with the significant subject. Among them, the size, position and other related features of the significant subject in the above-mentioned background image can be determined based on the size, position, etc. of the relevant significant subject in the original background image determined by the image semantic understanding of the corresponding original background image. Based on this, the present application can use the size, position, etc. of the relevant significant subject in the original background image, combined with the screen size of the electronic device, as the layout constraint conditions for adjusting the above-mentioned foreground elements on the set target interface, and adjust the preset size, position, color and other parameters of the foreground elements on the set target interface.

[0059] It can be understood that the screen size of the above-mentioned electronic device can constrain the interface size of the set target interface, for example, the aspect ratio of the set target interface can be the same as the screen aspect ratio. The above-mentioned foreground elements, for example, components displayed on the set target interface, include clock components, notification components, etc. Correspondingly, the display parameters of the foreground elements adjusted accordingly in this application can include the component size, component position, component color, etc. of the relevant components. The adjusted display parameters of the foreground elements, such as the component size, component position, component color and other parameters of the relevant components, can control the layout of the relevant components on the background image so that the relevant components and the above-mentioned significant subjects produce an interactive effect. In addition, in order to enhance the interactive effect, the cropped area corresponding to the above-mentioned background image can also be optimized based on aesthetic scoring to improve the aesthetics of the composition of the set target interface formed by the layout of the above-mentioned foreground elements on the background image.

[0060] In this way, the target interface that sets the background image based on the display method provided by this application can reflect the interactivity and interest between the prominent subject in the original background image and the related components. Furthermore, when adjusting the display parameters of foreground elements such as related components, optimizing the cropped area of ​​the background image corresponding to the original background image through methods such as aesthetic scoring can also help improve the aesthetics of the composition of the set target interface.

[0061] Among them, the above-mentioned significant subjects may include people, animals, plants, food, buildings or scenery, etc. Correspondingly, the features of the significant subjects determined based on the image semantic information may include the movement characteristics of people or animals, the morphological characteristics of plants, the color characteristics of food, and the features of representative buildings or scenery, etc., which are not limited here. The above-mentioned representative buildings may be the parts of the multiple buildings in the original background image whose representativeness is higher than the corresponding threshold, and the above-mentioned representative scenery or multiple landscape components whose representativeness is higher than the corresponding threshold. In some embodiments, when there are a large number of significant subjects such as people or animals in the original background image, one of the subjects such as people or animals may be determined as the significant subject for extracting relevant features by calculating the representativeness of each subject. The above-mentioned representativeness can be obtained by processing the original background image using a saliency detection algorithm or other methods, which will not be elaborated here.

[0062] In an embodiment of the present application, image semantic information can be used by a digital computer or a machine controlled by a digital computer to understand the content included in an image. For example, when the prominent subject in the original background image is a person, the user can understand that the image is an athlete kicking a football based on the person's actions and the object of the action; similarly, the machine can also extract information such as the person information as "athlete", the person action as "kick", the action object as "football", and the color of the action object based on the image semantic information of the original background image. Therefore, image semantic information can include but is not limited to color features, texture features, shape features, etc., and can express feature information corresponding to the content that is close to the user's understanding.

[0063] It is understood that when adjusting the display parameters of foreground elements that can affect the layout of the set target interface, the present application uses the screen size and the size and position of the salient subject in the original background image as layout constraints. The foreground elements on the set target interface can be integrated with some action-related features, morphology-related features, color features, and representative features of the salient subject, thereby producing a highly interactive and aesthetically pleasing interface display effect. In some embodiments, the above-mentioned integration process can be implemented using a pre-trained machine learning model or neural network model, etc., which is not limited here.

[0064] The above-mentioned morphological related features may include, for example, the shape of the significant subject and the direction of its extension, such as the shape of branches and leaves of a tree and the direction of their extension, etc., which are not limited here.

[0065] It is understood that the target interface to which the display method provided in this application is applicable may include the lock screen interface or desktop displayed on electronic devices such as mobile phones. In other embodiments, the target interface may also include other interfaces suitable for adjusting display parameters using the above display method, which is not limited here.

[0066] As an example, FIG. 2 a to FIG. 2 d show a schematic diagram of an interface corresponding to a display of a mobile phone when a display method is applied according to an embodiment of the present application.

[0067] As shown in reference figure 2a, on the lock screen interface 210 displayed by the mobile phone 100, the cropped image set as the lock screen wallpaper is the above-mentioned "athlete kicking football" image. At this time, based on the display method provided by the present application, the mobile phone 100 crops the image of the relevant part of the "athlete kicking football" in the original background image to obtain the cropped image, and then adapts the clock component 211 and lock screen notification 212 that need to be displayed on the lock screen interface 210 to the top of the "kick" action of the "athlete" 213. In this way, it is possible to avoid the clock component, lock screen notification, etc. from blocking the significant subjects such as the "athlete" on the wallpaper, and it is also possible to make the lock screen interface 210 displayed by the mobile phone 100 show the interactive effect of the "athlete" 213 "kicking" the football, the lock screen notification 212 and the clock component 211, thereby improving interactivity and fun, thereby facilitating the user's visual experience.

[0068] In other embodiments, the cropped image used as the lock screen wallpaper may also be other characters as the prominent subject, and the subject may exhibit other forms of body movements. Referring to FIG2b , on the lock screen interface 220 displayed by the mobile phone 100, the cropped image set as the lock screen wallpaper may also be an image of a “character crossing a cliff”. At this time, the mobile phone 100, based on the display method provided by the present application, may adapt the clock components and lock screen notifications that need to be displayed on the lock screen interface 220 to the “character” waving his arms when crossing. In this way, it is possible to avoid the clock components, lock screen notifications, etc. from blocking the prominent subjects such as the “character” on the lock screen wallpaper, and it is also possible to make the lock screen interface 220 displayed by the mobile phone 100 show the interactive effect of the “character” waving his arms to hold up the lock screen notification and clock components, thereby improving interactivity and fun, and thus facilitating the user’s visual experience.

[0069] In some embodiments, the prominent subject in the cropped image set as the lock screen wallpaper can also be a plant or a landscape. Referring to FIG2c, on the lock screen interface 230 displayed by the mobile phone 100, the cropped image set as the lock screen wallpaper can be a landscape picture with a "big tree" and a "flock of birds". At this time, the mobile phone 100 can adapt the clock components and lock screen notifications that need to be displayed on the lock screen interface 230 to be above the prominent subject of the "big tree" based on the display method provided by the present application. In this way, it can avoid the clock components, lock screen notifications, etc. from blocking the prominent subjects such as the "big tree" on the lock screen wallpaper, and it can also make the lock screen interface 230 displayed by the mobile phone 100 show the representative feature of the "big tree", the "treetop", and the interactive effect of the lock screen notifications and clock components above the "treetop", thereby improving interactivity and fun, thereby facilitating the user's visual experience.

[0070] In some embodiments, the mobile phone 100 may also be configured to display a desktop or other application interface when implementing the display method provided herein. Correspondingly, the components to be displayed on the desktop or other application interface may also include card components, application icons, etc., without limitation.

[0071] As shown in FIG2d, the cropped image set as the desktop wallpaper on the desktop 240 displayed by the mobile phone 100 continues to use the above-mentioned "athlete playing football" image as an example. The mobile phone 100 can display the WeChat image that needs to be displayed on the desktop 240 based on the display method provided by this application. TM The icons of application 241 and theme application 242 are adapted to the positions above the ball “kicked” by the “athlete” when kicking the football, below the “swinging” arm, etc. In this way, it is possible to avoid the application icons or some card components from blocking the significant subjects such as the “athlete” on the desktop wallpaper, and also to make the desktop 240 displayed by the mobile phone 100 show the “athlete” kicking the football and the “WeChat” TM Application 241" and the interactive effect of "theme application 242" enhance interactivity and fun, thereby facilitating the user's visual experience.

[0072] As an example, FIG3 shows a system framework and a schematic diagram of an implementation principle capable of implementing the above-mentioned display method according to an embodiment of the present application.

[0073] It is understood that for the lock screen interface, the components 401 to be displayed may include the above-mentioned clock components and lock screen notifications, etc. For the desktop, the components 401 to be displayed on the mobile phone 100 may include clock cards, suggestion cards, application cards and other card components providing different services, as well as some application icons, etc., which are not limited here.

[0074] Continuing with reference to FIG3 , the wallpaper service 320 may have an image semantic understanding function, capable of performing tasks such as “① semantic understanding” and “② semantic segmentation” on the original background image 300 as shown in FIG3 . The “① semantic understanding” process may, for example, determine that the semantic information corresponding to the original background image 300 is “athletes playing football” or the like. Accordingly, the “② semantic segmentation” process may segment the original background image 300, and then obtain the location information of the areas corresponding to the fields such as “athletes” and “football”. In this way, the fields such as “athletes” and “football” in the semantic information may be associated with the location information of the corresponding areas.

[0075] It is understandable that the above-mentioned semantic understanding function can be based on some image semantic understanding systems or models, such as an image processing model that generates harmonious text layout on natural images through deep aesthetic learning. The model can be trained based on a convolutional neural network (CNN) model or a recurrent neural network (RNN) model, and there is no limitation here.

[0076] For the semantic information and segmented fields identified by the aforementioned image semantic understanding function, wallpaper service 320 can perform intent recognition and slot filling based on a preconfigured natural language understanding (NLU) model, such as "③ Aesthetic Composition NLU" shown in Figure 3. Specifically, it determines the intent and slots expressed by the semantic information corresponding to original background image 300, as shown in "Intent {}" and "Slot {}" in Figure 3.

[0077] Continuing with FIG3 , the "intent {}" and "slot {}" identified by the NLU model are used as layout constraints. Combined with the layout rules for component 401 provided by desktop application 310, parameters such as component position, size, and color in the layout rules for component 401 are replaced. This determines recommended layout parameters that position component 401 to interact with the prominent subject of original background image 300. Ultimately, based on the determined recommended layout parameters, mobile phone 100 can display the corresponding interface layout effect.

[0078] Specifically, the specific implementation process of the display method shown in FIG3 can be referred to the detailed description below in combination with the relevant embodiments and drawings, and will not be described in detail here.

[0079] It can be understood that the display method provided in the embodiments of the present application is applicable to electronic devices including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, netbooks, as well as augmented reality (AR) or virtual reality (VR) devices, smart TVs, smart watches and other wearable devices, car equipment, portable game consoles, portable music players, reader devices, and electronic devices with one or more display screens and one or more processors.

[0080] Based on the application scenario shown in FIG1 above, the specific implementation process of the display method provided by the present application is introduced below in combination with specific embodiments.

[0081] Example 1

[0082] The embodiment of the present application introduces the specific implementation process of a display method in the application scenario shown in Figure 1 above. This implementation process can combine the relevant features of the prominent subject in the original background image to reasonably arrange the positions and sizes of the various components to be displayed on the set target interface, forming a set target interface layout that is interactive and aesthetically pleasing. In addition, the specific implementation process of the display method provided by the embodiment of the present application can also perform aesthetic scoring on the original background image set as the background image of the set target interface to screen suitable cropping areas, which is conducive to improving the aesthetics of the composition of the set target interface.

[0083] Specifically, Figure 4 shows a schematic diagram of an implementation process of a display method according to an embodiment of the present application. It is understood that the execution subject of each step in the implementation process shown in Figure 4 can be the above-mentioned mobile phone, tablet computer and other electronic devices, without limitation.

[0084] The following continues to use the mobile phone 100 as an example of the execution subject, and combines Figure 4 to describe in detail the specific implementation process of the display method provided by this application.

[0085] As shown in Figure 4, the implementation process may include the following steps:

[0086] 401: A user operation of setting the original background image as the target interface background image is detected.

[0087] Exemplarily, the above-mentioned original background image may be an image stored in the mobile phone 100, including photos taken by the user using the mobile phone 100, downloaded images or images in videos, etc., which are not limited here. Correspondingly, the above-mentioned set target interface may include the background image and the interfaces of one or more components displayed above the background image, such as the lock screen interface or desktop displayed by the mobile phone 100. In other embodiments, the above-mentioned set target interface may also be a chat interface of an instant messaging application, etc., which are not limited here. Correspondingly, the user operation of setting the background image may include setting a lock screen wallpaper, a desktop wallpaper, or setting a chat background of a chat interface, etc.

[0088] As an example, FIG5a to FIG5d show schematic diagrams of target interfaces set when a user sets a lock screen interface on a mobile phone.

[0089] 5a, when browsing images in the gallery of the mobile phone 100, the user can click on the control 511 on the image browsing interface 510 and select the "Set as" option 512, so that the mobile phone 100 displays the setting interface 520 shown in FIG5b.

[0090] 5b, the user can click on the "wallpaper" option 521 on the setting interface 520 to set the current image as wallpaper. At this time, the mobile phone 100 responds to the user's setting operation and can display the setting interface 530 shown in FIG5c.

[0091] Continuing with reference to FIG5c , the user can click the “Apply” control 531 on the settings interface 530 to set the current image as the “lock screen wallpaper” and / or as the “desktop wallpaper.” For example, the user can click the “Set as Lock Screen” option 532 on the settings interface 530 to set the current image as the lock screen wallpaper. For another example, the user can also click the “Set as Desktop” option 533 on the settings interface 530 to set the current image as the desktop wallpaper. For another example, the user can also click the “Set Both” option 534 on the settings interface 530 to set the current image as both the lock screen wallpaper and the desktop wallpaper. This is not a limitation.

[0092] Accordingly, when the mobile phone 100 detects the user's setting operation, it can execute the following steps 402 to 410 to complete the process of drawing, rendering, and synthesizing the lock screen interface corresponding to the setting operation, and finally display the lock screen interface 540 shown in Figure 5d. The specific display effect of the lock screen interface after the setting can be referred to the relevant description of step 410 below, which is not repeated here.

[0093] 402: Perform image semantic understanding on the original background image to obtain image semantic information.

[0094] For example, when the mobile phone 100 detects the above setting operation performed by the user, it can perform image semantic understanding on the original background image to be set as the background image, that is, the original background image, and then obtain image semantic information corresponding to the original background image.

[0095] It is understood that the above-mentioned image semantic understanding is natural language processing (NLP) based on the image content. In the embodiments of the present application, the implementation process of the above-mentioned image semantic understanding can be implemented using some image semantic understanding systems or models. A complete image understanding system can generally include the following four layers: data layer, description layer, cognitive layer and application layer.

[0096] The data layer is used to obtain image data, including binary images, grayscale images of color photos, color images, and depth images, and is mainly involved in the image compression and transmission process. The main operation object of this layer is pixels.

[0097] The description layer extracts features and measures the similarity between them, expressed as the distance between feature vectors. This measurement process can employ data dimensionality reduction methods such as subspace analysis, including independent component analysis (ICA) and principal component analysis (PCA). After dimensionality reduction, the image pixel feature data can be further symbolized to facilitate subsequent image understanding calculations. The main task of this layer is to symbolize pixel representations.

[0098] The cognitive layer is responsible for image understanding, specifically learning and inference from the symbolized pixels. It's understandable that the cognitive layer is the "engine" of the image understanding system, and its tasks are complex, including the establishment of relevant databases. The cognitive layer primarily processes symbols, such as the symbolized pixel representations mentioned above, but this is not a limitation here.

[0099] The application layer can design corresponding classifiers and learning algorithms based on task requirements, such as classification, recognition, and detection. These classifiers and learning algorithms ultimately output classification results and image semantic information for the image being understood. This section will not be elaborated on in detail.

[0100] In the embodiment of the present application, there is no specific limitation on the system or model used to implement the image semantic understanding corresponding to step 402. The electronic device, such as the mobile phone 100, can use a pre-trained image semantic understanding system or model to perform image semantic understanding on the original background image set by the user as the target interface background image, and obtain image semantic information corresponding to the image.

[0101] 403: Extract relevant features of salient subjects based on image semantic information.

[0102] Exemplarily, the mobile phone 100 performs semantic segmentation on the image semantic information obtained corresponding to the above-mentioned image semantic understanding process, and extracts fields in the semantic information that can express the location information of different regions. The fields determined corresponding to the semantic segmentation process may include fields describing significant subjects in the original background image, such as people, animals, plants, food, buildings, or scenery. In addition, the fields determined corresponding to the semantic segmentation process may also include relevant features of the significant subject, such as action-related features "kick," "run," "jump," "hold," "support," etc., as well as the significant subject.

[0103] Optionally, when there are multiple prominent subjects in the original background image, the mobile phone 100 can determine a representative prominent subject among them as a constraint condition for subsequent interface composition based on preset prominent subject screening rules. In some embodiments, the representative prominent subject can include more common prominent subjects, such as people corresponding to faces frequently recognized in the historical records of face recognition performed by the mobile phone 100. In other embodiments, the representative prominent subject can also include a subject with a higher degree of prominence, such as a subject that is centered or stands closer to the front among multiple prominent subjects, or a subject that has the largest movement amplitude among multiple prominent subjects, and there is no limitation here.

[0104] Optionally, when there are many relevant features of the salient subject in the original background image, for example, when the human subject in the original background image has multiple action-related features including raising arms, raising legs, and kicking a ball, the mobile phone 100 can determine the representative features among them as the constraint conditions for subsequent interface composition according to the preset relevant feature screening rules. In some embodiments, the representative feature can be, for example, a feature with a larger movement amplitude among multiple action-related features. In other embodiments, the representative feature can also include, for example, a feature in which the object of the action is larger or is located closer to the front of the picture, which is not limited here.

[0105] 404: Get the preset layout rules of one or more components to be displayed on the set target interface.

[0106] For example, the target interface set by the above user operation may include one or more components that need to be displayed, including a clock component and a notification component on the lock screen interface, a card component on the desktop, and a message bubble component on the chat interface, etc., without limitation. Since each component needs to display different content, the display position (i.e., component position), component size (e.g., aspect ratio, etc.), and component color of each component on the set target interface may all be different.

[0107] For example, in the operating system of an electronic device such as mobile phone 100, corresponding layout rules can be preset for the clock component, notification component, etc. displayed on the lock screen interface. For example, the above layout rules include displaying the clock component in the center of the upper area of ​​the set target interface, displaying the clock information in a preset font size, and displaying the clock component in a preset component color, aspect ratio, and other related parameters. The above layout rules can also include displaying the lock screen notification below the clock component, displaying the notification information in a preset font size, and the window color, font color, and window aspect ratio of the lock screen notification, etc. Related parameters are not repeated here.

[0108] 405: Determine the composition layout parameters of the target interface according to the relevant features of the significant subject and the preset layout rules of one or more components.

[0109] For example, after determining the relevant features of a prominent subject in the original background image set as the background image and the preset layout rules of one or more components to be displayed, the mobile phone 100 can determine the composition layout parameters of the set target interface by comprehensively considering the relevant features of the prominent subject, component color, component position, component size, and screen size. This determination process can be achieved, for example, through methods such as intent recognition and slot replacement.

[0110] It is understood that a subfield of NLP may include natural language understanding (NLU). Among them, the two most critical tasks of NLU include intent recognition and slot filling. Among them, intent recognition is the process of identifying the purpose of the significant subject expressed in the image semantic information corresponding to the above-mentioned original background image. For example, when the significant subject is a person or an animal, the intent recognized by the intent recognition process may include some common human or animal behaviors, such as kicking, straddling, supporting, holding, hugging, looking, eating, etc. Among them, each intent can be configured with one or more slots.

[0111] Slots refer to the key information that the NLU model needs to extract from the semantic information in the image. For example, the action subject and object corresponding to the action-related features identified as "intent." Slots can include attributes such as slot values. Slot values ​​refer to the specific parameters of the slot, also known as the slot entity.

[0112] For the image semantic information corresponding to the image shown in FIG5a or FIG5b , the intent recognition result achieved by running the NLU model on the mobile phone 100 may include:

[0113] In other embodiments, for action-related features of different significant subjects or other features with interactive intentions, the mobile phone 100 can also identify other intention and operation combinations based on the NLU model, such as the combination of the intention "holding" and the action subject "hand", the action object "rapeseed flower", etc., which is not limited here.

[0114] For the above intent recognition results, after adaptively expanding the "category, position, color" and other attribute fields based on the content of the above original background image as the foreground and background of the set target interface, the slot values ​​corresponding to the corresponding fields can be added to obtain the following intent recognition results including slot values:

[0115] Among them, the slot [920,810,100,100] may represent the relative coordinates of a specified position on the original background image. For example, the position coordinates may represent the relative coordinates corresponding to the position where the "kick" action of the action subject "foot" can interact, such as the relative coordinates corresponding to position 513 above the soccer ball "kicked" by the athlete in FIG5a . It is understood that the relative coordinates may be coordinate parameters relative to a corner point of the original background image, or may be coordinate parameters relative to the image preview origin preset in the mobile phone 100, etc., without limitation herein.

[0116] Correspondingly, [600,200,300,400] in the above slot represents the relative coordinates of the position of the action object "football"; [0,400,920,1120] represents the relative coordinates of the position of the foreground object "athlete" to which the action subject "foot" belongs; [0,0,1080,330] represents the relative coordinates of the position of the background "football field", etc., which will not be repeated here.

[0117] It will be appreciated that in specific applications, intent categories and slots assigned to each intent category can be pre-set. For example, for images containing prominent subjects such as people and animals, intent categories corresponding to action-related features such as kicking, stepping over, supporting, holding, hugging, looking, and eating can be pre-set. For landscape images containing prominent subjects such as buildings, delicious food, representative attractions, or distinctive objects, intent categories corresponding to the subject's up, down, left, and right orientation, or other interactive features, can be pre-set, without limitation.

[0118] Based on the above intention recognition results including slot values, slot replacement is performed, and then the composition layout parameters corresponding to the significant subject and the relevant components can be obtained. For example, based on the intention recognition results of the above example, the mobile phone 100 can replace the component position coordinate parameters in the preset layout rules of the relevant components with [920,810,100,100] in the above "action subject" slot, so as to modify the preset component position corresponding to the relevant component to [920,810,100,100] corresponding to the position indicated on the original background image. As mentioned above, at this position, the component can interact with the action subject in the original background image, that is, the "foot" of the "athlete". At the same time, the component color preset in the preset layout rules of the relevant components can also be replaced with the color channel parameter corresponding to "black" in the above "action subject" slot.

[0119] In this way, an interactive relationship can be established between the relevant components and the significant subjects of the relevant images, thereby improving the interactivity and interest between the components and the background image on the set target interface.

[0120] Based on the composition layout parameters determined in step 405 , the following further describes the process of performing aesthetic scoring on the composition determined based on the composition layout parameters and screening out recommended layout parameters corresponding to the recommended composition, which is performed in steps 406 to 408 .

[0121] 406: Determine a plurality of cropping areas in the original background image based on the cropping frame size parameters in the composition layout parameters.

[0122] Exemplarily, the cropping frame size parameters may include parameters such as the length and width of the cropping frame. By implementing the recommended layout parameters determined in step 405 above, mobile phone 100 may determine the size and position of the prominent subject, the size and position of related components, and the aspect ratio of the display screen. Based on these parameters, mobile phone 100 may determine the size parameters of the cropping frame used to crop the original background image. The length of the cropping frame may be less than or equal to the original length of the original background image, and the width of the cropping frame may be less than or equal to the width of the original background image. Furthermore, the aspect ratio of the cropping frame may be the same as the aspect ratio of the screen of mobile phone 100.

[0123] It can be understood that when the above-mentioned cropping frame is located at different positions on the original background image, the cropping area selected by the corresponding frame will also change. Therefore, based on the size parameters of the above-mentioned cropping frame, the mobile phone 100 can determine one or more cropping areas in the original background image. Among them, the image in each cropping area can be used as the background image of the set target interface, and combined with related components to form a composition result. However, the aesthetics of the composition formed by the combination of different cropping areas and related components are different. For details, please refer to the relevant description in the following step 408, which will not be repeated here.

[0124] 407: Determine a composition formed by laying out each component in each cutting area based on the component-related parameters in the composition layout parameters.

[0125] For example, the component-related parameters in the recommended layout parameters may include, without limitation, component size, component position, component color, and other related parameters. Based on the above component-related parameters, the mobile phone 100 can determine the position and size of each component on the image of each cropped area, that is, it can determine the composition formed by the background image and components in each cropped area.

[0126] It is understood that, during the process of laying out the components within the aforementioned cropping area to determine the target interface composition, the size of the components displayed on the target interface can be fixed to the size of the components in the preset layout rules, i.e., the size parameters of the relevant components can remain unchanged. In other embodiments, during the process of determining the recommended layout parameters corresponding to step 405, the mobile phone 100 can also resize the components displayed on the target interface, for example, by adaptively adjusting the size of the components based on the size of the prominent subject, etc., which is not limited here.

[0127] 408: Performing an aesthetic score on the composition formed by the layout of the relevant components in each cutout area.

[0128] Exemplarily, the above-mentioned composition is formed by combining images of various cropped areas associated with the prominent subject of the original background image with related components. System services of mobile phone 100, such as wallpaper services, may provide the above-mentioned aesthetic scoring functionality. For example, the wallpaper service of mobile phone 100 may, in response to invocations from applications such as desktop or wallpaper, perform aesthetic scoring on the multiple compositions formed by the layout of the above-mentioned related components within the cropped areas. Furthermore, based on the aesthetic scoring results, mobile phone 100 may proceed to step 409, selecting the composition layout parameters with the highest score or other parameters that meet pre-set scoring criteria.

[0129] 409: Based on the aesthetic scoring result, the composition layout parameters corresponding to the composition that meets the scoring screening condition are selected as recommended layout parameters.

[0130] Exemplarily, the above-mentioned scoring screening conditions may include, for example, the conditions for screening out the highest score in the aesthetic scoring sorting, or the conditions for screening out the aesthetic score higher than a preset scoring threshold, etc. Among them, the preset scoring threshold may include a single score, or may include multiple score and multi-level threshold conditions, etc., which are not limited here. Based on the preset scoring screening conditions, the mobile phone 100 can screen out the composition that meets the scoring screening conditions from the aesthetic scoring results, that is, the recommended composition. The composition layout parameters corresponding to the recommended composition can be provided as recommended layout parameters to a desktop application or wallpaper application, etc., and continue to execute the following step 410 for drawing, rendering and synthetic display.

[0131] It is understood that the process of executing steps 406 to 409 by the mobile phone 100 can be implemented by integrating constraint-based saliency detection and a composition scoring model. Specifically, the composition scoring model can screen out compositions with higher aesthetics based on the aesthetic scores of each composition formed by the layout of relevant components within each cropping area, and recommend them to the mobile phone 100 for further rendering and composite display.

[0132] Specifically, the mobile phone 100 implements the process of aesthetically grading the relevant composition and determining the recommended layout parameters corresponding to the recommended composition based on the above-mentioned composition scoring model in steps 406 to 409. This process will be described in detail below in conjunction with the relevant drawings and will not be repeated here.

[0133] 410 : In response to a user operation of locking the screen or switching back to the desktop, displaying a target interface rendered and synthesized according to the selected recommended layout parameters.

[0134] For example, after determining the recommended layout parameters that ensure both interactivity and aesthetics, the mobile phone 100 can then, in response to a user's lock screen operation, display a lock screen interface rendered and synthesized based on the recommended layout parameters. Alternatively, the mobile phone 100 can also display a desktop rendered and synthesized based on the recommended layout parameters in response to a user's operation of switching back to the desktop.

[0135] As an example, based on the results of executing steps 401 to 410 above, mobile phone 100 may display the lock screen interface shown in Figure 5d. Referring to Figure 5d, lock screen notification 542 may be displayed at position 541 above the soccer ball being "kicked" by the athlete on lock screen 540, and a clock component 543 and the like may be displayed near notification 542. Thus, lock screen notification 542, clock component 543, and the like interact with the "kicking" action performed by the athlete's "foot" in the background image of lock screen 540, making lock screen 540 interactive and aesthetically pleasing.

[0136] For the process of determining composition layout parameters and performing aesthetic scoring on relevant compositions to determine recommended layout parameters corresponding to the recommended composition, which is performed in steps 405 to 409 , reference may be made to the implementation process shown in FIG. 6 .

[0137] Specifically, FIG6 illustrates a schematic diagram of a process by which a composition scoring model determines recommended layout parameters based on the layout constraints of an input model, according to an embodiment of the present application. It will be appreciated that the process illustrated in FIG6 can be implemented using a composition scoring model or related algorithm pre-installed in the mobile phone 100. For example, the composition scoring model can be a neural network model, such as one trained using a CNN or RNN.

[0138] As shown in Figure 6, the input end of the composition scoring model 600 may include an embedding layer 610, also known as a mapping layer. In an embodiment of the present application, this layer can convert (or map) the intent and slot fields (referred to as fields) determined based on the understanding of image semantic information, intent recognition, and slot filling processes into vectors of a fixed size that can be recognized by the computer. In addition, the embedding layer can also convert a one hot vector into a dense vector, or convert a high-dimensional sparse feature vector into a low-dimensional dense feature vector, etc., which is not limited here.

[0139] Continuing to refer to Figure 6, the fields of the embedding layer 610 input into the composition scoring model 600 may include at least the component size and the screen size, and may also include the component position, component color, action-related features of the salient subject (i.e., the intention field) and the action object, etc., without limitation here. Among them, the screen size may include the aspect ratio of the screen, such as 16:9 or other ratios, and the component size may also include the length and width of the component, such as [200,800], etc. It can be understood that the aspect ratio of the component and the aspect ratio of the screen can be the same or similar to improve the aesthetics of the composition. The process of embedding mapping for each constraint condition, for example, includes mapping the above-mentioned component position, screen aspect ratio, component color, action-related features of the salient subject, etc. into an N-dimensional vector, such as a 1024-dimensional vector, etc.

[0140] After the various fields input to the embedding layer are mapped to vectors of corresponding dimensions, they can be further output as layout constraints to the adaptive weight module 620 of the composition scoring model 600 to calculate corresponding weights for each layout constraint. The adaptive weight module 620 can acquire the ability to calculate corresponding weights for each layout constraint through algorithmic or neural network training.

[0141] It is understood that the above process of calculating the corresponding weights for each layout constraint condition may include adaptively configuring higher weight values ​​for more important layout constraints, such as component size, component position, screen size, and action-related features; and may also include adaptively configuring lower weight values ​​for less important layout constraints, such as component color, action object, etc., without limitation. After the weight values ​​corresponding to each layout constraint condition are determined, the adaptive weight module 620 may output each layout constraint condition and the corresponding weight value to the scoring network 630 of the composition scoring model 600. Among them, the weight value corresponding to each layout constraint condition may affect the aesthetic scoring result of the composition scoring model 600 for the relevant composition. For details, please continue to refer to the relevant description below.

[0142] Continuing with Figure 6 , mobile phone 100 may encode original background image 300 using an encoder and provide the resulting encoded information to a wallpaper service configured with the aforementioned composition scoring model 600. Subsequently, mobile phone 100 running the wallpaper service may decode the received encoded information using a decoder and use it as input to the scoring network 630 of composition scoring model 600.

[0143] Furthermore, the scoring network 630 can integrate the aforementioned layout constraints and their corresponding weights to analyze the saliency of the salient subject in the decoded original background image, determine a cropping region associated with the salient subject and matching the cropping box size, and calculate an aesthetic score for the associated composition. Ultimately, the cropping region that meets the scoring criteria and the corresponding recommended layout parameters are selected. The recommended layout parameters are the parameters determined by the mobile phone 100 in step 409.

[0144] In other embodiments, the content executed corresponding to the above steps 402 to 403 can be described as an image semantic understanding (NLP) process and a natural language understanding (NLU) process for image semantic information. The above processes can determine the intention information and slot information related to the salient subject in the original background image.

[0145] Referring to Figure 7, for the original background image 300, the image semantic information corresponding to the above-mentioned NLP process and determined may be, for example, "the athlete kicks the football" as shown in Figure 7, and the intention information corresponding to the above-mentioned NLU process and determined may include, for example, "subject: {name: athlete, foot}" as shown in Figure 7, and the slot information may include, for example, "action: {name: kick}, slot: {name: football}" as shown in Figure 7.

[0146] The content executed corresponding to the above steps 404 to 407 can be described as a slot replacement process. This process can use the slot information determined and expanded by the above NLU process to replace the layout parameters related to the components to be displayed on the set target interface, so as to determine the interactive relationship between each component and the significant subject. In this slot replacement process, the corresponding determined composition layout parameters can define the component position, component size, component color, etc. of each component in the set target interface, and can also define the size of the cropping box used to crop a portion of the cropping area image containing the significant subject in the original background image. Referring to Figure 7, the above slot replacement process can, for example, adjust the relative coordinates of the component position in the preset layout rules of the relevant components based on the relative coordinates of the reference position in the replacement area shown in Figure 7, and adjust the component size of the relevant components based on the recommended size of the replacement area, thereby forming the component area shown in Figure 7.

[0147] The content executed in step 408 to step 409 can be described as a composition recommendation process. This process can perform aesthetic scoring on the composition result determined corresponding to the composition layout parameters determined after the slot replacement. The aesthetic scoring process can be performed simultaneously with the process of mapping the cropping area on the original background image based on the cropping box size in the composition layout parameters. The aesthetic scoring process can be to output the recommended composition that meets the scoring screening conditions in a timely manner according to the preset scoring threshold, or it can be to comprehensively sort the aesthetic scores corresponding to the various cropping areas and determine the recommended composition that meets the scoring screening conditions. There is no limitation here.

[0148] 7 , the composition recommendation process may, for example, cut out a cropping frame of the background image from the original background image 300. The cropping region corresponding to the cropping frame, which serves as the background image, may be the cropping region corresponding to the composition with the highest aesthetic score.

[0149] Based on the above description in combination with the relevant drawings, it can be understood that the display method provided by the embodiment of the present application can optimize the layout parameters of at least one component to be displayed on the set target interface on the background image based on the relevant features of the significant subject in the original background image set as the background image of the set target interface, so that an interactive connection is generated between the component on the set target interface and the significant subject, making the set target interface interactive and interesting. In addition, in the process of determining the recommended layout parameters, the display method provided by the embodiment of the present application can screen out a more aesthetically pleasing cropping area based on the results of the aesthetic score to compose the set target interface, which is also conducive to improving the aesthetics of the display of the set target interface.

[0150] It is understood that in other embodiments, the display method provided by the present application may also include other implementations, which will be described below in conjunction with another embodiment 2.

[0151] Example 2

[0152] The present application embodiment describes a specific implementation process of a display method in the application scenario shown in Figure 1. This implementation process not only rationally arranges the positions and sizes of various components to be displayed on the set target interface to form an interactive and aesthetically pleasing target interface layout, but also reduces the computational processing of using a scoring network to perform saliency analysis on the original background image and aesthetic scoring of related compositions, thereby improving the efficiency of mobile phones and other electronic devices in responding to user operations.

[0153] Specifically, Figure 8 shows a schematic diagram of an implementation process of a display method according to an embodiment of the present application. It is understood that the execution subject of each step in the implementation process shown in Figure 8 can be the above-mentioned mobile phone, tablet computer and other electronic devices, without limitation.

[0154] The following continues to use the mobile phone 100 as an example of the execution subject, and combines Figure 8 to introduce in detail the specific implementation process of the display method provided by this application.

[0155] As shown in Figure 8, the implementation process may include the following steps:

[0156] 801: A user operation of setting an original background image as a target interface background image is detected.

[0157] 802: Perform image semantic understanding on the original background image to obtain image semantic information.

[0158] 803: Extract relevant features of salient subjects based on image semantic information.

[0159] 804: Obtain preset layout rules of one or more components to be displayed on the set target interface.

[0160] The execution process of the above steps 801 to 804 is the same as that of steps 401 to 404 in the above embodiment 1. For details, please refer to the relevant description of the above steps 401 to 404, which will not be repeated here.

[0161] 805: Determine recommended layout parameters of the target interface according to the relevant features of the salient subject and the preset layout rules of one or more components.

[0162] Exemplarily, after determining the relevant features of the prominent subject in the original background image set as the background image, and the preset layout rules of one or more components that need to be displayed, the mobile phone 100 can comprehensively consider the relevant features of the prominent subject, component color, component position, component size, and screen size, etc., and then determine the recommended layout parameters of the set target interface.

[0163] It is understandable that in the recommended layout parameters, the cropped area of ​​the background image can be an area obtained by centering the prominent subject on the original background image, or an area cropped based on other preset cropping rules, which is not limited here.

[0164] 806 : In response to a user operation of locking the screen or switching back to the desktop, displaying the target interface rendered and synthesized according to the recommended layout parameters.

[0165] For example, after determining the recommended layout parameters that ensure both interactivity and aesthetics, the mobile phone 100 can then, in response to a user's lock screen operation, display a lock screen interface rendered and synthesized based on the recommended layout parameters. Alternatively, the mobile phone 100 can also display a desktop rendered and synthesized based on the recommended layout parameters in response to a user's operation of switching back to the desktop.

[0166] It can be understood that in the embodiment of the present application, the above-mentioned recommended layout parameters do not need to go through the calculation process such as embedding mapping by the composition scoring model, calculating the weights corresponding to each layout constraint condition, and performing aesthetic scoring on the relevant composition to screen out the recommended layout parameters corresponding to the recommended composition. Therefore, based on the implementation of the above-mentioned steps 801 to 806, the embodiment of the present application can achieve a rapid response to the user's operation of setting the background image of the set target interface, and improve the processing efficiency of cropping the relevant original background image and forming the set target interface composition. Moreover, based on the above-mentioned steps 801 to 806, the embodiment of the present application can also improve the interactivity and fun between the components in the set target interface and the prominent subject of the background image.

[0167] FIG9 shows a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0168] In some embodiments of the present application, the electronic device may be the mobile phone 100. In other embodiments, the electronic device may also be other electronic devices such as a tablet computer, which is not limited here.

[0169] As shown in Figure 9, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identity module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0170] It should be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown, or some components may be combined or separated, or arranged differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0171] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0172] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0173] In an embodiment of the present application, the processor 110 of the mobile phone 100 can read instructions related to the execution content of steps 401 to 410 in the above-mentioned embodiment 1, or instructions related to the execution content of steps 801 to 806 in the above-mentioned embodiment 2 through the above-mentioned controller, and control the execution of relevant instructions to implement the display method provided by the present application.

[0174] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0175] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a universal serial bus (USB) interface.

[0176] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the mobile phone 100 and to transfer data between the mobile phone 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0177] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0178] The charging management module 140 is configured to receive charging input from a charger. While charging the battery 142 , the charging management module 140 can also provide power to the electronic device through the power management module 141 .

[0179] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160.

[0180] The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0181] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0182] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G / 5.5G applied on the mobile phone 100.

[0183] The wireless communication module 160 can provide wireless communication solutions for application on the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0184] Mobile phone 100 implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0185] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel.

[0186] Mobile phone 100 implements a camera function through an ISP, camera 193, video codec, GPU, display 194, and application processor. The ISP processes data fed back by camera 193. Camera 193 is used to capture still images or video. The digital signal processor processes digital signals, including digital image signals and other digital signals. The video codec compresses or decompresses digital video.

[0187] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0188] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the mobile phone 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0189] The mobile phone 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0190] Keys 190 include a power button, a volume button, etc. Keys 190 may be mechanical keys or touch keys. Mobile phone 100 may receive key inputs and generate key signal inputs related to user settings and function control of mobile phone 100.

[0191] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0192] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0193] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone 100 by inserting or removing the SIM card into or from the SIM card interface 195.

[0194] FIG10 shows a schematic diagram of the software structure of an operating system of an electronic device according to an embodiment of the present application.

[0195] It is understood that the operating system of the mobile phone 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the system software structure of the mobile phone 100.

[0196] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Android TM The system is divided into four layers, from top to bottom: application layer, application framework layer, Android TM Runtime (Android TM runtime) and system libraries, as well as the kernel layer.

[0197] As shown in Figure 10, the application layer may include a series of application packages. The application packages may include desktop, gallery, wallpaper, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0198] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0199] The application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and wallpaper service.

[0200] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0201] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0202] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0203] The phone manager is used to provide communication functions of the mobile phone 100, such as management of call status (including answering, hanging up, etc.).

[0204] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0205] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0206] In an embodiment of the present application, the notification manager can also display the notification message of the application in the form of a lock screen notification on the lock screen interface. The display position of the lock screen notification on the lock screen interface can be determined based on the display scheme provided in this application.

[0207] The wallpaper service can provide the desktop application, wallpaper application, etc. of the application layer with the function of processing the original background image into wallpaper and determining the target interface composition to be set. In response to the call of the desktop application or wallpaper application, the wallpaper service can implement the relevant contents of steps 402 to 409 in Example 1 of this application, or implement the relevant contents of steps 802 to 805 in Example 2 of this application. The specific implementation process can refer to the relevant description of the relevant steps above and is not repeated here.

[0208] Android TM Runtime includes core libraries and virtual machines. Android TM runtime is responsible for Android TM System scheduling and management.

[0209] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0210] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0211] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0212] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. The media library supports a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0213] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0214] The embodiments of the present application also provide a computer program product for implementing the display methods provided in the above embodiments.

[0215] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as computer program modules or module codes executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0216] A computer program module or module code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0217] Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0218] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0219] References in the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one exemplary implementation or technique disclosed according to the embodiment of the present application. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0220] The disclosure of the embodiments of the present application also relates to an operating device for executing the text. The device can be constructed specifically for the required purpose or it can include a general-purpose computer that is selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC) or any type of medium suitable for storing electronic instructions, and each can be coupled to a computer system bus. In addition, the computer mentioned in the specification can include a single processor or can be an architecture involving multiple processors for increased computing power.

[0221] In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present disclosure of embodiments is intended to illustrate, not to limit, the scope of the concepts discussed herein.

Claims

1. A display method, applied to an electronic device, characterized in that: include: Interface display instructions are detected; In response to the interface display instruction, a first interface is displayed, wherein the first interface includes at least one first foreground element and a first background image area in a first background image, wherein: The display parameters of the first foreground element in the first background image area include position parameters of the first foreground element, and the position parameters are related to semantic information of at least one display element in the first background image area.

2. The method according to claim 1, characterized in that The semantic information includes one or more of the following features: Action-related features; Morphology-related features.

3. The method according to claim 2, characterized in that The action-related features include one or more of the following features: Used to describe the characteristics of an action; Characteristics of an object used to describe an action.

4. The method according to claim 2, characterized in that: The morphology-related features include one or more of the following features: Used to describe the characteristics of the morphology; Characteristics used to describe the direction of extension of a morphology.

5. The method according to any one of claims 1 to 4, characterized in that Before displaying the first interface, the method further includes: Performing image semantic understanding on the first background image to obtain semantic information related to the at least one display element and regional position information corresponding to features included in the semantic information; Determine first region position information corresponding to a first feature among the features included in the semantic information; The position parameter of the first foreground element is determined according to the first region position information.

6. The method according to claim 5, characterized in that The first feature is a feature of an object describing the action, and, The determining the position parameter of the first foreground element according to the first region position information includes: The first region position information is determined as a position parameter of the first foreground element.

7. The method according to any one of claims 1 to 4, characterized in that The display parameters also include color parameters of the first foreground element, and, The color parameters are associated with one or more of the following colors: an initial color of the first foreground element; The background color of the area where the features of the object describing the action are located; Background color of the area containing features that describe the direction of extension of the morphology; The color of a first display element in the at least one display element.

8. The method according to claim 5, characterized in that The method of determining the first background image area includes: Determining size parameters of a first cropping frame for cropping the first background image area on the first background image based on a screen size of the electronic device; determining, according to first semantic information of a first display element, position information of the first display element in the first background image, and size parameters of the first display element, at least one cropping region that satisfies the size parameters of the first cropping frame and includes the first display element, wherein the at least one display element includes the first display element; A first cropping area among the at least one cropping area is determined as the first background image area.

9. The method according to claim 8, characterized in that The size parameter of the first cropping box is also related to the size parameter of the first foreground element, and the method of determining the first background image area includes: Based on the screen size of the electronic device and the size parameter of the first foreground element, the size parameter of the first cropping box is determined.

10. The method according to claim 8 or 9, characterized in that: The at least one cutting area is a plurality of cutting areas, and, The step of determining a first cropping area among the at least one cropping area as the first background image area comprises: An aesthetic score is performed on the at least one cropped area, and a first cropped area whose aesthetic score satisfies a third condition is determined as the first background image area, wherein: The third condition includes the aesthetic score being higher than the second threshold, or the highest value among the aesthetic scores.

11. The method according to any one of claims 1 to 10, characterized in that The first interface includes a lock screen interface or a desktop, and the interface display instruction includes: A display instruction triggered by a first user operation indicating display of a lock screen interface; or, The second user operation indicating displaying the desktop corresponds to the triggered display instruction.

12. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the display method described in any one of claims 1 to 11.

13. A computer readable medium, characterized in that The readable medium stores instructions, which, when executed on a computer, enable the computer to execute the display method according to any one of claims 1 to 11.

Images