System and method for widget display management

Abstract

A system and method are provided for widget display management of wallpaper on a user equipment (UE). The method includes: detecting at least one of a current display position and a current style of at least one widget displayed on the wallpaper; detecting one or more objects present in multiple areas of the wallpaper; estimating one or more changes required for the at least one widget based on at least one of the current display position and the current style, and also based on the one or more objects; and changing at least one of the current display position and the current style of the at least one widget based on the estimated one or more changes.

Description

Technical Field

[0001] This disclosure relates to the field of managing widget displays. More specifically, this disclosure relates to a method and system for managing the widget display of wallpapers displayed on a user equipment (UE). Background Technology

[0002] With the advancement of technology and design patterns, there is a continuous drive to enhance user interfaces (UIs) to make them visually appealing and immersive. One area of ​​focus is the placement of widgets within the UI. Various solutions have been developed to determine the optimal positioning of widgets to maximize usability, visual appeal, and user engagement. One existing approach is depth-based widget placement.

[0003] The above information is presented as background information only to aid in understanding this disclosure. No determination or assertion is made regarding whether anything described above can be used as prior art in relation to this disclosure. Summary of the Invention

[0004] Technical solution The present disclosure aims to address at least the aforementioned problems and / or disadvantages, and to provide at least the following advantages. Therefore, one aspect of the present disclosure is to provide a method and system for managing the widget display of wallpapers displayed on a user equipment (UE).

[0005] Additional aspects will be set forth in part in the description which follows, and in part will be apparent from the description or may be learned by practice of the embodiments presented.

[0006] According to one aspect of this disclosure, a method for managing the display of widgets on a user device is provided. The method includes: detecting at least one of a current display position and a current style of at least one widget displayed on the wallpaper; detecting one or more objects present in multiple areas of the wallpaper; estimating one or more changes required for the at least one widget based on at least one of the current display position and the current style, and also based on the one or more objects; and changing at least one of the current display position and the current style of the at least one widget based on the estimated one or more changes.

[0007] According to another aspect of this disclosure, a system for managing the display of widgets on a user equipment (UE) wallpaper is provided. The system includes a memory storing one or more computer programs; and one or more processors communicatively coupled to the memory, wherein the one or more computer programs include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the system to: detect at least one of a current display position and a current style of at least one widget displayed on the wallpaper; detect one or more objects present in a plurality of areas of the wallpaper; estimate one or more changes required for the at least one widget based on at least one of the current display position and the current style, and also based on the one or more objects; and change at least one of the current display position and the current style of the at least one widget based on the estimated one or more changes.

[0008] According to another aspect of this disclosure, one or more non-transitory computer-readable storage media are provided, storing one or more computer programs including computer-executable instructions that, when executed individually or jointly by one or more processors of an electronic device, cause the electronic device to perform operations. The operations include: detecting at least one of a current display position and a current style of at least one widget displayed on a wallpaper; detecting one or more objects present in multiple areas of the wallpaper; estimating one or more necessary changes to the at least one widget based on at least one of the current display position and the current style, and also based on the one or more objects; and changing at least one of the current display position and the current style of the at least one widget based on the estimated one or more changes.

[0009] Other aspects, advantages, and salient features of this disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments disclosed in conjunction with the accompanying drawings. Attached Figure Description

[0010] The above and other aspects, features and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, wherein: Figure 1A and Figure 1B A schematic diagram depicting a problem associated with a solution for micro-part placement based on relevant technologies is shown; Figure 2 A schematic diagram depicting a micro-partition displayed using existing technology according to relevant techniques is shown; Figure 3A and Figure 3B A schematic diagram showing a depiction of a micro-part according to various embodiments of the present disclosure; Figure 4A block diagram depicting a system for widget display management according to an embodiment of the present disclosure is shown; Figure 5 A block diagram is shown depicting one or more modules of a system for widget display management according to embodiments of the present disclosure; Figure 6 A flowchart depicting the operation flow of a system for widget display management according to an embodiment of the present disclosure is shown; Figure 7 A schematic diagram depicting scene diagram determination by a scene understanding module according to an embodiment of the present disclosure is shown; Figure 8 A schematic diagram depicting the configuration features according to an embodiment of the present disclosure is shown; Figure 9A A flowchart depicting an operational flow for determining relevant objects according to embodiments of the present disclosure is shown; Figure 9B A schematic diagram depicting the determination of attention scores according to embodiments of the present disclosure is shown; Figure 9C A schematic diagram illustrating the determination of scene graph relevance scores according to embodiments depicting the present disclosure is shown. Figure 9D A schematic diagram illustrating an implementation of a modified widget generation module according to an embodiment depicting the present disclosure is shown. Figure 10 A schematic diagram depicting a modified micro-part according to an embodiment of the present disclosure is shown; and Figure 11A and Figure 11B A flowchart illustrating a method for widget display management according to various embodiments of the present disclosure is shown.

[0011] Throughout the accompanying drawings, it should be noted that the same reference numerals are used to depict the same or similar elements, features, and structures. Detailed Implementation

[0012] The following description, with reference to the accompanying drawings, is provided to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these are merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Additionally, for clarity and brevity, descriptions of well-known functions and constructions may be omitted.

[0013] The terms and words used in the following description and claims are not limited to their literal meaning, but are used by the inventors only to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the contents of this disclosure is for illustrative purposes only and is not intended to limit the disclosure as defined by the appended claims and their equivalents.

[0014] It should be understood that, unless the context clearly specifies otherwise, the singular form includes the plural indicator. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0015] Throughout this specification, references to "one aspect," "another aspect," or similar language mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of this disclosure. Therefore, the use of phrases such as "in one embodiment," "in another embodiment," and similar language throughout this specification may, but not necessarily, refer to the same embodiment.

[0016] It should be understood that, as used herein, terms such as “comprising,” “including,” and “having” are intended to mean that one or more listed features or elements are within the defined element, but the element is not necessarily limited to the listed features and elements, and additional features and elements may be included within the meaning of the defined element. Conversely, terms such as “consisting of” are intended to exclude features and elements not listed.

[0017] The embodiments described herein, along with their various features and advantageous details, are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques have been omitted to avoid unnecessarily obscuring the embodiments herein. Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein means non-exclusive. The examples used herein are intended only to help understand how the embodiments herein can be implemented and to further enable those skilled in the art to implement the embodiments herein. Therefore, the embodiments should not be construed as limiting the scope of the embodiments herein.

[0018] As is conventional in the art, embodiments can be described and illustrated from the perspective of blocks that perform one or more functions as described herein. These blocks, which may be referred to herein as units or modules, are physically implemented by analog or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuitry, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may optionally be driven by firmware and software. For example, the circuitry may be embodied in one or more semiconductor chips or on a substrate support such as a printed circuit board. The circuitry constituting a block may be implemented by dedicated hardware, a processor (e.g., one or more programmed microprocessors and associated circuitry), or a combination of dedicated hardware for performing certain functions of the block and a processor for performing other functions of the block. Without departing from the scope of this disclosure, each block of an embodiment may be physically divided into two or more interacting and discrete blocks. Similarly, without departing from the scope of this disclosure, the blocks of an embodiment may be physically combined into more complex blocks.

[0019] The accompanying drawings are provided to aid in the easy understanding of the various technical features, and it should be understood that the embodiments presented herein are not limited to the drawings. Therefore, except for those specifically set forth in the drawings, this disclosure should be construed as extending to any modifications, equivalents, and substitutions. Although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally used only to distinguish one element from another.

[0020] It should be understood that each block in a flowchart and combination of flowcharts can be executed by one or more computer programs including instructions. The entirety of one or more computer programs can be stored in a single memory device, or one or more computer programs can be divided into different parts stored in multiple different memory devices.

[0021] Any of the functions or operations described herein may be processed by a single processor or a combination of processors. A single processor or combination of processors is a circuit that performs processing and includes circuitry such as: an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, and Bluetooth. ® Chips, Global Positioning System (GPS) chips, Near Field Communication (NFC) chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio CODEC chips, Universal Serial Bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), system-on-a-chip (SoCs), ICs, etc.

[0022] Figure 1A and Figure 1B The diagram illustrates a problem associated with existing solutions for micro-part placement based on relevant technologies.

[0023] Reference Figure 1A and Figure 1B Depth-based widget placement techniques, according to relevant technologies, often fail to consider the perceived depth of content and surrounding elements, resulting in widgets that may appear disjointed or occluded. For example, Figure 1A Image 100 depicts two people fishing by a lake, with lush green trees further out. As shown, widgets (such as a time widget) are placed behind the trees when overlaid on the image using a depth-based widget placement method. The widgets do not blend well with the elements of the image, making them appear unattractive. Furthermore, even the full view of the widgets is obscured. Figure 1B A similar problem can be seen in image 100, which depicts a pair of sunglasses placed on a beach deck. When covered using a depth-based widget placement method, the time widget is behind the sunglasses. The view of the time widget is obscured, and the time widget appears out of place in the image. Therefore, existing methods present additional challenges leading to unattractive widget placement.

[0024] Therefore, an improved solution is needed to address the aforementioned problems.

[0025] As mentioned earlier, existing technologies for widget displays lack scene awareness when overlaying widgets onto a given wallpaper. More specifically, there is a lack of correlation between the widget and the wallpaper. Such technologies typically rely on the fixed positioning of the widget, thus limiting the widget's adaptability to different scenes.

[0026] Figure 2 A schematic diagram 200 depicting a display micro-device according to related technologies is shown.

[0027] Reference Figure 2 Widget 201 can be overlaid on an image 203 of a building displayed as wallpaper on the display 205 of the user device 207. Widget 201 appears to have been placed in a fixed position and does not appear to be part of the scene depicted in image 203. These solutions represent progress in making widget positioning more dynamic. However, even such methods lack scene awareness.

[0028] The purpose of this disclosure is to address the aforementioned limitations by providing a technique for displaying one or more widgets on wallpaper in such a manner that the widgets appear to blend into the scene depicted in the wallpaper and are context-aware with respect to the scene depicted in the wallpaper. According to embodiments of this disclosure, the wallpaper may be a user-preferred visual medium. In embodiments, the visual medium may be an image or a video.

[0029] This disclosure achieves the above objectives by providing a widget display management system and method. Regarding the disclosed system and method, this document describes a technique for enabling widget scene awareness to enhance user experience and improve the immersiveness of wallpaper and widget composition.

[0030] exist Figure 3A and Figure 3B The image depicts widget 201 on user device 207 and its display according to the described technique. Although the example used herein depicts only one widget, it should be noted that this technique can be implemented on multiple widgets that may exist on the wallpaper.

[0031] Figure 3A and Figure 3B A schematic diagram 300 is shown depicting a microdevice according to various embodiments of the present disclosure.

[0032] Reference Figure 3A When user device 207 includes system 301 for widget display management, widget 201 can be modified to appear as widget 201A. Widget 201A can be displayed on the wallpaper of image 203 in such a way that widget 201A appears as a giant banner hanging on the building shown in image 203. Figure 3B In another result depicted, widget 201 can be altered to appear as widget 201B. The altered widget 201B can be displayed in image 203 in such a way that widget 201B appears as a banner displayed on a billboard pole near the building shown in image 203.

[0033] The detailed method is explained in the following paragraphs of this disclosure.

[0034] Figure 4 A block diagram 400 is shown depicting a system 301 for widget display management according to an embodiment of the present disclosure.

[0035] Reference Figure 4 System 301 is configured to implement a method for managing widget displays in user device 207. User device 207 can be various types of devices, such as, but not limited to, personal digital assistants (PDAs), electronic photo frames, e-books, electronic notebooks, Moving Picture Experts Group (MPEG)-2 Audio Layer III (MP3) players, tablet PCs, laptops, monitors, kiosks, or tablet PCs.

[0036] System 301 includes one or more processors 401, memory 403, one or more modules 405, storage device 407 and display 205 coupled to each other.

[0037] Processor 401 may be a single processing unit or multiple units, all of which may include multiple computing units. Processor 401 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, logic processors, virtual processors, state machines, logic circuits, and / or any means of manipulating signals based on operating instructions. Among other capabilities, processor 401 may be configured to fetch and execute computer-readable instructions and data stored in memory 403.

[0038] Memory 403 may include any non-transitory computer-readable medium known in the art, including, for example, volatile memory (such as static random access memory (SRAM) and dynamic random access memory (DRAM)) and / or non-volatile memory (such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk and magnetic tape).

[0039] Module 405 may include programs, subroutines, portions of programs, software components, or hardware components capable of performing one or more functions, as discussed throughout this disclosure. As used herein, module 405 may be implemented independently of other modules on a hardware component (such as a server), or the module may reside on the same server as other modules, or within the same program. Module 405 may be implemented on a hardware component such as a processor, one or more microprocessors, a microcomputer, a microcontroller, a digital signal processor, a central processing unit, a state machine, logic circuitry, and / or any means of manipulating signals based on operating instructions. When executed by processor 401, module 405 may be configured to perform any of the functions described herein.

[0040] Storage device 407 can be implemented using integrated hardware and software. The hardware may include a hardware disk controller with programmable search capabilities or a software system running on general-purpose hardware. Examples of storage device 407 may include, but are not limited to, in-memory databases, cloud databases, distributed databases, embedded databases, etc. Storage device 407, among other functions, also serves as a repository for storing data processed, received, and generated by one or more processors and modules / engines / units.

[0041] Module(s) 405 may be implemented using one or more AI modules, including multiple neural network layers. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), and restricted Boltzmann machines (RBMs). According to some embodiments, module(s) 405 may be implemented using one or more generative AI modules, which may include variational autoencoders (VAEs), generative adversarial networks (GANs), flow-based generative models, autoregressive models, etc. Furthermore, "learning" in this disclosure can refer to a method for training a predetermined target device (e.g., a robot) using multiple learning data to enable, allow, or control the target device to make determinations or predictions. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. At least one of multiple CNNs, DNNs, RNNs, RMNs, VAEs, GANs, flow-based generative models, autoregressive models, etc., may be implemented to execute the mechanisms of this subject matter through an AI model or a generative AI model. Functions associated with the AI ​​module or generative AI model may be executed via non-volatile memory, volatile memory, and a processor. The processor may include one or more processors. One or more processors may be general-purpose processors (such as central processing units (CPUs), application processors (APs), graphics processing units only (such as graphics processing units (GPUs), vision processing units (VPUs)), and / or AI-specific processors (such as neural processing units (NPUs)). One or more processors or neural processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models or generative AI models stored in non-volatile and volatile memory. Predefined operating rules or AI models are provided through training or learning.

[0042] Display 205 can be configured to display wallpaper on user device 207 and display at least one widget on the wallpaper. Display 205 may include various types of display panels, such as liquid crystal display (LCD), organic light-emitting diode (OLED), electrophoretic display (EPD), electrochromic display (ECD), and plasma display panel (PDP).

[0043] Now combine Figures 5 to 8 , Figures 9A to 9D and Figure 10 Module 405 is described below (one or more).

[0044] Figure 5 A block diagram is shown depicting one or more modules of a system for widget display management according to embodiments of the present disclosure.

[0045] Figure 6A flowchart depicting the operation flow of a system 301 for widget display management according to an embodiment of the present disclosure is shown.

[0046] Reference Figure 5 and Figure 6 The description of operation procedure 600 will be explained in the following paragraphs. For ease of explanation and understanding, reference numerals have been kept the same for similar components throughout this disclosure.

[0047] One or more modules 405 may include a scene understanding module 501, a widget modification generation module 503, and a configuration generation module 505.

[0048] Initially, in operation 601, for a given widget 201 overlaid on wallpaper, at least one of the current display position and current style of widget 201 is detected. Visual media (e.g., image 203) may be displayed as wallpaper on user device 207. The current display position refers to the specific location of the widget on display 205. The style of the widget refers to the visual appearance and formatting characteristics of the widget. Such characteristics may include attributes, including but not limited to color, size, font, or any properties related to defining how the widget appears on display 205. Therefore, the current style of widget 201 refers to the current visual appearance and formatting characteristics of widget 201.

[0049] In operation 603, one or more objects are detected in multiple regions of the wallpaper (i.e., image 203 in this example) by generating a scene map associated with the scene depicted in image 203. For example, in image 203, the sky and roads may correspond to multiple regions, while buildings may correspond to objects in multiple regions.

[0050] In operation 605, one or more changes required for widget 201 are estimated, at least based on the detected current display position and current style. The required one or more changes may be based on one or more objects detected in multiple areas of the wallpaper (image 203). Scene understanding module 501 is described in more detail below.

[0051] The widget generation module 503 can modify at least one of the current display position and current style of widget 201 based on an estimated number of desired changes, to generate at least one modified widget. The at least one modified widget can be associated with, for example... Figure 2 The micro-parts 201A and 201B depicted correspond to each other. In operation 607, in response to one or more changes required by estimation, the micro-part generation module 503 identifies at least one relevant region from multiple regions for placing the micro-part 201.

[0052] In operation 609, based on at least one identified relevant region, the widget generation module 503 obtains a list of relevant widget styles based on one or more corresponding characteristics of the at least one relevant region. Widget 201 can be modified by changing at least one of its current display position and current style. In operation 611, the widget generation module 503 generates at least one modified widget, such that the at least one modified widget is defined by at least one of a changed display position and a changed style. The changed display position may refer to at least one relevant region. The changed style can be determined based on the obtained list of relevant widget styles and one or more corresponding characteristics of the at least one relevant region. (The following paragraphs / in conjunction with...) Figure 9A , Figure 9B , Figure 9C , Figure 9D and Figure 10 The changes to widget generation module 503 are described in more detail.

[0053] In operation 613, the configuration generation module 505 generates one or more component configurations based on at least one modified component, at least one associated region, and one or more corresponding characteristics associated with the at least one associated region. The configuration generation module 505 is described in more detail below.

[0054] In operation 615, the configuration generation module 505 renders the wallpaper by placing a preferred widget configuration onto the wallpaper. According to embodiments of this disclosure, a user can select a preferred widget configuration from at least one widget configuration generated by the configuration generation module 505.

[0055] The various modules (one or more) are described in more detail below 405.

[0056] The scene understanding module 501 can be configured to detect the current display position and current style of the widget 201. The scene understanding module 501 can also be configured to detect one or more objects present in multiple areas of the wallpaper (i.e., image 203). To detect one or more objects, the scene understanding module 501 can generate a scene graph associated with the scene depicted in image 203, such as... Figure 7 The image depicted in the painting.

[0057] Figure 7 A schematic diagram 700 is shown depicting a scene diagram determined by a scene understanding module according to an embodiment of the present disclosure.

[0058] Reference Figure 7 A scene graph represents one or more objects existing in a scene, as well as the relationships between those objects determined by their interactions. A scene graph is a hierarchical data structure used to represent the spatial and logical relationships between one or more objects in a scene.

[0059] In a scene graph, one or more objects are organized into a tree structure as shown, enabling an effective understanding of the scene, as illustrated in block 701. This understanding is important for determining the relevance of additional objects that may be included within the scene. Furthermore, the scene graph allows for the determination of how additional objects can interact with existing objects in the scene.

[0060] The scene understanding module 501 can also determine the segmentation map and depth map of the wallpaper (e.g., image 203). A predetermined segmentation model can be used to determine the segmentation map. According to embodiments of this disclosure, the predetermined segmentation model can be a SegmentAnything model. The SegmentAnything model is a well-known and widely used tool for image segmentation. This model uses a Semantic SegmentAnything engine (SSA engine) to provide rich semantic category annotations, enabling the generation of labeled segmentation maps for any given image. Therefore, the labeled segmentation maps provide accurate object detection and classification.

[0061] Furthermore, a predetermined framework can be used to determine the depth map. In an embodiment, the predetermined framework may be a predefined spatial channel multitasking cue framework based on a neural network model. This framework seamlessly combines the learning of task-specific and task-general representations and facilitates cross-task interactions at each layer of the neural network throughout the framework's architecture. This seamless combination is achieved by embedding task cues and patch tokens into the neural network model.

[0062] Task cues and word fragments are components used within the spatial-channel multitasking cues framework to guide and facilitate the learning of task-specific and task-general representations in neural network models. Task cues are transformed into spatial and channel task cues, enabling neural network models to learn spatial and channel interactions, which is crucial for generating dense predictions such as depth maps.

[0063] Furthermore, spatial and channel task cues, along with word fragments, play a crucial role in guiding the extraction of task-specific features for intensive tasks and contribute to multi-task prediction. Multi-task prediction refers to the ability of a neural network model to make predictions simultaneously within a single model. For example, when analyzing a busy road scene, a neural network model can simultaneously perform several related tasks based on an input image of the busy road scene. In the context of busy road scene analysis, these related tasks might include object detection, lane detection, and depth estimation. In general, the multi-task prediction capability of the spatial-channel multi-task cueing framework leverages shared information within the image (in this example, an image of a busy road) such as object edges, road markings, and the overall context of the scene to provide a comprehensive understanding of the scene.

[0064] The scene understanding module 501 can be configured to estimate one or more changes required for the widget 201 based on the widget 201's current display position, the widget 201's current style, and at least one of the detected objects. To estimate the required changes, the scene understanding module 501 can detect at least one of the following: overlap between the widget 201 and at least one of the detected objects, and a lack of correlation between the widget 201 and the wallpaper.

[0065] Overlap can be caused by at least one of the current display position and current style of at least one widget. A lack of correlation between at least one widget and the wallpaper can be detected based on one or more characteristics associated with multiple areas of the wallpaper. Finally, upon detecting at least one of overlap or a lack of correlation, one or more changes required to at least one of the current display position and current style of at least one widget are estimated.

[0066] One or more objects detected in the scene depicted in image 203, the relationships between one or more objects determined, the determined segmentation map, and the depth map can be used to generate a compositional feature map. The compositional feature map can be modified by widget generation module 503 to identify at least one relevant region from multiple regions of the wallpaper (image 203).

[0067] A compositional feature diagram can refer to one or more features of multiple areas of a wallpaper. For example, a compositional feature diagram can refer to one or more features of multiple areas of image 203. The features of multiple areas of the wallpaper may include material features, structural features, and scene features.

[0068] For a given wallpaper, such as image 203, material properties refer to the physical characteristics of multiple areas. For example, material properties describe physical characteristics (such as, but not limited to, reflective surface, transparency, translucency, color, coolness, heat, brightness, darkness, etc.). The material properties of one or more objects present in multiple areas can define physical characteristics (such as, but not limited to, the surface, material, texture, color, size, and shape of one or more objects). Physical characteristics associated with the surface of one or more objects can describe one or more objects as being rough or smooth, glossy or dull, hard or soft, flexible or rigid, etc.

[0069] Physical characteristics associated with the material of one or more objects can describe the construction of one or more objects, such as, but not limited to, wood, metal, plastic, stone, and glass. Therefore, physical characteristics associated with the material of one or more objects may include, but are not limited to, non-reflective, reflective, translucent, transparent, color, cold, hot, bright, and dark.

[0070] Structural characteristics can refer to the specifications of multiple regions. Specifications describe how one or more objects in multiple regions will look. Specifications can further describe what one or more objects are used for and what additional characteristics one or more objects will have in a real-world setting.

[0071] According to embodiments of this disclosure, structural characteristics may define specifications such as building characteristics, for example, a shopping mall with large billboards or banners. According to another embodiment of this disclosure, specifications may define road or street characteristics, such as narrowing roads, lane markings including painted lines, arrows, or symbols. According to yet another embodiment of this disclosure, specifications may define sky characteristics, for example, a clear blue sky or a cloudy sky.

[0072] Scene characteristics can refer to the dynamics of a scene depicted in a wallpaper (image 203) that includes multiple areas. Scene dynamics represent how one or more objects interact within the scene. For example, a small shop on a busy street, a shop on a lonely street, or an elderly person holding a child in an interior setting with a window and curtains behind them.

[0073] As described above, a compositional feature map is generated using one or more objects, the relationships between one or more objects, and the determination of the wallpaper's segmentation and depth maps. In embodiments, the compositional feature map may resemble a segmentation map. However, unlike a segmentation map, a compositional feature map may include labels, not limited to object categories. In addition to object categories, the compositional feature map may also capture labels associated with one or more features as described above.

[0074] Figure 8 A schematic diagram 800 is shown depicting the configuration features of an embodiment according to the present disclosure.

[0075] Reference Figure 8 For a given image 801, the compositional feature diagram 803 captures objects within the scene depicted in image 801, such as trees, sky, streets, people walking on the streets (not shown), shops, and street roads. In addition to these objects, the compositional feature diagram 803 also captures features including material, structural, and landscape characteristics, such as the shadows of trees, a clear sky, shops and sidewalks on the streets, and reflective street roads with shadows cast by pedestrians.

[0076] A predefined segmentation map estimation network can be implemented to generate a compositional feature map. For example, a segmentation map estimation network called SegVit can be implemented to generate a compositional feature map using the detection of one or more objects, the relationships between one or more objects, and the determination of the wallpaper's segmentation map and depth map. The compositional feature map is modified by the widget generation module 503 to identify one or more relevant regions from multiple regions of the wallpaper. The following is combined with... Figures 9A to 9D and Figure 10 The changes to widget generation module 503 are described in more detail.

[0077] Figure 9A A flowchart 900-1 is shown depicting an operational flow for determining relevant objects according to an embodiment of the present disclosure.

[0078] Reference Figure 9A The input image 901 is processed by following at least two series of operations described in blocks 903 and 905. The series of operations described in block 903 describes the determination of attention scores for each entity and object in image 901, and... Figure 9B It is described in more detail in the middle.

[0079] Figure 9B A schematic diagram 900-2 is shown depicting the determination of attention scores according to an embodiment of the present disclosure.

[0080] Reference Figure 9B The series of operations depicted in block 905 describes the determination of scene graph relevance scores associated with the scene depicted in image 901.

[0081] Figure 9C A schematic diagram 900-2 is shown depicting the determination of a scene graph relevance score according to an embodiment of the present disclosure. The following will now be described in conjunction with each other. Figure 9A , Figure 9B and Figure 9C .

[0082] Reference Figure 9A , Figure 9B and Figure 9C As shown in a series of operations 903, a descriptive text generator is used to obtain textual descriptions of the scene depicted in image 901. For example, refer to... Figure 9B For input image 901, a descriptive text generator can generate the descriptive text "An elderly person with a playful attitude is playing with a child." Subsequently, a predefined attention entity detection model is used to determine the attention score for each entity and object identified in image 901. The attention entity detection model determines the attention scores for entity-to-entity, object-to-object, and interactions between entities and objects. For example, for the descriptive text generated above, the attention entity detection model can determine the entity as an elderly person and its corresponding attention score. The model can also determine the object as a child and its corresponding attention score. If the attention score of an entity or object is below a predefined attention threshold, the corresponding entity or object is discarded. If the attention score of an entity or object is above the predefined attention threshold, the corresponding score of the entity or object can be added to the scene graph relevance score list. Finally, the model can determine the interaction as "playful attitude" or "playing."

[0083] As shown in a series of operations 905, a scene graph (or a feature graph) associated with the scene depicted in image 901 can be generated by a predetermined scene graph generator. Thereafter, the total in-degree and total out-degree for each node in the scene graph are calculated. For example, referring to... Figure 9C The total in-degree of the node representing "elderly" is 0, and the total out-degree of the node representing "elderly" is 3. Similarly, the total in-degree of the node representing "child" is 1, and the total out-degree of the node representing "child" is 1. Then, the maximum value of the in-degree and out-degree for each node is determined. The scene graph relevance score can be determined based on the determined maximum value of the in-degree and out-degree for each node. For example, the scene graph relevance score for "elderly" is 3, and the scene graph relevance score for "child" is 2. If the scene graph relevance score of an entity or object is lower than a predefined relevance threshold, the corresponding entity or object is discarded. If the scene graph relevance score of an entity or object is greater than the predefined relevance threshold, the corresponding scene graph relevance score of the corresponding entity or object can be added to the scene graph relevance score list.

[0084] Figure 9D Schematic diagram 900-4 illustrates an implementation of a modified widget generation module according to an embodiment of the present disclosure.

[0085] Reference Figure 9D The widget generation module 503 is configured to identify one or more relevant regions from multiple regions of the wallpaper (e.g., image 901) to place the widget 201 based on the size of multiple regions, the correlation of multiple regions, and the compositional characteristics associated with multiple regions of the wallpaper (Figure 901A).

[0086] To identify one or more related regions, the widget generation module 503 determines a set of contiguous regions from multiple regions based on segmentation information associated with the wallpaper's segmentation map and depth information associated with the depth map. For example, in image 901, window 907, curtain 909, and wall 911 can be identified as contiguous regions.

[0087] In addition, the widget generation module 503 determines the relevance score of each region in the set of continuous regions 907, 909, 911 based on one or more objects (elderly person 913, child 915, and window glass 917) and the set of continuous regions 907, 909, 911.

[0088] At least one relevant object may refer to the largest or most important object in the scene. At least one relevant object may be determined based on textual descriptions associated with the scene depicted in image 901 and a relevance score. For example, in Figure 9D In the data, elderly person 913 and child 915 can be identified as relevant individuals.

[0089] A relevance score can be determined by identifying at least one relevant object 913, 915 from one or more objects 913, 915, 917. In particular, the relevance score is determined by calculating the intersection-over-union ratio between at least one relevant object and each region in the set of continuous regions.

[0090] One or more candidate regions can be identified from the set of consecutive regions 907, 909, and 911 based on the determination of their relevance scores. Initially, the consecutive regions 907, 909, and 911 can be sorted based on the size and relevance score of each region in the set of consecutive regions. Then, one or more candidate regions are identified from the sorted set of consecutive regions such that the relevance scores of the identified one or more candidate regions are greater than a predefined relevance threshold. For example, the sorted consecutive regions could be 911, 909, or 907 based on the size and relevance score of each region. Curtain 909 and wall 911 could be identified as candidate regions.

[0091] At least one relevant region can be identified from one or more candidate regions 911, 909 based on the relationship between one or more objects and one or more candidate regions 911, 909. For example, a wall 911 can be identified as at least one relevant region that can be identified from one or more candidate regions 911, 909. Thereafter, a list of relevant widget styles can be obtained by modifying the widget generation module 503 based on one or more corresponding characteristics associated with the at least one relevant region.

[0092] The widget generation module 503 can be configured to obtain a list of relevant widget styles based on one or more characteristics of at least one identified relevant area (wall 911 in this example). A relevant widget style may refer to a modified widget. A modified widget may include a changed display position, a changed style, or both.

[0093] The changed display location may refer to at least one related area. The changed style can be determined based on a list of related widget styles and one or more corresponding characteristics associated with at least one related area. The list of related widget styles can be obtained from a widget database stored in storage device 407. A non-limiting list of related widget styles is depicted in Table I below.

[0094] [Table 1]

[0095] Figure 10 A schematic diagram 1000 is shown depicting a modified microdevice according to an embodiment of the present disclosure.

[0096] Reference Figure 10A modified widget 1001 can be generated by changing at least one of the widget's current display position and current style. The changed display position may correspond to at least one associated area. The changed style may be determined based on a list of associated widget styles and one or more corresponding properties associated with at least one associated area.

[0097] For example, for image 901, the modified widget 1001 can appear as an analog clock placed on wall 911. In this example, the appearance of the analog clock corresponds to the modified style of the modified widget 1001, and the corresponding position of the analog clock on wall 911 corresponds to the modified position of widget 1001. Subsequently, the configuration generation module 505 generates one or more widgets based on at least one modified widget.

[0098] The configuration generation module 505 can be configured to generate one or more configurations of micro-parts based on at least one modified micro-part, at least one associated region, and one or more corresponding characteristics associated with at least one associated region.

[0099] Each of one or more widget compositions may have an aesthetic score. One widget composition may have a higher aesthetic score than another. Furthermore, a suitable widget composition may have an aesthetic score greater than a predefined aesthetic threshold. The aesthetic score of a composition may depend on various factors, including but not limited to the location of the modified widget, the size of the modified widget, the relevance of the modified widget, and the color scheme of both the scene and the modified widget. The aesthetic score can be evaluated to indicate the overall compositional quality of the widget composition in terms of aesthetic value. In embodiments, predefined techniques such as, but not limited to, the Widget Composition Evaluation Dataset (WCAD) and the Saliency Enhancement Multimodal Pooling (SAMP) module can be used to evaluate the aesthetic score.

[0100] In a scene, one or more widget compositions may have an aesthetic score greater than a predefined aesthetic threshold. In this case, the user can select a preferred widget composition. Subsequently, the preferred widget composition can be displayed on a wallpaper rendered on the display 205 of the user device 207.

[0101] The composition generation module 505 can render a wallpaper by placing preferred widgets onto it. In an additional embodiment, the rendering module recognizes changed widgets in the rendered wallpaper as active components of the wallpaper and identifies the wallpaper as a static component of the wallpaper.

[0102] Active components can be configured to be refreshed when the content of a widget needs to be updated. For example, in image 901, the changed widget 1001 can be identified as an active component, while the other components of image 901 (such as wall 911, curtain 909, window 907, elderly person 913, and child 915) can be identified as static components.

[0103] According to another embodiment of this disclosure, the visual media displayed as wallpaper may be a video comprising one or more scenes. In such a scene, the composition generation module 505 may be configured to determine a span associated with each of the one or more scenes and a corresponding scene span distance. A technique based on a predefined neural network (NN) may be used to determine the span. The NN-based technique may be temporal span network video visual relationship detection. Finally, when the scene span distance is less than a predefined distance threshold, the composition generation module 505 may determine to generate a widget composition for the video. In an alternative embodiment, the composition generation module 505 may generate a predetermined widget composition when the scene span is greater than the predefined distance threshold.

[0104] To generate widget compositions for a video (rendered as wallpaper), the composition module 505 generates at least one widget composition for each frame of the video. Subsequently, the composition module 505 selects the most frequently occurring widget composition from the corresponding widget compositions associated with each frame of the video. Furthermore, the composition module 505 determines widget drift between one or more neighboring frames based on the difference between the coordinates of the center of at least one changed widget in consecutive frames.

[0105] Furthermore, the widget drift is compared to a predefined drift threshold, and when the widget drift is greater than the predefined drift threshold, the position of the selected widget in the current frame is modified based on its position in previous neighboring frames. Finally, when the widget drift is less than the predefined drift threshold, the selected widget is applied to every frame of the video.

[0106] In another embodiment, the techniques described in this disclosure can also be implemented in a visual perspective (VST) device. In such an embodiment, wallpaper can be applied to a scene viewed through a VST device.

[0107] The following is combined Figure 11A and Figure 11B Describes the publicly disclosed methods for widget display management.

[0108] Figure 11A and Figure 11B A flowchart illustrating a method for widget display management according to various embodiments of the present disclosure is shown.

[0109] Reference Figure 11A and Figure 11B Method 1100 includes a series of operations 1101 to 1121 performed by one or more components (particularly processor 401) of system 301 of user device 207.

[0110] In operation 1101, processor 401 detects at least one of the current display position and current style of at least one widget displayed on the wallpaper device.

[0111] In operation 1103, processor 401 detects one or more objects present in multiple areas of the wallpaper. A scene graph is used to detect the one or more objects. The scene graph represents one or more objects and the initial relationships between them determined based on interactions between these objects.

[0112] In operation 1105, processor 401 estimates one or more changes required for at least one widget based on at least one of the current display position and current style, and also based on one or more objects.

[0113] Estimating one or more changes may include: detecting at least one of the following: overlap between at least one widget and at least one of one or more objects, wherein the overlap is caused by at least one of the current display position and current style of at least one widget; and a lack of correlation between at least one widget and the wallpaper (203) regarding one or more characteristics associated with multiple areas of 203, 901 (203, 901). Furthermore, estimating one or more changes includes: when overlap and / or a lack of correlation are detected, estimating one or more changes required for at least one of the current display position and current style of at least one widget.

[0114] In operation 1107, processor 401 identifies at least one associated region for placing at least one widget from the multiple regions based on the size of the multiple regions, the correlation of the multiple regions, and a compositional feature map associated with the multiple regions of the wallpaper, such that the compositional feature map indicates one or more features associated with the multiple regions.

[0115] In this embodiment, a composition feature map is generated based on the determination of one or more objects, a first relationship between one or more objects, and the segmentation map and depth map of the wallpaper (203, 901).

[0116] In an embodiment, one or more characteristics correspond to material characteristics associated with the physical features of multiple regions, structural characteristics associated with the specifications of multiple regions, and scene characteristics dynamically associated with the scene depicted in the wallpaper (203, 901) which includes multiple regions.

[0117] In an embodiment, in order to identify at least one relevant region, processor 401 determines a set of continuous regions from multiple regions based on the segmentation map and depth map of the wallpaper (203, 901), determines a relevance score for each region in the set of continuous regions based on one or more objects and the set of continuous regions, identifies one or more candidate regions from the set of continuous regions based on the segmentation map and depth map, wherein the relevance score of one or more candidate regions is greater than a predefined relevance threshold, and identifies at least one relevant region from one or more candidate regions based on a second relationship between one or more objects and one or more candidate regions.

[0118] In an embodiment, determining the relevance score includes identifying at least one relevant object from one or more objects and determining the relevance score based on the intersection-union ratio between the at least one relevant object and each region in a set of contiguous regions.

[0119] In an embodiment, identifying at least one related object includes determining text descriptions and compositional feature diagrams associated with the scene depicted in the wallpaper, and identifying at least one related object based on the determined text descriptions and compositional feature diagrams.

[0120] In operation 1109, processor 401 obtains a list of associated widget styles based on one or more corresponding characteristics associated with at least one associated region.

[0121] In operation 1111, processor 401 changes at least one of the current display position and current style of at least one widget based on one or more estimated changes.

[0122] In operation 1113, processor 401 generates one or more component configurations based on at least one modified component, at least one associated region, and one or more corresponding characteristics associated with at least one associated region.

[0123] In operation 1115, processor 401 determines the aesthetic score associated with each of the generated one or more widget compositions.

[0124] In operation 1117, processor 401 provides at least one micro-component based on selection from one or more generated micro-components, wherein the aesthetic score associated with the at least one micro-component is greater than a predefined aesthetic threshold.

[0125] In operation 1119, processor 401 renders the wallpaper by placing a preferred widget configuration on the wallpaper, wherein the preferred widget configuration is selected by the user from at least one widget configuration.

[0126] In operation 1121, processor 401 identifies at least one changed widget in the rendered wallpaper as an active component of the wallpaper and identifies one or more objects of the rendered wallpaper as static components of the wallpaper, such that the active component is refreshed when the content of the widget is updated.

[0127] Based on the above, this topic offers at least the following advantages: The method described in this paper generates widget compositions by considering the material, scene, and structural characteristics of the wallpaper and uniquely defining the position and presentation of the widgets. Therefore, the described method enhances the user experience and improves the immersiveness of the widget compositions.

[0128] While the subject matter has been described in specific language, it is not intended to impose any limitation. It will be apparent to those skilled in the art that various working modifications can be made to the method to achieve the inventive concept as taught herein. The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements can be well combined into a single functional element. Alternatively, certain elements may be divided into multiple functional elements. Elements from one embodiment may be added to another embodiment.

[0129] It will be understood that various embodiments of the present disclosure as described in the claims and specification may be implemented in hardware, software, or a combination of hardware and software.

[0130] Any such software may be stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), the one or more computer programs including computer-executable instructions that, when executed individually or jointly by one or more processors of an electronic device, cause the electronic device to perform the methods of this disclosure.

[0131] Any such software may be stored in the form of volatile or non-volatile memory, such as memory like read-only memory (ROM), whether erasable or rewritable, or in the form of memory such as random access memory (RAM), memory chips, devices, or integrated circuits, or stored on optical or magnetically readable media such as optical discs (CDs), digital versatile optical discs (DVDs), magnetic disks, or magnetic tapes. It will be understood that storage devices and storage media are various embodiments of non-transitory machine-readable memory suitable for storing one or more computer programs including instructions that, when executed, implement various embodiments of this disclosure. Therefore, various embodiments provide programs including code for implementing the apparatus or methods claimed as any one of the claims of this specification, and non-transitory machine-readable memory for storing such programs.

[0132] Although this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for managing the display of wallpapers on a user equipment (UE), the method comprising: Detect at least one of the current display position and current style of at least one widget displayed on the wallpaper; Detect one or more objects present in multiple areas of the wallpaper; Based on at least one of the current display position and current style, and also based on the one or more objects, estimate one or more changes required for the at least one widget; as well as Based on the estimated one or more changes, change at least one of the current display position and current style of the at least one widget.

2. The method according to claim 1, wherein, Estimating the one or more changes includes: Detect at least one of the following: The overlap between the at least one widget and at least one of the one or more objects, wherein the overlap is caused by at least one of the current display position and current style of the at least one widget, and The correlation between the at least one widget and the wallpaper regarding one or more characteristics associated with the plurality of areas is missing; and When the overlap and / or lack of relevance are detected, estimate the required changes to at least one of the current display position and current style of the at least one widget.

3. The method according to claim 1, in, The one or more objects are detected using a scene graph, wherein the scene graph represents the one or more objects and a first relationship between the one or more objects determined based on the interactions between the one or more objects.

4. The method according to claim 3, wherein, In response to the one or more changes required for estimation, the method further includes: Based on the size of the plurality of regions, the correlation of the plurality of regions, and a compositional characteristic map associated with the plurality of regions of the wallpaper, at least one relevant region for placing the at least one micro-part is identified from the plurality of regions, wherein the compositional characteristic map indicates the one or more characteristics associated with the plurality of regions; and A list of relevant widget styles is obtained based on one or more corresponding characteristics associated with the at least one relevant region. The changes include: Generate at least one modified widget. The at least one modified widget includes at least one of a modified display position and a modified style. The changed display position corresponds to the at least one related area, and The modified style is determined based on a list of relevant widget styles and one or more corresponding characteristics associated with the at least one relevant region.

5. The method according to claim 4, wherein, The feature map is generated based on the determination of the one or more objects, the first relationship between the one or more objects, and the segmentation map and depth map of the wallpaper.

6. The method according to claim 4, wherein, The one or more characteristics correspond to material properties associated with the physical characteristics of the plurality of regions, structural properties associated with the specifications of the plurality of regions, and scene characteristics associated with the dynamics of the scene depicted in the wallpaper that includes the plurality of regions.

7. The method according to claim 5, wherein, Identifying the at least one relevant region includes: A set of continuous regions is determined from the plurality of regions based on the segmentation map and depth map of the wallpaper; Determine the relevance score of each region in the set of continuous regions based on the one or more objects and the set of continuous regions; One or more candidate regions are identified from a set of contiguous regions based on segmentation maps and depth maps, wherein the relevance scores of the one or more candidate regions are greater than a predefined relevance threshold; and Based on the second relationship between the one or more objects and the one or more candidate regions, the at least one relevant region is identified from the one or more candidate regions.

8. The method according to claim 7, wherein, Determine the relevance score, including: Identify at least one related object from the one or more objects; and The relevance score is determined based on the cross-union ratio between the at least one relevant object and each region in the set of continuous regions.

9. The method according to claim 8, wherein, Identifying the at least one related object includes: Determine the text descriptions and compositional features associated with the scene depicted in the wallpaper; and The at least one related object is identified based on the defined text description and the composition feature diagram.

10. The method of claim 4, further comprising: One or more micro-components are generated based on the at least one modified micro-component, the at least one associated region, and one or more corresponding characteristics associated with the at least one associated region; Determine the aesthetic score associated with each of the generated one or more micro-components; as well as At least one micro-component is provided based on the selection from the generated one or more micro-components, wherein the aesthetic score associated with the at least one micro-component is greater than a predefined aesthetic threshold.

11. The method of claim 10, further comprising: The wallpaper is rendered by placing preferred widgets onto it, wherein the preferred widgets are selected by the user from at least one widget; and The at least one modified widget in the rendered wallpaper is identified as an active component of the wallpaper, and the one or more objects in the rendered wallpaper are identified as static components of the wallpaper, wherein the active component is refreshed when the content of the widget is updated.

12. The method according to claim 10, in, The wallpaper is a video comprising one or more scenes, and The method further includes: Determine the span associated with each of the one or more scenarios; Determine the scene span distance; and When the scene span is less than a predefined distance threshold, the composition of the generated widget for the video is determined.

13. The method of claim 12, further comprising: When the scene span is greater than a predefined distance threshold, a pre-determined component composition is generated.

14. The method of claim 12, further comprising: Generate the at least one widget configuration for each frame of the video; Select the most frequently occurring widget configuration from the corresponding widget configurations associated with each frame of the video; The micro-device drift between one or more adjacent frames is determined based on the difference between the coordinates of the center of the at least one changed micro-device in consecutive frames; Compare the widget drift with a predefined drift threshold; When the widget drift exceeds a predefined drift threshold, the position of the selected widget in the current frame is modified based on the position of the selected widget in the previous neighboring frame; as well as When the widget drift is less than a predefined drift threshold, the selected widget configuration is applied to each frame of the video.

15. A system for managing the widget display of wallpapers on a user equipment (UE), the system comprising: Memory, which stores one or more computer programs; as well as One or more processors are communicatively coupled to the memory. The one or more computer programs include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the system to perform the following operations: Detect at least one of the current display position and current style of at least one widget displayed on the wallpaper. Detect one or more objects present in multiple areas of the wallpaper. Based on at least one of the current display position and current style, and also based on the one or more objects, estimate one or more changes required for the at least one widget, and Based on the estimated one or more changes, change at least one of the current display position and current style of the at least one widget.

Images