A method for intelligent selection of a region in AI drawing of a mobile terminal

Abstract

The application discloses a kind of mobile terminal AI drawing intelligent selection area method, comprising: obtaining image to be processed, the semantic segmentation is carried out to the image to be processed, obtains several color block regions;Establish the spatial subordination of each color block region and construct binary tree index based on the spatial subordination;After user moves or zooms the image to be processed, according to the binary tree index, the effective area displayed in screen is rendered;If user carries out selection area operation, then judge whether the point selection area of user overlaps with multiple color block regions, if yes, then according to the color block region overlapping with the point selection area, secondary selection request is sent to user, so as to realize selection area.The method overcomes the problem that the mobile terminal drawing is frequently rendered operation leads to the lag and point selection operation is not accurate on small screen, achieves the effect of optimizing drawing interaction experience, accurate positioning local operation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a method for intelligent selection of drawing areas using mobile AI. Background Technology

[0002] In contemporary times, intelligent tools such as text-to-image and image-to-image have emerged, where artificial intelligence replaces human drawing. Most AI drawing tools are available on PCs, where users perform operations on computers. However, with industry and product development, the demand for AI drawing on mobile devices, such as large-screen smartphones and tablets, will increase significantly. But unlike PCs where keyboard and mouse operations are used, mobile devices rely heavily on gestures for interaction. This presents challenges for AI drawing, which often involves frequent image movement and scaling, as well as various selection and editing operations. For example, dragging back and forth can lead to rendering delays and interface lag, and the small screen size on mobile devices makes gesture-based selection and editing inconvenient.

[0003] Specifically, during movement and scaling, the selected area needs to be rendered because various editing operations are performed on it, such as coloring, resizing, and repositioning. Therefore, every time the screen is moved or scaled, the screen needs to be re-rendered. Under normal circumstances, every user operation (moving the screen, scaling, etc.) requires drawing the entire selection area, wasting system resources, causing system lag, long user wait times, and a very unfriendly experience. In densely populated areas, tapping or editing selections with a finger on the screen easily leads to accidental selections, requiring users to repeatedly cancel and reselect, affecting the smoothness of the operation and resulting in a very unfriendly user experience. Summary of the Invention

[0004] The purpose of this application is to provide a method for intelligent selection of drawing areas using mobile AI, in order to solve the technical problems existing in related technologies: rendering is required even for invalid selection areas, which is time-consuming and has poor performance; gesture operation is not precise, has a high rate of accidental touches, and is not user-friendly.

[0005] According to a first aspect of the embodiments of this application, a method for intelligent selection of drawing regions using mobile AI is provided, comprising:

[0006] (1) Obtain the image to be processed, and perform semantic segmentation on the image to be processed to obtain several color block regions;

[0007] (2) Establish the spatial hierarchy of each color block region and construct a binary tree index based on the spatial hierarchy;

[0008] (3) After the user moves or zooms the image to be processed, the effective area displayed on the screen is rendered according to the binary tree index.

[0009] (4) If the user performs a selection operation, it is determined whether the user's selected area overlaps with multiple color block areas. If so, a secondary selection request is sent to the user based on the color block areas that overlap with the selected area, thereby realizing the selection.

[0010] Furthermore, a binary tree index is constructed based on the spatial subordination relationship, specifically as follows: for a color patch region A, its corresponding node in the binary tree is node(A), then the left child node of node(A) corresponds to the color patch region belonging to A, and the right child node of node(A) corresponds to the color patch region that is parallel to A in terms of subordination relationship.

[0011] Further, rendering the valid area displayed on the screen according to the binary tree index includes:

[0012] Traverse the binary tree starting from the root node and determine whether the node is on the screen.

[0013] If the root node is not in the screen, no rendering will be performed;

[0014] If a node exists on the screen, the structure on the screen is saved to obtain an active binary tree, and the nodes in the active binary tree are rendered.

[0015] Further, in step (4), by traversing the activated binary tree, it is determined whether the color block area corresponding to the node overlaps with the user's selected area. The overlapping nodes are saved to obtain the selected binary tree. If there is more than one node in the selected binary tree, the user's selected area overlaps with multiple color block areas.

[0016] Further, in step (4), a secondary selection request is sent to the user based on the color block area overlapping the selected area, specifically as follows:

[0017] Based on the color of the color block area that overlaps with the selected area, a color pop-up option is displayed on the screen for the user to make a secondary selection.

[0018] According to a second aspect of the embodiments of this application, a mobile AI drawing intelligent selection device is provided, comprising:

[0019] The semantic segmentation module is used to acquire the image to be processed, perform semantic segmentation on the image to be processed, and obtain several color block regions.

[0020] The binary tree construction module is used to establish the spatial hierarchy of each color block region and construct a binary tree index based on the spatial hierarchy.

[0021] The rendering module is used to render the effective area displayed on the screen according to the binary tree index after the user moves or scales the image to be processed.

[0022] The selection module is used to determine whether the user's selected area overlaps with multiple color block areas when the user performs a selection operation. If so, it sends a secondary selection request to the user based on the color block areas that overlap with the selected area, thereby realizing the selection.

[0023] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0024] One or more processors;

[0025] Memory, used to store one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0027] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0028] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0029] As can be seen from the above embodiments, this application divides the image into several color block regions through semantic segmentation, establishes a binary tree index for each color block region using the spatial belonging relationship of objects in the image, and then renders and accurately selects the effective region based on the binary tree index. This overcomes the problems of lag caused by frequent rendering operations on mobile devices and inaccurate selection operations on small screens, and achieves the effect of optimizing the drawing interaction experience and accurately locating local operations.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0032] Figure 1 This is a flowchart illustrating a mobile AI drawing intelligent selection method according to an exemplary embodiment.

[0033] Figure 2 This is a schematic diagram of an image to be processed according to an exemplary embodiment.

[0034] Figure 3 This is a schematic diagram of a semantically segmented image according to an exemplary embodiment.

[0035] Figure 4 This is a schematic diagram of a binary tree structure according to an exemplary embodiment.

[0036] Figure 5 This is a schematic diagram illustrating a binary tree structure numbering according to an exemplary embodiment.

[0037] Figure 6 This is a diagram of a mobile editing interface according to an exemplary embodiment.

[0038] Figure 7 This is a schematic diagram of a point-and-click interface according to an exemplary embodiment.

[0039] Figure 8 This is a scaled-down image of a mobile editing interface according to an exemplary embodiment.

[0040] Figure 9 This is a schematic diagram illustrating a secondary selection according to an exemplary embodiment.

[0041] Figure 10 This is a block diagram illustrating a mobile AI drawing intelligent selection device according to an exemplary embodiment.

[0042] Figure 11 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0044] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0046] Figure 1 This is a flowchart illustrating a method for intelligent operation of mobile AI drawing selection area according to an exemplary embodiment, such as... Figure 1 As shown, this method can be applied to various smart mobile terminals (such as mobile phones, tablets, etc.).

[0047] The following is combined with the example of Figure 2 The example shown illustrates how to make localized adjustments to the original design, such as redrawing parts, changing colors, adjusting sizes, and adding text. On mobile devices, users frequently perform operations such as moving, scaling, selecting, canceling, and editing images.

[0048] (1) Obtain the image to be processed, and perform semantic segmentation on the image to be processed to obtain several color block regions;

[0049] First, as follows Figure 1 The image shown is subjected to semantic segmentation (such as SAM) to obtain descriptions of different color block regions, such as... Figure 2 As shown, the entire image is marked with different colors, representing the division of the original image's various objects, elements, background, etc., into numerous color blocks of varying sizes.

[0050] Semantic segmentation is a classification at the pixel level. Pixels belonging to the same category are grouped together. Therefore, semantic segmentation understands images at the pixel level. For example, in a design draft, pixels belonging to the floor are grouped into one category, pixels belonging to the coffee table are grouped into another, and so on. Pixels contained in curtains, walls, sofas, etc., will also be categorized. Furthermore, note that even the same object, composed of different parts or with different pixel shapes due to varying lighting angles, will be semantically segmented into different categories. For instance, a sofa stool might be divided into a frame, cushion 1, and cushion 2, represented by red, purple, and pink blocks respectively, and so on.

[0051] (2) Establish the spatial hierarchy of each color block region and construct a binary tree index based on the spatial hierarchy;

[0052] Specifically, each color block region has a coordinate space in planar space. Coordinate spaces with spatial hierarchical relationships are semantically merged to obtain a higher-level coordinate space. In this embodiment, there are objects such as a coffee table, a large sofa, a small sofa, and wooden chairs on the floor. Therefore, the coffee table and other objects belong to the floor, and the spatial coordinates of the floor are the union of the spatial coordinates of all the objects, including the coffee table, the large sofa, the small sofa, and the wooden chairs. These spatial hierarchical relationships can be established manually or through computer vision, with different methods selected depending on the scenario. For example:

[0053] 1) In cases where the scene is relatively fixed, such as an interior decoration scene, subordinate relationships can be manually established based on the interior modeling;

[0054] 2) In cases where the scenario is not fixed, such as when a user uploads any photo, the subordinate relationship can be determined through computer vision.

[0055] These divided color blocks are indexed using a binary tree and associated according to the spatial ownership of objects. Each block corresponds to a node in the binary tree structure, with the block at the highest level being the root node. The left subtree contains blocks belonging to the root node, and the right subtree contains blocks parallel to the root node. In this embodiment, the living room consists of a front wall, side walls, floor, ceiling, and windows. The living room is considered the root node, and the front wall, side walls, floor, ceiling, and windows are considered child nodes. The front wall is the left subtree of the living room, and the side walls are the right subtree. The left subtree has an inclusion relationship with the root node of the living room, while the right subtree has a parallel but non-inclusion relationship with the root node. There is also a picture frame on the front wall, which is the left subtree of the front wall. On the floor is a coffee table with a saucer, vase, plant, and teapot. The coffee table is the left subtree of the floor, the saucer is the left subtree of the coffee table, the vase and teapot are the left and right subtrees of the saucer respectively, and the plant is the left subtree of the vase, and so on. This forms the following structure: Figure 4 The binary tree structure shown.

[0056] The items are labeled to form a tree-like node numbering system. In this embodiment, the storage nodes are numbered sequentially according to a preorder traversal to obtain... Figure 5 The binary tree shown.

[0057] (3) If the user moves or scales the image to be processed, the effective area displayed on the screen is rendered according to the binary tree index after the move or scale operation.

[0058] It should be noted that the image to be processed can be the original image or the image after semantic segmentation. When users operate AI drawing on their mobile devices, they continuously drag, scale, rotate and perform other operations on the image. After each operation, the system needs to perform a rendering operation on the entire image, causing interface lag, long waiting time and affecting the user's drawing operation. In addition, invalid areas outside the screen also need to be rendered, resulting in wasted performance.

[0059] In this application, each time the user drags or zooms, a preorder traversal is performed starting from the root node of the binary tree structure built according to spatial hierarchy. If the root node is not on the screen, the entire image does not need to be rendered. If the root node is on the screen, the positional relationship of all child nodes to the screen is checked sequentially. If the left subtree node 2 is not on the screen, then child node 2 and its subtree are discarded. Then, the positional relationship of the right subtree node 3 to the screen is checked. If child node 3 is on the screen, the positional relationship of its subtree nodes to the screen is checked. This process continues until the entire binary tree structure is traversed. Finally, the block structure of the current screen is saved, which is called the currently active binary tree. During rendering, only the nodes in the active binary tree need to be rendered; invalid selections that do not belong to the active binary tree are not rendered. This enables rapid location of the area displayed on the screen, reducing its time complexity to below O(n) and approaching O(log(n)) under binary tree balancing conditions. Furthermore, it eliminates rendering of invalid selections, rendering only the valid areas on the screen, which reduces the likelihood of mobile device lag and provides users with instant response and a better drawing experience.

[0060] (4) If the user performs a selection operation, it is determined whether the user's selected area overlaps with multiple color block areas. If so, a secondary selection request is sent to the user based on the color block areas that overlap with the selected area, thereby realizing the selection.

[0061] During gesture operations, users can select or deselect one or more areas of an object, allowing them to perform actions such as changing the color of the selection, expanding or shrinking the selection, and editing the selection.

[0062] Mobile devices (phones and tablets) have smaller screens compared to desktop computer screens, and touch gestures are not as precise as a mouse. When selection areas are closely arranged, it is very easy to make mistakes when trying to select a specific area. This causes users to repeatedly click and cancel selections, affecting the smoothness of drawing operations and resulting in a very poor user experience.

[0063] This application quickly determines which objects are selected when a user taps on the screen. Based on previous screen dragging, zooming, and other operations, the active binary tree structure of the current screen has already been obtained. Further, in the active binary tree structure of the current screen, a preorder traversal method is used again to determine whether the color block area corresponding to the node overlaps with the user's selected area. Nodes with overlap are saved to obtain the selected binary tree. If there are more than one node in the selected binary tree, the user's selected area overlaps with multiple color block areas, and a second tap is required.

[0064] When a user makes a selection, the system first renders the blocks on the screen based on previous operations, determining which objects are currently displayed. Then, it traverses the currently active binary tree again, checking each node for overlap with the area pressed by the finger. If overlap is found, it's recorded and saved as a secondary confirmation option (i.e., stored in the selected binary tree structure). This process continues until all nodes in the active binary tree have been traversed. The nodes stored in the binary tree structure then represent all the candidate blocks corresponding to the user's finger press. Users can then select a node from the stored binary tree for secondary confirmation, resulting in a quick and accurate selection.

[0065] For example, now we want to configure the coffee table plate on the coffee table, i.e., child node 12, corresponding to... Figure 6 Use the light cyan area shown to select and edit. The mobile interface is as follows: Figure 7 As shown, the scaled-down user interface is as follows: Figure 8 As shown. The portion within the rectangular frame includes the coffee table and tray to be manipulated, along with the surrounding coffee table (child node 11), vase (child node 13), and teapot (node ​​15). From Figure 8 It is clearly visible that even when the image is zoomed in, it is difficult to accurately select the light blue area corresponding to the coffee table box with a finger, and it is very easy to mistakenly select other nearby objects.

[0066] Therefore, after the user selects a target, a secondary selection request is sent to the user based on the color block area involved in the selected area. Specifically, the color block area overlapping with the selected area will appear on the screen as a color pop-up option for the user to confirm the operation a second time. Figure 9 As shown, after the user selects the coffee table plate (light cyan area), because the finger simultaneously touches the coffee table (cyan area), vase (purple area), and teapot (light pink area), the system will pop up a secondary confirmation box. The confirmation box will be marked with four options: light cyan, cyan, purple, and light pink, corresponding to the coffee table, coffee table plate, vase, and teapot. At this time, the user can quickly and accurately select the Xth light cyan box with their finger, and thus select the coffee table plate object.

[0067] Corresponding to the aforementioned embodiments of the mobile AI drawing intelligent selection method, this application also provides embodiments of a mobile AI drawing intelligent selection device.

[0068] Figure 10 This is a block diagram of a mobile AI drawing intelligent selection device according to an exemplary embodiment. (Refer to...) Figure 10 The device may include:

[0069] Semantic segmentation module 21 is used to acquire the image to be processed, perform semantic segmentation on the image to be processed, and obtain several color block regions;

[0070] Binary tree construction module 22 is used to establish the spatial hierarchy of each color block region and construct a binary tree index based on the spatial hierarchy.

[0071] Rendering module 23 is used to render the effective area displayed on the screen according to the binary tree index after the user moves or scales the image to be processed.

[0072] The selection module 24 is used to determine whether the user's selected area overlaps with multiple color block areas if the user performs a selection operation. If so, it sends a secondary selection request to the user based on the color block areas that overlap with the selected area, thereby realizing the selection.

[0073] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0074] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0075] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the mobile AI drawing intelligent selection method as described above. Figure 11 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an embodiment of the mobile AI drawing intelligent selection device provided by the present invention. (Except for...) Figure 11 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0076] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the mobile AI drawing intelligent selection method described above. The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0077] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0078] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for intelligent selection of a region in a mobile terminal AI drawing, characterized in that, include: (1) Obtain the image to be processed, and perform semantic segmentation on the image to be processed to obtain several color block regions; (2) Establish the spatial hierarchy of each color block region and construct a binary tree index based on the spatial hierarchy; (3) After the user moves or zooms the image to be processed, the effective area displayed on the screen is rendered according to the binary tree index; (4) If the user performs a selection operation, it is determined whether the user's selected area overlaps with multiple color block areas. If so, a secondary selection request is sent to the user based on the color block areas that overlap with the selected area, thereby realizing the selection. The rendering of the valid area displayed on the screen based on the binary tree index includes: Traverse the binary tree starting from the root node and determine whether the node is on the screen. If the root node is not in the screen, no rendering will be performed; If a node exists on the screen, the structure on the screen is saved to obtain an active binary tree, and the nodes in the active binary tree are rendered. In step (4), by traversing the activated binary tree, it is determined whether the color block area corresponding to the node overlaps with the user's selected area. The overlapping nodes are saved to obtain the selected binary tree. If there is more than one node in the selected binary tree, the user's selected area overlaps with multiple color block areas. In step (4), a secondary selection request is sent to the user based on the color block area overlapping the selected area, specifically as follows: Based on the color of the color block area that overlaps with the selected area, a color pop-up option is displayed on the screen for the user to make a secondary selection.

2. The method according to claim 1, characterized in that, A binary tree index is constructed based on the spatial hierarchy, specifically: for a color patch region A, its corresponding node node(A) in the binary tree, the left child node of node(A) corresponds to the color patch region belonging to A, and the right child node of node(A) corresponds to the color patch region that is parallel to A in terms of hierarchy.

3. A mobile AI drawing intelligent selection device, characterized in that, include: The semantic segmentation module is used to acquire the image to be processed, perform semantic segmentation on the image to be processed, and obtain several color block regions. The binary tree construction module is used to establish the spatial hierarchy of each color block region and construct a binary tree index based on the spatial hierarchy. The rendering module is used to render the effective area displayed on the screen according to the binary tree index after the user moves or scales the image to be processed. The selection module is used to determine whether the user's selected area overlaps with multiple color block areas when the user performs a selection operation. If so, it sends a secondary selection request to the user based on the color block areas that overlap with the selected area, thereby realizing the selection. The rendering of the valid area displayed on the screen based on the binary tree index includes: Traverse the binary tree starting from the root node and determine whether the node is on the screen. If the root node is not in the screen, no rendering will be performed; If a node exists on the screen, the structure on the screen is saved to obtain an active binary tree, and the nodes in the active binary tree are rendered. In the selection module, by traversing the activation binary tree, it is determined whether the color block area corresponding to the node overlaps with the user's selected area. The overlapping nodes are saved to obtain the selection binary tree. If there is more than one node in the selection binary tree, the user's selected area overlaps with multiple color block areas. In the selection module, a secondary selection request is sent to the user based on the color block area that overlaps with the selected area. Specifically: Based on the color of the color block area that overlaps with the selected area, a color pop-up option is displayed on the screen for the user to make a secondary selection.

4. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-2.

5. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-2.

Images