Method and device for determining interface elements

Abstract

The embodiment of the invention provides an interface element determination method and device, and the method comprises the steps: displaying a design page which comprises at least one editable user interface and a search function control; in response to a trigger operation for a target user interface in the at least one editable user interface, displaying an element selection area in the target user interface, the element selection area comprising target interface elements in the target user interface; and in response to a trigger operation for the search function control, displaying at least one element search area in the at least one editable user interface, the element search area comprising similar interface elements associated with the target interface elements, and the similar interface elements being determined according to visual information and position information of the target interface elements. The similar interface elements are determined according to the visual information and the position information, so that the searching accuracy of the similar interface elements is improved, the tedious operation of secondary screening of the searching result by the user is reduced, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of image processing, and in particular to a method and apparatus for determining interface elements. BACKGROUND

[0002] With the development of computer technology, design software emerges as the times require. Various design software is used to draw pages to provide users with more standardized and clearer design pages. When a user designs, typesets, and arranges elements in a design draft, the user often needs to perform batch operations on multiple design draft elements. These design draft elements can exist in the same design draft or multiple different design drafts. The user needs to manually select the design draft elements to be operated one by one, which leads to cumbersome user operations and is not conducive to the output efficiency of the design draft. Therefore, how to provide a more convenient element batch selection method for the user is a problem that needs to be solved at present. SUMMARY

[0003] In view of this, the embodiments of the present specification provide a method for determining interface elements. One or more embodiments of the present specification also provide a device for determining interface elements, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0004] According to a first aspect of the embodiments of the present specification, a method for determining interface elements is provided, including: displaying a design page, the design page including at least one editable user interface and a search function control; in response to a triggering operation on a target user interface in the at least one editable user interface, displaying an element selection area in the target user interface, the element selection area including a target interface element in the target user interface; in response to a triggering operation on the search function control, displaying at least one element search area in the at least one editable user interface, the element search area including similar interface elements associated with the target interface element, the similar interface elements being determined according to visual information and position information of the target interface element.

[0005] According to a second aspect of the embodiments of the present specification, a device for determining interface elements is provided, including: a first display module configured to display a design page, the design page including at least one editable user interface and a search function control; a second display module configured to, in response to a triggering operation on a target user interface in the at least one editable user interface, display an element selection area in the target user interface, the element selection area including a target interface element in the target user interface. A third display module is configured to, in response to a triggering operation on the search function control, display at least one element search area in the at least one editable user interface, the element search area including similar interface elements associated with the target interface element, the similar interface elements being determined according to visual information and position information of the target interface element.

[0006] According to a third aspect of an embodiment of the present specification, a computing device is provided, comprising: a memory and a processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which, when executed by the processor, implement the steps of the above interface element method.

[0007] According to a fourth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, which, when executed by a processor, implement the steps of the above interface element method.

[0008] According to a fifth aspect of an embodiment of the present specification, a computer program product is provided, comprising a computer program or instructions, which, when executed by a processor, implement the steps of the above interface element method.

[0009] One embodiment of the present specification realizes that when a user designs at least one editable user interface in a design page, the user can quickly search and select interface elements by using a search function control. Through a triggering operation of the user on a target user interface, an element selection area including a target interface element is displayed in the target user interface, and the user is shown the selected target interface element. Through a triggering operation of the user on the search function control, at least one element search area is displayed in at least one editable user interface, and the element search area includes similar interface elements associated with the target interface element, and the user is shown similar interface elements found according to visual information and position information of the target interface element, thereby providing a convenient and fast similar matching selection method for interface elements for the user, and facilitating the user to perform further batch operations on the selected similar interface elements. Furthermore, the similar interface elements are determined according to the visual information and the position information, which improves the search accuracy of the similar interface elements, reduces the cumbersome operation of the user to perform secondary screening on the search results, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A flowchart of a method for determining an interface element is shown according to one embodiment of the present specification; Figure 2 A schematic diagram of an editable user interface is shown according to one embodiment of the present specification; Figure 3 Fig. 1 shows an interface diagram of an element selection area according to an embodiment of the present specification; Figure 4 Fig. 2 shows a diagram of an element search area according to an embodiment of the present specification; Figure 5 Fig. 3 shows a diagram of a layer display area according to an embodiment of the present specification; Figure 6 Fig. 4 is a flow chart of a method for determining an interface element according to an embodiment of the present specification; Figure 7 Fig. 5 is a structural diagram of a device for determining an interface element according to an embodiment of the present specification; Figure 8 Fig. 6 is a structural block diagram of a computing device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0011] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, and it is understood that the present specification can also be practiced with other apparatuses and can include or exclude other elements. In other instances, well-known methods, procedures, components, and circuits have not been described in detail since it is understood that knowledgeable persons within the art will be able to utilize such processes without the need for further detail.

[0012] The terminology used in the one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the present specification. As used in the one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in the one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or any combination thereof.

[0013] It is to be understood that the terms first, second, etc. can be employed in this specification and may

[0014] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0015] The meanings of some terms mentioned in the specification will be explained first. The term "in response to" as referred to in the specification means the state in which the corresponding event occurs or the condition is met. The execution time of the subsequent action performed in response to the event or condition is not necessarily strongly associated with the time when the event occurs or the condition is met. For example, in some cases, the subsequent action can be performed immediately after the event occurs or the condition is met; in other cases, the subsequent action can be performed after a period of time after the event occurs or the condition is met.

[0016] The term "trigger operation" as referred to in the specification refers to an operation performed on information (such as a control) provided by a computer device, through which a corresponding instruction can be issued to the computer device to trigger the computer device to perform the next task. The next task triggered by different trigger operations performed on different information can be pre-set in the program. The trigger operation can be performed manually by the user, such as clicking, double-clicking, long-pressing, sliding, etc. operations performed by the user on the content displayed on the screen of the computer device can all belong to trigger operations. In some cases, the trigger operation can also be performed by the computer device based on the set program.

[0017] The term "event" as referred to in the specification refers to a specific action or change that occurs in a system or application, which can be triggered by a user (such as clicking a button, keyboard input) or triggered by the system (such as a timer expiring, a file download completing). The program usually listens to these events and executes the corresponding code when a specific event occurs.

[0018] "Control" refers to any element in a computer user interface that a user can directly interact with. These elements allow users to input data, select options, or trigger certain actions. Controls can be graphical (such as buttons, text boxes, checkboxes, etc.) or more abstract forms (such as voice commands). Buttons are typically used to perform an action or open a new interface. Text boxes allow users to input or edit text information. Checkboxes are used to represent a Boolean value (yes / no), which users can select or deselect. Dropdown lists provide a list of options for users to choose from, usually displaying only one selected value. List boxes display multiple options, and users can select one or more items. Labels display static text, often used to explain the function of other controls, and can also be used as a button. Scrollbars enable users to navigate through larger datasets, such as browsing long documents or lists. Images display static pictures and can sometimes be used as buttons. Comboboxes combine the functionality of text boxes and dropdown lists, allowing users to either input text or select from a list. These controls are usually provided with standard styles by the operating system or development toolkit, and developers can customize their appearance and behavior as needed. Different operating systems and programming languages (such as Java, C#, Python, etc.) have their own sets of controls, and provide corresponding APIs to create and manage them.

[0019] The AI model referred to in this specification can be a large language model, which refers to a large pre-trained natural language model based on deep learning, with strong context understanding and generation capabilities, capable of processing complex and diverse natural language tasks under the guidance of prompt information.

[0020] Design draft: The design draft is a visualized plan that a designer turns his ideas into. Each design draft is composed of multiple elements, each element containing specific attributes, and all elements form a tree structure to build the design draft. The design draft uses graphics, text annotations, and other methods to fully display the final sample of the product, such as the layout of the mobile APP page, button position, color matching, etc.

[0021] Design draft element: The design draft element is an interface element in the design draft, and the design draft element constitutes an independent module of interface design, including buttons, icons, text, and other visual elements. Each design draft element carries multiple layers of information such as color, font, shadow parameters, size, element name, etc.

[0022] Currently, when a designer user designs a user interface, if he wants to find similar elements or perform batch operations on the selected interface elements, he needs to manually compare elements in the design draft interface and select them manually, which takes a certain amount of time for element comparison and selection. To solve this problem, the similarity between elements is usually calculated using element attribute information, and similar elements are determined according to the similarity, thereby providing the user with convenient batch selection, including selecting elements with similar names according to element names, such as comparing element name similarity according to edit distance, and elements with similarity meeting a preset threshold are selected as similar elements; selecting elements according to component type, such as selecting buttons, if a component library is used in the design tool, the component type is defined in the component library, and all button type elements in the design draft can be selected as similar elements according to the component type; selecting elements according to element size; calculating similarity according to element attribute information, extracting attribute information from the elements selected by the user, extracting attribute information from all other elements in the design draft, calculating the similarity between elements according to the weighted sum of the attribute information, and selecting elements with similarity meeting a preset threshold as similar elements; calculating similarity according to element attribute information and vision, extracting attribute information from the elements selected by the user, and identifying the category of the selected elements such as buttons, drop-down boxes, etc. through image recognition, then identifying the category of all elements through image recognition, and selecting similar elements with the same category and high attribute similarity.

[0023] The above methods of automatically selecting similar elements can calculate the similarity between elements according to element attribute information, and can also calculate the similarity according to the category identified by vision. However, these methods all have the problem of inaccurate identification results, for example, the types or categories of two elements are both buttons, but one is in a product card and the other is in a navigation bar, then according to the above method, the two elements will also be recalled as similar elements; again, for example, two elements are both text elements, but one is a first-level title and the other is a second-level title, then according to the above method, the two elements will also be recalled as similar elements. Therefore, according to the above method, the search for similar elements may not meet the user's expectations, and the user may need to further manually filter the search results, reducing design efficiency.

[0024] Therefore, in this specification, a method for determining interface elements is provided, which aims to provide a more convenient and efficient element search function for users, and can search for elements from two dimensions of vision and position, thereby improving the accuracy of search results. This specification also relates to a device for determining interface elements, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0025] ReferenceFigure 1 , Figure 1 A flowchart of a method for determining an interface element is shown, which comprises the following steps.

[0026] Step 102: display a design page, the design page comprising at least one editable user interface and a search function control.

[0027] Step 104: in response to a triggering operation on a target user interface in the at least one editable user interface, display an element selection area in the target user interface, the element selection area comprising a target interface element in the target user interface.

[0028] Step 106: in response to a triggering operation on the search function control, display at least one element search area in the at least one editable user interface, the element search area comprising similar interface elements associated with the target interface element, the similar interface elements being determined according to visual information and position information of the target interface element.

[0029] The method for determining an interface element provided by the embodiments of the present specification can be applied to the interface design scenario, and the method is deployed in a design product or software of a UI interface to provide a function of quickly searching and matching an interface element for a user, so as to facilitate the user to automatically select the interface element for the designed interface element, thereby realizing batch operation of the interface element.

[0030] In actual application, when a user uses a design software to design a UI user interface, the user will perform a related editing operation in a design page displayed by the design software, and therefore the design page comprises at least one editable user interface. The design page is an operation page of the design software facing a designer, and the design page can comprise various design function controls, and the user can perform an editing operation on the UI user interface by operating the design function controls. The editable user interface can be understood as a user interface designed or drawn by the user, and the user interface can comprise interface elements arranged or added by the user. Referring to Figure 2 , Figure 2 A schematic diagram of an editable user interface provided by an embodiment of the present specification is shown, Figure 2 A UI user interface designed by a user for game login, and the user interface comprises many arranged elements, such as a game icon, a game name text, a text input box, a login button control, and the like. The user can continue to edit the user interface, such as adjusting the size and position of the interface elements, or adding and deleting the interface elements in the user interface.

[0031] When the design user wants to edit the same interface elements in one or more editable user interfaces at the same time, he needs to select the interface elements in one or more editable user interfaces in sequence before he can perform batch editing operations. For this purpose, the design page includes a search function control, that is, the design software provided in the embodiment of this specification provides the user with a search function for interface elements, and the search function control is the control corresponding to the search function in the design page, and the user can use the search function of the corresponding interface element through the search function control. Before the user searches for the interface element, it is necessary to select the reference interface element as the target interface element. In specific implementation, the user will click and select the target user interface in at least one editable user interface in the design page. For example, there are editable user interface A, editable user interface B and editable user interface C in the design page. The user selects editable user interface A as the target user interface by clicking on editable user interface A, and the target user interface is the user interface selected by the user. Correspondingly, the user can further perform target interface element selection operations in the target user interface. For example, if the user clicks on an element in the target user interface to select it, the selected element will be used as the target interface element, and the target interface element will be used as the reference interface element for subsequent searches for similar interface elements. That is, the target interface element can be understood as the interface element selected by the user for search, and similar interface elements found subsequently will be searched according to the characteristics or features of the target interface element. After the user selects the target interface element in the target user interface, an element selection area will be displayed in the target user interface, and the selected target interface element will be displayed to the user using the element selection area. See Figure 3 , Figure 3 FIG. 1 shows an interface diagram of an element selection area provided by an embodiment of the present specification, wherein the element selection area is displayed according to the position of the target interface element in the target user interface, such as Figure 3 In , if the user selects the text box control element for account input as the target interface element, the element display area is located in the text box control element. Figure 3 As shown, the element selection area will use a line frame to select the target interface element to highlight the selection of the target interface element to the user, preventing the user from selecting the wrong interface element and affecting subsequent element search.

[0032] When the user determines the target interface element in the target user interface, the element search function can be used by triggering the search function control. Specifically, the user can use the element search function by clicking the search function control. Then, the design software will search for similar elements according to the target interface element determined by the user. In actual application, the user can also determine the search range, that is, determine which user interfaces need to be searched. The user can manually specify the user interfaces that need to be searched, or the system can use the editable user interfaces that have been opened in the design page as the search range. After searching, at least one element search area is displayed in the editable user interfaces belonging to the search range, and each element search area includes similar interface elements associated with the target interface element. The similar interface elements are interface elements found according to the visual information and position information of the target interface element. In specific implementation, a user interface can include multiple similar interface elements to the target interface element, and the user interface can include multiple element search areas. Referring to Figure 4 , Figure 4 FIG. 1 shows a schematic diagram of an element search area provided by an embodiment of the present specification, and Figure 4 If the editable user interface in

[0033] In actual application, the similar interface elements need to be determined according to the visual information and position information of the target interface element. The visual information can be understood as the visual information obtained from the image corresponding to the interface element through image recognition. The position information can be understood as the position information determined according to the related attributes of the interface element. Since the visual information is extracted from the image corresponding to the interface element through image recognition technology, the visual information can be regarded as related information in the visual dimension. The visual information can include the color, shadow parameter, size, shape and pattern of the image element. Since the position information is determined according to the related attributes of the interface element to determine the specific position of the interface element in the design page, the position information can be regarded as related information in the position dimension. The position information can include the coordinate position, size and hierarchical structure of the image element. By combining the above visual information and position information, the system can more accurately identify and match similar interface elements. For example, when judging whether two buttons are similar, in addition to considering their visual features such as color, font and size, the relative position of the buttons in the page and the context environment in which the buttons are located (such as whether the buttons are located in the navigation bar) also need to be considered. This multi-level analysis method ensures that even in different scenarios, as long as the visual and position information are close enough, the similar elements can be correctly identified.

[0034] In summary, the method for determining interface elements provided in an embodiment of the present specification provides an element search function for users in design software or products, so that users can achieve automatic search of interface elements by simple operations such as selecting target interface elements, user interfaces to be searched, and clicking search function controls. In addition, the visual information and position information of the interface elements are comprehensively considered during the search process, so that the similar interface elements obtained by the search are more in line with user needs, and the search accuracy is improved.

[0035] The method for determining interface elements provided in the present specification is further explained and described through various embodiments.

[0036] Further, the design page further includes a layer display area, and the layer display area includes layer information corresponding to the at least one editable user interface; after displaying the design page, the method further includes: in response to a triggering operation on the layer display area, displaying target layer information corresponding to the target user interface in the layer display area; and in response to a triggering operation on the target layer information, displaying an element selection area in the target user interface, and the element selection area includes target interface elements in the target user interface.

[0037] In the design page, a layer display area can also be included, which can be understood as a display area for displaying layer information of each editable user interface in the design page. Referring to Figure 5 , Figure 5 A schematic diagram of a layer display area provided in an embodiment of the present specification is shown, and the layer display area includes the name corresponding to each editable user interface. The user can expand the specific layer information of the editable user interface by clicking the layer display area, and each layer includes the name of the elements contained in the layer, such as the editable user interface of “login interface C”, which includes elements 1, 2, 3, and 4 in the interface layer.

[0038] In actual application, the user can determine the target user interface and the target interface element through the layer display area. The user can select the editable user interface to be searched in the layer display area, and take the user interface as the target user interface. After selecting the target user interface, the user can expand the target layer information of the target user interface by clicking operation, and select the interface element in the target layer information as the target interface element by operation on the target layer information. Correspondingly, after the user selects the target interface element, the target interface element is also displayed in the target user interface in the form of an element selection area.

[0039] In a specific embodiment of the present specification, the method for determining interface elements provided in the present specification is applied to a design software, and the design software includes a design page. Figure 5For example, the user clicks "login interface C" in the layer display area, and the target layer information corresponding to "login interface C" is displayed in the layer display area. The user selects the name of the target interface element in the expanded target layer information. At this time, the target interface element is highlighted in the user interface corresponding to "login interface C" by using a highlight line, thereby forming the element selection area.

[0040] Based on this, the layer display area is introduced in the design page, which not only enhances the user's management and operation ability of the interface element, but also provides a more intuitive and flexible design experience, so that the user can browse the layer information of each user interface in the layer display area, and also allows the user to determine the target interface element based on the layer information, thereby providing a more convenient element selection method for the user.

[0041] Further, in response to the triggering operation of the search function control, at least one element search area is displayed in the at least one editable user interface, including: in response to the triggering operation of the search function control, an interface selection guide control is displayed in the design page; in response to the triggering operation of the interface selection guide control in the design page, a user interface to be searched is determined in the at least one editable user interface; and at least one element search area is displayed in the user interface to be searched.

[0042] When the user triggers the search function control, it means that the user wants to use the element search function. At this time, the interface selection guide control is displayed in the design page. The interface selection guide control can be understood as a floating window, a drop-down menu or a real form interactive component. The function of the interface selection guide control is to help the user select a specific interface for similar element search from multiple editable user interfaces.

[0043] In actual application, the interface selection guide control can list all editable user interfaces in the current design page, and each editable user interface usually has a name or identifier for identification. In order to facilitate the user to quickly find a specific user interface, the interface selection guide control can also include a filter box or a search bar to allow the user to input keywords. The user selects a specific interface for similar element search through the interface selection guide control. The selected user interface is the user interface to be searched, and the similar element search will be performed in the user interface to be searched. Subsequently, the element search area is displayed in the user interface to be searched.

[0044] In a specific implementation, based on the options provided by the interface selection guide control, the user can select one or more user interfaces that the user wants to find similar elements in as the user interface to be found by clicking, checking, or other methods. For example, if the user only wants to find similar elements to a certain button in the "login interface C", the user only needs to select this interface. Once the user has completed the selection, the user can submit the selection result by clicking the confirmation button or other methods, and the system will continue to perform the subsequent steps based on the user's selection. It should be noted that the target user interface can also be used as the user interface to be found, that is, the user wants to find similar elements in the user interface where the target interface element is located.

[0045] In a specific implementation of the present specification, after the user selects to use the search function by clicking the search function control, an interface selection guide control is displayed in the design page. The control can be in the form of a text prompt, and the prompt content includes "Please select the user interface to be found". After the user selects the user interface to be found according to the prompt and submits the determination, the system will perform a search for similar elements in the user interface submitted by the user.

[0046] Therefore, through the interface selection guide control, the user can accurately specify the user interface in which the user wants to find similar elements, avoiding unnecessary calculations and displays in the entire project range and improving efficiency. The user can flexibly select a single or multiple user interfaces for searching according to actual needs, increasing the flexibility and control of the operation. For large-scale design projects that include numerous user interfaces, this method greatly simplifies the task of finding and managing similar elements and reduces the workload of manual searching.

[0047] Further, after determining the user interface to be found in the at least one editable user interface, the method further includes: determining a target interface graph corresponding to the target user interface, a user interface to be found graph corresponding to the user interface to be found, and element attribute information corresponding to the target interface element, wherein the element attribute information includes visual information and position information; inputting the target interface graph, the user interface to be found graph, and the element attribute information into an element matching model to obtain predicted element coordinates output by the element matching model, wherein the predicted element coordinates are used to determine similar interface elements associated with the target interface element.

[0048] The target interface picture corresponding to the target user interface can be understood as a design draft picture corresponding to the target user interface, and the to-be-searched interface picture corresponding to the to-be-searched user interface can be understood as a design draft picture corresponding to the to-be-searched user interface. After the target interface element selected by the user and the to-be-searched user interface in the search range are determined, image extraction is performed on the target user interface and the to-be-searched user interface, so that interface pictures corresponding to the user interfaces are obtained. The interface pictures can not only contain the interface elements themselves, but also contain structural relationships such as relative position relationships, combination relationships, parent-child relationships, and context information between the interface elements.

[0049] In actual application, the target interface picture, the to-be-searched interface picture, and the element attribute information of the target interface element are used as data required for searching. The element attribute information includes visual information such as color, font, and position information such as element coordinates and size, and the element attribute information can more accurately define the position of the target interface element in the user interface, thereby providing a basis for subsequent element matching. These data can be input into an element matching model, which can be a pre-trained multi-modal large model. The element matching model can simultaneously process and understand image and text information.

[0050] In specific implementation, the element searching in an embodiment of the present specification is implemented by using a multi-modal large model. For this purpose, the multi-modal large model needs to be trained in advance. First, training data needs to be constructed, and the multi-modal large model is trained, so that the multi-modal large model receives attribute information of a selected element, a design draft picture in which the selected element is located, and a design draft picture in which a similar element needs to be searched, and inputs coordinate information of the similar element of the selected element.

[0051] In an embodiment of the present specification, the construction of the model training data can include first collecting a batch of design files, each carrying corresponding element ID, size, position coordinates and the like information. Similar elements are manually screened out in the same or different design files for each element, and the position and size information of the similar elements are corresponded to obtain a training data. Since the multi-modal large model supports inputting multiple pictures, each training data can contain different picture data. If there is only one editable user interface corresponding to a design file in the user design page, and the selected element and the similar element are in the user interface, there can be only one picture in the input data; if the user design page contains multiple editable user interfaces corresponding to multiple design files, and the selected element is in one of the user interfaces, the input data includes multiple pictures. The sample data in the training data is the coordinates of the selected element, the design file picture where the selected element is located and other design file pictures to be searched for similar elements (optional), and the label data is the coordinates of the similar element. If there are multiple similar elements, multiple coordinates can be output, connected by line breaks. The coordinate format is "(x1, y1), (x2, y2)", (x1, y1) is the left upper corner coordinate of the element, and (x2, y2) is the right lower corner coordinate of the element, which is calculated according to the size information of the element. After the training data is artificially constructed, the data can be input into the multi-modal large model for training. After the model training is completed, an element matching model can be obtained, and corresponding search data is input into the element matching model. The element matching model can automatically output predicted element coordinates, which can be understood as the coordinates of the similar elements predicted by the model. Subsequently, the similar interface elements associated with the target interface elements can be determined according to the predicted element coordinates.

[0052] Therefore, by comprehensively considering visual information and position information, corresponding interface graphs and element attribute information of target interface elements are input into the element matching model, and predicted element coordinates are automatically determined by the element matching model, so that users do not need to manually search for similar elements, and similar elements can be accurately identified by using the model. In the application scenario of complex design files with multiple hierarchical structures and a large number of elements or a large number of design files, the required similar elements can also be quickly and accurately found, and the design efficiency of the user interface is improved.

[0053] Further, after obtaining the predicted element coordinates output by the element matching model, the method further includes: determining the to-be-searched element coordinates corresponding to the to-be-searched interface element in the to-be-searched user interface; calculating element position overlap information according to the to-be-searched element coordinates and the predicted element coordinates; and determining the similar interface elements associated with the target interface elements in the to-be-searched interface element according to the element position overlap information.

[0054] After obtaining the predicted element coordinates of the model output, in order to determine the interface element corresponding to each predicted element coordinate in the corresponding user interface, positioning of the interface element according to the predicted element coordinates is required. Specifically, the element coordinates corresponding to the interface element to be searched in the user interface to be searched need to be searched, which can be understood as the element coordinates corresponding to each interface element to be searched in the user interface to be searched. The element position overlap information can be calculated according to the searched element coordinates and the predicted element coordinates. The element position overlap information is the element overlap degree calculated based on two element coordinates, and can be used to quantify the overlap degree between the predicted box and the real box, so as to locate the specific interface element in the user interface to be searched based on the predicted element coordinates, thereby achieving screening of the predicted element coordinates with prediction errors.

[0055] In actual application, the element position overlap information is IoU (Intersection over Union), and the overlap degree between the predicted element coordinates and the real element coordinates is determined through the IoU index, so as to determine the similar interface element associated with the target interface element based on the overlap degree in the interface element to be searched.

[0056] In an embodiment of the present disclosure, the element coordinates corresponding to each interface element to be searched in the user interface to be searched are determined, which can include coordinate information and size information, representing the specific positions of the elements in the user interface. The element position overlap information IoU is calculated according to the searched element coordinates and the predicted element coordinates. An IoU threshold is set according to actual application requirements, and the element position overlap information is compared with the threshold. When the element position overlap information is greater than the set threshold, it indicates that the actual interface element corresponding to the preset element coordinates is the interface element corresponding to the searched element coordinates, and the interface element to be searched corresponding to the searched element coordinates is taken as the similar interface element.

[0057] Therefore, by introducing the IoU index to quantify the overlap degree between the predicted box and the real box, the actual interface element can be found based on the predicted element coordinates and the searched element coordinates, so as to facilitate subsequent display of the found interface element to the user through the element search area, and facilitate the user to browse the search result.

[0058] Further, after determining the similar interface element associated with the target interface element, the method further includes: determining similar element coordinate information and similar element size information corresponding to the similar interface element; and generating and displaying an element search area corresponding to the similar interface element in the user interface to be searched according to the similar element coordinate information and the similar element size information.

[0059] The similar interface element is an element similar to the target interface element, and the similar element coordinates and the similar element size information are coordinate information and size information of the similar interface element, respectively. According to the coordinate information and the size information, the specific position of the similar interface element in the user interface to be searched can be determined, so as to generate and display the corresponding element search area based on the specific position of the element.

[0060] In actual application, after the similar interface elements are screened out by the method provided in the above embodiment, in order to facilitate the user to intuitively view, select or batch operate the similar interface elements, the similar interface elements can be highlighted by the element search area. Specifically, the specific occupied area of the similar interface element in the user interface to be searched can be determined according to the similar element coordinate information and the similar element size information, and a visual area, usually a high-light line frame with a frame, is drawn in the user interface according to the size of the determined area, which is used to indicate the position and range of the similar interface element in the user interface.

[0061] In specific implementation, each similar interface element corresponds to an independent element search area, and is displayed in a unified style, such as a dashed line frame, color differentiation, etc., so as to facilitate the user to view the interface elements in the element search area in different user interfaces. Correspondingly, the element search area not only supports visual prompt, but also supports click, multi-selection, drag and other interactive behaviors, for example, the user can click a certain element search area to individually select the similar interface element, and the frame selection of multiple element search areas is also supported to realize batch operation. After the user modifies the similar interface element (such as moving, adjusting size), the system can automatically update the coordinate and size information thereof, and synchronously refresh the corresponding element search area, to ensure real-time and accuracy.

[0062] In a specific embodiment of the present specification, the similar interface elements are determined to be located in the “registration interface D” and the “home page E”, respectively, and the high-light element search areas are generated in the user interfaces to which the similar interface elements belong according to the coordinate and size information of the similar interface elements, respectively. The user can perform click, batch modification and other operations in these areas without manually searching one by one.

[0063] Therefore, by generating the element search area, the user can intuitively see which elements are similar to the target element, which improves the friendliness and ease of use of interface interaction. The user can directly click or batch operate in the element search area, avoiding the cumbersome process of repeatedly switching interfaces or manually searching, and significantly improving the work efficiency.

[0064] Further, the method further comprises: in response to a trigger operation for the similar interface element, determining an element adjustment instruction corresponding to the similar interface element; and adjusting the similar interface element based on the element adjustment instruction, and displaying the adjusted similar interface element in the user interface to be searched.

[0065] The element adjustment instruction corresponding to the similar interface element can be understood as an instruction for adjusting the similar interface element, and the element adjustment instruction can be used to adjust the size, position, and the like of the element. After the similar interface element is adjusted based on the element adjustment instruction, the adjusted similar interface element is displayed in the user interface to be searched.

[0066] In an embodiment of the present specification, after the similar interface elements are determined, the user can perform a unified adjustment operation such as modifying the size, position, style, and the like on the elements in addition to viewing the elements, so as to improve the design efficiency and consistency. The user can select one or more similar interface elements in the generated element search area by clicking, box selecting, right-clicking a menu, and the like. The user can issue an element adjustment instruction by dragging, inputting or modifying a value, applying a preset style template, and the like. After the system receives the element adjustment instruction, the similar interface elements are adjusted according to the element adjustment instruction.

[0067] Based on this, by adjusting the similar interface elements found by the user, the purpose of automatically finding similar elements and quickly adjusting the similar elements is achieved, and the operation efficiency and accuracy of the user in the user interface design are improved.

[0068] Further, the element search area further comprises a text editing area and a feedback control; and the method further comprises: in response to an input event of target feedback content in the text editing area and a trigger operation for the feedback control, wherein the target feedback content is used to optimize the element search model; and displaying at least one updated element search area in the at least one editable user interface.

[0069] The text editing area can be understood as an area for inputting text by the user. The text editing area is used for the user to describe feedback opinions on the current search result, such as inconsistent button styles, misselected icons, and the like. The feedback control can be understood as a control for submitting feedback by the user, and is used to trigger the uploading and processing process of the feedback information. The target feedback content is the feedback opinion input by the user in the text editing area. The target feedback content can be used to optimize the element search model, so that the element search model can improve the subsequent search accuracy.

[0070] In actual application, the determined similar interface elements may be misselected, indicating that the element search model has an inaccurate output problem. The user can feed back the problem according to the search result output, input the target feedback content in the text editing area and click the feedback control, upload the target feedback content to the system, and the system can retrain the element search model according to the target feedback content to achieve the optimization purpose. Through this feedback mechanism, the element search model can output more accurate search results in the subsequent element determination process.

[0071] Correspondingly, the target feedback content of the user this time can be used to update the current similar interface elements, and an updated element search area is obtained.

[0072] Based on this, by providing a feedback entrance to make the user feel a sense of participation and control, the product affinity and user satisfaction are improved, and based on the user feedback mechanism, the system can continuously accumulate error cases and improvement suggestions in real use scenarios, and improve the model generalization ability.

[0073] The method provided in the specification for determining an interface element includes displaying a design page, the design page including at least one editable user interface and a search function control; in response to a trigger operation on a target user interface in the at least one editable user interface, displaying an element selection area in the target user interface, the element selection area including a target interface element in the target user interface; in response to a trigger operation on the search function control, displaying at least one element search area in the at least one editable user interface, the element search area including similar interface elements associated with the target interface element, the similar interface elements being determined according to visual information and position information of the target interface element. When the user designs the at least one editable user interface in the design page, the search function control can be used to quickly search and select the interface elements. By the trigger operation of the user on the target user interface, the element selection area including the target interface element is displayed in the target user interface, and the selected target interface element is displayed to the user. By the trigger operation of the user on the search function control, at least one element search area is displayed in the at least one editable user interface, the element search area including similar interface elements associated with the target interface element, and the similar interface elements searched according to the visual information and position information of the target interface element are displayed to the user, providing a convenient and fast similar matching selection method for the interface elements, facilitating the user to further perform batch operations on the selected similar interface elements. And the similar interface elements are determined according to the visual information and position information, improving the search accuracy of the similar interface elements, reducing the cumbersome operation of the user for secondary screening of the search results, and improving the user experience.

[0074] The following describes the method for determining the interface element in combination with the accompanying drawings Figure 6 The method for determining the interface element is further described by taking the application of the method for determining the interface element in batch deleting elements as an example. In this method, Figure 6 A flowchart of a process of a method for determining an interface element is shown, which includes the following steps.

[0075] Step 602: Display a design page, the design page including at least one editable user interface and a search function control.

[0076] In an implementable manner, the user opens multiple editable UI interfaces through a design page of the interface design software, and the design page includes a search function control.

[0077] Step 604: In response to a triggering operation on a target user interface in the at least one editable user interface, display an element selection area in the target user interface, the element selection area including a target interface element in the target user interface.

[0078] In an implementable manner, the user selects a target user interface from the multiple UI interfaces and selects a text box as a target interface element in the target user interface, and the element selection area is used to highlight the text box in the target user interface.

[0079] In response to a triggering operation on the layer display area, display target layer information corresponding to the target user interface in the layer display area; and in response to a triggering operation on the target layer information, display an element selection area in the target user interface, the element selection area including a target interface element in the target user interface.

[0080] In an implementable manner, the user can also click to expand the target layer information of the target user interface in the layer display area and select the target interface element by selecting the target layer information.

[0081] Step 606: In response to a triggering operation on the search function control, display an interface selection guide control in the design page.

[0082] Step 608: In response to a triggering operation of the interface selection guide control in the design page, determine a user interface to be searched for in the at least one editable user interface.

[0083] In an implementable manner, the user selects a user interface to be searched for according to the guiding prompt of the interface selection guide control.

[0084] Step 610: determine a target interface graph corresponding to the target user interface, a to-be-searched interface graph corresponding to the to-be-searched user interface, and element attribute information corresponding to the target interface element, wherein the element attribute information includes visual information and position information.

[0085] Step 612: input the target interface graph, the to-be-searched interface graph, and the element attribute information into the element matching model to obtain the predicted element coordinates output by the element matching model.

[0086] Step 614: determine to-be-searched element coordinates corresponding to a to-be-searched interface element in the to-be-searched user interface; calculate element position overlap information according to the to-be-searched element coordinates and the predicted element coordinates; and determine a similar interface element of the target interface element in the to-be-searched interface element according to the element position overlap information.

[0087] Step 616: determine similar element coordinate information and similar element size information corresponding to the similar interface element; and generate and display an element search area corresponding to the similar interface element in the to-be-searched user interface according to the similar element coordinate information and the similar element size information.

[0088] In an implementable manner, the element search area is highlighted according to the coordinate information and the size information of the similar interface element in the corresponding to-be-searched user interface.

[0089] Step 618: in response to a trigger operation on the similar interface element, determine an element adjustment instruction corresponding to the similar interface element; and adjust the similar interface element based on the element adjustment instruction, and display the adjusted similar interface element in the to-be-searched user interface.

[0090] In an implementable manner, the user selects the similar interface element and issues an element deletion instruction, and a batch of similar interface elements are deleted according to the element deletion instruction, and a user interface after the similar interface elements are deleted is displayed in the to-be-searched user interface.

[0091] The method provided by the specification for determining an interface element enables quick searching and selection of the interface element by using a search function control when a user designs a page and edits at least one user interface. By triggering an operation of the user on a target user interface, an element selection area including a target interface element is displayed on the target user interface, and the user is shown the selected target interface element. By triggering an operation of the user on the search function control, at least one element search area is displayed on the at least one editable user interface, and the element search area includes similar interface elements associated with the target interface element, and the user is shown the similar interface elements searched according to the visual information and position information of the target interface element, thereby providing a convenient and quick similar matching selection method of the interface element for the user, and facilitating the user to perform further batch operations on the selected similar interface elements. Moreover, the similar interface elements are determined according to the visual information and position information, the searching accuracy of the similar interface elements is improved, the user is saved from the cumbersome operation of secondary screening of the search results, and the user experience is improved.

[0092] Corresponding to the method embodiments described above, the specification also provides a device for determining an interface element, Figure 7 A structure diagram of a device for determining an interface element is shown in an embodiment of the specification. As shown in the figure, Figure 7 The device includes: A first display module 702 configured to display a design page, the design page including at least one editable user interface and a search function control; A second display module 704 configured to display an element selection area in a target user interface in response to a triggering operation of the target user interface in the at least one editable user interface, the element selection area including a target interface element in the target user interface; A third display module 706 configured to display at least one element search area in the at least one editable user interface in response to a triggering operation of the search function control, the element search area including similar interface elements associated with the target interface element, the similar interface elements being determined according to visual information and position information of the target interface element.

[0093] Optionally, the design page further includes a layer display area, the layer display area including layer information corresponding to the at least one editable user interface; the device further includes: in response to a triggering operation of the layer display area, displaying target layer information corresponding to the target user interface in the layer display area; and in response to a triggering operation of the target layer information, displaying an element selection area in the target user interface, the element selection area including a target interface element in the target user interface.

[0094] Optionally, the apparatus comprises displaying an interface selection guide control in the design page in response to a triggering operation for the search function control; determining a to-be-searched user interface in the at least one editable user interface in response to a triggering operation of the interface selection guide control in the design page; and displaying at least one element search area in the to-be-searched user interface.

[0095] Optionally, the apparatus further comprises determining a target interface graph corresponding to the target user interface, a to-be-searched interface graph corresponding to the to-be-searched user interface, and element attribute information corresponding to the target interface element, wherein the element attribute information comprises visual information and position information; inputting the target interface graph, the to-be-searched interface graph, and the element attribute information into an element matching model to obtain the predicted element coordinates output by the element matching model, wherein the predicted element coordinates are used to determine a similar interface element associated with the target interface element.

[0096] Optionally, the apparatus further comprises determining to-be-searched element coordinates corresponding to a to-be-searched interface element in the to-be-searched user interface; calculating element position overlap information according to the to-be-searched element coordinates and the predicted element coordinates; and determining a similar interface element associated with the target interface element in the to-be-searched interface element according to the element position overlap information.

[0097] Optionally, the apparatus further comprises determining similar element coordinate information and similar element size information corresponding to the similar interface element; and generating and displaying an element search area corresponding to the similar interface element in the to-be-searched user interface according to the similar element coordinate information and the similar element size information.

[0098] Optionally, the apparatus further comprises determining similar element coordinate information and similar element size information corresponding to the similar interface element; and generating and displaying an element search area corresponding to the similar interface element in the to-be-searched user interface according to the similar element coordinate information and the similar element size information.

[0099] Optionally, the apparatus further comprises determining an element adjustment instruction corresponding to the similar interface element in response to a triggering operation for the similar interface element; and adjusting the similar interface element based on the element adjustment instruction and displaying the adjusted similar interface element in the to-be-searched user interface.

[0100] Optionally, the apparatus further comprises inputting target feedback content into the text editing area in response to an input event and a triggering operation for the feedback control, wherein the target feedback content is used to optimize the element search model; and displaying at least one updated element search area in the at least one editable user interface.

[0101] The above is a schematic solution of the interface element determination apparatus of the embodiment. It should be noted that the technical solution of the interface element determination apparatus is the same as the technical solution of the interface element determination method described above, and the details of the technical solution of the interface element determination apparatus not described in detail can be referred to the description of the technical solution of the interface element determination method.

[0102] Figure 8 A structural block diagram of a computing device 800 is shown according to one embodiment of the present specification. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected with the memory 810 through a bus 830, and a database 850 is used to save data.

[0103] The computing device 800 further includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 can include one or more of any type of network interface (e.g., a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC).

[0104] In one embodiment of the present specification, the above-mentioned components of the computing device 800 and other components not shown in the Figure 8 may be connected to each other, for example, through a bus. It should be understood that Figure 8 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0105] The computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 can also be a mobile or stationary server.

[0106] The processor 820 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for determining interface elements described above.

[0107] The above is a schematic solution of the computing device of the embodiment. It should be noted that the technical solution of the computing device and the technical solution of the method for determining interface elements described above belong to the same concept, and the details of the technical solution of the computing device that are not described in detail can be referred to the description of the technical solution of the method for determining interface elements.

[0108] An embodiment of the present specification further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method for determining interface elements described above.

[0109] The above is a schematic solution of the computer-readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the method for determining interface elements described above belong to the same concept, and the details of the technical solution of the storage medium that are not described in detail can be referred to the description of the technical solution of the method for determining interface elements.

[0110] An embodiment of the present specification further provides a computer program product, which includes a computer program or instructions, and the computer program or instructions, when executed by a processor, implement the steps of the method for determining interface elements described above.

[0111] The above is a schematic solution of the computer program product of the embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the method for determining interface elements described above belong to the same concept, and the details of the technical solution of the computer program product that are not described in detail can be referred to the description of the technical solution of the method for determining interface elements.

[0112] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims that follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable features.

[0113] The computer readable medium can include any entity or apparatus that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0114] It should be noted that for the foregoing method embodiments, the acts described can be performed in a different order from that described, and that various steps can be performed in parallel or with no breaks in between acts. Also, the various steps described with respect to the method embodiments can be combined in any order, can be performed with fewer than all of the steps described, or can be performed using methods having more than all of the described steps. Further, the method embodiments described above can be implemented using any combination of software code or routines, firmware code or routines, and / or hardware components.

[0115] In the above embodiments, the description of each embodiment is focused on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0116] The above-disclosed specification of the present application is only used to help explain the present application. The alternative embodiments do not describe all the details, nor limit the application to only the specific embodiments described. Obviously, according to the content of the embodiments of the present application, many modifications and changes can be made. The present application selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present application, so that those skilled in the art can well understand and use the present application.

Claims

1. A method for determining an interface element, characterized in that: include: Displaying a design page, wherein the design page includes at least one editable user interface and a search function control; In response to a triggering operation on a target user interface in the at least one editable user interface, displaying an element selection area in the target user interface, wherein the element selection area includes a target interface element in the target user interface; In response to a triggering operation on the search function control, at least one element search area is displayed in the at least one editable user interface, and the element search area includes similar interface elements associated with the target interface element, and the similar interface elements are determined based on the visual information and position information of the target interface element.

2. The method according to claim 1, characterized in that The design page further includes a layer display area, wherein the layer display area includes layer information corresponding to the at least one editable user interface; After the design page is displayed, it also includes: In response to a triggering operation on the layer display area, displaying target layer information corresponding to the target user interface in the layer display area; In response to a triggering operation on the target layer information, an element selection area is displayed in the target user interface, where the element selection area includes a target interface element in the target user interface.

3. The method according to claim 1, characterized in that In response to a triggering operation on the search function control, displaying at least one element search area in the at least one editable user interface includes: In response to a triggering operation on the search function control, displaying an interface selection guide control in the design page; In response to a triggering operation of the interface selection guide control in the design page, determining a user interface to be found in the at least one editable user interface; At least one element search area is displayed in the to-be-searched user interface.

4. The method according to claim 3, characterized in that After determining the user interface to be found in the at least one editable user interface, the method further includes: Determine a target interface image corresponding to the target user interface, a to-be-searched interface image corresponding to the to-be-searched user interface, and element attribute information corresponding to the target interface element, wherein the element attribute information includes visual information and position information; The target interface image, the interface image to be searched and the element attribute information are input into an element matching model to obtain predicted element coordinates output by the element matching model, wherein the predicted element coordinates are used to determine similar interface elements associated with the target interface element.

5. The method according to claim 4, characterized in that After obtaining the predicted element coordinates output by the element matching model, the method further includes: Determining coordinates of an element to be found corresponding to an interface element to be found in the user interface to be found; Calculating element position overlap information according to the coordinates of the element to be found and the coordinates of the predicted element; Similar interface elements associated with the target interface element are determined in the interface elements to be found according to the element position overlap information.

6. The method according to claim 5, characterized in that After determining similar interface elements associated with the target interface element, the method further includes: Determining similar element coordinate information and similar element size information corresponding to the similar interface elements; An element search area corresponding to the similar interface element is generated and displayed in the to-be-searched user interface according to the similar element coordinate information and the similar element size information.

7. The method according to claim 6, characterized in that The method further comprises: In response to a triggering operation on the similar interface element, determining an element adjustment instruction corresponding to the similar interface element; The similar interface elements are adjusted based on the element adjustment instruction, and the adjusted similar interface elements are displayed in the to-be-searched user interface.

8. The method according to claim 4, characterized in that The element search area further includes a text editing area and a feedback control; the method further includes: responding to an input event of target feedback content in the text editing area and a triggering operation on the feedback control, wherein the target feedback content is used to optimize the element search model; At least one updated element search area is displayed in the at least one editable user interface.

9. A device for determining interface elements, characterized in that: include: A first display module is configured to display a design page, wherein the design page includes at least one editable user interface and a search function control; a second display module configured to, in response to a triggering operation on a target user interface in the at least one editable user interface, display an element selection area in the target user interface, wherein the element selection area includes a target interface element in the target user interface; The third display module is configured to display at least one element search area in the at least one editable user interface in response to a trigger operation on the search function control, wherein the element search area includes similar interface elements associated with the target interface element, and the similar interface elements are determined based on the visual information and position information of the target interface element.

10. A computing device, characterized in that include: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for determining the interface elements described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that It stores computer-executable instructions, which, when executed by a processor, implement the steps of the method for determining interface elements described in any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method for determining the interface elements according to any one of claims 1 to 8.

Images