Design manuscript element identification method and device

Abstract

The embodiment of the invention provides a design manuscript element recognition method and device, and the method comprises the steps: extracting a group of candidate elements from each design manuscript associated with at least two design manuscript types, and screening each group of candidate elements according to an element screening rule, and obtaining a plurality of groups of candidate screen elements; and inputting at least two candidate screen elements included in the multiple groups of candidate screen elements into the design draft identification model to obtain element identification information corresponding to the at least two candidate screen elements. Selecting a target screen element from the at least two candidate screen elements according to the at least two pieces of element identification information and the element level information associated with the at least two candidate screen elements, firstly performing primary screening on the plurality of candidate elements extracted from the design draft, and identifying the screened candidate screen elements by utilizing a design draft identification model, precise selection of the target screen elements is realized, and the recognition efficiency of the target screen elements is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a method and device for identifying design draft elements. Background Art

[0002] A design draft is a visual plan that a designer translates their creative ideas into. Each design draft consists of multiple elements, each with specific attributes. Design draft elements are independent modules that make up the interface design, and design draft elements within a design draft can often be reused.

[0003] In the prior art, when extracting design elements from a design draft, attribute conditions for the design draft elements, such as the width and height of the design draft, are usually pre-set. The size of the design draft element is determined based on its attributes, such as width, height, and area, and the design draft element is identified within the design draft. However, the restrictive conditions set by this design draft element recognition method have significant limitations and do not analyze the design draft from a visual perspective, resulting in low recognition accuracy and efficiency. Therefore, a more effective design draft recognition method is urgently needed to address the above issues. Summary of the Invention In view of this, embodiments of this specification provide a method for identifying design elements. One or more embodiments of this specification also relate to a design element identification device, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a method for identifying design draft elements is provided, comprising: Extracting a group of candidate elements from each design draft associated with at least two design draft types, and screening each group of candidate elements according to element screening rules to obtain multiple groups of candidate screen elements; Inputting at least two candidate screen elements included in the multiple groups of candidate screen elements into the design draft recognition model to obtain element identification information corresponding to the at least two candidate screen elements respectively; A target screen element is selected from the at least two candidate screen elements according to the at least two pieces of element identification information and the element level information associated with the at least two candidate screen elements.

[0005] Optionally, extracting a group of candidate elements from each design draft associated with at least two design draft categories includes: Determining a design draft set that includes design drafts associated with at least two design draft categories and having an element hierarchical structure; Elements are extracted from each design draft according to the preset element hierarchy to obtain multiple groups of candidate elements.

[0006] Optionally, the filtering each group of candidate elements according to the element filtering rule to obtain multiple groups of candidate screen elements comprises: determining element type information, element size information and element display information according to the element filtering rule; filtering at least two groups of candidate screen elements matching the element type information, the element size information and the element display information from the multiple groups of candidate elements.

[0007] Optionally, the inputting at least two candidate screen elements contained in the multiple groups of candidate screen elements into the design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements respectively comprises: converting the at least two candidate screen elements contained in the multiple groups of candidate screen elements into at least two candidate screen images, and inputting the at least two candidate screen images into the design draft recognition model; splitting the at least two candidate screen images into multiple candidate screen image blocks by using the design draft recognition model, and converting the multiple candidate screen image blocks into candidate screen vectors; processing the candidate screen vectors associated with the multiple candidate screen image blocks by using an attention layer of the design draft recognition model, and outputting the element recognition information corresponding to the at least two candidate screen elements respectively by an output layer of the design draft recognition model.

[0008] Optionally, the selecting a target screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information and element hierarchical information associated with the at least two candidate screen elements comprises: selecting a target candidate screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information; determining whether there are at least one pair of associated screen elements having a hierarchical association relationship in the target candidate screen element; if yes, taking screen elements in the target candidate screen element except the at least one pair of associated screen elements as a first target screen element, and selecting a second target screen element from the at least one pair of associated screen elements according to the hierarchical relationship of the at least one pair of associated screen elements, and taking the first target screen element and the second target screen element as the target screen element; if no, taking the target candidate screen element as the target screen element.

[0009] Optionally, the selecting a target candidate screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information comprises: determining target element recognition information corresponding to a target type from the at least two pieces of element recognition information; selecting a target candidate screen element corresponding to an element identifier contained in the target element recognition information from the at least two candidate screen elements.

[0010] Optionally, after selecting the target screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information and element hierarchy information associated with the at least two candidate screen elements, the method further includes: extracting a target design draft in a target design draft format corresponding to the target screen element from a plurality of design drafts according to an element identifier of the target screen element.

[0011] Optionally, the training of the design draft recognition model includes: extracting sample candidate elements from a plurality of design drafts included in a sample design draft set; screening the sample candidate elements to obtain sample target elements, and adding design draft labels to the sample target elements according to element types to construct training samples; training an initial design draft recognition model using the training samples to obtain the design draft recognition model.

[0012] According to a second aspect of an embodiment of the present specification, a design draft element recognition device is provided, including: a screening module configured to extract a group of candidate elements from each design draft associated with at least two design draft categories, and screen each group of candidate elements according to an element screening rule to obtain a plurality of groups of candidate screen elements; an input module configured to input at least two candidate screen elements included in the plurality of groups of candidate screen elements to a design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements, respectively; a selection module configured to select a target screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information and element hierarchy information associated with the at least two candidate screen elements.

[0013] According to a third aspect of an embodiment of the present specification, a computing device is provided, including: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which, when executed by the processor, implement the steps of the design draft element recognition method described above.

[0014] According to a fourth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, which, when executed by a processor, implement the steps of the design draft element recognition method described above.

[0015] According to a fifth aspect of an embodiment of the present specification, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the design draft element recognition method described above.

[0016] The design draft element identification method provided by one embodiment of the specification extracts a group of candidate elements in each design draft associated with at least two design draft categories, and screens each group of candidate elements according to element screening rules to obtain multiple groups of candidate screen elements. At least two candidate screen elements contained in the multiple groups of candidate screen elements are input into a design draft identification model to obtain element identification information corresponding to the at least two candidate screen elements respectively. According to the at least two pieces of element identification information and element hierarchical information associated with the at least two candidate screen elements, a target screen element is selected from the at least two candidate screen elements. The multiple candidate elements extracted from the design draft are screened once, and the candidate screen elements screened are identified by using the design draft identification model, so as to realize accurate selection of the target screen element and improve the identification efficiency of the target screen element. When the target screen element is selected, the element hierarchical relationship is considered, and accurate extraction of the target screen element is realized from the perspective of vision and element integrity. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a flowchart of a design draft element identification method according to an embodiment of the specification; Figure 2 FIG. 2 is a process flowchart of a design draft element identification method according to an embodiment of the specification; Figure 3 FIG. 3 is an element structure diagram of a design draft element identification method according to an embodiment of the specification; Figure 4 FIG. 4 is a structural diagram of a design draft element identification device according to an embodiment of the specification; Figure 5 FIG. 5 is a structural block diagram of a computing device according to an embodiment of the specification. DETAILED DESCRIPTION

[0018] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the specification. However, the specification can be practiced without the specific details, other than those described herein, and it will be readily understood by those skilled in the art that the specification can be practiced without the specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to unnecessarily obscure aspects of the specification.

[0019] The terms used in one or more embodiments of the specification are merely for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the specification. The singular forms "a", "said" and "the" used in one or more embodiments of the specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the specification means and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that, although the terms first, second, etc. can be employed in describing various information in one or more embodiments of the present specification, the information should not be limited to such terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, in the absence of departing from the scope of one or more embodiments of the present specification, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "upon" or "in response to determining".

[0021] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0022] First, the terms involved in one or more embodiments of the present specification are explained.

[0023] Design draft: Design draft is a visual solution that designers turn their ideas into. Each design draft is composed of multiple elements, each element contains specific attributes, and all elements form a tree structure to constitute a design draft, which displays the final appearance of a product in a complete manner, such as the page layout, button position, color matching of a mobile phone APP. In the design tool Motiff, the design draft not only contains static visual effects, but also demonstrates button click effects, page jump logic, just like a moving user manual, helping the development team accurately restore the design and ensure that every detail of the product from conception to landing is accurate.

[0024] Motiff: A smart design tool for designers and product managers, focusing on the understanding and automatic analysis of UI design drafts, especially in design draft analysis, component identification, interaction restoration, code generation, etc.

[0025] Design draft element: Design draft element is an independent module that constitutes interface design, including buttons, icons, text, etc. Each element carries multiple layers of information: such as color, font, shadow parameters, size, element name, id, etc. These elements form a tree structure to constitute a design draft.

[0026] Screen extraction: In the design tool, designers design applications suitable for mobile phones, web pages, iPads, TVs and other terminals. Designers add many design schemes in the design tool to form design drafts. Each design draft may be a complete terminal screen content, or it may be a part of the terminal screen content, or it may contain multiple screen contents in a design draft. Since most of the subsequent functions of the Motiff product are based on complete screen operations, a scheme is needed to extract the complete screen from the design draft. This extraction scheme is called screen extraction.

[0027] VIT: VIT model is a visual conversion model (Vision Transformer, ViT for short) in the field of artificial intelligence. In the VIT model, the input image is divided into a series of small image blocks, and each image block is converted into a corresponding vector through a linear transformation layer. Then all these vectors are sent to a series of transformer layers for processing. The biggest advantage of this method is that the VIT model can capture global image information, that is, the relationship between each image block and all other image blocks, thereby improving the performance of image recognition tasks.

[0028] FRAME type: FRAME generally refers to a container or framework that defines an independent area or view on the screen. This area can contain other UI components such as buttons, text boxes, etc., and the positions of these components relative to the FRAME are fixed. In a design draft, if a part is identified as a FRAME type, it represents a complete screen or an independent page in the application.

[0029] GROUP type: a more flexible organization method, mainly used for logically related element collections. Grouping (GROUP) multiple elements can help keep the design draft clean and simplify element selection and operation. If a group of elements closely cooperate to form a complete information display or interactive unit, the GROUP may also be considered as a part of a complete screen.

[0030] In this specification, a design draft element identification method is provided. This specification also relates to a design draft element identification device, a computing device, a computer readable storage medium and a computer program product, which are described in detail in the following embodiments.

[0031] Reference Figure 1 , Figure 1 A flowchart of a design draft element identification method according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0032] Step 102: extract a set of candidate elements in each design draft of the at least two design draft categories, and screen each set of candidate elements according to the element screening rule to obtain multiple sets of candidate screen elements.

[0033] Specifically, a design draft is a visual scheme that a designer converts from a creative idea. Each design draft is composed of multiple elements, each element contains specific attributes, and all elements form a tree structure to constitute a design draft. The design draft completely displays the interface of a product in the form of graphics, text annotations, etc., such as the page layout, button position, color matching, etc. of a mobile phone APP. The design draft categories include but are not limited to complete screen design drafts, non-complete screen design drafts, and multi-screen collection design drafts. The complete screen design draft refers to a design draft corresponding to the design of a complete user interface, such as a complete recommendation page of a shopping APP or a complete music playing page of a music APP. The non-complete screen design draft is a part of the complete screen design draft, such as a design draft obtained by intercepting a complete recommendation page of a shopping APP. The multi-screen collection design draft can be a design draft that collects at least two complete screen design drafts, or at least two non-complete screen design drafts, or a complete screen design draft and a non-complete screen design draft. The design draft elements contained in each design draft can be used as a set of candidate elements corresponding to the design draft. The element screening rule is used to filter the sub-elements in each set of candidate elements according to the preset element attributes. The multiple sets of candidate screen elements are the candidate screen elements obtained after screening each set of candidate elements, which may correspond to a complete screen design draft.

[0034] Based on this, in the scenario of identifying design drafts, design draft elements can be identified from the multiple design drafts associated with at least two design draft categories prepared in advance. In each design draft corresponding to each design draft category, a set of candidate elements is extracted, and each set of candidate elements is screened according to the element screening rule, i.e. the sub-elements in each set of candidate elements that do not meet the preset element attributes are filtered out, and multiple sets of candidate screen elements are obtained.

[0035] Further, considering that design drafts are usually composed of design draft elements drawn in a hierarchical nested manner, the drawing of design draft elements usually has certain rules and orders, and when extracting candidate elements in a design draft, the elements can also be extracted according to the element hierarchical structure of the design draft, which is implemented as follows: Determine the design drafts associated with at least two design draft categories and having element hierarchical structures contained in the design draft set; extract elements from each design draft according to the preset element hierarchy to obtain multiple sets of candidate elements.

[0036] Based on this, a design draft set is determined, and the design draft set includes multiple design drafts associated with at least two design draft categories and having an element hierarchical structure. For each design draft, element extraction can be performed on each design draft according to a preset element hierarchy, and a preset number of elements are extracted to form a group of candidate elements corresponding to each design draft.

[0037] For example, in the case of a design draft for which design draft element identification is performed, the design draft is a design draft related to a shopping APP. The design draft set can include a design draft corresponding to a complete recommendation page of the shopping APP, a design draft corresponding to part of the recommendation page of the shopping APP, and a design draft composed of multiple recommendation pages of the shopping APP. For each design draft corresponding to a recommendation page, the design draft elements are arranged in a tree structure according to the element hierarchy. When performing element extraction on each design draft, the first five element levels of design draft elements can be extracted. For the design draft of the shopping APP page, the elements corresponding to the first layer of the product card list, the elements corresponding to the second layer of the product, the elements corresponding to the third layer of the product description, the button elements in the fourth layer of the product description, and the icon and text elements in the fifth layer of the button elements are extracted. The five layers of elements form a group of candidate elements.

[0038] In summary, element extraction is performed on each design draft according to a preset element hierarchy, and a preset number of elements are extracted to form a group of candidate elements corresponding to each design draft, ensuring that each group of candidate elements has a perfect element tree structure.

[0039] Further, considering that the purpose of identifying design draft elements in a design draft is to identify design draft elements corresponding to a complete screen design draft, and a complete screen design draft usually has a certain size and type to match the size of a terminal screen such as a mobile phone or a computer, each group of candidate elements can be filtered in multiple dimensions such as size and type, and the specific implementation is as follows: According to the element filtering rule, determine element type information, element size information, and element display information; and filter at least two groups of candidate screen elements that match the element type information, the element size information, and the element display information from the multiple groups of candidate elements.

[0040] Specifically, the element type information refers to a preset element type, indicating the existence form of the element. The element size information includes, but is not limited to, the width, height, and area of the element. The element display information refers to the display attribute of whether it is originally displayed (visible) in the design draft.

[0041] Based on this, the element type information, the element size information, and the element display information are determined according to the element screening rule. Each group of candidate elements is screened based on the element type information, the element size information, and the element display information, and at least two groups of candidate screen elements that match the element type information, the element size information, and the element display information are screened out from the multiple groups of candidate elements. The sub-elements that do not match the element type information, the element size information, and the element display information are filtered out from each group of candidate elements, and multiple groups of candidate screen elements are obtained.

[0042] In the above example, after multiple groups of candidate elements are extracted from the multiple design drafts, the sub-elements included in the multiple groups of candidate elements can be screened according to the element type information, the element size information, and the element display information in the dimensions of element type, element size, and element display. Since the size of the sub-elements corresponding to the complete screen is not too small, the element size can be preset, and the sub-elements smaller than the preset element size (width*height>50000) can be filtered out. Since the complete screen is usually of the FRAME and GROUP types (in user interface design and development, FRAME and GROUP are two common ways to organize and manage design elements), the sub-elements of each group of candidate elements that are not of the FRAME and GROUP types can be filtered out. In addition, the sub-elements of each group of candidate elements whose element properties are invisible can be filtered out, and multiple groups of candidate screen elements are obtained.

[0043] In summary, at least two groups of candidate screen elements that match the element type information, the element size information, and the element display information are screened out from the multiple groups of candidate elements, and the sub-elements corresponding to the complete screen with a low probability are filtered out from each group of candidate elements. This reduces the workload of subsequent element recognition and improves the element recognition efficiency.

[0044] Step 104: Input at least two candidate screen elements included in the multiple groups of candidate screen elements into a design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements, respectively.

[0045] Specifically, after a set of candidate elements is extracted from each design draft associated with at least two design draft categories and each set of candidate elements is screened according to the element screening rule to obtain multiple sets of candidate screen elements, at least two candidate screen elements included in the multiple sets of candidate screen elements can be input into a design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements respectively. The design draft recognition model can be a VIT model. In the VIT model, an input image is divided into a series of small image blocks, and each image block is converted into a corresponding vector through a linear transformation layer. Then all the vectors are sent to a series of transformer layers for processing. The biggest advantage of this method is that the VIT model can capture global image information, that is, the relationship between each image block and all other image blocks, thereby improving the performance of the image recognition task. The element recognition information corresponding to the at least two candidate screen elements respectively is the recognition result of each candidate screen element, which is "non-complete screen" or "complete screen".

[0046] Based on this, after a set of candidate elements is extracted from each design draft associated with at least two design draft categories and each set of candidate elements is screened according to the element screening rule to obtain multiple sets of candidate screen elements, at least two candidate screen elements included in the multiple sets of candidate screen elements are input into a design draft recognition model, and each candidate screen element is recognized by using the design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements respectively.

[0047] Further, considering that the candidate screen elements are extracted from the design draft in the form of design draft elements, and the recognition object of the design draft recognition model is an image, it is necessary to convert the at least two candidate screen elements into at least two candidate screen images before inputting them into the design draft recognition model for processing. The specific implementation is as follows: The at least two candidate screen elements included in the multiple sets of candidate screen elements are converted into at least two candidate screen images, and the at least two candidate screen images are input into the design draft recognition model. The design draft recognition model splits the at least two candidate screen images into multiple candidate screen image blocks, and converts the multiple candidate screen image blocks into candidate screen vectors. The attention layer of the design draft recognition model processes the candidate screen vectors associated with the multiple candidate screen image blocks, and the output layer of the design draft recognition model outputs element recognition information corresponding to the at least two candidate screen elements respectively.

[0048] Specifically, the design draft editing software supports one-key export of design draft elements corresponding images, so as to obtain at least two candidate screen images corresponding to at least two candidate screen elements. The candidate screen image block is an image block obtained by cutting the candidate screen image by using the design draft recognition model. The candidate screen vector is an encoding vector obtained by encoding the candidate screen image block. The attention layer and the output layer are model layers of the design draft recognition model. The attention layer can correspond to a transformer layer, which is used to capture the relationship between the candidate screen vectors and obtain the relationship between the image blocks, thereby improving the performance of image recognition.

[0049] Based on this, the design draft software is used to convert at least two candidate screen elements contained in multiple groups of candidate screen elements into at least two candidate screen images, and input the at least two candidate screen images into the design draft recognition model. For each candidate screen image, the design draft recognition model is used to split into multiple candidate screen image blocks, and the multiple candidate screen image blocks are converted into candidate screen vectors. The attention layer of the design draft recognition model is used to process the candidate screen vectors associated with the multiple candidate screen image blocks corresponding to each candidate screen image, identify the relationship between the candidate screen image blocks, and output element recognition information corresponding to the at least two candidate screen elements through the output layer of the design draft recognition model.

[0050] In summary, after converting the at least two candidate screen elements into at least two candidate screen images, the design draft recognition model is used for processing, thereby improving the performance of image recognition.

[0051] Step 106: selecting a target screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information and element hierarchical information associated with the at least two candidate screen elements.

[0052] Specifically, after the above-mentioned at least two candidate screen elements contained in multiple groups of candidate screen elements are input into the design draft recognition model, and at least two pieces of element recognition information corresponding to the at least two candidate screen elements are obtained, a target screen element can be selected from the at least two candidate screen elements according to the at least two pieces of element recognition information and element hierarchical information associated with the at least two candidate screen elements. The element hierarchical information associated with the at least two candidate screen elements refers to the hierarchical relationship between the at least two candidate screen elements. There can be two candidate screen elements having a hierarchical association relationship in the at least two candidate screen elements.

[0053] Based on this, after the above at least two candidate screen elements containing multiple groups of candidate screen elements are input into the design draft recognition model, and the element recognition information corresponding to the at least two candidate screen elements is obtained, the target screen element is selected from the at least two candidate screen elements according to the at least two pieces of element recognition information and the element hierarchy information associated with the at least two candidate screen elements. The at least two candidate screen elements can be filtered according to the element recognition information, and the candidate screen element recognized as a complete screen is filtered out. Then, the target screen element is selected from the at least two candidate screen elements according to the element hierarchy information associated with the at least two candidate screen elements.

[0054] Further, after the at least two candidate screen elements are determined, the target candidate screen element predicted as a complete screen by the design draft recognition model can be selected from the at least two candidate screen elements according to the at least two pieces of element recognition information, and the specific implementation is as follows: According to the at least two pieces of element recognition information, the target candidate screen element is selected from the at least two candidate screen elements; it is judged whether there is at least one pair of associated screen elements having a hierarchical association relationship in the target candidate screen element; if yes, the screen elements in the target candidate screen element except the at least one pair of associated screen elements are taken as a first target screen element, and a second target screen element is selected from the at least one pair of associated screen elements according to the hierarchical relationship of the at least one pair of associated screen elements, and the first target screen element and the second target screen element are taken as the target screen element; if not, the target candidate screen element is taken as the target screen element.

[0055] Specifically, the target candidate screen element is the candidate screen element whose element recognition information is determined as a complete screen in the at least two candidate screen elements. The hierarchical association relationship is the parent-child relationship between elements, for example, the element of the first layer is the parent element, and the element of the second layer drawn on the basis of the parent element is the child element. The parent element and the child element have a hierarchical association relationship of parent-child class, and are regarded as a pair of associated screen elements. The second target screen element is the parent element selected from the pair of associated screen elements.

[0056] According to the at least two pieces of element identification information, a target candidate screen element that is identified as a complete screen is selected from the at least two candidate screen elements. It is judged whether there are at least one pair of associated screen elements with a hierarchical association relationship in the target candidate screen element. If there are, it indicates that there are two target candidate screen elements with a parent-child relationship in the target candidate screen element. Then, screen elements in the target candidate screen element except the at least one pair of associated screen elements are taken as first target screen elements, and a second target screen element that is a parent element is selected from the at least one pair of associated screen elements according to the hierarchical relationship of the at least one pair of associated screen elements. The first target screen elements and the second target screen element are taken as the target screen element. If there are not, it indicates that there are not two target candidate screen elements with a parent-child relationship in the target candidate screen element, and the target candidate screen element can be taken as the target screen element.

[0057] According to the above example, a target candidate screen element that is identified as a complete screen is selected from the at least two candidate screen elements according to the at least two pieces of element identification information. In a case where there are two candidate screen elements of a shopping APP recommendation page and a commodity card display list with a parent-child relationship in the target candidate screen element, since the two candidate screen elements of the shopping APP recommendation page and the commodity card display list have a parent-child relationship and come from the same design draft, the candidate screen element of the commodity card display list as a child element is deleted and is not taken as the target screen element.

[0058] In summary, according to the at least two pieces of element identification information and the element hierarchical information associated with the at least two candidate screen elements, a target screen element is selected from the at least two candidate screen elements, the accurate selection of the target screen element is realized, and the problem of repeated selection is avoided.

[0059] Further, before the target candidate screen element is selected from the at least two candidate screen elements, the at least two pieces of element identification information can be filtered, and target element identification information corresponding to complete screen information is selected from the at least two pieces of element identification information. The specific implementation is as follows: Target element identification information corresponding to a target type is determined from the at least two pieces of element identification information. A target candidate screen element corresponding to an element identifier contained in the target element identification information is selected from the at least two candidate screen elements.

[0060] According to this, target element identification information corresponding to a target type, i.e., a complete screen, is determined from the at least two pieces of element identification information. An element identifier contained in the target element identification information is determined, and a target candidate screen element corresponding to the element identifier contained in the target element identification information is selected from the at least two candidate screen elements. The target candidate screen element is a candidate screen element predicted as a complete screen by a design draft identification model.

[0061] In the above example, when the design draft recognition model predicts at least two candidate screen elements, it outputs two types of element recognition information, i.e., complete screen and incomplete screen. The element recognition information corresponding to the complete screen can be selected from the at least two types of element recognition information as the target element recognition information.

[0062] In summary, the target element recognition information corresponding to the complete screen information is selected from the at least two types of element recognition information, and the target candidate screen element predicted as the candidate screen is determined based on the target element recognition information, thereby realizing the screening of the candidate screen element corresponding to the complete screen.

[0063] Further, after selecting the target screen element from the at least two candidate screen elements, the target design draft corresponding to the complete screen can be determined in the multiple design drafts of multiple categories according to the element identifier of the target screen element, which is specifically implemented as follows: After selecting the target screen element from the at least two candidate screen elements, the target design draft corresponding to the target design draft format is extracted from the multiple design drafts according to the element identifier of the target screen element.

[0064] Specifically, the element identifier of the target screen element can be the marking information used to mark the design draft of the target screen element, and the element identifier can also be the element ID of the target screen element, from which the design draft to which the target screen element belongs can be known.

[0065] In practical applications, the element identifier of the target screen element is determined, the target design draft is determined from the multiple design drafts according to the element identifier of the target screen element, and the target design draft format of the target design draft is extracted. This facilitates subsequent design, modification, and other processing based on the target design draft. It realizes accurate and efficient determination of complete screen design drafts from multiple design drafts of multiple design draft categories.

[0066] Further, the training of the design draft recognition model includes: The sample candidate elements are extracted from the multiple design drafts contained in the sample design draft set. The sample target elements are obtained by screening the sample candidate elements, and the training samples are constructed by adding design draft labels to the sample target elements according to the element types. The initial design draft recognition model is trained using the training samples to obtain the design draft recognition model.

[0067] Specifically, the sample design draft set includes complete screen design drafts, incomplete screen design drafts, and multi-screen set design drafts. The sample candidate elements refer to at least two design draft elements extracted from each sample design draft, and the at least two design draft elements corresponding to each design draft are arranged in a tree structure. The sample target elements are selected from the multiple groups of sample candidate elements according to the element size, element type, and element display state. The design draft label of whether the sample target element is a complete screen is added to each sample target element by identifying whether each sample target element corresponds to a complete screen. The training sample is composed of the sample target elements and the design draft label.

[0068] In actual applications, the sample candidate elements are extracted from multiple design drafts included in the sample design draft set, and a group of sample candidate elements with a tree structure is extracted for each sample design draft. The sample target elements are obtained by screening each group of sample candidate elements to filter out elements with small areas and elements with types not conforming to the preset element type. The training sample is constructed by adding the design draft label to the sample target element according to the element type. Each sample target element in the training sample corresponds to a design draft label of whether it is a complete screen. The design draft recognition model is trained using the training sample, and the sample target element input to the initial design draft recognition model is in a picture format, and the output content is 0 or 1, where 0 represents an incomplete screen and 1 represents a complete screen.

[0069] The design draft element recognition method provided in an embodiment of the present specification extracts a group of candidate elements in each design draft associated with at least two design draft categories, and screens each group of candidate elements according to the element screening rule to obtain multiple groups of candidate screen elements. At least two candidate screen elements included in the multiple groups of candidate screen elements are input to the design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements, respectively. The target screen element is selected from the at least two candidate screen elements according to the at least two pieces of element recognition information and the element hierarchical information associated with the at least two candidate screen elements. The target screen element is accurately selected by first screening the multiple candidate elements extracted from the design draft and identifying the screened candidate screen elements using the design draft recognition model, thereby improving the recognition efficiency of the target screen element. When selecting the target screen element, the element hierarchical relationship is considered to accurately extract the target screen element from the perspective of visual angle and element integrity.

[0070] The following describes the design draft element recognition method provided in the present specification in conjunction with the accompanying drawings. Figure 2 The design draft element recognition method provided in the present specification is further described by taking the application of the design draft element recognition method in design draft screen extraction as an example. Among them, Figure 2 FIG. 1 shows a processing process flowchart of a design draft element recognition method provided in an embodiment of the present specification, which specifically includes the following steps: Step 202: Determine a set of design files.

[0071] A design is a scheme in which a designer converts a creative idea into a visualization. Each design is composed of a plurality of elements, each element contains specific attributes, and all elements form a tree structure to constitute the design. When training a design recognition model for recognizing a complete screen, a batch of design files can be collected to form a set of design files.

[0072] Step 204: Determine a plurality of designs in the set of design files, and extract a set of candidate elements corresponding to each design.

[0073] Each design in the set of design files can be a complete screen, an incomplete screen, or a multi-screen set. For each design, the design elements are extracted according to the element editing hierarchy, and a set of candidate elements in a tree arrangement can be extracted for each design.

[0074] In the case of a design file corresponding to a shopping APP, a set of candidate elements extracted for a complete screen design can be seen in (a) of Figure 3 As shown in (a) of Figure 3 , a set of candidate elements are arranged in a tree form, the first layer contains the "shopping APP recommendation page" element; the second layer contains the "navigation bar", "search box", "goods card display list", and "icon bar" elements; the third layer, under the "navigation bar" element, contains the "recommendation", "digital", and "sports" elements; under the "search box" element, there are "input box" and "search icon" elements; under the "goods card display list" element, there are "goods card 1" and "goods card 2" elements; under the "icon bar" element, there are "shopping cart icon" and "message icon" elements.

[0075] A set of candidate elements extracted for a complete screen set design can be seen in (b) of Figure 3 As shown in (b) of Figure 3 , a set of candidate elements are arranged in a tree form, the first layer contains the "shopping APP page set" element; the second layer contains the "recommendation page", "goods detail page", and "payment page" elements.

[0076] Step 206: Element filtering is performed on each set of candidate elements to obtain a plurality of sets of candidate screen elements.

[0077] Since each group of candidate elements is a tree structure, and in practical applications, the complete screen will not be nested too deep, therefore, all elements of the top five layers of each design draft are extracted. Since the complete screen is of the FRAME and GROUP types, therefore, when filtering each group of candidate elements, only FRAME and GROUP type elements are retained. Since the complete screen size will not be too small, therefore, when filtering each group of candidate elements, only elements with a width*height>50000 are retained. In addition, all visible elements need to be retained. Through the above rules, most elements can be filtered, and the remaining elements also constitute multiple groups of candidate screen elements in a tree structure.

[0078] In practical applications, by filtering the group of tree-shaped candidate elements shown in (a) of FIG. 6 according to the above rules, a group of tree-shaped candidate screen elements shown in (c) of FIG. 6 can be obtained, which includes the "shopping APP recommendation page" element and the "product card display list" element. Figure 3 Figure 3 In practical applications, by filtering the group of tree-shaped candidate elements shown in (a) of FIG. 6 according to the above rules, a group of tree-shaped candidate screen elements shown in (c) of FIG. 6 can be obtained, which includes the "shopping APP recommendation page" element and the "product card display list" element.

[0079] Step 208: For each group of candidate screen elements, candidate screen element extraction is performed to obtain multiple candidate screen elements.

[0080] For each group of candidate screen elements, screen elements are extracted and converted into a picture format to obtain multiple candidate screen elements.

[0081] Step 210: Element labels are added to each candidate screen element to construct training samples.

[0082] When adding element labels to each candidate screen element, a label indicating whether it is a complete screen is added. The labeled data and the pictures are one-to-one corresponding, which constitutes the training data.

[0083] Step 212: The initial design draft recognition model is trained using the training samples to obtain a trained design draft recognition model.

[0084] The initial design draft recognition model can be a VIT model. In the VIT model, the input image is divided into a series of small image blocks, and each image block is converted into a corresponding vector through a linear transformation layer. Then all these vectors are sent to a series of transformer layers for processing. The biggest advantage of this method is that the VIT model can capture global image information, i.e., the relationship between each image block and all other image blocks, thereby improving the performance of the image recognition task.

[0085] Step 214: Determine multiple to-be-predicted elements, and input the to-be-predicted elements into the design draft recognition model to obtain prediction information of each to-be-predicted element.

[0086] ​After obtaining the trained design draft recognition model, inference can be performed. The to-be-predicted elements that need to be inferred are determined, and multiple to-be-predicted elements are input to the design draft recognition model at a time, so that the recognition result of whether each to-be-predicted element is a complete screen can be obtained.

[0087] Step 216: determining a target element corresponding to a complete screen in the multiple to-be-predicted elements according to the prediction information.

[0088] According to the recognition result of whether it is a complete screen, a target element predicted as a complete screen can be determined in the multiple to-be-predicted elements.

[0089] Step 218: filtering the target element according to the element association relationship to obtain a complete screen element.

[0090] It is considered that in the target element, there can be two target elements having an element association relationship, that is, a parent-child relationship, so that the child element needs to be deleted to obtain the final complete screen element.

[0091] In summary, the complete screen element is recognized based on the combination of vision and element attributes, the inference time of the design draft recognition model is reduced, and accurate classification of design draft elements is realized. Nested screen extraction is supported. If there are multiple screens in the design draft, only the screens inside the design draft can be recognized, rather than the entire design draft being recognized as a screen.

[0092] Corresponding to the method embodiments described above, the present specification also provides design draft element recognition device embodiments, Figure 4 A structural schematic diagram of a design draft element recognition device provided by an embodiment of the present specification is shown. As Figure 4 shown, the device includes: The screening module 402 is configured to extract a group of candidate elements in each design draft associated with at least two design draft categories, and screen each group of candidate elements according to an element screening rule to obtain multiple groups of candidate screen elements; The input module 404 is configured to input at least two candidate screen elements contained in the multiple groups of candidate screen elements to a design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements, respectively; The selection module 406 is configured to select a target screen element from the at least two candidate screen elements according to at least two pieces of element recognition information and element hierarchical information associated with the at least two candidate screen elements.

[0093] An optional embodiment, the screening module 402 is further configured to: determine a design draft associated with at least two design draft categories and having an element hierarchical structure in a design draft set; extract elements from each design draft according to a preset element hierarchy to obtain multiple groups of candidate elements.

[0094] In an optional embodiment, the screening module 402 is further configured to: determine element type information, element size information, and element display information according to the element screening rule; screen at least two groups of candidate screen elements that match the element type information, the element size information, and the element display information from the multiple groups of candidate screen elements.

[0095] In an optional embodiment, the input module 404 is further configured to: convert at least two candidate screen elements included in the multiple groups of candidate screen elements into at least two candidate screen images, and input the at least two candidate screen images into the design draft recognition model; split the at least two candidate screen images into multiple candidate screen image blocks by using the design draft recognition model, and convert the multiple candidate screen image blocks into candidate screen vectors; process the candidate screen vectors associated with the multiple candidate screen image blocks by using an attention layer of the design draft recognition model, and output element recognition information corresponding to the at least two candidate screen elements respectively by using an output layer of the design draft recognition model.

[0096] In an optional embodiment, the selection module 406 is further configured to: select a target candidate screen element from the at least two candidate screen elements according to the at least two pieces of element recognition information; determine whether at least one pair of associated screen elements having a hierarchical association relationship exist in the target candidate screen element; if yes, take screen elements in the target candidate screen element other than the at least one pair of associated screen elements as a first target screen element, and select a second target screen element from the at least one pair of associated screen elements according to a hierarchical relationship of the at least one pair of associated screen elements, and take the first target screen element and the second target screen element as the target screen element; if no, take the target candidate screen element as the target screen element.

[0097] In an optional embodiment, the selection module 406 is further configured to: determine target element recognition information corresponding to a target type from the at least two pieces of element recognition information; select a target candidate screen element corresponding to an element identifier included in the target element recognition information from the at least two candidate screen elements.

[0098] In an optional embodiment, the selection module 406 is further configured to: According to the element identifier of the target screen element, a target design draft in a target design draft format is extracted from a plurality of design drafts.

[0099] In an optional embodiment, the selection module 406 is further configured to: extracting sample candidate elements from a plurality of design drafts contained in a sample design draft set; screening the sample candidate elements to obtain sample target elements, and constructing a training sample by adding a design draft label to the sample target elements according to element types; training an initial design draft recognition model by using the training sample to obtain the design draft recognition model.

[0100] In an embodiment of the present specification, a design draft element recognition device is provided. A set of candidate elements is extracted from each design draft of at least two design draft categories, and each set of candidate elements is screened according to an element screening rule to obtain a plurality of sets of candidate screen elements. At least two candidate screen elements contained in the plurality of sets of candidate screen elements are input into a design draft recognition model to obtain element recognition information corresponding to the at least two candidate screen elements, respectively. According to the at least two pieces of element recognition information and element hierarchical information associated with the at least two candidate screen elements, a target screen element is selected from the at least two candidate screen elements. The plurality of candidate elements extracted from the design draft are screened once, and the candidate screen elements screened are recognized by using the design draft recognition model, so as to realize accurate selection of the target screen element and improve the recognition efficiency of the target screen element. When the target screen element is selected, the element hierarchical relationship is considered, and accurate extraction of the target screen element is realized from the perspective of vision and element integrity.

[0101] The above is a schematic scheme of the design draft element recognition device of the present embodiment. It should be noted that the technical scheme of the design draft element recognition device belongs to the same concept as the technical scheme of the design draft element recognition method described above. The technical scheme of the design draft element recognition device is not described in detail, and the description of the technical scheme of the design draft element recognition method is referred to.

[0102] Figure 5 A structural block diagram of a computing device 500 according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 through a bus 530, and a database 550 is used to save data.

[0103] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 540 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0104] In one embodiment of the present specification, the above-mentioned components of the computing device 500 and other components not shown in the Figure 5 may be connected to each other, for example, through a bus. It should be understood that Figure 5 The computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0105] The computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 can also be a mobile or stationary server.

[0106] The processor 520 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the above-described design element recognition method.

[0107] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the design draft element identification method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the design draft element identification method.

[0108] An embodiment of the present specification further provides a computer readable storage medium, which stores computer executable instructions. The computer executable instructions are executed by a processor to implement the steps of the design draft element identification method.

[0109] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the design draft element identification method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the design draft element identification method.

[0110] An embodiment of the present specification further provides a computer program product, which includes a computer program or instructions. The computer program or instructions are executed by a processor to implement the steps of the design draft element identification method.

[0111] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the design draft element identification method described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the design draft element identification method.

[0112] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order described in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0113] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0114] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present specification.

[0115] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0116] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited only by the claims and their entire scope and equivalents.

Claims

1. A method for identifying design draft elements, characterized in that: include: Extracting a group of candidate elements from each design draft associated with at least two design draft types, and screening each group of candidate elements according to element screening rules to obtain multiple groups of candidate screen elements; Inputting at least two candidate screen elements included in the multiple groups of candidate screen elements into the design draft recognition model to obtain element identification information corresponding to the at least two candidate screen elements respectively; A target screen element is selected from the at least two candidate screen elements according to the at least two pieces of element identification information and the element level information associated with the at least two candidate screen elements.

2. The design draft element recognition method according to claim 1, characterized in that: The step of extracting a set of candidate elements from each design draft associated with at least two design draft types includes: Determining a design draft set that includes design drafts associated with at least two design draft categories and having an element hierarchical structure; Elements are extracted from each design draft according to the preset element hierarchy to obtain multiple groups of candidate elements.

3. The design draft element recognition method according to claim 1, characterized in that: The step of screening each group of candidate elements according to the element screening rules to obtain multiple groups of candidate screen elements includes: Determine element type information, element size information, and element display information according to element screening rules; At least two groups of candidate screen elements that match the element type information, the element size information, and the element display information are screened out from the multiple groups of candidate elements.

4. The design draft element recognition method according to claim 1, characterized in that: The step of inputting at least two candidate screen elements included in the plurality of groups of candidate screen elements into the design draft recognition model to obtain element identification information corresponding to the at least two candidate screen elements includes: converting at least two candidate screen elements included in the multiple groups of candidate screen elements into at least two candidate screen images, and inputting the at least two candidate screen images into the design draft recognition model; Using the design draft recognition model, the at least two candidate screen images are split into a plurality of candidate screen image blocks, and the plurality of candidate screen image blocks are converted into candidate screen vectors; The candidate screen vectors associated with the plurality of candidate screen image blocks are processed using the attention layer of the design draft recognition model, and the element identification information corresponding to the at least two candidate screen elements is output through the output layer of the design draft recognition model.

5. The design draft element recognition method according to claim 1, characterized in that: The selecting a target screen element from the at least two candidate screen elements according to the at least two pieces of element identification information and the element level information associated with the at least two candidate screen elements includes: selecting a target candidate screen element from the at least two candidate screen elements according to the at least two pieces of element identification information; Determining whether there is at least one pair of associated screen elements having a hierarchical association relationship among the target candidate screen elements; If so, select the screen elements other than the at least one pair of associated screen elements in the target candidate screen elements as the first target screen element, and select a second target screen element from the at least one pair of associated screen elements based on the hierarchical relationship of the at least one pair of associated screen elements, and select the first target screen element and the second target screen element as the target screen elements; If not, the target candidate screen element is used as the target screen element.

6. The design draft element recognition method according to claim 5, characterized in that: The selecting a target candidate screen element from the at least two candidate screen elements according to the at least two pieces of element identification information includes: Determining target element identification information corresponding to the target type from the at least two pieces of element identification information; A target candidate screen element corresponding to the element identifier included in the target element identification information is selected from the at least two candidate screen elements.

7. The design draft element recognition method according to claim 1, characterized in that: After selecting a target screen element from the at least two candidate screen elements based on the at least two pieces of element identification information and the element hierarchy information associated with the at least two candidate screen elements, the method further includes: A target design draft corresponding to a target design draft format is extracted from a plurality of design drafts according to the element identifier of the target screen element.

8. The design draft element recognition method according to claim 1, characterized in that: The training of the design draft recognition model includes: Extracting sample candidate elements from a plurality of design drafts included in the sample design draft set; Screening the sample candidate elements to obtain sample target elements, and constructing training samples by adding design draft labels to the sample target elements according to element types; The initial design draft recognition model is trained using the training samples to obtain the design draft recognition model.

9. A design draft element recognition device, characterized in that: include: a screening module configured to extract a group of candidate elements from each design draft associated with at least two design draft types, and screen each group of candidate elements according to an element screening rule to obtain multiple groups of candidate screen elements; An input module is configured to input at least two candidate screen elements included in the multiple groups of candidate screen elements into the design draft recognition model to obtain element identification information corresponding to the at least two candidate screen elements respectively; The selection module is configured to select a target screen element from the at least two candidate screen elements according to at least two pieces of element identification information and element level information associated with the at least two candidate screen elements.

10. A computing device, characterized in that include: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that It stores computer-executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or instructions, which implements the steps of the method according to any one of claims 1 to 8 when executed by a processor.

Images