Component classification method and device based on graph attention network

Abstract

The application provides a component classification method and device based on a graph attention network, wherein the component classification method based on the graph attention network comprises the following steps: receiving an identification instruction for a design draft to be identified, wherein the design draft to be identified comprises at least one interface component; identifying the design draft to be identified to obtain interface component graph information corresponding to the design draft to be identified; inputting the design draft to be identified and the interface component graph information into a component classification model to obtain a component classification result corresponding to each interface component, wherein the component classification model comprises a neural network model based on a graph attention network. Through the method provided in the embodiment of the application, the interface component graph information corresponding to the design draft is generated, and the processing speed of the model is improved and the prediction accuracy is improved through the component classification model based on the graph attention network.

Description

Technical Field

[0001] This application relates to the field of interface design technology, and in particular to a component classification method based on graph attention networks. This application also relates to a component classification apparatus based on graph attention networks, a computing device, and a computer-readable storage medium. Background Technology

[0002] Front-end development is the process of creating web pages or apps (applications) and presenting them to users. During front-end development, dedicated designers create user interfaces (UIs). These design drafts are the drafts created by these designers for the UI, and the system can generate the corresponding UI based on these drafts.

[0003] In the process of generating a UI interface, it is necessary to generate corresponding pages based on the components in the design draft. However, there are many types of components in the design draft, and some components are quite similar. If the components are classified incorrectly, it will cause problems in the generated UI interface. Therefore, how to accurately identify the component classification has become an urgent problem for technical personnel to solve. Summary of the Invention

[0004] In view of this, embodiments of this application provide a component classification method based on graph attention networks. This application also relates to a component classification apparatus based on graph attention networks, a computing device, and a computer-readable storage medium, to solve the aforementioned problems existing in the prior art.

[0005] According to a first aspect of the embodiments of this application, a component classification method based on graph attention networks is provided, including:

[0006] Receive an identification instruction for a design draft to be identified, wherein the design draft to be identified includes at least one interface component;

[0007] Identify the design draft to be identified and obtain the interface component diagram information corresponding to the design draft to be identified;

[0008] The design draft to be identified and the interface component graph information are input into the component classification model to obtain the component classification results corresponding to each interface component. The component classification model includes a neural network model based on graph attention network.

[0009] According to a second aspect of the embodiments of this application, a component classification apparatus based on graph attention networks is provided, comprising:

[0010] The receiving module is configured to receive a recognition instruction for a design draft to be recognized, wherein the design draft to be recognized includes at least one interface component;

[0011] The recognition module is configured to recognize the design draft to be recognized and obtain the interface component diagram information corresponding to the design draft to be recognized;

[0012] The classification module is configured to input the design draft to be identified and the interface component graph information into the component classification model to obtain the component classification results corresponding to each interface component. The component classification model includes a neural network model based on graph attention network.

[0013] According to a third aspect of the embodiments of this application, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the component classification method based on graph attention network.

[0014] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the component classification method based on graph attention networks.

[0015] The component classification method based on graph attention networks provided in this application receives a recognition instruction for a design draft to be identified, wherein the design draft to be identified includes at least one interface component; identifies the design draft to be identified to obtain interface component graph information corresponding to the design draft to be identified; inputs the design draft to be identified and the interface component graph information into a component classification model to obtain component classification results corresponding to each interface component, wherein the component classification model includes a neural network model based on graph attention networks.

[0016] The component classification method based on graph attention networks provided in this application uses a component classification model based on graph attention networks in the process of classifying interface components. In the feature encoding layer of the component classification model, a graph attention network is used. By using parallel processing in the graph attention mechanism and referring to the weight values ​​of neighboring nodes, the component feature information of the target interface component is extracted. Then, a classifier is used to classify the components according to the component feature information. The parallel processing improves the processing speed of the model, and the reference to the weight values ​​of neighboring nodes enriches the feature information of the interface component, thereby improving the prediction accuracy of the target interface component. Attached Figure Description

[0017] Figure 1 This is a flowchart of a component classification method based on graph attention networks provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of interface component diagram information provided in an embodiment of this application;

[0019] Figure 3 This is a flowchart of a component classification method based on graph attention network for APP interface generation provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a component classification device based on a graph attention network according to an embodiment of this application;

[0021] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0022] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0023] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0024] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0025] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0026] This application provides a component classification method based on graph attention networks. This application also relates to a component classification apparatus based on graph attention networks, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0027] Figure 1 The flowchart illustrates a component classification method based on graph attention networks according to an embodiment of this application, specifically including the following steps:

[0028] Step 102: Receive an identification instruction for the design draft to be identified, wherein the design draft to be identified includes at least one interface component.

[0029] In this application, the design draft mentioned in the embodiments can be a design draft of an information technology (IT) product. The content of the design draft can include the front-end interface content of the product. The content of the design draft can include text content and non-text content. Non-text content can include images, lines, etc.

[0030] The design draft to be identified specifically refers to the design draft that needs to be processed in the method provided in this application. In practical applications, when developers design an application, they create a design draft for that application. The design draft includes the application's interface design, interface components, interaction relationships, etc. By identifying the design draft, the interface components contained in the design draft are obtained, and then the corresponding interface components are searched from the component library, thereby generating the application interface based on the design draft.

[0031] The design draft clearly identifies various interface components for the user, but the terminal cannot determine what these components are. Therefore, it is necessary to identify the component categories of each interface component to facilitate querying the corresponding interface components from the component library in subsequent processing, and finally generate the application interface from the design draft.

[0032] Based on this, the design draft that needs to be identified is the design draft to be identified. The method provided in this application is applied to electronic devices, such as those that have a display screen and support information browsing, such as personal mobile terminals like mobile phones, tablets, personal computers, smart wearable devices, etc.

[0033] Users operate electronic terminals to manipulate the design drafts to be identified on the electronic terminals. In practical applications, users can directly create the design drafts to be identified on the electronic terminals, or obtain the design drafts to be identified from other electronic terminals or the network by means of copying, downloading, etc. In this application, there is no limitation on how the electronic terminal obtains the design drafts to be identified.

[0034] Furthermore, users can load the design draft to be identified through the application and send identification instructions through the application so that the application can identify the design draft, thereby obtaining each interface component in the design draft, finding each interface component from the component library, and then generating the application interface corresponding to the design draft based on the layout of each interface component in the design draft.

[0035] In one specific embodiment provided in this application, taking design draft A as an example, the identification instruction for design draft A is received. Design draft A includes multiple interface components, such as button 1, text box 2, pop-up window 3, selection box 4, etc.

[0036] Step 104: Identify the design draft to be identified and obtain the interface component diagram information corresponding to the design draft to be identified.

[0037] The design draft to be identified is the interface information of the software application designed and developed by the developers. The design draft to be identified contains relevant information of each interface component. By identifying the design draft to be identified, the component information of each interface component can be obtained, thereby generating the interface component diagram information corresponding to the design draft to be identified.

[0038] Specifically, the interface component diagram information refers to the diagram information of each interface component in the design draft to be identified. For example, see... Figure 2 , Figure 2 This illustration shows a schematic diagram of interface component diagram information provided in an embodiment of this application, such as... Figure 2 As shown, a design draft A to be identified contains 5 interface components: Interface Component 1, Interface Component 2, Interface Component 3, Interface Component 4, and Interface Component 5. Interface Component 1 is associated with Interface Components 2 and 3; Interface Component 2 is associated with Interface Components 4 and 5; Interface Component 3 is associated with Interface Components 4 and 5; and Interface Component 4 is associated with Interface Component 5. Based on the design draft A to be identified, the following can be obtained: Figure 2 The interface component diagram shown.

[0039] In one specific embodiment provided in this application, identifying the design draft to be identified and obtaining the interface component diagram information corresponding to the design draft to be identified includes:

[0040] Identify the design draft to be identified, and obtain at least one interface component and the component attribute information corresponding to each interface component;

[0041] The interface component graph information is constructed based on the component attribute information corresponding to each interface component.

[0042] In practical applications, designers will design different interface components in the design draft during the design process. At the same time, they will set corresponding meta-information for each interface component. Meta-information is information about information, used to describe the structure, semantics, user, usage, etc. of the interface component. In practical applications, by parsing the design draft to be identified, the component meta-information corresponding to each interface component can be obtained. Then, from the component meta-information of each interface component, the component attribute information corresponding to each interface component can be obtained.

[0043] It should be noted that the component attribute information in this application may specifically include the color, border length, border width, text content, coordinates, area, and relationship with other interface components of the interface component, etc.

[0044] After obtaining the component attribute information of each interface component, the interface component diagram information can be created based on the component attribute information of each interface component.

[0045] Specifically, UI component graph information is constructed based on the component attribute information corresponding to each UI component, including:

[0046] Get the first component attribute information of the first interface component and the second component attribute information of the second interface component;

[0047] Based on the first component attribute information and the second component attribute information, determine the component association relationship between the first interface component and the second interface component;

[0048] Generate UI component graph information based on component relationships.

[0049] The first interface component and the second interface component are any two interface components among the interface components. The attribute information of the first component is the component attribute information of the first interface component, and the attribute information of the second component is the component attribute information of the second interface component. After the above steps, the component attribute information of each interface component can be obtained. Here, it is only necessary to determine the first interface component and the second interface component among the interface components to obtain the attribute information of the first component and the attribute information of the second component.

[0050] After obtaining the attribute information of the first component and the attribute information of the second component, it is possible to further determine whether there is a component association relationship between the first interface component and the second interface component based on the two component attribute information. If there is a component association relationship between the first interface component and the second interface component, a component relationship edge can be constructed between them; if there is no component association relationship between them, then there is no component relationship edge between them.

[0051] After traversing each interface component, the component relationships between each interface component can be determined, and then interface component graph information can be generated based on the component relationships between each interface component.

[0052] Interface component diagrams provide a more intuitive representation of the relationships between various interface components, directly and accurately identifying reference interface components associated with the target component. This facilitates more accurate reference information during subsequent component classification, leading to more accurate final prediction results.

[0053] Step 106: Input the design draft to be identified and the interface component graph information into the component classification model to obtain the component classification results corresponding to each interface component. The component classification model includes a neural network model based on graph attention network.

[0054] Specifically, the component classification model refers to a neural network model based on graph attention networks, which is used to decode the input design draft to be identified and the graph information of interface components, thereby obtaining the final component classification result of the interface components.

[0055] Graph Attention Networks (GAT) are a neural network architecture based on graph-structured data. They utilize hidden self-attention layers to address the shortcomings of previous graph convolution-based feature extraction. In GAT, through stacked layers, the target node can participate in the features of its neighbors. Different weights can be assigned to different nodes in the neighborhood, and no complex matrix operations or prior knowledge of the graph structure are required. In GAT, the operations between nodes are processed in parallel, resulting in high efficiency and fast processing speed.

[0056] Neural network models based on graph attention networks have the following characteristics:

[0057] 1. The operations of the self-attention layers in graph attention networks can be processed in parallel across all edges, and the calculation of the output features of all nodes can also be processed in parallel. Therefore, the neural network model of graph attention networks has high computational efficiency.

[0058] 2. Graph attention-based neural network models allow implicit assignment of different importance to nodes of the same neighbor, thereby improving model capacity.

[0059] 3. The attention mechanism is applied to the edges of the graph in a shared manner, without relying on prior access to the global graph structure or all its nodes. This allows neural network models based on graph attention networks to be used for inductive tasks.

[0060] Furthermore, the design draft to be identified and the interface component diagram information are input into the component classification model to obtain the component classification results corresponding to each interface component, including:

[0061] Identify the design draft to be identified, and obtain the component attribute information and component structure information corresponding to each interface component;

[0062] The component attribute information, component structure information, and interface component graph information corresponding to each interface component are input into the component classification model to obtain the component classification results corresponding to each interface component.

[0063] In practical applications, inputting the design draft to be identified into the component classification model actually requires inputting the component attribute information and component structure information corresponding to each interface component in the design draft into the component classification model. Each interface component is then represented using its corresponding component attribute information and component structure information. Finally, in the graph attention network-based component classification model, each interface component is classified according to its component attribute information and component structure information.

[0064] Component structure information specifically refers to the hierarchical information within a UI component itself. This includes details such as the number of layers within a component and the information contained in each layer. For example, a UI component might contain several child nodes, such as text, icons, buttons, checkboxes, etc. Component structure information essentially refers to the hierarchical information of components within a UI component.

[0065] Specifically, the component classification model includes an embedding layer, a feature encoding layer, and a classifier;

[0066] The embedding layer encodes various types of information into features that the terminal can process, making it easier for the model to process these types of information. The feature encoding layer extracts features related to the final component classification from the features of various types of information and analyzes the relationships between the features. The classifier outputs the component classification results based on the relationships between the features.

[0067] The component attribute information, component structure information, and interface component graph information are input into the component classification model to obtain the component classification results corresponding to each interface component, including S1060-S1068:

[0068] S1060. Determine the target interface component among the various interface components, and obtain the target component attribute information and target component structure information corresponding to the target interface component.

[0069] In one specific embodiment provided in this application, a single interface component is used as an example for explanation; the same processing method is applied to other interface components. In this embodiment, the component used for component classification and identification is called the target interface component, which is any one of the various interface components.

[0070] After identifying the target interface component, obtain the target component's attribute information and target component's structure information.

[0071] S1062. Determine the reference interface component corresponding to the target interface component in the interface component diagram information, and obtain the reference component attribute information and reference component structure information corresponding to the reference interface component.

[0072] In the interface component diagram information, determine the reference interface component corresponding to the target interface component, and at the same time obtain the reference component attribute information and reference component structure information of each reference interface component.

[0073] In practical applications, determining the reference interface component corresponding to the target interface component in the interface component diagram information specifically means determining the interface component that has a component association relationship with the target interface component in the interface component diagram information as the reference interface component.

[0074] The component attribute information and component structure information of each interface component obtained in the above steps can be used to obtain the reference component attribute information and reference component structure information after the reference interface component is determined.

[0075] S1064. Input the target component attribute information, target component structure information, reference component attribute information, and reference component structure information into the embedding layer to obtain the target component attribute feature information, target component structure feature information, reference component attribute feature information, and reference component structure feature information.

[0076] After acquiring the target component's attribute information, target component's structure information, and reference component's attribute information and structure information, the relevant information of the target interface component is input into the embedding layer. The embedding layer then performs embedding processing on the target component's attribute information, target component's structure information, reference component's attribute information, and reference component's structure information to obtain the target component's attribute feature information, target component's structure feature information, reference component's attribute feature information, and reference component's structure feature information. Through the embedding layer, the natural language component information is converted into feature information that can be recognized by machines.

[0077] S1066. Input the target component attribute feature information, the target component structural feature information, the reference component attribute feature information, and the reference component structural feature information into the feature encoding layer to obtain the target component encoded feature information.

[0078] After obtaining the target component's attribute features, target component's structural features, and reference component's attribute features and structural features, the feature information of the target interface component can be concatenated to obtain concatenated feature information, which represents the target component's feature information. Then, the feature encoding layer extracts features from the target component's feature information.

[0079] Specifically, taking a component attribute feature information of 50*728 dimensions and a component structural feature information of 60*728 dimensions as an example, multiple feature information can be concatenated to obtain concatenated feature information of 220*728 dimensions. This concatenated feature information is the target component feature information of the target interface component. Concatenating multiple feature information together for feature encoding can integrate the relationships between multiple feature information, making the extracted features more accurate, and thus more accurate in subsequent classification prediction.

[0080] When processing the component feature information of each interface component, the feature encoding layer determines the neighboring components j corresponding to any interface component i, and calculates the similarity coefficient between each neighboring component j and interface component i. Then, the similarity coefficients between interface component i and each neighboring component j are normalized to obtain the attention coefficient a corresponding to interface component i.

[0081] After calculating the attention coefficient 'a' corresponding to interface component i, the component feature information of interface component i and the component feature information of neighboring component j are weighted and summed according to the attention coefficient 'a' corresponding to interface component i, thereby obtaining the target component encoding feature information of interface component i.

[0082] After performing the same processing operation on each interface component in sequence, the feature extraction of each interface component is completed in the feature encoding layer, and the component encoding feature information corresponding to each interface component is obtained.

[0083] S1068. Input the target component encoding features into the classifier to obtain the target component classification result corresponding to the target interface component.

[0084] After determining the target component encoding features of the target interface component, the target component encoding features are input into the classifier, which classifies the target component based on the target component encoding features and outputs the target component classification result corresponding to the target interface component.

[0085] Furthermore, the classifier can be an XGBoost classifier. The basic building block of an XGBoost classifier is a decision tree. The decision trees that make up an XGBoost classifier are arranged in a specific order. The generation of a later decision tree takes into account the prediction results of the previous decision tree. That is, the later decision tree takes into account the bias of the previous decision tree. The data used by each decision tree when making predictions are the target component encoded feature information.

[0086] In the XGBoost classifier, there are multiple decision trees, each corresponding to a prediction. Finally, the predictions from each decision tree are summed to obtain the prediction result of the XGBoost classifier.

[0087] In one specific embodiment provided in this application, the component classification model is trained through the following steps:

[0088] Obtain training sample pairs, wherein the training sample pairs include sample design drafts, sample interface component diagram information corresponding to the sample design drafts, and sample component classification results;

[0089] The sample design draft and the sample interface component diagram information are input into the component classification model to obtain the predicted component classification result output by the component classification model;

[0090] The model loss value is calculated based on the classification results of the sample components and the classification results of the predicted components.

[0091] Adjust the model parameters of the component classification model based on the model loss value, and continue training the component classification model until the training stop condition is met.

[0092] In a specific embodiment provided in this application, a training method for a component classification model is also provided. In this embodiment, the component classification model is trained using supervised training. First, training sample pairs are obtained. The training sample pairs specifically include sample design drafts, sample interface component diagram information corresponding to the sample design drafts, and sample component classification results.

[0093] The sample design draft and sample interface component diagram information are input into the component classification model to be trained. After processing by the embedding layer, feature encoding layer and classifier in the component classification model, the predicted component classification result output by the component classification model is obtained.

[0094] Since the component classification model at this stage is not yet fully trained, there will be a discrepancy between the predicted component classification results and the actual sample component classification results. Therefore, the model parameters need to be adjusted. Specifically, the model loss value is calculated based on the sample component classification results and the predicted component classification results. In practical applications, the loss function used to calculate the model loss value can be the 0-1 loss function, absolute value loss function, squared loss function, cross-entropy loss function, etc. In this application, the choice of loss function is not limited and is determined by practical application. Here, the cross-entropy loss function is used as an example for explanation; see Formula 1 below:

[0095]

[0096] Where C represents the model loss value, x represents the sample design draft and sample interface component diagram information, y represents the sample component classification result, a represents the predicted component classification result, and n represents the total number of samples. The model parameters of the component classification model are adjusted through backpropagation based on the loss value. Specifically, the learning rate parameter, batch_size parameter, etc., can be adjusted in the component classification model. The model is then trained again using the next batch of sample data until the model training stops.

[0097] Specifically, the model training stopping conditions include the model loss value being less than a preset threshold and / or the number of training epochs reaching a preset number of epochs.

[0098] In one specific embodiment provided in this application, the training stopping condition is taken as the model loss value being less than a preset threshold. The preset threshold is 0.3. When the model loss value is less than 0.3, the component classification model is considered to have completed training.

[0099] In another specific embodiment provided in this application, taking a preset number of training rounds as the training stopping condition as an example, the preset number of training rounds is 20 rounds. When the training rounds of the sample data reach 20 rounds, the component classification model is considered to have completed training.

[0100] In another specific embodiment provided in this application, two training stopping conditions are set: a preset threshold and a preset number of training rounds. At the same time, the loss value and the number of training rounds are monitored. When either the model loss value or the number of training rounds meets the training stopping condition, the component classification model is considered to have completed training.

[0101] An embodiment of this application provides a component classification method based on graph attention networks, comprising receiving an identification instruction for a design draft to be identified, wherein the design draft to be identified includes at least one interface component; identifying the design draft to be identified to obtain interface component graph information corresponding to the design draft to be identified; inputting the design draft to be identified and the interface component graph information into a component classification model to obtain component classification results corresponding to each interface component, wherein the component classification model includes a neural network model based on graph attention networks.

[0102] The component classification method based on graph attention networks provided in this application uses a component classification model based on graph attention networks in the process of classifying interface components. In the feature encoding layer of the component classification model, a graph attention network is used. By using parallel processing in the graph attention mechanism and referring to the weight values ​​of neighboring nodes, the component feature information of the target interface component is extracted. Then, a classifier is used to classify the components according to the component feature information. The parallel processing improves the processing speed of the model, and the reference to the weight values ​​of neighboring nodes enriches the feature information of the interface component, thereby improving the prediction accuracy of the target interface component.

[0103] The following is in conjunction with the appendix Figure 3 Taking the component classification method based on graph attention networks provided in this application as an example of an application generated from an APP interface, the component classification method based on graph attention networks will be further explained. Figure 3 The present application provides a flowchart of a component classification method based on graph attention networks for APP interface generation, which includes the following steps:

[0104] Step 302: Obtain the interface design draft of the APP interface, which includes multiple interface components.

[0105] In the embodiments provided in this application, the developer designed an interface design draft for the APP, which includes multiple interface components.

[0106] Step 304: Identify the design draft to be identified and obtain at least one interface component and the component attribute information corresponding to each interface component.

[0107] In the embodiments provided in this application, the interface components and component attribute information of each interface component are determined by parsing the design draft to be identified.

[0108] Step 306: Obtain the first component attribute information of the first interface component and the second component attribute information of the second interface component.

[0109] In the embodiments provided in this application, an arbitrary first interface component and a second interface component are determined among the various interface components, and the first component attribute information of the first interface component and the second component attribute information of the second interface component are determined at the same time.

[0110] Step 308: Determine the component association relationship between the first interface component and the second interface component based on the first component attribute information and the second component attribute information.

[0111] In the embodiments provided in this application, the presence or absence of a component association relationship between the first interface component and the second interface component is determined based on the first component attribute information and the second component attribute information. Specifically, the component association relationship includes whether the first interface component and the second interface component are related or unrelated.

[0112] Step 310: Generate UI component graph information based on component relationships.

[0113] In the embodiments provided in this application, each interface component is used as a node in the graph. If there is a component association relationship between two interface components, it means that there is an edge between the corresponding nodes of the two interface components; if there is no component association relationship between two interface components, it means that there is no edge between the corresponding nodes of the two interface components. Interface component graph information is generated based on the component association relationships between the interface components.

[0114] Step 312: Identify the design draft to be identified and obtain the component structure information corresponding to each interface component.

[0115] In the embodiments provided in this application, the component structure information corresponding to each interface component can be obtained by parsing the design draft to be identified.

[0116] Step 314: Input the component attribute information, component structure information and interface component diagram information corresponding to each interface component into the component classification model to obtain the component classification results corresponding to each interface component.

[0117] In the embodiments provided in this application, the component attribute information and component structure information corresponding to each interface component are input into the component classification model. In the component classification model, feature extraction and feature classification are performed on the component attribute information and component structure information corresponding to each interface component based on graph attention and interface component graph information, and finally the component classification result corresponding to each interface component is obtained. In practical applications, the component classification result can be multiple prediction results and the prediction probability of each prediction result. Finally, the prediction result with the highest probability is selected as the component classification result corresponding to the interface component.

[0118] Step 316: Based on the component classification results corresponding to each interface component, obtain each interface component from the preset component library and generate the APP interface corresponding to the interface design draft.

[0119] In the embodiments provided in this application, based on the component classification results corresponding to each interface component, each interface component is obtained from the preset component library to generate the APP interface corresponding to the interface design draft.

[0120] An embodiment of this application provides a component classification method based on graph attention networks, comprising receiving an identification instruction for a design draft to be identified, wherein the design draft to be identified includes at least one interface component; identifying the design draft to be identified to obtain interface component graph information corresponding to the design draft to be identified; inputting the design draft to be identified and the interface component graph information into a component classification model to obtain component classification results corresponding to each interface component, wherein the component classification model includes a neural network model based on graph attention networks.

[0121] The component classification method based on graph attention networks provided in this application uses a component classification model based on graph attention networks in the process of classifying interface components. In the feature encoding layer of the component classification model, a graph attention network is used. By using parallel processing in the graph attention mechanism and referring to the weight values ​​of neighboring nodes, the component feature information of the target interface component is extracted. Then, a classifier is used to classify the components according to the component feature information. The parallel processing improves the processing speed of the model, and the reference to the weight values ​​of neighboring nodes enriches the feature information of the interface component, thereby improving the prediction accuracy of the target interface component.

[0122] Corresponding to the above method embodiments, this application also provides an embodiment of a component classification device based on graph attention networks. Figure 4 This diagram illustrates a structural schematic of a component classification device based on a graph attention network according to an embodiment of this application. Figure 4 As shown, the device includes:

[0123] The receiving module 402 is configured to receive a recognition instruction for a design draft to be recognized, wherein the design draft to be recognized includes at least one interface component;

[0124] The recognition module 404 is configured to recognize the design draft to be recognized and obtain the interface component diagram information corresponding to the design draft to be recognized;

[0125] The classification module 406 is configured to input the design draft to be identified and the interface component graph information into the component classification model to obtain the component classification results corresponding to each interface component, wherein the component classification model includes a neural network model based on graph attention network.

[0126] Optionally, the identification module 404 is further configured to:

[0127] Identify the design draft to be identified, and obtain at least one interface component and the component attribute information corresponding to each interface component;

[0128] The interface component graph information is constructed based on the component attribute information corresponding to each interface component.

[0129] Optionally, the identification module 404 is further configured to:

[0130] Get the first component attribute information of the first interface component and the second component attribute information of the second interface component;

[0131] Based on the first component attribute information and the second component attribute information, determine the component association relationship between the first interface component and the second interface component;

[0132] Generate UI component graph information based on component relationships.

[0133] Optionally, the classification module 406 is further configured to:

[0134] Identify the design draft to be identified, and obtain the component attribute information and component structure information corresponding to each interface component;

[0135] The component attribute information, component structure information, and interface component graph information corresponding to each interface component are input into the component classification model to obtain the component classification results corresponding to each interface component.

[0136] Optionally, the component classification model includes an embedding layer, a feature encoding layer, and a classifier;

[0137] The classification module 406 is further configured as follows:

[0138] The target interface component is identified among the various interface components, and the target component attribute information and target component structure information corresponding to the target interface component are obtained.

[0139] In the interface component diagram information, determine the reference interface component corresponding to the target interface component, and obtain the reference component attribute information and reference component structure information corresponding to the reference interface component;

[0140] The target component attribute information, target component structure information, reference component attribute information, and reference component structure information are input into the embedding layer to obtain the target component attribute feature information, target component structure feature information, reference component attribute feature information, and reference component structure feature information.

[0141] The target component attribute feature information, the target component structural feature information, the reference component attribute feature information, and the reference component structural feature information are input into the feature encoding layer to obtain the target component encoded feature information;

[0142] The target component's encoded features are input into the classifier to obtain the target component classification result corresponding to the target interface component.

[0143] The classification module 406 is further configured as follows:

[0144] The interface components that have a component association relationship with the target interface component are identified in the interface component diagram information as reference interface components.

[0145] Optionally, the device further includes a training module configured to:

[0146] Obtain training sample pairs, wherein the training sample pairs include sample design drafts, sample interface component diagram information corresponding to the sample design drafts, and sample component classification results;

[0147] The sample design draft and the sample interface component diagram information are input into the component classification model to obtain the predicted component classification result output by the component classification model;

[0148] The model loss value is calculated based on the classification results of the sample components and the classification results of the predicted components.

[0149] Adjust the model parameters of the component classification model based on the model loss value, and continue training the component classification model until the training stop condition is met.

[0150] An embodiment of this application provides a component classification device based on graph attention networks, comprising receiving a recognition instruction for a design draft to be identified, wherein the design draft to be identified includes at least one interface component; identifying the design draft to be identified to obtain interface component graph information corresponding to the design draft to be identified; inputting the design draft to be identified and the interface component graph information into a component classification model to obtain component classification results corresponding to each interface component, wherein the component classification model includes a neural network model based on graph attention networks.

[0151] The component classification device based on graph attention network provided in this application uses a component classification model based on graph attention network in the process of classifying interface components. In the feature encoding layer of the component classification model, a graph attention network is used. By using parallel processing in the graph attention mechanism and referring to the weight values ​​of neighboring nodes, the component feature information of the target interface component is extracted. Then, a classifier is used to classify the components according to the component feature information. The parallel processing improves the processing speed of the model, and the reference to the weight values ​​of neighboring nodes enriches the feature information of the interface component, thereby improving the prediction accuracy of the target interface component.

[0152] The above is an illustrative scheme of a component classification device based on graph attention networks according to this embodiment. It should be noted that the technical solution of this component classification device based on graph attention networks belongs to the same concept as the technical solution of the component classification method based on graph attention networks described above. For details not described in detail in the technical solution of the component classification device based on graph attention networks, please refer to the description of the technical solution of the component classification method based on graph attention networks described above.

[0153] Figure 5 A structural block diagram of a computing device 500 according to an embodiment of this application is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0154] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0155] In one embodiment of this application, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0156] Computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). Computing device 500 can also be a mobile or stationary server.

[0157] Specifically, when the processor 520 executes the computer instructions, it implements the steps of the component classification method based on graph attention networks.

[0158] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the component classification method based on graph attention networks described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the component classification method based on graph attention networks described above.

[0159] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the component classification method based on graph attention networks as described above.

[0160] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the component classification method based on graph attention networks described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the component classification method based on graph attention networks described above.

[0161] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0162] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0163] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0164] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0165] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A component classification method based on graph attention networks, characterized in that, include: Receive an identification instruction for a design draft to be identified, wherein the design draft to be identified includes at least one interface component; Identify the design draft to be identified and obtain the interface component diagram information corresponding to the design draft to be identified; The process involves identifying the design draft to be identified, obtaining component attribute information and component structure information corresponding to each interface component; a component classification model including an embedding layer, a feature encoding layer, and a classifier; determining the target interface component among the interface components, and obtaining the target component attribute information and target component structure information corresponding to the target interface component; determining the reference interface component corresponding to the target interface component in the interface component graph information, and obtaining the reference component attribute information and reference component structure information corresponding to the reference interface component; inputting the target component attribute information, target component structure information, reference component attribute information, and reference component structure information into the embedding layer to obtain target component attribute feature information, target component structure feature information, reference component attribute feature information, and reference component structure feature information; inputting the target component attribute feature information, target component structure feature information, reference component attribute feature information, and reference component structure feature information into the feature encoding layer to obtain target component encoding feature information; and inputting the target component encoding feature into the classifier to obtain the target component classification result corresponding to the target interface component. The component classification model includes a neural network model based on a graph attention network.

2. The method as described in claim 1, characterized in that, Identify the design draft to be identified, and obtain the interface component diagram information corresponding to the design draft to be identified, including: Identify the design draft to be identified, and obtain at least one interface component and the component attribute information corresponding to each interface component; The interface component graph information is constructed based on the component attribute information corresponding to each interface component.

3. The method as described in claim 2, characterized in that, The UI component graph is constructed based on the component attribute information corresponding to each UI component, including: Get the first component attribute information of the first interface component and the second component attribute information of the second interface component; Based on the first component attribute information and the second component attribute information, determine the component association relationship between the first interface component and the second interface component; Generate UI component graph information based on component relationships.

4. The method as described in claim 1, characterized in that, Determining the reference interface component corresponding to the target interface component from the interface component diagram information includes: The interface components that have a component association relationship with the target interface component are identified in the interface component diagram information as reference interface components.

5. The method as described in claim 1, characterized in that, The component classification model is trained through the following steps: Obtain training sample pairs, wherein the training sample pairs include sample design drafts, sample interface component diagram information corresponding to the sample design drafts, and sample component classification results; The sample design draft and the sample interface component diagram information are input into the component classification model to obtain the predicted component classification result output by the component classification model; The model loss value is calculated based on the classification results of the sample components and the classification results of the predicted components. Adjust the model parameters of the component classification model based on the model loss value, and continue training the component classification model until the training stop condition is met.

6. A component classification device based on graph attention networks, characterized in that, include: The receiving module is configured to receive a recognition instruction for a design draft to be recognized, wherein the design draft to be recognized includes at least one interface component; The recognition module is configured to recognize the design draft to be recognized and obtain the interface component diagram information corresponding to the design draft to be recognized; A classification module is configured to identify the design draft to be identified and obtain component attribute information and component structure information corresponding to each interface component. The component classification model includes an embedding layer, a feature encoding layer, and a classifier. It identifies a target interface component among the interface components and obtains the target component attribute information and target component structure information corresponding to the target interface component. It also identifies a reference interface component corresponding to the target interface component in the interface component graph information and obtains the reference component attribute information and reference component structure information corresponding to the reference interface component. The target component attribute information, target component structure information, reference component attribute information, and reference component structure information are input to the embedding layer to obtain target component attribute feature information, target component structure feature information, reference component attribute feature information, and reference component structure feature information. The target component attribute feature information, target component structure feature information, reference component attribute feature information, and reference component structure feature information are input to the feature encoding layer to obtain target component encoding feature information. The target component encoding feature information is input to the classifier to obtain the target component classification result corresponding to the target interface component. The component classification model includes a neural network model based on a graph attention network.

7. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer instructions, it implements the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium storing computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1-5.

Images