Model training method and device for classifying design draft components

Abstract

The embodiment of the invention provides a training method and device for a model for classifying design draft components. The training method for the model for classifying the design draft components comprises the steps that a design draft picture is acquired; processing the design draft picture to obtain structured data corresponding to the design draft picture; the structured data at least comprises region position information and component category information of each component in the design draft picture; obtaining training data according to the design draft picture and the structured data; training an initial model by using the training data to obtain a trained component classification model; the trained component classification model is used for generating classification information of components in the design draft picture.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a training method, device, equipment, medium and product for a model for classifying design draft components. Background Art

[0002] With the rapid development of science and technology, many tasks can be processed by models. Natural language processing models can understand and generate natural language text, which can be applied to classify design draft components. Components in design draft images can be classified based on design draft images. However, in the classification process, due to the diversity of component layouts in the design drafts, some components have high similarities, and the accuracy of classifying components in the design drafts is poor.

[0003] Therefore, how to provide a model that can accurately classify design draft components is a technical problem that needs to be solved urgently. Summary of the invention In view of this, an embodiment of this specification provides a training method for a model for classifying design draft components. One or more embodiments of this specification also relate to a training device for a model for classifying design draft components, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a training method for a model for classifying design draft components is provided, comprising: Get the design draft picture; Processing the design draft image to obtain structured data corresponding to the design draft image; the structured data at least includes regional location information and component category information of each component in the design draft image; Obtaining training data according to the design draft image and the structured data; The training data is used to train the initial model to obtain a trained component classification model; the trained component classification model is used to generate classification information of components in the design draft image.

[0005] According to a second aspect of an embodiment of this specification, a training device for a model for classifying design draft components is provided, comprising: The picture acquisition module is configured to acquire the design draft picture; An image processing module is configured to process the design draft image to obtain structured data corresponding to the design draft image; the structured data at least includes regional location information and component category information of each component in the design draft image; A training data acquisition module is configured to obtain training data according to the design draft image and the structured data; The model training module is configured to train the initial model using the training data to obtain a trained component classification model; the trained component classification model is used to generate classification information of components in the design draft image. According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the training method for the model for classifying design draft components are implemented.

[0006] According to a fourth aspect of an embodiment of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned training method for a model for classifying design draft components.

[0007] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned training method for a model for classifying design draft components.

[0008] At least one embodiment provided in this specification can achieve the following beneficial effects: by processing the design draft image, structured data containing component category information of each component in the design draft image is obtained, the design draft image and the structured data are composed of training data, and the initial model is trained using the training data to obtain a trained component classification model, thereby improving the recognition and classification capabilities of the trained component classification model, so that the trained component classification model can capture the fine-grained differences between similar design draft components, and further improve the accuracy of the component category information generated using the trained component classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of a training method for a model for classifying design draft components provided by an embodiment of this specification; Figure 2 It is a schematic diagram of structured data provided by an embodiment of this specification; Figure 3 It is a structural schematic diagram of a training device for a model for classifying design draft components provided by an embodiment of this specification; Figure 4 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0010] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0011] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0012] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0013] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0014] First, the terms involved in one or more embodiments of this specification are explained.

[0015] Design draft: During the product development process, the visual draft designed by the designer for the product can be called a design draft. The design draft can be composed of a series of different components according to a customized layout, which can be used to show the appearance, layout, color, size and other elements of the product. Design draft can be applied to many fields, including graphic design, web design, UI design, industrial design and architectural design.

[0016] Design Draft Component: A design draft component can represent a customer service element used to build an interface or product during the design process. It is understandable that a design draft component is a reusable design element that can be created once in a design draft and then used multiple times in different pages or frames. This feature can help designers improve the consistency and efficiency of their designs. When designers design visual drafts for products in different fields, the design draft components used can be different. For example, components in graphic design can include specific graphic elements, icons, typography styles, etc.; design draft components in UI design can include buttons, input boxes, tabs, drop-down menus, pop-ups, etc.; design draft components in architectural design can include beams, columns, windows, etc.

[0017] Fine-tune: Fine-tune is a technique in the field of machine learning and deep learning that can be used to optimize the performance of pre-trained models to adapt to new tasks and data sets. You can choose a model that has been trained on a large-scale dataset as the base model, and prepare to fine-tune the pre-trained model with a dataset that is more focused on solving specific problems or covering a specific field. Specifically, you can use the weights and parameters in the pre-trained model as a starting point and train it again on new tasks and datasets to improve the model's accuracy and generalization performance on new tasks.

[0018] Multimodal large model: A multimodal machine learning model with large-scale parameters or large-scale data training can process and generate multiple types of data, and can simultaneously process two or more different forms of data, such as pictures, text, audio, video, structured data, unstructured data, etc. The "large" here is reflected in the complexity of the model, the number of parameters, and the required computing resources.

[0019] GPT-4: GPT-4 is one of a series of generative pre-trained transform models (GPT) developed by OpenAI. It has been trained on a large amount of Internet text and is a large language model that can generate coherent sentences or paragraphs in a human style.

[0020] In this specification, a training method for a model for classifying design draft components is provided. This specification also relates to a training device for a model for classifying design draft components, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0021] See also Figure 1 , Figure 1A flowchart of a method for training a model for classifying design draft components according to an embodiment of the present specification is shown. From a program perspective, the execution subject of the process can be a program installed on a server or training platform. From a hardware perspective, the execution subject of the process can be a server or training platform capable of model training. Figure 1 As shown, the specific steps include the following steps.

[0022] Step 102: Obtain a design draft image.

[0023] The design draft images in the embodiments of this specification may be design draft images of one or more design types such as graphic design, web design, UI design, industrial design, and architectural design. The design draft image may be a complete design draft image or a design draft image of a local area in a complete design draft image. The design draft image may be obtained from a database for storing historical design draft images, or may be a design draft image generated using a model, or may be provided by other users, or may be searched from other platforms, such as web pages. The method for obtaining the design draft image is not specifically limited here.

[0024] Step 104: Process the design draft image to obtain structured data corresponding to the design draft image.

[0025] The structured data at least includes the regional location information and component category information of each component in the design draft image.

[0026] In the embodiment of this specification, structured data can be a data format pre-set by the user based on actual business needs. The regional position of the component in the design draft can be expressed in a variety of formats, such as xyxy format, xywh format, cxcywh format, etc., or in other forms of regional position expression formats. The expression format of the regional position of the specific component can be determined based on the model to be trained. For example, the format output or used by the model to be trained is the xyxy format, and the regional positions of the components in the design draft picture can be set to the xyxy format. The xyxy format can represent a rectangular area, the first xy can represent the coordinates of the upper left corner of the rectangular area, and the second xy can represent the coordinates of the lower right corner of the rectangular area. If the format output or used by the model to be trained is the xywh format, the regional positions of the components in the design draft picture can be set to the xywh format, the x in xywh can represent the horizontal coordinate of the upper left corner of the rectangular area, y can represent the vertical coordinate of the upper left corner of the rectangular area, w can represent the width of the rectangle, and h can represent the height of the rectangle. The formats of the regional positions are no longer listed one by one here. It can be understood that no matter which format is used, xy refers to the coordinate position in the design draft image.

[0027] In the embodiments of this specification, the component category information may include at least one of the component categories such as graphic elements, icons, typesetting styles, buttons, input boxes, tabs, drop-down menus, pop-ups, lists, navigation bars, etc. When determining the component category information, you may also first determine the type of the design draft image, and determine the category information of each component in the design draft image based on the design draft component category corresponding to the design draft image type; you may also directly determine the category information of each component in the design draft image, which is not specifically limited here. In practical applications, structured data may also include information such as the relationship between each component, so that the components in the design draft image can be distinguished based on text data, avoiding the situation where components with high visual similarity cannot be accurately distinguished when visually distinguishing components in the design draft image.

[0028] Step 106: Obtain training data based on the design draft image and the structured data.

[0029] In the embodiments of this specification, the design draft image and the corresponding structured data can be directly spliced ​​to obtain training data. Specifically, the design draft image can be spliced ​​before the structured data, or after the structured data, which is not specifically limited here.

[0030] Step 108: Use the training data to train the initial model to obtain a trained component classification model; the trained component classification model is used to generate classification information of components in the design draft image.

[0031] In the embodiments of this specification, the initial model can be a pre-trained model, such as a Large Language and Vision Assistant (LLAVA) model, a YunLLM 70B model, a BERT model, a RoBERTa model, an XLNet model, an Electra model, etc., or a model with 7b pre-trained weights. The 7b pre-trained weights can represent a machine learning model that has been pre-trained with a large data set, especially a deep learning model, with a parameter volume of approximately 7 billion. The "b" here is an abbreviation for the English word "billion". Models of this scale belong to the category of large language models. It is understandable that the pre-training process of the pre-trained model refers to the use of a large amount of text data to train the model without specific task adjustment, in order to allow the model to learn general knowledge of the language, such as grammar, factual information, and contextual understanding. Such a model can be used as a basis, and then fine-tuned to adapt to specific tasks such as sentiment analysis, question-answering systems, or text generation.

[0032] In the embodiment of this specification, using training data to train the initial model can mean using training data related to the classification of design draft components to fine-tune the initial model so that the fine-tuned model can focus more on the classification of design draft components, capture the fine-grained differences between similar components, and improve the accuracy of classification.

[0033] In practical applications, training parameters can be configured for fine-tuning training, such as learning rate, batch size, number of sub-training rounds, total number of training rounds, and other information. If the number of sub-training rounds for fine-tuning training is multiple rounds, all training data can be grouped, such as by random grouping, equal grouping, etc.; the training data of each round after grouping can contain the same training data or different numbers of training data, which is not specifically limited here. If there are 2,000 training data and 10 rounds of training are required, they can be evenly divided into 200 training data per round; or 300, 400, etc. numbers of training data can be randomly allocated to each round. If the total number of training rounds for fine-tuning training is 3 times, the training data corresponding to the above sub-training rounds can be used to train the pre-trained model 3 times.

[0034] In practical applications, the initial model can also be a model with a structural framework but not pre-trained, or it can be trained with a large amount of training data related to the classification of design draft components to obtain a trained component classification model, so that the trained component classification model is more professional and can be more focused on classifying components in design draft images, thereby improving the accuracy and effectiveness of design draft component classification.

[0035] In practical applications, the trained component classification model can also be evaluated through the loss function. If the loss function meets the preset requirements, the trained component classification model can be put into use; if the loss function does not meet the preset requirements, the model can continue to be trained using the training data until the loss function of the trained component classification model meets the preset requirements.

[0036] In practical applications, in order to improve the accuracy of problem-solving model training, a large number of design draft images can be used to train the initial model to improve the classification accuracy of the trained model. For example, the number of design draft images can be 1000, 2000, 500, etc., and there is no specific limitation here.

[0037] In the embodiments of this specification, the design draft image is processed to obtain structured data containing component category information of each component in the design draft image, the design draft image and the structured data are combined into training data, and the initial model is trained using the training data to obtain a trained component classification model, thereby improving the recognition and classification capabilities of the trained component classification model and further improving the accuracy of the component category information generated using the trained component classification model.

[0038] based on Figure 1 The method, the examples of this specification also provide some specific implementation plans of the method, which are described below.

[0039] As an implementation mode, optionally, the processing of the design draft image in the embodiments of this specification to obtain structured data corresponding to the design draft image may specifically include: The design draft image is processed using the target detection model to obtain the regional position information of each component in the design draft image and the corresponding component category information; Determine the level relationship between the components in the design draft image according to the regional position information of the components in the design draft image; By using preset rules and based on the level relationship, the regional position information of the components in the design draft image and the corresponding component category information are structured to obtain the structured data.

[0040] In the embodiments of this specification, the design draft image can be input into the target detection model, so that the target detection model can output the regional location information and component category information of the preset format corresponding to each component in the design draft image. Before the design draft image is processed by the target detection model, a preset number of design draft images can be manually annotated to annotate the regional location information and component category information of the preset format that conforms to the input data format of the target detection model, so that the target detection model can be trained using the annotated data and the design draft image, thereby improving the accuracy of the regional location information and component category information of the design draft components output by the target detection model.

[0041] In the embodiments of this specification, the preset rules can represent the rules for generating structured data. For example, a line of text in the generated structured data can correspond to a component in the design draft component; the left part of the text in a line of text can be the component category information or name information of the component; the right part can be the coordinate box identification information and regional location information of the component; if the next line of text is indented by some characters relative to the previous line of text, it can be said that the component corresponding to the previous line of text includes the component corresponding to the next line of text, or the component corresponding to the next line of text is the next level component of the component corresponding to the previous line of text. It can be understood that the text of the components at the same level is left-aligned. The preset rules can be set based on actual needs, which are not listed one by one.

[0042] The structured data in the embodiments of this specification may be data in DSL format. In order to more clearly explain the structured data, Figure 2 A schematic diagram of structured data provided by an embodiment of this specification. Figure 2As shown in the figure, a line of text can correspond to a component in the design draft image. The characters on the left can represent the name or category of the component, such as Navigation bar for navigation bar, Button for button, Tag for tag, List for list, Horizontal Card for horizontal card, and Icon for icon. The right part is the area location information. <box> (262,684),(491,784)< / box> The box in the column is the coordinate box identification information, and the numbers in the column are the regional position coordinates of the component in the xyxy format. The component corresponding to the second line of text and the component corresponding to the third line of text can be the next level component of the component corresponding to the first line of text; the component corresponding to the fourth line of text can be the same level component as the component corresponding to the first line of text; the component corresponding to the last line of text can be the next level component of the component corresponding to the last line of text; Figure 2 It can be determined that the text corresponding to the same level component is in the left-aligned format, and the text corresponding to the next level component is in the format of indenting the preset characters to the right relative to the text corresponding to the previous level component. In actual applications, other formats of data can also be used, such as XML, JSON and other formats.

[0043] In practical applications, before using the target detection model to process the design draft image, the design draft image can also be normalized. Specifically, the width and height of the design draft image can be normalized. For example, the width and height of the image are image_w and image_h respectively. The coordinates of the region position of component 1 before normalization are x1, y1, x2, y2, and the coordinates after normalization are x1 / image_w, y1 / image_h, x2 / image_w, y2 / image_h. If the image size is 750x3506, the coordinates of the region position of a component before normalization are "(262,684), (491,784)", and after normalization, they can be "(0.349, 0.195), (0.654, 0.223)", thereby improving the data processing efficiency.

[0044] As an implementation manner, optionally, the design draft image in the embodiment of this specification includes at least a first component and a second component; the determining the level relationship between the components in the design draft image according to the regional position information of each component in the design draft image may specifically include: Acquire first area location information of the first component; Acquire second area location information of the second component; determining whether the first area location information includes the second area location information; If the first area location information includes the second area location information, the second component is determined as a next-level component of the first component.

[0045] In the embodiment of this specification, if the first area position information does not include the second area position information, the second component can be determined as the same level component of the first component. The first area position information of the first component is (x1, y1), (x2, y2); the area position of the second component can be (x3, y3), (x4, y4), if (x1, y1) is less than or equal to (x3, y3), (x2, y2) is greater than or equal to (x4, y4), the second component can be determined as the next level component of the first component. Considering that there may be errors in image segmentation, a certain error in the position information can be allowed. If it is within a certain error range and meets the above conditions, the first component can also be determined as the upper level component of the second component. For example, if the error is 5 mm, the first component is (0, 87), (750, 207), and the second component is (48, 88), (248, 208). Although 208 is greater than 207, it is within the error range. Therefore, the second component can also be determined as the next level component of the first component. That is to say, if within the preset error range, the first area position information of the first component includes the second area position information of the second component, the second component can also be determined as the next-level component of the first component; if within the preset error range, the first area position information of the first component still does not include the second area position information of the second component, the second component can be determined as the same-level component of the first component.

[0046] In actual applications, a structure tree can be constructed based on the containment relationship between components to reflect the containment relationship. The nodes in the structure tree can represent the design draft components, and the edges in the structure tree can represent the containment relationship between two design draft components. The containment relationship can reflect the local hierarchical relationship between two components in the design draft; the structure tree can reflect the overall hierarchical relationship between all elements in the design draft.

[0047] By using preset rules, the regional position information of the components in the design draft image and the corresponding component category information are structured based on the hierarchical relationship to obtain the structured data. Specifically, the structure tree can be traversed in the order from the root node to the leaf node or from the leaf node to the root node to obtain a structured text reflecting the hierarchical relationship between the various design draft components; wherein, if the first design draft component in each design draft is included in the second design draft component, then the starting position of the first structured text information for recording the first design draft component in the structured text is indented by a preset character relative to the starting position of the second structured text information for recording the second design draft component; in the structured text, the first structured text information and the second structured text information are located in different rows or columns.

[0048] In the embodiments of the present specification, for the constructed structure tree, the structure tree is traversed in a preset order, and the relevant information of each design draft component traversed is recorded in sequence, and the relevant information of each design draft component can be recorded in an independent row, and the relevant information of different design draft components can be recorded in different rows, so as to obtain a structured text for representing the design draft. The relevant information may include at least one of the location information, category information, text information and semantic information of the design draft component. The preset order may include: for the up and down direction of the structure tree, the order is from the root node to the leaf node of the structure tree, or from the leaf node to the root node of the structure tree; for the left and right direction of the structure tree, the direction is from left to right or from right to left.

[0049] In the embodiments of this specification, for two design draft components having an inclusion relationship, such as a first design draft component and a second design draft component, if the first design draft component is directly included in the second design draft component, the relevant information of the first design draft component is recorded in the next line of the line used to record the relevant information of the second design draft component, and the starting position of recording the first design draft component needs to be indented by a preset character relative to the starting position of recording the second design draft component. The indentation of the preset character can be achieved by adding a tab or a space at the starting position of recording the relevant information of the first design draft component.

[0050] It should be noted that, in the above-mentioned embodiment, if the second design draft component includes the third design draft component on the basis of including the first design draft component, and there is no inclusion relationship between the first design draft component and the third design draft component, the relevant information of the first design draft component and the relevant information of the third design draft component can be recorded in different rows, and the starting position of recording the first design draft component and the starting position of recording the third design draft component are aligned. If the relevant information of the first design draft component is recorded in the next row of the row used to record the relevant information of the second design draft component, the relevant information of the third design draft component is recorded in the next row of the row used to record the relevant information of the first design draft component.

[0051] It should be noted that the relevant information of each design draft component can also be recorded in an independent column, and the relevant information of different design draft components can be recorded in different columns. At this time, the recording method of using columns to record the relevant information of each design draft component can refer to the above-mentioned recording method of using rows to record the relevant information of each design draft component.

[0052] As an implementation mode, optionally, the method described in the embodiments of this specification may further include: Obtaining prompt word information from the prompt word set; According to the design draft image and the structured data, training data is obtained, including: Training data is obtained according to the design draft image, the structured data and the prompt word information.

[0053] In the embodiments of this specification, the prompt word set may include several different prompt word information. The prompt word information is used to prompt the model to complete the task, and one can be randomly selected from the prompt word set to form training data together with the design draft image and structured data. The training data can be several training data, and the several training data may include training data with different prompt words but the same design draft image and structured data; the several training data may also include training data with the same prompt words but different design draft images and structured data; the several training data may also include training data with different prompt words, different design draft images and structured data.

[0054] The training data can be obtained by splicing the design draft image, structured data and prompt word information. Specifically, it can be spliced ​​into <design draft image, structured data, prompt word information>; it can also be spliced ​​into <design draft image, prompt word information, structured data>; it can also be spliced ​​into <prompt word information, design draft image, structured data>; it can also be spliced ​​into combinations of other orders, which are not specifically limited here.

[0055] In the embodiments of this specification, the above method can be used to obtain several training data, so as to train the pre-trained model with the training data. In practical applications, the multimodal large model LLAVA large model can be used as the initial model, and the LLAVA large model can at least include three structures such as a visual encoder, a mapping layer, and a large language model. The training data, for example, <design draft picture, prompt word information, structured data>, can be input into the LLAVA large model. The model first uses the visual encoder to encode the design draft picture. During the encoding process, the picture needs to be compressed to a picture with a resolution of 224x224; the encoded information is input into the mapping layer, so that the visual features can be converted into language features; then the prompt word information is encoded, that is, the prompt word information is divided into smaller parts to obtain token information, and the token information and the language features output by the mapping layer are spliced ​​and input into the LLAVA large model together to obtain the final prediction label. All the training data can be used to train the LLAVA large model for 3 epochs, and the loss value is calculated according to the final prediction label and structured data. If the loss value meets the requirements, the final model can be obtained and the model can be put into use. The model can be maintained regularly, that is, the model in use can be trained periodically with training data to improve the robustness of the trained model and the accuracy of component classification.

[0056] As an implementation mode, optionally, the generation method of the prompt word set described in the embodiments of this specification can be as follows: Get sample prompt word information; The sample prompt word information is amplified using a natural language model to obtain a prompt word set.

[0057] In the embodiments of this specification, the sample prompt words may be prompt words used for examples. The natural language model may be a GPT-4 model, or other natural language models, such as a recurrent neural network, a GloVe model, a Sentence-BERT model, and the like. For example, the sample prompt word information may be "please use DSL language to describe this screenshot", and the natural language model may be a GPT-4 model, and "please use DSL language to describe this screenshot" may be transformed into "please use DSL language to describe this screenshot" and extended, and the transformed sentence may be used as the input of the GPT-4 model, so that the GPT-4 model may extend the sample prompt words, and may obtain prompt word information with different sentences but the same meaning, and store the obtained multiple prompt word information in a prompt word set, so that prompt words may be randomly selected from the prompt word set to train the large model, improve the generalization ability of the large model, and thus avoid using a single prompt word information to train the large model, so that when the prompt word information changes, the large model cannot understand and cause output errors.

[0058] In practical applications, multiple sample prompt word information can be obtained, and several different prompt word information can be generated for each sample prompt word information, so that the prompt word information can include as many expression forms as possible, thereby improving the generalization ability of the large model.

[0059] As an implementation mode, optionally, the method described in the embodiments of this specification may further include: Get the design draft pictures to be classified; Acquire target prompt word information for instructing the trained component classification model to process the design draft image to be classified; Inputting the design draft image to be classified and the target prompt word information into the trained component classification model, so that the trained component classification model performs recognition processing on the design draft image to be identified based on the target prompt word information to obtain the target structured data; The target structured data is parsed and processed to obtain component category information of each target component in the design draft image to be classified.

[0060] In the embodiments of this specification, the design draft images to be classified can be design draft components that the user needs to process, such as design drafts that need to be maintained, or design draft images for reference, etc. The target prompt word information can be determined by the user based on actual needs. Specifically, the design draft images to be classified and the target prompt word information can be input by the user into the big model. For example, if the user needs to classify a certain UI design draft image, the certain UI design draft image can be input into the big model, and "Please classify this UI design draft image" can be input into the big model as the target prompt word information.

[0061] In the embodiment of this specification, the trained component classification model performs recognition processing on the design draft image to be recognized based on the target prompt word information. Specifically, the large model converts the design image to be classified into text data, and segments the target prompt word information to obtain keywords, and processes the converted text data based on the keywords in the target prompt word information to obtain target structured data corresponding to the design image to be classified. In practical applications, the target structured data can be parsed based on the preset rules for generating structured data to obtain component category information of each target component.

[0062] As an implementation manner, optionally, the target structured data in the embodiment of this specification includes multiple structured texts; the method may also include: For any structured text among the multiple structured texts, determining whether any structured text among the multiple structured texts meets a preset condition; If any structured text in the plurality of structured texts does not meet the preset condition, any structured text in the plurality of structured texts is deleted to obtain modified structured data; The parsing process of the target structured data may specifically include: The modified structured data is parsed.

[0063] In the embodiment of this specification, if any of the multiple structured texts meets the preset conditions, any of the multiple structured texts can be retained. In practical applications, the corrected structured data can include data of all components in the design draft image to be classified; it can also include data of some components in the design draft image to be classified.

[0064] In the embodiments of this specification, the judgment of whether any of the multiple structured texts meets the preset conditions can be made during the process of generating the data structure, such as once for each structured text generated; or after all the data structures are generated, such as judging each structured text in the order displayed in the structured data; or judging all the structured texts at the same time; or judging a preset number of structured texts at the same time. The specific judgment method can be set by the user based on actual needs and the functions of the large model, and is not specifically limited here.

[0065] As an implementation manner, optionally, the parsing and processing of the target structured data in the embodiments of this specification may specifically include: For any structured text in the target structured data, extract component name information contained in any structured text in the target structured data; The component name information is determined as component category information of a component targeted by any structured text in the target structured data.

[0066] In the embodiments of the present specification, the structured text that has been parsed in the target structured data can be marked, such as adding a first identifier to indicate that the parsing has been completed; or the structured text that has not been parsed in the target structured data may contain a second identifier to indicate that the parsing has not been completed, and after the structured text is parsed, the second identifier in the structured text is deleted.

[0067] In the embodiments of the present specification, an association relationship between component name information and component category information can be established, and then the component category information corresponding to the component name information can be determined through the association relationship; keyword information in the component name information can also be searched, and the component category information can be determined based on the keyword information. For example, if the component name information is a yellow button, a keyword button can be found, and the component corresponding to the component name information can be classified into the button component category.

[0068] In practical applications, it is also possible to extract the coordinate box identification information of the component targeted by any structured text in the target structured data, and determine the coordinate information between the two coordinate box identification information as the regional position information, and then determine the rectangular area of ​​the component in the design draft image to be classified according to the regional position information. The coordinate box identification information is "box", and there is a "box" as the first coordinate box identification information before the coordinate information, and there is a "box" as the second coordinate box identification information after the coordinate information, so as to identify the coordinate information based on the "box" to avoid confusion with the garbled code generated by the large model. In practical applications, after the analysis of the design draft image to be classified is completed, it can be stored in the form of tree nodes based on the hierarchical relationship of each component in the design draft.

[0069] As an implementation manner, optionally, any structured text among the multiple structured texts described in the embodiments of this specification that does not meet the preset condition may specifically include at least one of the following situations: Any structured text in the plurality of structured texts does not include component name information of the component targeted by any structured text in the plurality of structured texts; Any structured text among the multiple structured texts does not contain the coordinate frame identification information of the component targeted by any structured text among the multiple structured texts; Any structured text among the multiple structured texts does not contain the regional location information of the component targeted by any structured text among the multiple structured texts; The multiple structured texts contain any structured text among the multiple structured texts that is repeated; The regional position information of the component targeted by any structured text among the multiple structured texts exceeds the design draft image to be classified.

[0070] In the embodiment of the present specification, if a structured text does not contain component name information, the component category of the component targeted by the structured text cannot be parsed, and the structured text that does not contain component name information can be deleted.

[0071] A structured text may contain two coordinate frame identification information. If either one of them is not contained, in order to avoid the parsed coordinate information being inconsistent with the rectangular area of ​​the component in the design draft image to be classified, the structured text that does not contain either one of the coordinate frame identification information may be deleted.

[0072] If the multiple structured texts contain at least two identical structured texts, one of them may be retained and the other duplicate structured texts may be deleted.

[0073] If the rectangular area corresponding to the area position information in the structured text is outside the design draft image to be classified, the structured text with the area position information outside the design draft image can be deleted.

[0074] In practical applications, structured text that does not meet the preset conditions can be deleted during the process of parsing structured data; structured text that does not meet the preset conditions can be deleted before parsing structured data and after generating structured data; structured text that does not meet the preset conditions can be deleted during the process of generating structured data. There is no specific limitation on the timing of deleting structured text that does not meet the preset conditions.

[0075] In the embodiments of this specification, the large model can be trained through the design draft images and the corresponding structured data to improve the accuracy of the trained component classification model in classifying the design draft components, and a variety of different training data can be used for training. While improving the professionalism of the trained component classification model, it can also improve the generalization ability of the trained component classification model.

[0076] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for a model for classifying design draft components. Figure 3 FIG. 1 is a schematic diagram showing a structure of a training device for a model for classifying design draft components provided by an embodiment of the present specification. Figure 3 As shown, the device comprises: The picture acquisition module 302 is configured to acquire a design draft picture; The image processing module 304 is configured to process the design draft image to obtain structured data corresponding to the design draft image; the structured data at least includes the regional location information and component category information of each component in the design draft image; The training data obtaining module 306 is configured to obtain training data according to the design draft image and the structured data; The model training module 308 is configured to train the initial model using the training data to obtain a trained component classification model; the trained component classification model is used to generate classification information of components in the design draft image.

[0077] Optionally, the image processing module may be specifically configured as follows: The image processing unit is configured to process the design draft image using the target detection model to obtain the regional position information of each component in the design draft image and the corresponding component category information; A level relationship determination unit, configured to determine the level relationship between the components in the design draft image according to the regional position information of the components in the design draft image; The structured processing unit is configured to perform structured processing on the regional position information of the components in the design draft image and the corresponding component category information based on the level relationship using preset rules to obtain the structured data.

[0078] Optionally, the design draft image includes at least a first component and a second component; the level relationship determination unit may be specifically configured as follows: Acquire first area location information of the first component; Acquire second area location information of the second component; determining whether the first area location information includes the second area location information; If the first area location information includes the second area location information, the second component is determined as a next-level component of the first component.

[0079] Optionally, the structured data is data in DSL format.

[0080] Optionally, the device may also be configured as: Obtaining prompt word information from the prompt word set; According to the design draft image and the structured data, training data is obtained, including: Training data is obtained according to the design draft image, the structured data and the prompt word information.

[0081] Optionally, the prompt word set is generated as follows: Get sample prompt word information; The sample prompt word information is amplified using a natural language model to obtain a prompt word set.

[0082] Optionally, the device may also be configured as: Get the design draft pictures to be classified; Acquire target prompt word information for instructing the trained component classification model to process the design draft image to be classified; Inputting the design draft image to be classified and the target prompt word information into the trained component classification model, so that the trained component classification model performs recognition processing on the design draft image to be identified based on the target prompt word information to obtain the target structured data; The target structured data is parsed and processed to obtain component category information of each target component in the design draft image to be classified.

[0083] Optionally, the target structured data includes a plurality of structured texts; the device may also be configured as follows: For any structured text among the multiple structured texts, determining whether any structured text among the multiple structured texts meets a preset condition; If any structured text in the plurality of structured texts does not meet the preset condition, any structured text in the plurality of structured texts is deleted to obtain modified structured data; The parsing process of the target structured data specifically includes: The modified structured data is parsed.

[0084] Optionally, the device may also be configured as: For any structured text in the target structured data, extract component name information contained in any structured text in the target structured data; The component name information is determined as component category information of a component targeted by any structured text in the target structured data.

[0085] Optionally, any structured text among the plurality of structured texts does not meet the preset condition, specifically including at least one of the following situations: Any structured text in the plurality of structured texts does not include component name information of the component targeted by any structured text in the plurality of structured texts; Any structured text among the multiple structured texts does not contain the coordinate frame identification information of the component targeted by any structured text among the multiple structured texts; Any structured text among the multiple structured texts does not contain the regional location information of the component targeted by any structured text among the multiple structured texts; The multiple structured texts contain any structured text among the multiple structured texts that is repeated; The regional position information of the component targeted by any structured text among the multiple structured texts exceeds the design draft image to be classified.

[0086] The above is a schematic scheme of a training device for a model for classifying design draft components in this embodiment. It should be noted that the technical scheme of the training device for a model for classifying design draft components and the technical scheme of the training method for a model for classifying design draft components mentioned above belong to the same concept. The details of the technical scheme of the training device for a model for classifying design draft components that are not described in detail can all be found in the description of the technical scheme of the training method for a model for classifying design draft components mentioned above.

[0087] Figure 4 The block diagram of a computing device provided by one embodiment of the present specification is shown. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0088] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0089] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0090] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 may also be a mobile or stationary server.

[0091] Among them, the processor 420 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned training method for the model for classifying design draft components.

[0092] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the training method of the model for classifying the design draft components are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the training method of the model for classifying the design draft components.

[0093] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned training method for the model for classifying design draft components.

[0094] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the training method of the model for classifying the design draft components are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the training method of the model for classifying the design draft components.

[0095] An embodiment of the present specification also provides a computer program product, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned training method for the model for classifying design draft components.

[0096] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the training method of the model for classifying design draft components are of the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the training method of the model for classifying design draft components.

[0097] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0099] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0100] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A training method for a model for classifying design draft components, characterized in that: include: Get the design draft picture; Processing the design draft image to obtain structured data corresponding to the design draft image; The structured data at least includes the regional location information and component category information of each component in the design draft image; Obtaining training data according to the design draft image and the structured data; The training data is used to train the initial model to obtain a trained component classification model; the trained component classification model is used to generate classification information of components in the design draft image.

2. The method according to claim 1, characterized in that The processing of the design draft image to obtain structured data corresponding to the design draft image specifically includes: The design draft image is processed using the target detection model to obtain the regional position information of each component in the design draft image and the corresponding component category information; Determine the level relationship between the components in the design draft image according to the regional position information of the components in the design draft image; By using preset rules and based on the level relationship, the regional position information of the components in the design draft image and the corresponding component category information are structured to obtain the structured data.

3. The method according to claim 2, characterized in that The design draft image includes at least a first component and a second component; and determining the level relationship between the components in the design draft image according to the regional position information of each component in the design draft image specifically includes: Acquire first area location information of the first component; Acquire second area location information of the second component; determining whether the first area location information includes the second area location information; If the first area location information includes the second area location information, the second component is determined as a next-level component of the first component.

4. The method according to claim 1, characterized in that: The structured data is data in DSL format.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining prompt word information from the prompt word set; According to the design draft image and the structured data, training data is obtained, including: Training data is obtained according to the design draft image, the structured data and the prompt word information.

6. The method according to claim 5, characterized in that The generation method of the prompt word set is as follows: Get sample prompt word information; The sample prompt word information is amplified using a natural language model to obtain a prompt word set.

7. The method according to claim 1, characterized in that The method further comprises: Get the design draft pictures to be classified; Acquire target prompt word information for instructing the trained component classification model to process the design draft image to be classified; Inputting the design draft image to be classified and the target prompt word information into the trained component classification model, so that the trained component classification model performs recognition processing on the design draft image to be identified based on the target prompt word information to obtain the target structured data; The target structured data is parsed and processed to obtain component category information of each target component in the design draft image to be classified.

8. The method according to claim 7, characterized in that The target structured data includes a plurality of structured texts; the method further includes: For any structured text among the multiple structured texts, determining whether any structured text among the multiple structured texts meets a preset condition; If any structured text in the plurality of structured texts does not meet the preset condition, any structured text in the plurality of structured texts is deleted to obtain modified structured data; The parsing process of the target structured data specifically includes: The modified structured data is parsed.

9. The method according to claim 7, characterized in that: The parsing process of the target structured data specifically includes: For any structured text in the target structured data, extract component name information contained in any structured text in the target structured data; The component name information is determined as component category information of a component targeted by any structured text in the target structured data.

10. The method according to claim 7, characterized in that Any structured text among the plurality of structured texts does not meet the preset condition, specifically including at least one of the following situations: Any structured text in the plurality of structured texts does not include component name information of the component targeted by any structured text in the plurality of structured texts; Any structured text among the multiple structured texts does not contain the coordinate frame identification information of the component targeted by any structured text among the multiple structured texts; Any structured text among the multiple structured texts does not contain the regional location information of the component targeted by any structured text among the multiple structured texts; The multiple structured texts contain any structured text among the multiple structured texts that is repeated; The regional position information of the component targeted by any structured text among the multiple structured texts exceeds the design draft image to be classified.

11. A training device for a model for classifying design draft components, characterized in that: include: The picture acquisition module is configured to acquire the design draft picture; An image processing module is configured to process the design draft image to obtain structured data corresponding to the design draft image; The structured data at least includes the regional location information and component category information of each component in the design draft image; A training data acquisition module is configured to obtain training data according to the design draft image and the structured data; The model training module is configured to train the initial model using the training data to obtain a trained component classification model; the trained component classification model is used to generate classification information of components in the design draft image.

12. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the training method for a model for classifying design draft components described in any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the training method of a model for classifying design draft components as described in any one of claims 1 to 10.

14. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the training method of a model for classifying design draft components as described in any one of claims 1 to 10.

Images