Method and device for automatically generating program code from design drawing, medium and product
By performing deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings, multimodal features are generated and decoded into program code, solving the problems of poor code generation quality and low efficiency in existing technologies, and realizing automated and efficient code generation.
Patent Information
- Application Number
- CN202411415781.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-11-29
AI Technical Summary
Existing technologies for converting design drawings into HTML code suffer from poor code generation quality, low efficiency, and high levels of human intervention, especially when dealing with complex and diverse design drawings.
By performing deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings, semantic features, layout features, and visual features are obtained. After preprocessing, these features are fused into multimodal features and then converted into preset vectors for decoding to generate a sequence of program code.
It enables automated and rapid generation of high-quality program code, reduces the need for manual coding, improves code accuracy and consistency, shortens the development cycle, and avoids human error.
Smart Images

Figure CN119045821A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, medium, and product for automatically generating program codes from design drawings. Background Art
[0002] With the continuous development of Web technology, Web front-end development has become increasingly important in various industries. In Web front-end development, design drawings are a key part of the development process. Design drawings are usually created by designers during the design phase and then handed over to front-end developers for implementation. However, when converting design drawings into actual HTML code, traditional methods often require a lot of manual work, which is not only time-consuming and labor-intensive, but also prone to errors.
[0003] Currently, there are some tools and plug-ins designed to simplify the front-end development process, some of which can automatically generate basic HTML structures based on design drawings. However, these existing technologies are often limited to fixed templates or rules and cannot flexibly adapt to the complexity and diversity of various design drawings. In addition, these tools often cannot effectively handle design drawings with a large number of layers and complex layouts, resulting in poor quality of generated HTML code, which requires subsequent manual adjustment and correction. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide a method, device, medium, and product for automatically generating program code from a design drawing, which can solve the problems of poor code generation quality, low efficiency, and high manual participation in the prior art.
[0005] In a first aspect, an embodiment of the present disclosure provides a method for automatically generating program codes from a design drawing, the method specifically comprising:
[0006] Perform deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings to obtain semantic features, layout features, and visual features;
[0007] Preprocessing the semantic features, the layout features, and the visual features, and fusing the preprocessed semantic features, the layout features, and the visual features to obtain multimodal features;
[0008] Convert the multimodal features into a preset vector, decode the preset vector, and obtain program YY+242206P
[0009] Code sequence.
[0010] Optionally, the step of performing deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawing to obtain semantic features, layout features, and visual features includes:
[0011] Performing deep semantic segmentation on the design drawing to obtain all interface elements;
[0012] Performing semantic conversion on the interface elements to obtain semantic features;
[0013] Performing layout structural analysis on the design drawing to obtain all layout elements and structural relationships;
[0014] The structural relationship includes the spatial relationship of elements, the hierarchical relationship between elements, and the alignment logic information;
[0015] Based on the layout elements and the structural relationships, obtaining layout features;
[0016] Extracting all visual elements of the design;
[0017] Features of the visual elements at different scales are captured to obtain visual features.
[0018] Optionally, the method for obtaining the semantic feature includes:
[0019] Extracting an image of the design drawing based on the first preset model, and classifying each pixel in the image to obtain a corresponding category;
[0020] Based on the preset semantic information library and all the categories, determine all the interface elements;
[0021] Convert all of the interface elements into corresponding semantic features based on a second preset model;
[0022] The interface elements include one or more of buttons, text boxes, pictures, navigation bars, input boxes, drop-down menus, check boxes, sliders, switches, tabs, lists, grids, modal windows, tooltips, icons, dividing lines, progress bars, warning boxes, forms, search boxes, menus, and scroll bars.
[0023] Optionally, performing image extraction on the design drawing based on the first preset model and classifying each pixel in the image to obtain a corresponding category includes:
[0024] Determine a first initial model; the first initial model includes one or more of a convolutional neural network model, a full convolutional network model, and a U-Net model;
[0025] Training the first initial model based on the first historical database, and using the trained first initial model as the first preset model;
[0026] The first historical database includes pixel information of preset types and corresponding category information;
[0027] Based on the first preset model, the design drawing is gradually abstracted from low-level features to high-level features, each pixel is identified and classified, and a category corresponding to each pixel is obtained;
[0028] The low-level features include one or more of edges and textures, and the high-level features include one or more of shapes and objects.
[0029] Optionally, based on the second preset model, all the interface elements are converted into corresponding semantics YY+242206P
[0030] Features include:
[0031] Determine a second initial model; the second initial model includes one or more of a Word2Vec model or a BERT model, a GloVe model, a FastText model, and an ELMo model;
[0032] Training the second initial model based on the second historical database, and using the trained second initial model as the second preset model;
[0033] The second history database includes phonetic relationship information between different words;
[0034] Extracting the category and related attributes of each of the interface elements to obtain high-dimensional semantic information;
[0035] The attributes include one or more of text content and text status;
[0036] Converting the high-dimensional semantic information into a low-dimensional embedding vector based on the second preset model;
[0037] Perform a nonlinear transformation on the embedding vector to obtain the corresponding semantic feature.
[0038] Optionally, the method for obtaining the layout feature includes:
[0039] Determine a third initial model; the third initial model includes one or more of a graph neural network model, a graph convolutional network model, and an image-to-image conversion model;
[0040] Training the third initial model based on the third historical data, and using the trained third initial model as the third preset model;
[0041] The third historical data includes layout structure information of different design drawings;
[0042] Performing a high-level layout structure analysis on the design drawing based on the third preset model and the target detection algorithm to obtain all layout elements and structural relationships;
[0043] The target detection algorithm includes one or more of the YOLO algorithm, the Faster R-CNN algorithm, and the SSD algorithm;
[0044] Analyze the layout elements and the structural relationships based on a graph convolution operation to obtain layout features;
[0045] The layout structure includes one or more of a grid layout, a Flexbox layout, and absolute positioning;
[0046] The layout elements include one or more of a grid system, a Flexbox layout, absolute positioning, floating, a multi-column layout, a viewport unit, a CSS custom attribute, a CSS shape, and an asynchronous layout.
[0047] Optionally, the method for obtaining the visual features includes:
[0048] Determine a fourth preset model; the fourth preset model includes one or more of a ResNet model, an EfficientNet model, a VGGNet model, an Inception model, a DenseNet model, and a MobileNet model;
[0049] Training the fourth initial model based on fourth historical data, and using the trained fourth initial model as a fourth preset model;
[0050] The fourth historical data includes style information of different design drawings;
[0051] Identify edges and contours in the design drawing based on a fourth preset model, and determine corresponding visual elements based on the edges and contours;
[0052] Performing changes to the visual elements at different scales to capture the features of the visual elements at different scales;
[0053] Detecting key points in the visual element and generating a descriptor for each key point;
[0054] Obtaining the visual feature based on the features and descriptors in the visual element;
[0055] The visual features include one or more of color, font, size, margin between elements, shadow, element padding, border, and transparency.
[0056] Optionally, the method for obtaining the multimodal features includes:
[0057] Dynamically assign feature weights based on task requirements and feature importance;
[0058] Aligning the semantic features, the layout features, and the visual features, converting them into the same feature space, and obtaining a first feature vector, a second feature vector, and a third feature vector;
[0059] Performing weighted summation on the first feature vector, the second feature vector, and the third feature vector based on the feature weight to obtain a comprehensive feature vector;
[0060] The comprehensive feature vector is subjected to multi-layer nonlinear transformation decoding to obtain multi-modal features.
[0061] Optionally, the method further includes: scanning the program code sequence to obtain abnormal information, and correcting the abnormal information to obtain corrected code information;
[0062] The revised code information is semantically analyzed to identify the logical structure in the code, and the label selection of the code is optimized based on a preset code library to obtain optimized revised code information.
[0063] In a second aspect, the present application discloses a method for automatically generating program code from a design drawing, comprising:
[0064] Configure a plug-in architecture that is seamlessly integrated with various UI design software;
[0065] The preset window based on the plug-in architecture receives the design drawings uploaded by the user;
[0066] Perform deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings to obtain semantic features, layout features, and visual features;
[0067] Preprocessing the semantic features, the layout features, and the visual features, and fusing the preprocessed semantic features, the layout features, and the visual features to obtain multimodal features;
[0068] The multimodal features are converted into preset vectors, and the preset vectors are decoded to obtain a program code sequence.
[0069] In a third aspect, the embodiments of the present disclosure further provide a system for automatically generating program codes from design drawings, including:
[0070] The analysis module is used to perform deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings to obtain semantic features, layout features, and visual features;
[0071] A processing module, used for preprocessing the semantic features, the layout features, and the visual features, and fusing the preprocessed semantic features, the layout features, and the visual features to obtain multimodal features;
[0072] The code acquisition module is used to convert the multimodal features into preset vectors, decode the preset vectors, and obtain a program code sequence.
[0073] In a fourth aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical solution:
[0074] The computer device comprises:
[0075] at least one processor; and,
[0076] a memory communicatively connected to the at least one processor; wherein,
[0077] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned methods for automatically generating program codes from design drawings.
[0078] In a fifth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned methods for automatically generating program codes from design drawings.
[0079] In a sixth aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.
[0080] The method for automatically generating program code from a design drawing disclosed in the present application first performs deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawing to obtain semantic features, layout features, and visual features. Different types of features can be extracted from multiple angles to obtain complete information of the design drawing to ensure the comprehensiveness and accuracy of the generated code; then the semantic features, layout features, and visual features are preprocessed, and the preprocessed semantic features, layout features, and visual features are processed to obtain multimodal features. The combination of preprocessing and feature processing can improve the efficiency of feature extraction and processing, reduce redundant information, and optimize computing resources; finally, the multimodal features are converted into preset vectors, and the preset vectors are decoded to obtain a program code sequence. By converting into preset vectors, a large amount of information can be effectively condensed into a vector of fixed dimension, which can reduce the computational complexity and storage requirements. By decoding the preset vectors, the complex features of the design drawing can be accurately converted into the corresponding program code, ensuring that the code generation meets the design intent and requirements, and realizing automatic code generation, that is, realizing the automatic generation of program code from the design drawing, effectively reducing the need for manual code writing, saving time and labor, and at the same time, the automatically generated code can ensure consistency, avoid human errors, and has high accuracy and strong practicality.
[0081] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0083] Figure 1 A flowchart of a method for automatically generating program codes from a design drawing provided in an embodiment of the present disclosure.
[0084] Figure 2 A schematic flow chart of a method for analyzing a design drawing provided in an embodiment of the present disclosure.
[0085] Figure 3 A flowchart of a method for obtaining semantic features provided in an embodiment of the present disclosure.
[0086] Figure 4 A flowchart of a method for obtaining a category provided in an embodiment of the present disclosure.
[0087] Figure 5 A flowchart of a method for converting interface elements provided in an embodiment of the present disclosure.
[0088] Figure 6 A schematic flow chart of a method for obtaining layout features provided in an embodiment of the present disclosure.
[0089] Figure 7 A flowchart of a method for obtaining visual features provided in an embodiment of the present disclosure.
[0090] Figure 8 A flowchart of a method for obtaining multimodal features provided in an embodiment of the present disclosure.
[0091] Fig. 9 A flowchart diagram of another embodiment of a method for automatically generating program codes from a design drawing provided in an embodiment of the present disclosure.
[0092] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0093] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0094] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0095] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0096] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0097] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0098] Reference Figure 1 In a first embodiment, the present application discloses a method for automatically generating program codes from a design drawing, the method specifically comprising the following steps:
[0099] S100, deep semantic segmentation, layout structure analysis, and visual information extraction are performed on the design drawing to obtain semantic features, layout features, and visual features.
[0100] The design drawings are in compressed form; specifically, an efficient image compression algorithm can be used to compress the design drawings to reduce the image file size, which can not only reduce the system reasoning cost and storage cost, but also improve the response speed of the model. The compressed image file (i.e., the design drawing) can be transmitted to the cloud storage system through a high-speed network to ensure the integrity and security of data transmission, while providing a high-quality data set for subsequent model training.
[0101] In this step, the design drawings can first be subjected to deep semantic segmentation to obtain semantic features, then the layout structure analysis of the design drawings can be performed to obtain layout features, and finally the visual information extraction of the design drawings can be performed to obtain visual features. These three steps can be performed simultaneously or one by one, and this application does not impose any restrictions on this.
[0102] In this step, through deep semantic segmentation, different components in the design drawing can be accurately distinguished to help understand the function and properties of each component; through layout structure analysis, layout information can be extracted to identify the relationship between components and provide structured information for subsequent code generation; through visual information extraction, details in the design drawing can be captured to help generate program code that better meets the design requirements.
[0103] S200, preprocessing the semantic features, layout features, and visual features, and processing the preprocessed semantic features, layout features, and visual features to obtain multimodal features.
[0104] In this step, the multimodal features are obtained as a fused comprehensive feature.
[0105] In this step, preprocessing can improve the processing efficiency of features and reduce redundant information, so that features can be better used for subsequent model training and prediction; through feature processing, that is, fusing different types of features through multimodal learning methods, the meaning of the design drawing can be more fully understood, thereby generating more accurate program code.
[0106] S300, converting the multimodal features into preset vectors, decoding the preset vectors, and obtaining program code sequences.
[0107] Specifically, the fused multimodal features can be converted into a preset vector through an encoder (such as a neural network encoder), that is, an encoder-decoder architecture can be used to encode the fused multimodal features into a vector of a fixed length to represent the high-level features of the design drawing.
[0108] A decoder (such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer decoder) can then be used to convert the preset vector into a program code sequence, and the generated code sequence can be formatted and optimized according to the preset programming language and framework.
[0109] In this step, the program code sequence refers to the HTML code.
[0110] Specifically, the powerful sequence modeling capability of the Transformer model can be used as a decoder, and an autoregressive mechanism can be introduced to generate HTML code tokens one by one while referring to the generated code context information.
[0111] In this step, high-dimensional, multimodal features are compressed into vectors of fixed dimensions, which helps reduce computational complexity and improve the accuracy of generated code. By converting vectors into code through a decoder, structured code that meets design requirements can be generated, facilitating subsequent development and maintenance by developers.
[0112] The method for automatically generating program code from a design drawing disclosed in the present application first performs deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawing to obtain semantic features, layout features, and visual features. Different types of features can be extracted from multiple angles to obtain complete information of the design drawing to ensure the comprehensiveness and accuracy of the generated code; then the semantic features, layout features, and visual features are preprocessed, and the preprocessed semantic features, layout features, and visual features are processed to obtain multimodal features. The combination of preprocessing and feature processing can improve the efficiency of feature extraction and processing, reduce redundant information, and optimize computing resources; finally, the multimodal features are processed. Convert it into a preset vector, decode the preset vector, and obtain a program code sequence. By converting it into a preset vector, a large amount of information can be effectively condensed into a vector of fixed dimension, which can reduce the computational complexity and storage requirements. By decoding the preset vector, the complex features of the design drawing can be accurately converted into the corresponding program code, ensuring that the code generation meets the design intent and requirements, and realizing automatic code generation, that is, automatically and quickly generating program code from the design drawing, effectively reducing the need for manual code writing, shortening the development cycle, saving time and labor, and at the same time, the automatically generated code can ensure consistency, avoid human errors, have high accuracy, and strong practicality.
[0113] Furthermore, users can choose to generate HTML code for the entire page (i.e., the overall design) or a specific area (i.e., a local design) as needed, which improves the flexibility and applicability of the system. At the same time, the method disclosed in this application is not limited to and does not rely on any software, and only requires a picture to generate the corresponding HTML code.
[0114] Reference Figure 2 The method for analyzing the design drawing specifically includes the following steps, namely, performing deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawing to obtain semantic features, layout features, and visual features, specifically including the following steps:
[0115] S110, performing deep semantic segmentation on the design drawing to obtain all interface elements.
[0116] Through this step, all interface elements in the design drawing can be accurately identified. The automated segmentation process reduces the need for manual labeling and improves efficiency.
[0117] S120, performing semantic conversion on the interface elements to obtain semantic features.
[0118] In this step, the interface elements are converted into standardized semantic features, making subsequent processing more consistent and accurate, and making the meaning and function of the interface elements easier to be understood and utilized by the system.
[0119] S130, performing layout structure analysis on the design drawing to obtain all layout elements and structural relationships.
[0120] The structural relationship preferably includes the spatial relationship of elements, the hierarchical relationship between elements, and the alignment logic information.
[0121] In this step, an organizational structure of the elements in the design drawing is provided, which helps to generate program code that meets the design intent and enables the generated code to accurately reflect the layout and structure of the design drawing.
[0122] S140, obtaining layout features based on layout elements and structural relationships.
[0123] Specifically, the types and structural relationships of layout elements may be comprehensively considered, and layout features or layout feature descriptions (such as CSS styles and parameters of a layout framework) may be generated using rules or models.
[0124] S150, extracting all visual elements in the design drawing.
[0125] In this embodiment, the visual element refers to style information; in this step, the visual details in the design drawing can be accurately extracted and described, so that the generated code can better restore the visual effect of the design drawing and improve the visual consistency of the automatically generated code.
[0126] S160, capturing features of visual elements at different scales to obtain visual features.
[0127] In this step, the details of visual elements at different sizes can be captured to ensure that the generated code can maintain good display effects on different devices and screen sizes, which can improve the adaptability and responsiveness of the code.
[0128] In addition, visual features can also be obtained by extracting all visual elements in interface elements and layout elements, and then capturing features of the visual elements at different scales to obtain visual features.
[0129] In the method disclosed in this embodiment, deep semantic segmentation and layout structure analysis ensure that all interface elements and layout structures in the design drawing are accurately restored in the generated code; semantic conversion and layout feature extraction can accurately map the design intent and reduce the deviation between design and implementation; through multi-scale feature capture, the generated code can adapt to different screen sizes and devices, improving the user experience; at the same time, the automated processing and generation process saves time and resources, making the design and development process more efficient.
[0130] Reference Figure 3 , the method for obtaining semantic features specifically includes the following steps:
[0131] A100, extract the image of the design drawing based on the first preset model, and classify each pixel in the image to obtain a corresponding category.
[0132] In this step, each pixel in the image is classified, that is, pixel-by-pixel classification is performed, so as to accurately identify various interface elements in the design drawing, effectively reduce the workload of manual labeling and classification, and improve processing efficiency. This is different from traditional image classification, which only gives a category label to the entire image, while semantic segmentation classifies each pixel.
[0133] A200,determines all interface elements based on the preset semantic information base,all categories.
[0134] Specifically, a semantic information library containing all possible interface elements and their attributes can be built, which includes the definitions and characteristics of elements such as buttons, text boxes, pictures, and navigation bars; then each pixel category obtained from A100 is matched with the elements in the semantic information library to determine the specific interface element corresponding to each area; based on the matching results, a detailed list of all interface elements in the design drawing is generated, including information such as their position, size, and type.
[0135] The preset semantic information library ensures accurate identification and classification of each interface element, while covering the definitions of various interface elements so that all common interface elements can be identified and processed.
[0136] In this embodiment, the interface elements preferably include one or more of buttons, text boxes, pictures, navigation bars, input boxes, drop-down menus, check boxes, sliders, switches, tabs, lists, grids, modal windows, tooltips, icons, dividing lines, progress bars, warning boxes, forms, search boxes, menus, and scroll bars.
[0137] A300 converts all interface elements into corresponding semantic features based on the second preset model.
[0138] The method for obtaining semantic features disclosed in this embodiment can provide accurate classification at the pixel level, ensuring that each interface element in the design drawing is accurately identified. At the same time, it can accurately convert the interface elements into standardized semantic features, reduce errors in the conversion process, and increase automation efficiency. The entire process has a high degree of automation, effectively reducing dependence on manual labeling and classification, and improving processing efficiency. The automated process can quickly process a large number of design drawings and is suitable for large-scale design and development projects.
[0139] Through these steps, the solution can efficiently and accurately extract and transform semantic features from design drawings, improve the consistency of design and implementation and code quality, and ultimately achieve fast and high-quality development and maintenance.
[0140] The method further includes: obtaining the position and size of each interface element based on the interface element and the extracted image.
[0141] Reference Figure 4 The method for obtaining the category specifically includes the following steps, namely, extracting the image of the design drawing based on the first preset model, and classifying each pixel in the image to obtain the corresponding category, specifically including the following steps:
[0142] A110, determining a first initial model.
[0143] Among them, the first initial model preferably includes one or more of a convolutional neural network model (such as VGG, ResNet), a full convolutional network model, and a U-Net model.
[0144] In this step, the selected model architecture is obtained, and the corresponding computing resources and software environment are prepared to support model training and reasoning. At the same time, it has strong flexibility and can select a suitable model according to specific needs and resource conditions. The use of existing mature model architectures can accelerate the development and training process.
[0145] A120, training a first initial model based on the first historical database, and using the trained first initial model as a first preset model;
[0146] The first history database includes pixel information of preset types and corresponding category information.
[0147] The first historical database contains a large amount of labeled pixel information and corresponding category information, which is used to train the model to identify different types of interface elements. The first initial model is trained using the data in the first historical database, and the parameters of the model are iteratively optimized so that it can accurately predict the corresponding category from the pixel information; during the training process, the model is evaluated using the validation set to ensure that its performance and accuracy meet expectations.
[0148] By training with a large amount of labeled data, the model can learn more accurate classification boundaries and feature representations. The trained model can better generalize to unseen data and improve classification accuracy.
[0149] A130, based on the first preset model, gradually abstracts the design drawing from low-level features to high-level features, identifies and classifies each pixel, and obtains the category corresponding to each pixel.
[0150] Among them, low-level features include one or more of edges and textures, and high-level features include one or more of shapes and objects.
[0151] In this step, the design drawing is processed using the trained first preset model. The model first extracts low-level features (such as edges and textures), and then gradually abstracts high-level features (such as shapes and objects). Based on the extracted features, the model classifies each pixel and determines the category of the interface element to which it belongs. The model finally outputs the category information of each pixel to form a category map or category matrix. Through gradual abstraction from low to high levels, the model can more finely identify different types of interface elements and classify each pixel, ensuring that all elements in the design drawing can be accurately identified.
[0152] The method for obtaining categories disclosed in this embodiment can accurately identify various interface elements in the design drawing through the training of the deep learning model and pixel-level classification, and provide accurate basic data for subsequent semantic feature extraction and code generation; the entire process is highly automated, reducing the workload of manual intervention and annotation, and improving processing efficiency. At the same time, the trained model can quickly process new design drawings, further improving efficiency; through the training of a large amount of data, the model has good generalization ability and can process design drawings of various styles and layouts. The solution can be easily expanded to more types of interface elements and more complex design drawing processing tasks, and only needs to update or expand the historical database and retrain the model to adapt to new needs.
[0153] The solution disclosed in this embodiment provides a solid foundation for subsequent steps. Specifically, accurate pixel-level classification provides reliable basic data for subsequent steps such as semantic conversion, layout analysis, and visual feature extraction, thereby ensuring the accuracy and efficiency of the entire design drawing to code conversion process.
[0154] Reference Figure 5 The method for converting interface elements specifically includes the following steps, that is, converting all interface elements into corresponding semantic features based on the second preset model, specifically including the following steps:
[0155] A310, determining a second initial model.
[0156] In this embodiment, the second initial model preferably includes one or more of a Word2Vec model or a BERT model, a GloVe model, a FastText model, and an ELMo model.
[0157] In this step, according to the task requirements, one or more models such as Word2Vec, BERT, GloVe, FastText, ELMo, etc. are selected as the second initial model. These models perform well in natural language processing and text embedding.
[0158] In addition, you can also download the pre-trained weights of the selected model (if necessary) and prepare the corresponding computing resources and software environment to support further training and inference of the model.
[0159] In this step, the selected model is designed to process text and semantic information and can capture the complex relationships between words. Using a pre-trained model can accelerate the training process and improve model performance.
[0160] A320, training a second initial model based on a second historical database, and using the trained second initial model as a second preset model;
[0161] The second history database includes the phonetic relationship information between different words.
[0162] In this step, the second historical database contains rich information on phonetic relationships between words, such as synonyms, antonyms, contextual relationships, etc., which are crucial for training the model to understand semantics; the second initial model is fine-tuned or further trained using the data in the second historical database, with the goal of enabling the model to more accurately understand the semantic relationships between interface element categories and attributes; the performance of the model is evaluated through the validation set to ensure that it can accurately convert high-dimensional semantic information into low-dimensional embedding vectors.
[0163] Through training, the model can better adapt to the semantic relationships in specific fields (such as interface design), and the fine-tuned model is more accurate in processing the semantics of interface elements.
[0164] A330, extracts the category and related attributes of each interface element to obtain high-dimensional semantic information;
[0165] The attributes include one or more of text content and text status.
[0166] In this step, the category and related attributes of each element are extracted from the interface elements identified in the design drawing, and the extracted information is organized into a structured format for subsequent processing and analysis, ensuring that all important interface elements and attributes are taken into account, providing complete information for subsequent semantic conversion, and facilitating efficient data processing and analysis.
[0167] A340, converts high-dimensional semantic information into a low-dimensional embedding vector based on a second preset model.
[0168] The high-dimensional semantic information extracted in step A330 is input into the second preset model. The model converts the high-dimensional information into a low-dimensional embedding vector while retaining the key information of the original semantics, effectively reducing the complexity of the data while retaining important information for subsequent processing. The embedding vector can capture the key features of the original high-dimensional semantic information.
[0169] A350 performs nonlinear transformation on the embedded vector to obtain the corresponding semantic features.
[0170] Specifically, the embedded vector can be processed using nonlinear transformation methods (such as neural network layers, activation functions, etc.), and the semantic features of the interface elements can be extracted from the transformed vector. These features will be used for subsequent layout analysis and code generation.
[0171] In this step, nonlinear transformation can capture more complex semantic relationships, thereby generating richer feature representations. The transformation methods and parameters can be adjusted according to specific needs to adapt to different scenarios and tasks.
[0172] The method for converting interface elements disclosed in this embodiment uses a professional natural language processing model to perform semantic conversion of interface elements, which can more accurately capture the complex relationship between elements, improve the accuracy of semantic understanding, convert high-dimensional semantic information into low-dimensional embedding vectors, simplify the data processing process, and retain key information, thereby improving processing efficiency. At the same time, the selected model and method are flexible and can easily adapt to different interface design styles and element types. In addition, as data increases and models are updated, YY+242206P
[0173] The entire solution can be continuously optimized and expanded.
[0174] The solution disclosed in this embodiment lays the foundation for subsequent steps, that is, accurate semantic feature extraction provides a solid foundation for subsequent layout structure analysis, visual feature capture and code generation, ensuring the consistency and accuracy of the entire design drawing to code conversion process.
[0175] Reference Figure 6 , the method for obtaining the layout feature specifically includes the following steps:
[0176] B100, determine the third initial model.
[0177] In this embodiment, the third initial model preferably includes one or more of a graph neural network model, a graph convolutional network model, and an image-to-image conversion model.
[0178] Specifically, according to the task requirements, one or more models are selected as the third initial model. These models may include graph neural networks (GNNs), graph convolutional networks (GCNs), image-to-image conversion models (such as CycleGAN, etc.), configure the architecture and parameters of the selected models, and prepare computing resources for training and inference.
[0179] In this step, the selected model can handle complex graph structures and image data, which helps to capture the layout features in the design drawing. The most suitable model can be selected for layout feature extraction according to specific needs.
[0180] B200, training a third initial model based on third historical data, and using the trained third initial model as a third preset model;
[0181] The third historical data includes layout structure information of different design drawings.
[0182] In this step, the third historical data contains layout structure information of different design drawings, such as grid system, Flexbox layout, etc. These data are used to train the model to identify and understand various layout structures; the third initial model is trained using the third historical data, and the parameters of the model are optimized so that it can accurately identify the layout elements and structural relationships in the design drawings. The model performance is evaluated using the validation data set to ensure its effectiveness in actual layout recognition. Through training with a large amount of layout data, the model can more accurately understand and analyze different layout structures, and the trained model can adapt to various layout styles and complex design requirements.
[0183] B300, performs high-level layout structure analysis on the design drawing based on the third preset model and target detection algorithm to obtain all layout elements and structural relationships.
[0184] Among them, the target detection algorithm includes one or more of the YOLO algorithm, the Faster R-CNN algorithm, and the SSD algorithm.
[0185] In this embodiment, the layout elements include one or more of a grid system, a Flexbox layout, absolute positioning, floating, a multi-column layout, a viewport unit, a CSS custom property, a CSS shape, and an asynchronous layout.
[0186] Specifically, a suitable target detection algorithm (such as YOLO, Faster R-CNN, SSD, etc.) is selected to detect layout elements in the design drawing, such as grid system, floating layout, etc., and the target detection results are further analyzed using the third preset model to obtain detailed information on the layout elements and their structural relationships (for example, which elements are parent elements and which are child elements). The target detection algorithm can efficiently detect layout elements, while the third preset model further analyzes the relationship between these elements, and can also identify and analyze all layout elements and structural relationships in the design drawing to ensure that nothing is missed.
[0187] B400, analyzes layout elements and structural relationships based on graph convolution operations to obtain layout features;
[0188] The layout structure includes one or more of grid layout, Flexbox layout, and absolute positioning.
[0189] Specifically, a graph convolutional network (GCN) is used to process layout elements and their structural relationships. GCN can capture the relationship between nodes in the graph and extract layout features. Through graph convolution operations, layout features in the design drawing are extracted, including grid layout, Flexbox layout, absolute positioning, etc. GCN can effectively capture the complex relationship between nodes (layout elements), extract more meaningful layout features, understand and analyze the characteristics and advantages of different layout structures, and provide detailed layout information for subsequent operations and code generation.
[0190] In addition, a graph neural network (GNN) can be introduced to model the spatial relationship between design drawing elements. The design drawing elements are regarded as graph nodes, and the spatial relationships between elements (such as relative position, alignment, etc.) are regarded as graph edges. The context information of the nodes is learned through graph convolution operations to generate a feature vector containing layout information.
[0191] The method for obtaining layout features disclosed in this embodiment can accurately identify and analyze layout elements and their structural relationships in a design drawing, effectively extract layout features, and capture the complex relationships between layout elements, thereby improving the accuracy of layout analysis, through a third initial model and a target detection algorithm; the entire process has a high degree of automation, from model selection to feature extraction, effectively reducing manual intervention and improving processing efficiency.
[0192] In this embodiment, the object detection algorithm is able to quickly detect layout elements, while the graph convolutional network is able to efficiently process and analyze the relationships between these elements.
[0193] In addition, it also supports a variety of layout structures (such as grid, Flexbox, absolute positioning, etc.) and layout elements (such as grid system, floating, multi-column layout, etc.) to adapt to different design needs, and can expand and optimize the model according to specific design requirements and new layout structures.
[0194] Through this method, it is possible to accurately extract layout features from the design drawing, improve the efficiency of the design to implementation process, ensure the accuracy and consistency of the layout, and lay a solid foundation for subsequent development and maintenance.
[0195] Reference Figure 7 , the method for obtaining visual features specifically includes the following steps:
[0196] C100, determining a fourth preset model.
[0197] In this embodiment, the fourth preset model includes one or more of a ResNet model, an EfficientNet model, a VGGNet model, an Inception model, a DenseNet model, and a MobileNet model.
[0198] In this step, selecting a suitable model can improve the accuracy and efficiency of visual feature extraction.
[0199] C200, training a fourth initial model based on fourth historical data, and using the trained fourth initial model as a fourth preset model;
[0200] The fourth historical data includes style information of different design drawings.
[0201] In this step, the trained model can better adapt to specific visual feature extraction tasks and improve recognition accuracy; through training, the model can recognize multiple visual features, thereby supporting more complex design requirements.
[0202] C300, identifies edges and contours in the design drawing based on a fourth preset model, and determines corresponding visual elements based on the edges and contours.
[0203] In this step, the fourth preset model is used to analyze the design drawing to detect the edges and contours in the image. The convolution layer and pooling layer of the model can be used to extract these features. Based on the detected edges and contours, the visual elements in the design drawing, such as buttons, text boxes, icons, etc., are identified. The edges and contours in the design drawing can be accurately detected to help identify the visual elements, and the visual elements are structured based on the edges and contours to facilitate subsequent feature analysis.
[0204] C400 changes visual elements at different scales and captures the features of visual elements at different sizes.
[0205] In this step, the detected visual elements are scaled at different scales to generate images of various sizes. The fourth preset model is used to extract features from these images of different scales to capture the changes in visual elements at different sizes. The features of visual elements at different sizes can be captured, thereby improving the robustness and accuracy of the model and adapting to the design requirements of various screen sizes and resolutions.
[0206] Further, for key point detection, key points such as corner points, edge intersections, etc. can be identified in visual elements, which can be done using specific key point detection algorithms (such as SIFT, SURF, etc.).
[0207] C500, detects key points in visual elements and generates descriptors for each key point.
[0208] Specifically, for descriptor generation, a descriptor can be generated for each detected key point. These descriptors are vectors used to describe the features of the key points, such as SIFT descriptors or ORB descriptors. The key points and descriptors are robust to changes in perspective and scale.
[0209] C600,obtains visual features based on feature,descriptors in visual elements.
[0210] In this embodiment, the visual features include one or more of color, font, size, margin between elements, shadow, element padding, border, and transparency.
[0211] Specifically, various features (color, shape, size, etc.) extracted from visual elements and the descriptors of each key point can be aggregated, and weighted averaging or a specific fusion algorithm can be used to integrate this information into a unified set of visual features; the aggregated features are standardized to ensure a consistent input format in subsequent machine learning models or analyses.
[0212] In this embodiment, a high-dimensional visual feature expression is formed through feature fusion, which can be better used for subsequent pattern recognition, classification and analysis tasks.
[0213] The method for obtaining visual features disclosed in this embodiment can extract various visual features through the selection and training of multiple preset models, and can accurately identify and describe the different features of visual elements, providing rich information for subsequent design analysis and code generation; edge and contour detection can effectively detect the edges and contours in the design drawing to ensure that all visual elements are accurately identified; through feature extraction at different scales, the adaptability to design drawings of different sizes and resolutions is improved; multi-scale features can adapt to various screen sizes and resolutions, improving the robustness of the model; at the same time, the use of key points and descriptors enhances the stability and accuracy of visual features, and detailed visual feature extraction provides a basis for the conversion process from design drawings to codes, ensuring the consistency of the design and the accuracy of implementation.
[0214] Through this method, visual features can be extracted comprehensively and accurately, the accuracy of design drawing parsing can be improved, and efficient interface implementation and optimization can be supported.
[0215] Reference Figure 8 , the method for obtaining multimodal features specifically includes the following steps:
[0216] S210, dynamically assigning feature weights based on task requirements and feature importance.
[0217] Specifically, an attention mechanism is designed to dynamically assign weights according to the importance of different modal features.
[0218] Specifically, the weight of each feature is dynamically adjusted according to the task requirements and feature analysis results. For example, in text generation tasks, semantic features may be more important, while in image retrieval tasks, visual features may be more important. During model training, adaptive mechanisms or optimization algorithms (such as the Adam optimizer) are used to continuously adjust feature weights.
[0219] By dynamically adjusting feature weights, the model can pay more attention to the features that are most important to the task, thereby improving the performance of the model. It can automatically adapt to the importance of features according to task requirements, thereby improving the applicability and effectiveness of the model.
[0220] S220, aligning the semantic features, the layout features, and the visual features, converting them into the same feature space, and obtaining a first feature vector, a second feature vector, and a third feature vector.
[0221] Specifically, a mapping function (such as linear transformation, dimensionality reduction technique) can be used to transform the features of different modalities into the same feature space. For example, principal component analysis (PCA) or the projection YY+242206P layer in deep learning can be used.
[0222] For text (semantic features), images (visual features), and layout features (structural information), a common representation space is used for alignment, such as through a shared embedding layer in a deep learning network.
[0223] By converting the features of different modalities into the same feature space, the comparability and consistency between the features are ensured, which simplifies the subsequent feature fusion and analysis process and facilitates the comprehensive processing of features of different modalities.
[0224] S230, performing weighted summation on the first feature vector, the second feature vector, and the third feature vector based on feature weights to obtain a comprehensive feature vector.
[0225] Specifically, according to the feature weights determined in S210, the first feature vector, the second feature vector, and the third feature vector are weighted and summed, and the following formula may be used:
[0226] Comprehensive feature vector = w1·first feature vector + w2·second feature vector + w3·third feature vector, where w1, w2, and w3 are the weights of semantic features, layout features, and visual features, respectively; the weighted feature vectors are then merged into a comprehensive feature vector using a linear combination method.
[0227] By weighted summing and fusing information from different modalities, a comprehensive feature vector is formed, which can represent the data more comprehensively. At the same time, the weights can be adjusted according to the importance of the features to improve the sensitivity of the model to key features.
[0228] S240, performing multi-layer nonlinear transformation decoding on the comprehensive feature vector to obtain multi-modal features.
[0229] Specifically, a multi-layer neural network (such as a fully connected network, a convolutional neural network) can be used to perform nonlinear transformations on the comprehensive feature vector. For example, when using a deep feedforward neural network for feature decoding, the output of the intermediate layer can be transformed by applying a nonlinear activation function (such as ReLU, Sigmoid) to capture complex feature relationships.
[0230] Through the trained neural network, the comprehensive feature vector is decoded into the final multimodal feature for further analysis or task processing. Nonlinear transformation can capture the complex relationship between different modal features and improve the ability of feature representation; the multi-layer network structure enhances the expression ability of the model, making the final multimodal feature more discriminative.
[0231] The method for obtaining multimodal features disclosed in this embodiment effectively integrates semantic features, layout features and visual features to form a comprehensive feature representation, which can more comprehensively understand the data; by dynamically adjusting feature weights, the model can adapt to the needs of different tasks, thereby optimizing performance; aligning features of different modalities to the same feature space simplifies the feature processing flow; through weighted summation and nonlinear transformation, more discriminative multimodal features can be extracted, improving the performance of the model in practical applications; multi-layer nonlinear transformation can capture the complex relationship between features of different modalities, improving the model's understanding of dataYY+242206P
[0232] and processing power.
[0233] Furthermore, this solution can be applied to a variety of tasks, such as joint understanding of images and texts, cross-modal retrieval, recommendation systems, etc., and has strong versatility.
[0234] Through these steps, features can be efficiently extracted and fused from multiple modal data to improve the performance and application scope of the model.
[0235] Furthermore, the present application also includes: constraining the process of multi-layer nonlinear transformation decoding through grammatical constraints and structural constraints, that is, performing code structure constraints in order to ensure the correctness and completeness of the generated HTML code structure, specifically, such as tag closure, nesting rules, etc.
[0236] By introducing grammatical constraints and structural constraints to optimize the multimodal feature decoding process, the quality and reliability of the generated HTML code can be significantly improved. This not only ensures the correctness and completeness of the generated code, but also improves the maintainability and user experience of the code. At the same time, standard code structure plays an important role in improving web page performance and compatibility.
[0237] Introducing grammatical constraints and structural constraints in the decoding process, especially tag closure, nesting rules, etc., means that in the decoding process of processing data or information, certain grammatical and structural rules need to be followed to ensure the accuracy and consistency of the decoding results. This method is often used to process complex markup languages (such as HTML, XML) or structured data. The following is a specific explanation.
[0238] Syntax constraints refer to the grammatical rules that must be followed during data decoding. These rules define the legal structure and format of the data, ensuring that the data in the decoding process conforms to the expected grammatical specifications.
[0239] Example: In an XML document, tags must appear in pairs. For example, <tag> content< / tag> The syntax constraints ensure that every opening tag has a corresponding closing tag.
[0240] In programming languages, grammatical constraints define legal code structures, such as the pairing of brackets, the end of statements, etc.
[0241] Structural constraints refer to the organizational structure and hierarchical relationships of data. These constraints ensure that the data follows specific structural rules during the decoding process so that the data can be correctly parsed and understood.
[0242] Tag closure means that tags must be closed correctly. For example, in HTML, A tag must have a corresponding Label.
[0243] Nesting rules refer to the fact that nested tags must follow certain rules. For example, in HTML, a In the label Tags cannot be directly nested within other tags and must follow structure.
[0244] In specific application scenarios, for markup language processing, when parsing HTML or XML documents, the decoder needs to ensure that all tags are closed correctly and the nested structure conforms to the grammatical rules. This can prevent parsing errors and ensure that the document can be rendered or processed correctly.
[0245] When working with structured data, applying structural constraints can help identify and correct errors and inconsistencies in the data. Data validation tools use syntax and structural constraints to ensure that input data conforms to the expected format and structure, for example, checking whether JSON data conforms to a specific schema definition.
[0246] The following are examples of tag closures, such as incorrect closures: Some text .
[0247] Correct closure: Some text .
[0248] For the nested rule example, incorrect nesting: Item 1 Item 2 ; Correct nesting: Item 1 Item 2 .
[0249] Introducing these constraints ensures that the decoding process can not only parse the data correctly, but also maintain the consistency and integrity of the data.
[0250] Furthermore, the method for automatically generating program codes from design drawings disclosed in the present application also includes:
[0251] Scan the program code sequence to obtain exception information, and correct the exception information to obtain correction code information;
[0252] Perform semantic analysis on the revised code information, identify the logical structure in the code, optimize the code label selection based on the preset code library, and obtain the optimized revised code information.
[0253] Specifically, static analysis tools (such as Lint tools, SonarQube) can be used to scan the generated code to detect potential syntax errors, logical errors, or places that do not comply with coding standards, and identify abnormal information in the code, such as unclosed tags, missing semicolons, undefined variables, etc.
[0254] For exception correction, you can use automatic repair tools or algorithms to automatically correct the code based on the detected exception information. For example, you can automatically add missing closing tags, fix spelling errors, adjust code indentation, and manually review and repair complex exceptions that cannot be automatically corrected. After correcting the exception, the corrected code information is formed to prepare for further analysis and optimization.
[0255] Perform semantic analysis on the corrected code information to identify the logical structure in the code, including: constructing an abstract syntax tree of the code to identify the logical structure and hierarchical relationship of the code, which helps to understand the overall structure and logic of the code; analyzing the control flow, data flow, function call relationship, etc. of the code to identify the main logical structure in the code. Based on the preset code base or design pattern, optimize the label selection of the code, for example, replace labels that do not conform to best practices and use more efficient or more readable code structures. Match the corrected code information with the preset code template to ensure that the generated code conforms to the standardized design pattern.
[0256] Integrate the semantically analyzed and optimized codes to form the final optimized and corrected code information.
[0257] Through this solution, automatic scanning and correction steps can timely discover and fix errors in the code, reduce potential bugs, and improve the stability and reliability of the code; at the same time, semantic analysis and optimization can improve the logical structure of the code and enhance the organization and readability of the code.
[0258] By scanning, correcting exceptions, semantically analyzing and optimizing the generated program code, the quality, maintainability and performance of the code can be significantly improved. These steps not only reduce errors and defects in the code, but also ensure that the generated code complies with standards and best practices, thereby improving development efficiency and user experience; through optimization, the semantic level of the code can be improved.
[0259] Furthermore, the present application also includes: user interactive editing, that is, allowing users to interactively edit the generated HTML code, and feeding back the user's modifications to the model to improve the controllability and adaptability of the model. This function not only allows users to adjust the code according to actual needs, but also improves the controllability and adaptability of the model through the feedback mechanism, thereby reducing manual intervention and development costs, while supporting personalized customization and continuous improvement of the code.
[0260] Furthermore, this application also includes: by recording and analyzing the user's operational data during use (such as code modification records and design preferences), using machine learning algorithms to dynamically adjust and optimize the parameters of the AI model to make it more adaptable to the user's actual needs and improve the practicality and effectiveness of the generated code; providing multiple configuration options, allowing users to choose code indentation style, naming conventions, CSS preprocessors and other parameters according to their own coding habits and project requirements. This flexible configuration mechanism ensures that the generated code can better match the team's working methods and improve the consistency and maintainability of the code; building an open plug-in ecosystem to support third-party developers to create and share plug-ins and expand system functions. At the same time, deeply integrate mainstream version control systems and continuous integration / continuous delivery (CI / CD) tools to achieve automation of the entire process from code generation to testing and deployment, making front-end development more efficient and coherent.
[0261] Reference Fig. 9 In a second embodiment of the present application, the present application discloses a method for automatically generating program codes from a design drawing, which specifically includes the following steps:
[0262] S10, configures the plug-in architecture, which is seamlessly integrated with various UI design software.
[0263] Through this step, it can be ensured that the design drawings can be directly obtained from the design software. In addition, an API or interface can be provided to realize data exchange with the design software and support the upload and processing of the design drawings.
[0264] S20, a preset window based on the plug-in architecture receives a design drawing uploaded by a user.
[0265] Among them, the design drawing is in compressed format.
[0266] In this embodiment, multiple design drawing file formats are supported, such as PSD, SVG, PNG, etc., to ensure compatibility and flexibility.
[0267] S30, performing deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawing to obtain semantic features, layout features, and visual features.
[0268] S40, preprocessing the semantic features, layout features, and visual features, and processing the preprocessed semantic features, layout features, and visual features to obtain multimodal features.
[0269] S50, converting the multimodal features into preset vectors, decoding the preset vectors, and obtaining a program code sequence.
[0270] The method disclosed in this embodiment integrates the plug-in architecture with various UI design software, reducing the user's operation steps and realizing the automatic conversion from design drawings to codes; through deep semantic segmentation, layout analysis and visual information extraction, program code that meets the design requirements is automatically generated, reducing the workload of manual intervention and code writing; deep semantic segmentation and visual information extraction can accurately identify the elements and attributes in the design drawings, and improve the accuracy and consistency of the generated code; layout structure analysis ensures that the generated code can correctly reflect the layout and hierarchical relationship of the design drawings, and avoid layout errors in the generated code.
[0271] Through the plug-in architecture, users can directly upload design drawings in commonly used UI design software without switching tools, which improves the convenience of operation. Users can preview and modify the code after generating the code to verify the design effect more intuitively.
[0272] This embodiment supports multiple design drawing file formats, adapts to the output of different design software, and enhances the compatibility and flexibility of the system; through the comprehensive processing of semantic features, layout features and visual features, it can cope with complex design requirements and realize diversified design conversion.
[0273] The solution disclosed in this embodiment can automatically generate code, significantly improve development efficiency, shorten the cycle from design to implementation, reduce the repetitive work of manually writing code, and improve development efficiency and accuracy.
[0274] By configuring the plug-in architecture, receiving the design drawing, performing deep feature analysis and processing, and generating program code, the automation and accuracy of the code generation from the design drawing can be significantly improved. This method not only improves development efficiency and user experience, but also supports the processing of diverse design requirements and helps with subsequent system optimization and improvement.
[0275] The embodiments of the present invention can be integrated with various UI design software such as Sketch and Adobe XD in the form of software plug-ins, or provided as independent online services. Users can easily convert design drawings into HTML codes through simple operations, greatly improving the efficiency and quality of Web front-end development.
[0276] Through the above implementation steps, the present invention can efficiently and flexibly complete the automatic conversion of design drawings to HTML codes, ensure code quality and structural optimization, and significantly improve the efficiency and quality of Web front-end development.
[0277] It should be noted that all methods disclosed in the first embodiment of the present application are applicable to the solutions disclosed in the second embodiment of the present application, so they will not be described one by one here.
[0278] In a second aspect, the present application discloses a system for automatically generating program codes from design drawings, comprising:
[0279] The analysis module is used to perform deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings to obtain semantic features, layout features, and visual features;
[0280] A processing module, used for preprocessing the semantic features, layout features, and visual features, and processing the preprocessed semantic features, layout features, and visual features to obtain multimodal features;
[0281] The code acquisition module is used to convert the multimodal features into preset vectors, decode the preset vectors, and obtain the program code sequence.
[0282] It should be noted that all methods disclosed in the first embodiment of the present application are applicable to the scheme disclosed in the second aspect of the present application, so they will not be described one by one here.
[0283] The computer device according to the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.
[0284] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device performs all or part of the steps of the method for automatically generating program codes from design drawings in the aforementioned embodiments of the present disclosure.
[0285] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.
[0286] like Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown, which is a schematic diagram of the structure of a computer device suitable for implementing the embodiment of the present disclosure. Fig.10 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0287] like Fig.10 As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0288] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Fig.10 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.
[0289] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the method of automatically generating program code from a design drawing of an embodiment of the present disclosure are executed.
[0290] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.
[0291] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the method for automatically generating program codes from design drawings in the above-mentioned embodiments of the present disclosure are executed.
[0292] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).
[0293] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.
[0294] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0295] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.
[0296] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.
[0297] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0298] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.
[0299] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0300] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for automatically generating program code from a design drawing, characterized in that: include: Perform deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings to obtain semantic features, layout features, and visual features; Preprocessing the semantic features, the layout features, and the visual features, and fusing the preprocessed semantic features, the layout features, and the visual features to obtain multimodal features; The multimodal features are converted into preset vectors, and the preset vectors are decoded to obtain a program code sequence.
2. The method for automatically generating program code from a design drawing according to claim 1, characterized in that: The design drawings are subjected to deep semantic segmentation, layout structure analysis, and visual information extraction to obtain semantic features, layout features, and visual features, including: Performing deep semantic segmentation on the design drawing to obtain all interface elements; Performing semantic conversion on the interface elements to obtain semantic features; Performing layout structural analysis on the design drawing to obtain all layout elements and structural relationships; The structural relationship includes the spatial relationship of elements, the hierarchical relationship between elements, and the alignment logic information; Based on the layout elements and the structural relationships, obtaining layout features; Extracting all visual elements of the design; Features of the visual elements at different scales are captured to obtain visual features.
3. The method for automatically generating program code from a design drawing according to claim 2, characterized in that: The method for obtaining the semantic feature includes: Extracting an image of the design drawing based on the first preset model, and classifying each pixel in the image to obtain a corresponding category; Based on the preset semantic information library and all the categories, determine all the interface elements; Convert all of the interface elements into corresponding semantic features based on a second preset model; The interface elements include one or more of buttons, text boxes, pictures, navigation bars, input boxes, drop-down menus, check boxes, sliders, switches, tabs, lists, grids, modal windows, tooltips, icons, dividing lines, progress bars, warning boxes, forms, search boxes, menus, and scroll bars.
4. The method for automatically generating program code from a design drawing according to claim 3, characterized in that: The step of extracting an image of the design drawing based on the first preset model and classifying each pixel in the image to obtain a corresponding category includes: Determine a first initial model; the first initial model includes one or more of a convolutional neural network model, a full convolutional network model, and a U-Net model; Training the first initial model based on the first historical database, and using the trained first initial model as the first preset model; The first historical database includes pixel information of preset types and corresponding category information; Based on the first preset model, the design drawing is gradually abstracted from low-level features to high-level features, each pixel is identified and classified, and a category corresponding to each pixel is obtained; The low-level features include one or more of edges and textures, and the high-level features include one or more of shapes and objects.
5. The method for automatically generating program code from a design drawing according to claim 4, characterized in that: The converting all the interface elements into corresponding semantic features based on the second preset model includes: Determine a second initial model; the second initial model includes one or more of a Word2Vec model or a BERT model, a GloVe model, a FastText model, and an ELMo model; Training the second initial model based on the second historical database, and using the trained second initial model as the second preset model; The second history database includes phonetic relationship information between different words; Extracting the category and related attributes of each of the interface elements to obtain high-dimensional semantic information; The attributes include one or more of text content and text status; Converting the high-dimensional semantic information into a low-dimensional embedding vector based on the second preset model; Perform a nonlinear transformation on the embedding vector to obtain the corresponding semantic feature.
6. The method for automatically generating program code from a design drawing according to claim 2, characterized in that: The method for obtaining the layout feature comprises: Determine a third initial model; the third initial model includes one or more of a graph neural network model, a graph convolutional network model, and an image-to-image conversion model; Training the third initial model based on the third historical data, and using the trained third initial model as the third preset model; The third historical data includes layout structure information of different design drawings; Performing a high-level layout structure analysis on the design drawing based on the third preset model and the target detection algorithm to obtain all layout elements and structural relationships; The target detection algorithm includes one or more of the YOLO algorithm, the Faster R-CNN algorithm, and the SSD algorithm; Analyze the layout elements and the structural relationships based on a graph convolution operation to obtain layout features; The layout structure includes one or more of a grid layout, a Flexbox layout, and absolute positioning; The layout elements include one or more of a grid system, a Flexbox layout, absolute positioning, floating, a multi-column layout, a viewport unit, a CSS custom attribute, a CSS shape, and an asynchronous layout.
7. The method for automatically generating program code from a design drawing according to claim 2, characterized in that: The method for obtaining the visual features comprises: Determine a fourth preset model; the fourth preset model includes one or more of a ResNet model, an EfficientNet model, a VGGNet model, an Inception model, a DenseNet model, and a MobileNet model; Training the fourth initial model based on fourth historical data, and using the trained fourth initial model as a fourth preset model; The fourth historical data includes style information of different design drawings; Identify edges and contours in the design drawing based on a fourth preset model, and determine corresponding visual elements based on the edges and contours; Performing changes to the visual elements at different scales to capture the features of the visual elements at different scales; Detecting key points in the visual element and generating a descriptor for each key point; Obtaining the visual feature based on the features and descriptors in the visual element; The visual features include one or more of color, font, size, margin between elements, shadow, element padding, border, and transparency.
8. The method for automatically generating program code from a design drawing according to claim 1, characterized in that: The method for obtaining the multimodal features comprises: Dynamically assign feature weights based on task requirements and feature importance; Aligning the semantic features, the layout features, and the visual features, converting them into the same feature space, and obtaining a first feature vector, a second feature vector, and a third feature vector; Performing weighted summation on the first feature vector, the second feature vector, and the third feature vector based on the feature weight to obtain a comprehensive feature vector; The comprehensive feature vector is subjected to multi-layer nonlinear transformation decoding to obtain multi-modal features.
9. The method for automatically generating program code from a design drawing according to claim 1, characterized in that: The method further comprises: scanning the program code sequence to obtain abnormal information, and correcting the abnormal information to obtain corrected code information; The revised code information is semantically analyzed to identify the logical structure in the code, and the label selection of the code is optimized based on a preset code library to obtain optimized revised code information.
10. A method for automatically generating program code from a design drawing, characterized in that: include: Configure a plug-in architecture that is seamlessly integrated with various UI design software; The preset window based on the plug-in architecture receives the design drawings uploaded by the user; Perform deep semantic segmentation, layout structure analysis, and visual information extraction on the design drawings to obtain semantic features, layout features, and visual features; Preprocessing the semantic features, the layout features, and the visual features, and fusing the preprocessed semantic features, the layout features, and the visual features to obtain multimodal features; The multimodal features are converted into preset vectors, and the preset vectors are decoded to obtain a program code sequence.
11. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of automatically generating program code from a design drawing as described in any one of claims 1-9 or the method of automatically generating program code from a design drawing as described in claim 10.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the method of automatically generating program codes from a design drawing as described in any one of claims 1 to 9 or the method of automatically generating program codes from a design drawing as described in claim 10.
13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method of automatically generating program codes from design drawings as described in any one of claims 1 to 9 or the steps of the method of automatically generating program codes from design drawings as described in claim 10 are implemented.
Citation Information
Cited By
Method and system for generating AI graph recognition front-end code and computer storage medium
CN120122934A
Method and system for generating ai image recognition front-end code and computer storage medium
CN120122934B
Target embedded code generation method and device, electronic equipment and storage medium
CN120704694A
CSS file generation method and device, computer equipment and storage medium
CN121009251A
UI design drawing segmentation method based on front-end semantic understanding and related equipment
CN121353676A
