Design draft reduction method based on computer vision, deep learning and code generation

Abstract

The invention is applied to the technical field of software, and discloses a design draft reduction method based on computer vision, deep learning and code generation, comprising the following steps: step 1, information extraction, step 2, component identification, step 3, code conversion, and step 4, result display. According to the design manuscript restoration method based on computer vision, deep learning and code generation, support for D2CSchema is added through the information extraction module, that is, a data structure of a design manuscript generated by a designer through a D2C tool can be received instead of only a picture, so that the accuracy and efficiency of information extraction can be improved, and the design manuscript restoration method based on computer vision, deep learning and code generation can be realized. According to the method, image analysis and processing time and errors are reduced, support for an external component library is increased through the component identification module, that is, the most suitable component can be retrieved and matched from a plurality of different component libraries and frames instead of an internal component library, and thus the diversity and flexibility of the component can be increased.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to a design draft restoration method based on computer vision, deep learning and code generation. Background Art

[0002] In today's digital and information age, the demand for software and application development is growing. As an important starting point for software development, how to accurately and efficiently restore design drafts into actual executable codes has always been an important challenge faced by the software development field. Traditional design draft restoration methods mainly rely on manual coding. Developers need to manually write codes to implement corresponding functions and interfaces based on the visual elements and layout of the design drafts. This method is not only inefficient, but also prone to human errors, resulting in extended development cycles and increased costs. With the rapid development of computer technology, computer vision and deep learning technologies have gradually emerged. Computer vision technology enables computers to understand and process image information, while deep learning algorithms have shown excellent performance in image recognition and feature extraction. Using deep learning methods, neural network models are used to image or D2C Schema is used to analyze and process images, and information such as layers, components, and layouts are extracted from it for subsequent conversion. The advantage of this solution is that it can improve the accuracy and adaptability of information extraction. The disadvantage is that it requires a large amount of training data and computing resources, as well as model tuning and maintenance. The traditional image processing method is used to analyze and process the image of the design draft or D2CSchema using algorithms such as image segmentation, image recognition, and edge detection, and information such as layers, components, and layouts are extracted from it for subsequent conversion. The advantage of this solution is that the task of information extraction can be completed quickly. The disadvantage is that it may not be able to process complex design drafts, and there are certain requirements for the quality and format of the image. Tools such as imgCook are used to generate code for the design draft, input the image of the design draft or D2C Schema, and output the front-end code, including HTML, CSS, JavaScript, etc., as well as references to corresponding component libraries and frameworks, such as Ant. Design, React, etc. The advantage of this solution is that it can quickly convert design drafts into front-end codes, reduce development time and costs, and support output on multiple platforms and languages. The disadvantage of this solution is that the design drafts need to be annotated for tool recognition and parsing, and the annotation process may be cumbersome and time-consuming, and the quality of annotation will affect the effect of code generation. In addition, this solution cannot guarantee the quality and performance of the generated code, and may require subsequent optimization and adjustment. Summary of the invention

[0003] The purpose of the present invention is to provide a design draft restoration method based on computer vision, deep learning and code generation, so as to solve the problems proposed in the above background technology, such as the need for a large amount of training data and computing resources, as well as the tuning and maintenance of the model, the inability to handle complex design drafts, and the certain requirements on the quality and format of the image and the need to annotate and optimize the design draft.

[0004] To achieve the above object, the present invention provides the following technical solution: a design draft restoration method based on computer vision, deep learning and code generation, the restoration method comprising the following steps:

[0005] Step 1: Information extraction: This is done by the information extraction module, which uses computer vision technology to analyze and process the source data, extracting information such as layers, components, and layouts, and outputting them for subsequent conversion.

[0006] Step 2: Component identification: This is done by the component identification module, which uses deep learning technology to identify and classify source data, identify different component types, and output the corresponding component libraries and framework references for subsequent conversion.

[0007] Step 3: Code conversion: This is completed by the code conversion module, which uses code generation technology to convert the source data and convert the information in the design draft into front-end code, including HTML, CSS, JavaScript, etc., as well as adaptation to different browsers, devices, resolutions, etc., to ensure the restoration and compatibility of the front-end page;

[0008] Step 4: Result display: This is completed by the result display module, which displays the source data, renders the front-end code into a front-end page, and transmits it to the user through a browser or other means, so that the user can view and use the front-end page;

[0009] The source data received by the information extraction module in step 1 is a picture of the design draft or one of the D2C Schemas, and the output of the information extraction module is a data structure containing information such as layers, components, and layouts, such as in JSON format, which is used as input for the next step;

[0010] The source data received by the component identification module in step 2 is the output of the information extraction module, that is, a data structure containing information such as layers, components, and layouts, and the output of the component identification module is a data structure containing information such as component type, properties, component library, and framework reference, such as in JSON format, which serves as input for the next step;

[0011] The source data received by the code conversion module in step 3 is the output of the component identification module, that is, a data structure containing information such as component type, attributes, component library and framework reference, and the output of the code conversion module is the front-end code, such as HTML, CSS, JavaScript, etc., and the corresponding component library and framework reference, such as Ant Design, React, etc., which is used as the final result data;

[0012] The source data received by the result display module in step 4 is the output of the code conversion module, that is, the front-end code, and the corresponding component library and framework reference, and the output of the result display module is the front-end page, as well as user feedback and evaluation, which serves as the basis for subsequent optimization and improvement.

[0013] Preferably, in step 1, the information extraction module is responsible for image analysis and processing of the design draft, and the image of the design draft or the D2C Schema is converted into structured data through the information extraction module. The functions of the information extraction module include image segmentation, image recognition and layout analysis.

[0014] Preferably, the component identification module in step 2 is responsible for retrieving and matching the most suitable components from the internal component library or the external component library, as well as the references of the corresponding component library and framework according to the output of the information extraction module. The role of this module is to identify the component types and attributes in the design draft and connect with the existing component library and framework. The functions of this module include feature extraction, image classification, prediction probability and component matching.

[0015] Preferably, the role of the code conversion module in step 3 is to convert the structure and content in the design draft into executable code, and the functions of this module include annotation tools, code generation tools, code optimization tools and code adaptation tools.

[0016] Preferably, the code optimization tool and code adaptation tool optimize and adapt the generated front-end code to improve the quality and performance of the code, and ensure the restoration and compatibility of the front-end page to improve the quality and effect of the output of the data processing process.

[0017] Preferably, information extraction comprises the following steps:

[0018] Image segmentation: Divide the image of the design into several areas, each area corresponds to a layer or a component, and determine the boundaries and attributes of the area based on the shape, color, edge and other features of the area;

[0019] Image recognition: Identify each area and determine what type of component it belongs to, such as buttons, text boxes, icons, etc. Determine the component's label and attributes based on its shape, color, content and other features;

[0020] Layout analysis: Perform layout analysis on each area to determine its position and size in the design draft, as well as its relative relationship with other areas, such as up, down, left, right, center alignment, equidistant distribution, etc. Determine the layout parameters and rules based on the coordinates, size, spacing and other characteristics of the area.

[0021] Preferably, component identification comprises the following steps:

[0022] Feature extraction: Extract features from each region and convert them into a feature vector, which contains information such as the shape, color, content, and relative relationship with other regions of the region, and is used to represent the characteristics of the region;

[0023] Image classification: Use convolutional neural networks to classify images of each region into several categories. Each category corresponds to a type of component, such as buttons, text boxes, icons, etc., which are used to represent the category of the region.

[0024] Prediction probability: Using the output layer of the neural network, the probability that each region belongs to each category is calculated to represent the confidence level of the region;

[0025] Component matching: According to the category and confidence of the area, as well as the pre-set threshold, the most suitable component is matched, as well as the reference of the corresponding component library and framework, such as Ant Design, React, etc., which are used to represent the components of the area.

[0026] Preferably, the code conversion comprises the following steps:

[0027] Annotation: Use annotation tools to annotate each area in the design draft and generate a D2CSchema, which contains information such as the boundaries, attributes, labels, parameters, rules, components, references, etc. of the area, and is used to represent the structure and content of the design draft.

[0028] Code generation: Use code generation tools to generate front-end code based on D2C Schema, including HTML, CSS, JavaScript, etc., as well as references to corresponding component libraries and frameworks, such as Ant Design and React, to represent the structure and content of the front-end page;

[0029] Code optimization: Use code optimization tools to optimize the generated front-end code, including compression, formatting, annotation, verification, etc., to improve the quality and performance of the code;

[0030] Code adaptation: Use code adaptation tools to adapt the generated front-end code, including adaptation to different browsers, devices, resolutions, etc., to ensure the restoration and compatibility of the front-end page.

[0031] Preferably, the computer vision technology in step 1 is implemented using OpenCV and TensorFlow libraries, the deep learning technology in step 2 is implemented using Keras and TensorBoard libraries, and the code generation technology in step 3 is implemented using Sketch, imgCook, Ant Design and React tools or frameworks.

[0032] Preferably, the result display module in step 4 collects user feedback and evaluation as a data basis for subsequent optimization and improvement.

[0033] Compared with the prior art, the beneficial effects of the present invention are: the design draft restoration method based on computer vision, deep learning and code generation:

[0034] 1. Added support for D2C Schema through the information extraction module, that is, it can receive the data structure of the design draft generated by the designer using the D2C tool, rather than just the picture. This can improve the accuracy and efficiency of information extraction and reduce the time and error of image analysis and processing;

[0035] 2. Added support for external component libraries through the component identification module, that is, the most suitable components can be retrieved and matched from multiple different component libraries and frameworks, not just the internal component library, which can increase the diversity and flexibility of components to meet different design needs and styles;

[0036] 3. The code conversion module adds the functions of code optimization and adaptation, that is, the generated front-end code can be compressed, formatted, annotated, verified, etc. to improve the quality and performance of the code, as well as adapt to different browsers, devices, resolutions, etc. to ensure the restoration and compatibility of the front-end page. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process flow of restoring the design draft of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 The present invention provides a technical solution: a design draft restoration method based on computer vision, deep learning and code generation, the restoration method comprising the following steps:

[0040] Step 1: Information extraction: This is done by the information extraction module, which uses computer vision technology to analyze and process the source data, extracting information such as layers, components, and layouts, and outputting them for subsequent conversion.

[0041] Step 2: Component identification: This is done by the component identification module, which uses deep learning technology to identify and classify source data, identify different component types, and output the corresponding component libraries and framework references for subsequent conversion.

[0042] Step 3: Code conversion: This is completed by the code conversion module, which uses code generation technology to convert the source data and convert the information in the design draft into front-end code, including HTML, CSS, JavaScript, etc., as well as adaptation to different browsers, devices, resolutions, etc., to ensure the restoration and compatibility of the front-end page;

[0043] Step 4: Result display: This is completed by the result display module, which displays the source data, renders the front-end code into a front-end page, and transmits it to the user through a browser or other means, so that the user can view and use the front-end page;

[0044] Example

[0045] A web design draft designed by D2C tool, which includes a navigation bar, a carousel, a content area and a bottom bar;

[0046] The data structure of the design is a D2C Schema, which is a JSON file that contains information such as layers, components, and layouts in the design, as shown below:

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] Step 1: Information extraction. This step is completed by the information extraction module. The source data received by this module is the D2C Schema of the design draft. This module parses the source data and extracts information such as layers, components, and layouts from it to facilitate subsequent conversion. The output of this module is a data structure containing information such as layers, components, and layouts, such as JSON format. This data structure serves as the input of the next step.

[0054] Step 2: Component identification. This step is completed by the component identification module. The source data received by this module is the output of the information extraction module, that is, a data structure containing information such as layers, components, and layouts. This module identifies and classifies the source data, and identifies different component types and corresponding component libraries and framework references to facilitate subsequent conversion. The output of this module is a data structure containing information such as component type, properties, component libraries, and framework references, such as JSON format. This data structure serves as the input of the next step.

[0055] Step 3: Code conversion. This step is completed by the code conversion module. The source data received by the module is the output of the component identification module, that is, a data structure containing information such as component type, attributes, component library and framework reference. The module converts the source data and converts the information in the design draft into front-end code, including HTML, CSS, JavaScript, etc., as well as adaptation to different browsers, devices, resolutions, etc., to ensure the restoration and compatibility of the front-end page. The output of this module is the front-end code, such as HTML, CSS, JavaScript, etc., and the corresponding component library and framework reference, such as Ant Design, React, etc. This output is the final result data;

[0056] Step 4: Result display. This step is completed by the result display module. The source data received by this module is the output of the code conversion module, that is, the front-end code, as well as the corresponding component library and framework reference. This module displays the source data, renders the front-end code as a front-end page, and transmits it to the user through a browser or other means, so that the user can view and use the front-end page. The output of this module is the front-end page, as well as user feedback and evaluation. This output serves as the basis for subsequent optimization and improvement.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A design draft restoration method based on computer vision, deep learning and code generation, characterized in that: The restoration method comprises the following steps: Step 1: Information extraction: This is done by the information extraction module, which uses computer vision technology to analyze and process the source data, extracting information such as layers, components, and layouts, and outputting them for subsequent conversion. Step 2: Component identification: This is done by the component identification module, which uses deep learning technology to identify and classify source data, identify different component types, and output the corresponding component libraries and framework references for subsequent conversion. Step 3: Code conversion: This is completed by the code conversion module, which uses code generation technology to convert the source data and convert the information in the design draft into front-end code, including HTML, CSS, JavaScript, etc., as well as adaptation to different browsers, devices, resolutions, etc., to ensure the restoration and compatibility of the front-end page; Step 4: Result display: This is completed by the result display module, which displays the source data, renders the front-end code into a front-end page, and transmits it to the user through a browser or other means, so that the user can view and use the front-end page; The source data received by the information extraction module in step 1 is a picture of the design draft or one of the D2C Schemas, and the output of the information extraction module is a data structure containing information such as layers, components, and layouts, such as in JSON format, which is used as input for the next step; The source data received by the component identification module in step 2 is the output of the information extraction module, that is, a data structure containing information such as layers, components, and layouts, and the output of the component identification module is a data structure containing information such as component type, properties, component library, and framework reference, such as in JSON format, which serves as input for the next step; The source data received by the code conversion module in step 3 is the output of the component identification module, that is, a data structure containing information such as component type, attributes, component library and framework reference, and the output of the code conversion module is the front-end code, such as HTML, CSS, JavaScript, etc., and the corresponding component library and framework reference, such as Ant Design, React, etc., which is used as the final result data; The source data received by the result display module in step 4 is the output of the code conversion module, that is, the front-end code, and the corresponding component library and framework reference, and the output of the result display module is the front-end page, as well as user feedback and evaluation, which serves as the basis for subsequent optimization and improvement.

2. The design draft restoration method based on computer vision, deep learning and code generation according to claim 1, characterized in that: In step 1, the information extraction module is responsible for image analysis and processing of the design draft. The image of the design draft or the D2C Schema is converted into structured data through the information extraction module. The functions of the information extraction module include image segmentation, image recognition and layout analysis.

3. The design draft restoration method based on computer vision, deep learning and code generation according to claim 1, characterized in that: The component identification module in step 2 is responsible for retrieving and matching the most suitable components from the internal component library or the external component library, as well as the references of the corresponding component library and framework according to the output of the information extraction module. The role of this module is to identify the component types and attributes in the design draft and connect with the existing component library and framework. The functions of this module include feature extraction, image classification, prediction probability and component matching.

4. The design draft restoration method based on computer vision, deep learning and code generation according to claim 1, characterized in that: The role of the code conversion module in step 3 is to convert the structure and content in the design draft into executable code. The functions of this module include annotation tools, code generation tools, code optimization tools and code adaptation tools.

5. The design draft restoration method based on computer vision, deep learning and code generation according to claim 4, characterized in that: The code optimization tool and code adaptation tool optimize and adapt the generated front-end code to improve the quality and performance of the code, and ensure the restoration and compatibility of the front-end page to improve the quality and effect of the output of the data processing process.

6. The design draft restoration method based on computer vision, deep learning and code generation according to claim 2, characterized in that: Information extraction includes the following steps: Image segmentation: Divide the image of the design into several areas, each area corresponds to a layer or a component, and determine the boundaries and attributes of the area based on the shape, color, edge and other features of the area; Image recognition: Identify each area and determine what type of component it belongs to, such as buttons, text boxes, icons, etc. Determine the component's label and attributes based on its shape, color, content and other features; Layout analysis: Perform layout analysis on each area to determine its position and size in the design draft, as well as its relative relationship with other areas, such as up, down, left, right, center alignment, equidistant distribution, etc. Determine the layout parameters and rules based on the coordinates, size, spacing and other characteristics of the area.

7. The design draft restoration method based on computer vision, deep learning and code generation according to claim 3, characterized in that: Component identification includes the following steps: Feature extraction: Extract features from each region and convert them into a feature vector, which contains information such as the shape, color, content, and relative relationship with other regions of the region, and is used to represent the characteristics of the region; Image classification: Use convolutional neural networks to classify images of each region into several categories. Each category corresponds to a type of component, such as buttons, text boxes, icons, etc., which are used to represent the category of the region. Prediction probability: Using the output layer of the neural network, the probability that each region belongs to each category is calculated to represent the confidence level of the region; Component matching: According to the category and confidence of the area, as well as the pre-set threshold, the most suitable component is matched, as well as the reference of the corresponding component library and framework, such as Ant Design, React, etc., which are used to represent the components of the area.

8. The design draft restoration method based on computer vision, deep learning and code generation according to claim 4, characterized in that: The code conversion process includes the following steps: Annotation: Use annotation tools to annotate each area in the design draft and generate a D2CSchema, which contains information such as the boundaries, attributes, labels, parameters, rules, components, references, etc. of the area, and is used to represent the structure and content of the design draft. Code generation: Use code generation tools to generate front-end code based on D2C Schema, including HTML, CSS, JavaScript, etc., as well as references to corresponding component libraries and frameworks, such as Ant Design and React, to represent the structure and content of the front-end page; Code optimization: Use code optimization tools to optimize the generated front-end code, including compression, formatting, annotation, verification, etc., to improve the quality and performance of the code; Code adaptation: Use code adaptation tools to adapt the generated front-end code, including adaptation to different browsers, devices, resolutions, etc., to ensure the restoration and compatibility of the front-end page.

9. The design draft restoration method based on computer vision, deep learning and code generation according to claim 1, characterized in that: In step 1, the computer vision technology is implemented using OpenCV and TensorFlow libraries, in step 2, the deep learning technology is implemented using Keras and TensorBoard libraries, and in step 3, the code generation technology is implemented using Sketch, imgCook, Ant Design and React tools or frameworks.

10. The design draft restoration method based on computer vision, deep learning and code generation according to claim 1, characterized in that: The result display module in step 4 collects user feedback and evaluations as a data basis for subsequent optimization and improvement.

Images