Method and system for generating ai image recognition front-end code and computer storage medium

Abstract

The application provides an AI image recognition front-end code generation method and system and a computer storage medium. The generation method includes three main steps of image recognition, layout analysis and code generation. The original webpage prototype and its configuration information are processed by a convolutional neural network, an optical character recognition technology and a target detection algorithm to extract interface elements and determine the hierarchical structure. Then, attribute information extraction and rendering tree generation are performed. Finally, data tokenization processing is performed to map into HTML, CSS or JavaScript code, and a greedy search or clustering search strategy is flexibly selected according to user requirements to generate the final front-end code. The system supports multiple front-end frameworks, ensures efficient and accurate code generation, and allows user interaction to enhance the individuality and adaptability of code generation. The application relates to the field of artificial intelligence technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an AI image recognition front-end code generation method and system and a computer storage medium. BACKGROUND

[0002] With the development of the Internet and the improvement of user experience needs, front-end development plays an increasingly important role in software engineering. Traditionally, web design and development is a highly manual process, requiring designers to first create visual prototypes, and then front-end developers to convert these static images into interactive web codes (HTML, CSS and JavaScript). However, the inventors found some problems in the practice process. First, errors are easily introduced in the conversion process from design drafts to executable codes. Second, developers must have a deep understanding of the design intent to accurately achieve the expected effect, which often requires a lot of communication costs. Finally, in the face of rapidly changing design trends and technology stacks, traditional methods are difficult to respond efficiently.

[0003] In recent years, the progress of automation tools and AI technology has made it possible to simplify this process. Automated front-end code generation tools can reduce human intervention, improve efficiency and reduce the probability of errors. However, existing solutions are usually limited to simple layout and style generation, and lack support for complex interface elements and flexibility to adapt to diverse user needs and specific design styles. In addition, they often overlook the importance of human-machine collaboration and fail to fully utilize additional information provided by designers to guide the code generation process, resulting in inaccurate or unexpected generation results.

[0004] Therefore, the inventors designed an AI image recognition front-end code solution based on configuration to solve the above problems. SUMMARY

[0005] To solve the above problems, the present application provides an AI image recognition front-end code solution based on configuration, which aims to solve the problems of lack of support for complex interface elements, inaccurate generation results and inability to effectively utilize additional information provided by designers in the prior art.

[0006] To achieve the above purpose, the technical solution adopted by the present application is: an AI image recognition front-end code generation method, comprising the following steps:

[0007] S1: using a convolutional neural network for image recognition, the image recognition comprising the following steps:

[0008] S11: inputting a web prototype and its configuration information;

[0009] S12: preprocessing the prototype;

[0010] S13: Feature extraction is performed on the preprocessed prototype, interface elements are identified, a hierarchical structure is determined, and a preliminary hierarchical structure tree is generated;

[0011] S14: The classifier identifies and labels the user interface elements based on the text information on the prototype obtained through optical character recognition technology and the textual description in the input configuration information;

[0012] S15: A final preliminary hierarchical structure tree is generated based on the labeled user interface elements;

[0013] S2: Layout analysis, which includes the following steps:

[0014] S21: Boundary information of the prototype is detected using a target detection algorithm;

[0015] S22: Attribute information of the interface elements, including color, size, and position, is extracted;

[0016] S23: Through a loop search, it is ensured that all attributes are correctly extracted and processed, and steps S21 and S22 are repeated as necessary until completion;

[0017] S24: A final rendering tree is generated based on the extracted attribute information;

[0018] S3: Code generation, which includes the following steps:

[0019] S31: The data generated in steps S15 and S24 is tokenized by data preprocessing and dataset construction, and is mapped into HTML, CSS, or JavaScript code;

[0020] S32: The HTML, CSS, or JavaScript code is split and labeled according to syntax units to form a dataset pair of input and output sequences. In the decoder decoding process, greedy search or bundle search is used to compare the syntax labels generated by each sequence to gradually generate the front-end code label sequence;

[0021] S33: The generated code label sequence is spliced and parsed according to the syntax rules of the front-end code to obtain the final front-end code.

[0022] Based on the above, the beneficial effects of an AI-based configuration-based image recognition front-end code scheme are to solve the problems of lack of support for complex interface elements in existing front-end code automatic generation, inaccurate generation results, and inability to effectively utilize designer-provided auxiliary information. The main advantages are:

[0023] 1. This invention flexibly selects between greedy search and bundle search strategies during the decoder decoding process, depending on the clarity of the user's requirements. When the requirements are clear, greedy search can quickly generate basic sequences, thereby reducing computational costs. When the requirements are unclear, bundle search can maintain a candidate list of size k, select the k sequences with the highest probabilities to continue generating, and finally select the optimal solution from them. In summary, this not only speeds up the generation of basic sequences but also ensures high-quality and diverse outputs even when faced with ambiguous requirements, significantly improving the quality and flexibility of code generation. It also enables the effective use of auxiliary information provided by designers, improving the accuracy of the generated results.

[0024] 2. This invention uses convolutional neural networks for feature extraction and combines optical character recognition technology and target detection algorithms to accurately identify and process complex visual and textual information, thereby supporting the accurate parsing of more diverse and complex user interface elements;

[0025] 3. This invention significantly shortens the time cycle from design to implementation by automating the process of generating front-end code, reduces communication costs between designers and developers, and improves the efficiency of the entire development process. In particular, by using a smart search strategy of greedy search or cluster search, the system can generate code that meets the requirements more efficiently, further accelerating the development speed.

[0026] Furthermore, in step S11, the input of the webpage prototype image includes uploading a design draft or a temporary hand-drawn electronic draft. The input of configuration information includes text input control information and selection of a syntax template. If no syntax template is selected, the default template is automatically selected, thereby assisting in code generation and enhancing recognition accuracy and code generation accuracy.

[0027] Based on the above, by inputting a webpage prototype and configuration information, the system provides users with the ability to annotate control information and output descriptions on the prototype, thereby enhancing the system's recognition accuracy and ensuring that the final generated code is closer to the designer's original intent. This reduces errors caused by misunderstanding the design draft, enabling the system to more accurately understand the user's intent and generate front-end code that better meets expectations.

[0028] Furthermore, in step S15, it is determined whether the user has selected a syntax template. If yes, the process jumps to steps S21 and S22. If no, it continues to determine whether the user has marked control information on the prototype diagram. If no, the default syntax template is called, and it is re-determined whether the user has selected a syntax template. If yes, the database is searched to determine whether the control information is an unknown control. If yes, the default syntax template is called, and it is re-determined whether the user has selected a syntax template. If no, the process jumps to steps S21 and S22.

[0029] Further, in step S11, the input configuration information also includes user input descriptions to assist code generation, further customizing the generated code.

[0030] Based on the above, through user output descriptions, the system can adjust the code generation logic according to the user's special needs, generating more personalized and customized front-end code. For example, users can specify the behavior of certain components, style change rules, or responsive layout requirements through descriptions, making the generated code better adapt to the unique needs of the project. For special requirements or technical limitations in different projects, users can inform the system through descriptions, such as specific front-end frameworks, library usage habits, or company internal technical specifications, so that the generated code not only meets general standards but also meets special conditions in specific environments, enhancing the adaptability and practicality of the system.

[0031] Further, in step S32, when the system detects that the amount of information provided by the user meets the preset threshold and the complexity of the design draft is moderate, a greedy search is used to quickly generate the basic sequence; when the information amount is below the threshold or the design draft complexity is higher than the set standard, a beam search is used to increase the possibility and quality of the generated code, where the complexity is evaluated by a system-predefined algorithm.

[0032] Further, in step S32, the greedy search strategy selects the single token with the highest current probability at each iteration to construct the code token sequence; while the beam search strategy maintains a candidate list of fixed size k, where k is determined by the system's performance parameters and the user's specified accuracy requirements.

[0033] The present application also provides a system for implementing the AI image recognition front-end code generation method, comprising:

[0034] An image processing module for executing steps S11 and S12;

[0035] A feature extraction module for executing steps S13-S15;

[0036] A text analysis module for executing step S14;

[0037] A target detection module for executing steps S21 and S22;

[0038] A loop search module for executing step S23;

[0039] A rendering tree generation module for executing step S24;

[0040] And a code generation module for executing steps S31-S33.

[0041] Based on the above, each module is responsible for a specific task, has a clear functional boundary, so that the developer can focus on optimizing the performance of a single module without affecting other parts, when a certain function needs to be updated or repaired, only the corresponding module needs to be operated, reducing the maintenance cost of the whole system, thus, with the progress of technology, even if new algorithms and technologies can be easily introduced into the corresponding module, the whole system does not have to be completely reformed;

[0042] The modular design also divides different functions, so that even if a module fails, the fault range can be limited, and the normal operation of other modules is not affected, and the system can ensure basic functions when part of the components are abnormal, and the fault tolerance and stability of the system are improved.

[0043] Further, the AI image recognition front-end code generation method system further comprises a user interaction interface, which provides functions of uploading a prototype image, selecting a syntax template, inputting a prototype image resolution, adding additional annotations or descriptions, and viewing generated code.

[0044] Based on the above, the user interaction interface allows the designer to add annotations or descriptions directly when uploading the prototype image, which has the beneficial effect of being directly used to guide code generation, thereby greatly reducing the possibility of misunderstanding and rework; the user can select different syntax templates, which has the beneficial effect that the front-end code generated can meet the specific specifications according to the technical stack requirements of the project, and the generated code can be directly applied to the actual project, reducing the workload of subsequent modification.

[0045] Further, the AI image recognition front-end code generation method system supports multiple popular front-end frameworks and component libraries, including Element UI and Ant Design.

[0046] The application also provides a computer storage medium having instructions stored thereon, when the instructions are executed by a computer, the computer executes the AI image recognition front-end code generation method based on the configuration.

[0047] In order to make the above features of the application and the purposes to be achieved more clearly, the application will be further described below in conjunction with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0048] Fig. 1 : the front-end code step flow chart of the application;

[0049] Fig. 2 : the flow chart of the front-end code generation method of the application;

[0050] Fig. 3 : the module connection diagram of the front-end code generation method system of the application. DETAILED DESCRIPTION

[0051] Reference Figs. 1-3 ;

[0052] The embodiment discloses a method for generating AI image recognition front-end code, comprising the following steps:

[0053] S1: using a convolutional neural network for image recognition, which comprises the following steps:

[0054] S11: inputting a web prototype and its configuration information;

[0055] S12: preprocessing the prototype;

[0056] S13: feature extraction on the preprocessed prototype, interface element recognition, determination of the hierarchical structure, and generation of a preliminary hierarchical structure tree;

[0057] S14: the classifier recognizes and labels the user interface elements through the text information on the prototype obtained by the optical character recognition technology and the text description in the input configuration information;

[0058] S15: generating a final preliminary hierarchical structure tree according to the labeled user interface elements;

[0059] S2: layout analysis, which comprises the following steps:

[0060] S21: using a target detection algorithm to detect the boundary information of the prototype;

[0061] S22: extracting the attribute information of the interface elements, including color, size, and position;

[0062] S23: ensuring that all attributes are correctly extracted and processed through a loop search, and repeating steps S21 and S22 as necessary until completion;

[0063] S24: generating a final rendering tree according to the extracted attribute information;

[0064] S3: code generation, which comprises the following steps:

[0065] S31: tokenizing the data generated in steps S15 and S24 through data preprocessing and constructing a dataset, and mapping them into HTML, CSS, or JavaScript code;

[0066] S32: Split and mark the HTML, CSS or JavaScript code by syntax unit to form a dataset pair of input sequence and output sequence, and in the decoding process of the decoder, greedy search or bundle search is adopted to generate the front-end code mark sequence by comparing the syntax mark generated by each sequence;

[0067] S33: The generated code mark sequence is spliced and parsed according to the syntax rule of the front-end code to obtain the final front-end code.

[0068] In the embodiment, in step S11, the input mode of the web prototype includes uploading a design draft or a temporary hand-drawn electronic draft, and the input configuration information includes text input control information and a selected syntax template, and when no syntax template is selected, a default template is automatically selected, thereby assisting code generation and enhancing recognition accuracy and code generation accuracy.

[0069] In the embodiment, in step S15, it is judged whether the user has selected a syntax template, if yes, it is jumped to steps S21 and S22, if not, it is continuously judged whether the user has marked control information on the prototype, if not, a default syntax template is called, and it is re-judged whether the user has selected a syntax template, if yes, it is judged whether the control information is unknown control by searching the library, if yes, a default syntax template is called, and it is re-judged whether the user has selected a syntax template, if not, it is jumped to steps S21 and S22.

[0070] In the embodiment, in step S11, the input configuration information further includes user description through an input interface to assist code generation and further customize the generated code.

[0071] In the embodiment, in step S32, when the system detects that the information amount provided by the user meets a preset threshold and the complexity of the design draft is moderate, a greedy search is used to quickly generate a basic sequence, and when the information amount is lower than the threshold or the complexity of the design draft is higher than a set standard, a bundle search is used to increase the possibility and quality of the generated code, wherein the complexity is evaluated by a system pre-defined algorithm.

[0072] In the embodiment, in step S32, the greedy search strategy selects a single word with the highest current probability to construct a code mark sequence each time iteration, and the bundle search strategy maintains a candidate list with a fixed size k, wherein k is determined by the performance parameters of the system and the accuracy requirement specified by the user.

[0073] The application further discloses a system for implementing the front-end code generation method based on the configuration AI prototype recognition, and the system is characterized by comprising:

[0074] An image processing module is used to execute steps S11 and S12.

[0075] a feature extraction module for performing steps S13-S15;

[0076] a text parsing module for performing step S14;

[0077] a target detection module for performing steps S21 and S22;

[0078] a loop search module for performing step S23;

[0079] a render tree generation module for performing step S24;

[0080] and a code generation module for performing steps S31-S33.

[0081] In this embodiment, the system of the AI image recognition front-end code generation method further includes a user interaction interface, which provides functions of uploading a prototype image, selecting a syntax template, inputting a prototype image resolution, adding additional annotations or descriptions, and viewing generated code.

[0082] In this embodiment, the system of the AI image recognition front-end code generation method supports multiple popular front-end frameworks and component libraries, including Element UI and Ant Design, which facilitates project maintenance and iteration.

[0083] The application also discloses a computer storage medium having instructions stored thereon, which, when executed by a computer, cause the computer to execute the front-end code generation method based on the configured AI image recognition.

[0084] To sum up, the specific embodiments of the application are:

[0085] The preprocessing of the prototype image includes image scaling, cropping, padding, and normalization, specifically:

[0086] 1. The uploaded prototype image is uniformly scaled to 224x224 pixels;

[0087] 2. Cropping and padding: if the aspect ratio of the image is greater than 1 (i.e., the width is greater than the height), black padding bars can be added at the top and bottom to make the image height reach 224 pixels; if the aspect ratio is less than 1 (i.e., the height is greater than the width), black padding bars can be added on the left and right sides to make the image width reach 224 pixels, to ensure that the image maintains the correct aspect ratio and avoids recognition errors caused by distortion;

[0088] 3. Normalization: convert the pixel values of the image from 0-255 to floating-point numbers between 0-1, which is achieved by dividing by 255, to improve the convergence speed and performance of the model;

[0089] Feature extraction uses a convolutional neural network for VGGNet (Visual Geometry Group Net) to extract the features of the image. In image recognition, the image with clear labeled attributes is directly written into the code of the control; the one without clear labeled attributes is labeled with div, and then the style is extracted to achieve the same visual effect.

[0090] When extracting the attribute information of the interface element, if the control layout has no labeled attribute, the control is generated using the default attribute, wherein the recognized control structure is as follows:

[0091] {'type':'button',

[0092] 'text':'demo',

[0093] 'id':'demo',

[0094] 'class':'demo',

[0095] 'position':{'x':100,'y':100},

[0096] 'size':{'width':100,'height':50},

[0097] 'color':{'background':'#0000FF','text':'#FFFFFF'}

[0098] 'children':{} ....

[0100] }};

[0101] In steps S32 and S33, it should be understood that the input sequence in step S32 is input into the encoder of the trained Transformer model, and then on the decoder side, starting from the start token, according to the token probability distribution output by the model, the code token with high weight is selected in turn by using greedy search or beam search, the syntax token generated by each sequence is compared, and the front-end code token sequence is gradually generated, finally the token sequence is spliced and parsed according to the syntax rule of the front-end code to obtain the final front-end code, and then converted into a specific front-end framework syntax through a plug-in.

[0102] The above only describes the optimal solution embodiment of the present application, and is not used to limit the present application. Those skilled in the art can make various modifications or replacements to the present application without departing from the essence and protection scope of the present application, and such modifications or replacements should be within the protection scope of the present application.

Claims

1. A method for generating AI image recognition front-end code, characterized in that, Includes the following steps: S1: Image recognition is performed using a convolutional neural network, the image recognition comprising the following steps: S11: Input the webpage prototype and its configuration information; S12: Preprocess the prototype drawing; S13: Extract features from the preprocessed prototype image, identify interface elements, determine the hierarchical structure, and generate a preliminary hierarchical structure tree. S14: The classifier uses optical character recognition technology to identify and label user interface elements based on the text information on the prototype diagram and the text descriptions in the input configuration information. S15: Generate the final preliminary hierarchical structure tree based on the marked user interface elements; S2: Layout analysis, which includes the following steps: S21: Use an object detection algorithm to detect the boundary information of the prototype image; S22: Extract the attribute information of interface elements, including color, size, and position; S23: Ensure all attributes are correctly extracted and processed by looping through the search, repeating steps S21 and S22 as necessary until completion; S24: Generate the final rendering tree based on the extracted attribute information; S3: Code generation, which includes the following steps: S31: The data generated in steps S15 and S24 is tokenized by data preprocessing and dataset construction, and mapped into HTML, CSS or JavaScript code; S32: Split and label HTML, CSS, or JavaScript code according to syntactic units to form a dataset pair of input and output sequences. Input the input sequence into the encoder of the trained Transformer model. Then, at the decoder, starting from the initial label, according to the label probability distribution output by the model, greedy search or bundle search is used to select code labels with high weights in turn. By comparing the syntactic labels generated by each sequence, the front-end code label sequence is generated step by step. Specifically, when the system detects that the amount of information provided by the user meets the preset threshold and the complexity of the prototype is moderate, a greedy search is used to quickly generate the basic sequence; when the amount of information is lower than the threshold or the complexity of the design draft is higher than the set standard, a cluster search is used to increase the probability and quality of code generation, so that the result is not unique and unchanging each time the user uploads the same prototype. S33: The generated code tag sequence is concatenated and parsed according to the syntax rules of the front-end code to obtain the final front-end code.

2. The method for generating AI image recognition front-end code according to claim 1, characterized in that, In step S11, the input of the web page prototype image includes uploading a design draft or a temporary hand-drawn electronic draft. The input of configuration information includes text input control information and selection of a syntax template. If no syntax template is selected, the default template is automatically selected, thereby assisting in code generation and enhancing recognition accuracy and code generation accuracy.

3. The method for generating AI image recognition front-end code according to claim 2, characterized in that, In step S15, it is determined whether the user has selected a syntax template. If yes, the process jumps to steps S21 and S22. If no, it continues to determine whether the user has marked control information on the prototype diagram. If no, the default syntax template is called, and it is determined again whether the user has selected a syntax template. If yes, the library is searched to determine whether the control information is an unknown control. If yes, the default syntax template is called, and it is determined again whether the user has selected a syntax template. If no, the process jumps to steps S21 and S22.

4. The method for generating AI image recognition front-end code according to claim 3, characterized in that, In step S11, the input configuration information also includes user descriptions through the input interface to assist in code generation, further customizing the generated code.

5. The method for generating AI image recognition front-end code according to claim 4, characterized in that, In step S32, the greedy search strategy selects the single word with the highest probability at each iteration to construct the code tag sequence; while the cluster search strategy maintains a candidate list of fixed size k, where k is determined by the system's performance parameters and the accuracy requirements specified by the user.

6. A system for generating AI image recognition front-end code as described in claim 5, characterized in that, The system includes: The image processing module is used to perform steps S11 and S12; The feature extraction module is used to perform steps S13-S15; The text parsing module is used to execute step S14; The target detection module is used to perform steps S21 and S22; The loop search module is used to execute step S23; The rendering tree generation module is used to execute step S24; And a code generation module, used to execute steps S31-S33.

7. The system according to claim 6, characterized in that, The system for generating AI image recognition front-end code also includes a user interface that allows users to upload prototype images, select syntax templates, input prototype image resolution, add additional comments or descriptions, and view the generated code.

8. The system according to claim 7, characterized in that, The system for generating AI image recognition front-end code supports a variety of popular front-end frameworks and component libraries, including Element UI and Ant Design.

9. A computer storage medium storing instructions thereon, which, when executed by a computer, cause the computer to perform the method for generating AI image recognition front-end code as described in any one of claims 1 to 5.

Images