Method, system and equipment for automatically generating user interface code and medium

Abstract

The invention provides a method, a system, equipment and a medium for automatically generating mobile application codes based on user graphical interface design. According to the technical scheme provided by the invention, the mobile application code can be automatically generated based on the graphical user interface design by using hierarchical data structure conversion and generative artificial intelligence with a large language model. According to the scheme provided by the invention, the graphic user interface design is firstly converted into a layered data structure (such as a DOM tree structure), and then is further converted into a vector; the vector is compared to a pre-vectorized user interface design and its code in a database using a similarity analysis algorithm to find user interface design source code with similar features. Based on the best match, the graphical user interface design source code is generated using generative artificial intelligence and output to the user. And before the data is vectorized, structural information in the original graphic user interface design is reserved by using a hierarchical structure, so that the accuracy of the finally generated code is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a method, system, device and computer-readable medium for automatically generating code for rendering and presenting a graphical user interface based on machine learning. Background Art

[0002] Generating source code for a Graphical User Interface (GUI) is an important part of the software (e.g., mobile application) development process. In this process, software developers need to write relevant code based on the designer's design of the GUI, so as to create a user interface (UI) that not only reflects the designer's design of the GUI but also realizes the functionality of the GUI. The traditional development method is that the front-end developer first designs the GUI, then manually writes the front-end code and performs iterative testing until the code is written. Although code modularization can be used to improve efficiency in this process, modular components are often limited in functionality and require a lot of customization work based on actual conditions.

[0003] There are many other problems in manually converting GUI designs into relevant codes. For example, the manual writing process is inefficient and susceptible to human errors, which delays the progress of software development. In addition, the traditional software development process requires engineers or developers to code the elements of the user interface one by one, resulting in a lot of repetitive work in the coding process. For example, a certain interface design may contain parts that share certain design elements, so a lot of repetitive work will be generated in the process of manually coding it. Especially in some large organizations, there are often a large number of similar codes for the same or similar design elements stored in branches in various places. Failure to reuse these codes well often leads to a lot of waste of human and development resources.

[0004] Another challenge faced by researchers in the process of developing automatic code conversion is how to ensure the accuracy of code conversion. Conventional methods often lose structural information (e.g., component information, functional features, etc.) in the original GUI design, resulting in a decrease in the accuracy of the final GUI presentation.

[0005] Therefore, in the GUI design process, a technical solution is needed that can automatically reuse existing codes to minimize the manual repetitive labor used in the process of converting GUI designs into codes, improve the efficiency of the code writing process, reduce the rate of manual errors, and retain structured information as much as possible to ensure code accuracy. Summary of the invention

[0006] In order to overcome the problems existing in the related art, the present invention provides a method, system, computer device and computer-readable storage medium for automatically generating rendering user interface code based on machine learning. The method can use a large language model (LLM) to deduce usable graphical user interface rendering code from design elements in a static interface design, and then automatically deploy and apply it to various types of user interfaces.

[0007] In one aspect, the present invention provides a method for automatically generating application code based on a graphical user interface design, comprising: receiving an original data object representing a graphical user interface design; converting the original data object into a target hierarchical data object, wherein the hierarchical data object retains the structural information of the original data object; using a hash function to convert the target hierarchical data object into a target multidimensional vector, wherein the target multidimensional vector retains the structural information of the hierarchical data object; using a keyword search method to identify one or more groups of similar code pairs in a database based on the target hierarchical data object, the database including a plurality of pre-input code pairs in the form of multidimensional vectors; calculating the cosine similarity between the target multidimensional vector and each group of the similar code pairs to obtain a cosine similarity score of each group of the similar code pairs relative to the multidimensional vector, thereby determining an optimal similar code pair, wherein the optimal similar code pair has the highest cosine similarity score; constructing a prompt based on the optimal similar code pair; inputting the prompt and the original data object into a large language model for context learning and generating a text instruction set based on multiple context clues of the original data object; and outputting the generated text instruction set.

[0008] According to a further embodiment of the present invention, the method further includes packaging the output text instruction set, generating a machine language instruction set, and outputting the machine language instruction set.

[0009] According to a further embodiment of the present invention, the method further comprises linking the machine language instruction set to an executable binary file, wherein the executable binary file is a collection of the machine language instruction set; and outputting the executable binary file.

[0010] According to a further embodiment of the present invention, the method further comprises running the executable binary file to render the graphical user interface; and outputting the rendered graphical user interface.

[0011] According to a further embodiment of the present invention, the method further includes setting a first threshold of the highest cosine similarity score; determining whether the highest cosine similarity score is greater than or equal to the first threshold; if the highest cosine similarity score is greater than or equal to the first threshold, transmitting the generated text instruction set to the development test environment for packaging and execution.

[0012] According to a further embodiment of the present invention, the method further includes setting a second threshold of the highest cosine similarity score, which is higher than the above-mentioned first threshold; determining whether the highest cosine similarity score is greater than or equal to the second threshold; if the highest cosine similarity score is greater than or equal to the second threshold, deploying the generated instruction set to one or more servers accessible to end users.

[0013] According to a further embodiment of the present invention, the keyword search ends when a preset number of similar code pairs are identified.

[0014] According to a further embodiment of the present invention, the best similar code pair is manually selected from the one or more groups of similar code pairs.

[0015] According to a further embodiment of the present invention, the method further includes displaying the generated text instruction set to a manual auditor for review, and determining whether to add the generated text instruction set to a database; after the manual auditor's review is passed, a code pair corresponding to the generated text instruction set is created, and the code pair is added to the database.

[0016] According to a further embodiment of the present invention, the hierarchical data object is a text object model, which includes a tree structure composed of a plurality of nested nodes, wherein each nested node represents a data element of the original data object.

[0017] The present invention provides a system for automatically generating application code based on a graphical user interface design, comprising: a data receiving module configured to receive an original data object representing a graphical user interface design; a hierarchical data conversion module configured to convert the original data object into a target hierarchical data object, wherein the hierarchical data object retains the structural information of the original data object; a vector conversion module configured to use a hash function to convert the target hierarchical data object into a target multidimensional vector, wherein the target multidimensional vector retains the structural information of the hierarchical data object; a keyword retrieval module configured to use a keyword search method to identify one or more groups of similar code pairs in a database based on the target hierarchical data object, wherein the database The system comprises a plurality of pre-input code pairs, wherein the code pairs are in the form of multi-dimensional vectors; a similarity calculation module, configured to calculate the cosine similarity between the target multi-dimensional vector and each group of the similar code pairs to obtain the cosine similarity score of each group of the similar code pairs relative to the multi-dimensional vector, thereby determining the optimal similar code pair, wherein the optimal similar code pair has the highest cosine similarity score; a prompt construction module, configured to construct a prompt based on the optimal similar code pair; a large language model module, configured to input the prompt and the original data object into a large language model for context learning and generate a text instruction set based on multiple context clues of the original data object; and an output module, configured to output the generated text instruction set.

[0018] In some further embodiments, the hierarchical data object is a text object model, and the text object model includes a tree structure composed of a plurality of nested nodes, wherein each nested node represents a data element of the original data object. In addition, the optimal similar code pair can also be selected from the one or more groups of similar code pairs manually.

[0019] According to a further embodiment of the present invention, the output module can be further configured to encapsulate the output text instruction set to generate a machine language instruction set; and output the machine language instruction set. Moreover, the output module can be further configured to link the machine language instruction set to an executable binary file, wherein the executable binary file is a collection of machine language instruction sets; and the executable binary file.

[0020] According to a further embodiment of the present invention, the system may further include a running module configured to run the above executable binary file to render a graphical user interface; and output the rendered graphical user interface.

[0021] According to a further embodiment of the present invention, the similarity calculation module can be further configured to set a first threshold value of the highest cosine similarity score; determine whether the highest cosine similarity score is greater than or equal to the first threshold value; if the highest cosine similarity score is greater than or equal to the first threshold value, the generated text instruction set is transmitted to the development test environment for packaging and execution. It can be further configured to set a second threshold value of the highest cosine similarity score, which is higher than the above-mentioned first threshold value; determine whether the highest cosine similarity score is greater than or equal to the second threshold value; if the highest cosine similarity score is greater than or equal to the second threshold value, the generated instruction set is deployed to one or more servers accessible to end users.

[0022] According to a further embodiment of the present invention, the keyword search module ends the keyword search when a preset number of similar code pairs are identified.

[0023] According to a further embodiment of the present invention, the output module displays the generated text instruction set to the manual auditor for review, and determines whether to add the generated text instruction set to the database; after the manual auditor passes the review, a code pair corresponding to the generated text instruction set is created, and the code pair is added to the database.

[0024] On the other hand, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method of automatically generating application code based on graphical user interface design when executing the computer program.

[0025] In another aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for automatically generating application codes based on a graphical user interface design.

[0026] The present invention solves the problem of long time consumption and high error rate caused by manually converting the graphical user interface design into source code, which in turn delays the progress of development iterations. A challenge encountered in the development of the technical solution of the present invention is how to accurately capture the characteristics of the graphical user interface, including structural information, and generate related interactive components, as well as how to handle large-scale scaling adaptation problems. The mismatch between design and functional implementation will lead to increased development time and low accuracy of automatic code generation. In addition, due to the inconsistency of user interfaces on different platforms, the scalability and adaptability of mobile application development are also limited.

[0027] The improved technical solution proposed in the present invention can use a large language model to automatically generate mobile application code from a graphical user interface (GUI) design. This is an automatic process that converts the graphical user interface design into a hierarchical data structure (such as a document object model (DOM) tree structure) and then into a vector, which can be analyzed by a large language model for comparison with a database formed by other user interfaces and codes, etc., so as to find user interface designs and codes with similar features using a similarity search algorithm. Based on the best matching code snippet, the large language model can use generative artificial intelligence to generate code for similar user interface elements and then output it to the user. Using a hierarchical data structure (such as a DOM tree) before vectorization can retain the structural information of the original graphical user interface design input, thereby improving the accuracy of the converted graphical user interface and its retained structural information, thereby solving the above-mentioned challenges.

[0028] Some embodiments of the technical solution provided by the present invention may include, in practice, a design platform plug-in that converts a static design of a graphical user interface (whose functional features may be represented by static placeholders) into a user interface specification in JSON format, and a database that stores the mapping relationship between such specification and the corresponding mobile code. The database may store the code pairing or mapping relationship between such specification and the corresponding mobile code, and may also perform similarity searches to identify and call code snippets with user interface elements of similar designs, thereby enhancing the accuracy and relevance of the generated code. The system and method use a similarity search algorithm in the database to find design pages with similar features, and employ a generative artificial intelligence engine configured to automatically generate functional mobile code by adapting and modifying baseline code obtained from a code library. The present invention greatly improves the efficiency and accuracy of code generation, and achieves a seamless connection from graphical user interface design to code development, and even production environment.

[0029] Furthermore, some embodiments of the technical solution provided by the present invention use enhanced generative artificial intelligence to automatically generate mobile application code based on the identified similar designs, ensuring that the final generated code not only implements the function but also accurately reflects the design intent. At the same time, continuous data collection enables persistent updating and improvement of the database, allowing the system and method of the present invention to adapt to various page designs and continuously improve its ability to generate universal and accurate mobile code. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of an example method for automatically generating source code for a graphical user interface design according to an aspect of the present invention.

[0031] Figure 2is a block diagram of an example generative artificial intelligence system for automatically generating source code for a graphical user interface design according to some embodiments of the present invention.

[0032] Figure 3 is a flow chart of an example keyword search module for identifying similar code pairs according to some embodiments of the present invention.

[0033] Figure 4 is a flow chart of an example similarity comparison module for performing similarity comparisons between graphical user interface designs and similar code pairs according to some embodiments of the present invention.

[0034] Figure 5 is a schematic diagram of example code pairs according to some embodiments of the present invention.

[0035] Figure 6 is a flowchart of an example large language model for training based on code pairs and generating source code according to some embodiments of the present invention.

[0036] Figure 7 is a schematic diagram of example design input and example intermediate JSON data structure output according to some embodiments of the present invention.

[0037] Figure 8 is an example prompt according to some embodiments of the present invention, which has a portion where content can be inserted, and the inserted content is inserted by the generative user interface code system provided by the present invention for execution.

[0038] Fig. 9 is an architectural diagram of an exemplary system for automatically generating source code for a graphical user interface design according to an aspect of the present invention.

[0039] Fig.10 It is a schematic diagram of the structure of a computer device provided according to one aspect of the present invention. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the embodiments shown in the accompanying drawings are only examples, and the present invention can be implemented in various forms in practical applications and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0041] As basic preparation, some embodiments of the technical solution provided by the present invention are configured to prepare JSON format data structure-code pairing combinations (hereinafter referred to as "code pairs") and store them in a database for context learning by a large language model. At the beginning of the automatic encoding process, the system receives the original graphical user interface page and design, extracts the user interface specifications and structural information therein, and then converts them into a data structure in JSON format. The data structure in JSON format is then converted into a hierarchical tree structure, such as a document object model (DOM) tree structure, so that the structural features and layout information in the original graphical user interface page can be retained before being vectorized and cosine similarity calculation is performed through a hash function. The JSON format data structure is then embedded and vectorized for similarity analysis with the vector database.

[0042] In some embodiments, the JSON format data structure can reduce the token size and input load by using computational operations. For example, some embodiments may use the Figma plug-in API to recursively traverse the JSON format data structure, but extracting all data may result in the generation of a large amount of irrelevant information. In order to overcome this problem, some embodiments of the technical solution provided by the present invention can identify the corresponding user interface element type for each user interface node or component in the development environment, so that only the necessary information related to the specific element type can be extracted. For example, if the user interface node is identified as a text element, only its size, text content, text color, and font information will be extracted, and other unnecessary information will be filtered out. In some embodiments, designers will use drawing nodes (such as lines and rectangles) to create combined graphics. These drawing nodes can be generally identified as representing a single graphic node. Therefore, this method ultimately outputs only one graphic information entry in JSON, rather than outputting an entry for each node, thereby forming multiple entries. This method not only reduces the use of tags, but also improves the effectiveness of the conversion process.

[0043] Some embodiments of the technical solution provided by the present invention are configured to use keywords to search the above-mentioned database for user interface pages having components similar to the inputted graphical user interface page and perform cosine similarity calculation to generate a similarity score. The system and method further use the structural information retained in the DOM tree structure to perform DOM tree similarity calculation to identify similar graphical user interface pages based on page layout and component attributes. The components can be identified and extracted from the graphical user interface page, including but not limited to, for example, text boxes, navigation bars, cases, home page indicators, etc. Keyword retrieval in the vector database can be based on the identified components, thereby filtering out results that do not contain the target components.

[0044] Figure 1is a flow diagram of an example system for automatically generating source code for a graphical user interface design according to some embodiments of the present invention.

[0045] The system 100 includes a specific workflow for automatically generating mobile application code from a graphical user interface design using a generative artificial intelligence system. First, the system 100 receives a graphical user interface design page 102 provided by a user. The graphical user interface design page 102 design can be in the form of various data objects used by designers, including wireframe data objects, rendering data objects, and other various data object formats that can be applied to graphical interface design applications, such as bitmaps, data files, or vector files, which can be rendered and compiled into data objects. When designers design, they can use various tools for creative design, such as using drawing tools provided by certain design platforms to design related pages, and the data objects designed can also be various data objects including vector graphics. In these static designs, static placeholders can be used to indicate the control elements of the dynamic interactive user interface. For example, the graphical user interface design can be a Visio file. In some embodiments, the relevant graphical user interface design can be designed for mobile applications, such as being compatible with Android or iOS devices.

[0046] The system 100 extracts the GUI design specification from the GUI design page 102 and encodes it into a JSON format data structure 104. The JSON format data structure 104 includes key elements of the subsequent code generation process, such as page layout details, color schemes, and interactive component descriptions.

[0047] The JSON format data structure 104 is then input into the generative artificial intelligence system 106 for processing to synthesize a mobile application code 108 that is consistent with the target graphical user interface page design in syntax and semantics. The mobile application code 108 can be adapted for testing or design and rendered on a user interface rendering engine running on a computing device (e.g., a mobile device). The code 108 can also be provided to a hosting device (e.g., a network server) that dynamically generates an output interface using a network development script engine (e.g., a PHP interpreter). These hosting devices (e.g., network servers running in a model-view-controller architecture) serve the requesting computing device and render a graphical user interface on it. In this case, the generated code 108 can be a PHP code processed on a network server, and when the requesting computing device interacts with it, the network server provides the generated hypertext markup language (HTML) or other binary image data as a response to the hypertext transfer protocol to the requesting computing device for subsequent operations.

[0048] The generated source code 108 can then be packaged into a machine language instruction set and further linked to or incorporated into an executable binary file. Once the generated source code 108 is compiled and executed, the corresponding graphical user interface design can be rendered for manual review, or the system 100 can automatically apply the generated code in a development or production environment. This automatic process greatly improves the speed and accuracy of mobile application development, seamlessly bridging the conversion of graphical user interface design from design concept to deployable code.

[0049] In the technical solution provided by the present invention, the graphical user interface design data object is converted into a hierarchical data structure (e.g., a DOM tree structure) to obtain a related structural representation, thereby preparing for subsequent machine learning. Subsequently, the hierarchical data structure is converted into a vector in preparation for using a similarity search method to compare in a database including other user interface code fragments to find user interface designs and corresponding codes with similar features. Subsequently, based on the best matching code fragment, the large language model can use generative artificial intelligence to automatically generate code for generating related user interface elements, and then output the code to the user. In the generation process, the generated code can be adapted based on the characteristics and limitations of the actual device and rendering engine. The large language model uses the baseline code identified from many code fragments in the database, combined with the above-mentioned vector and / or other input instruction sets to generate new code that can be deployed immediately for testing or designing a production environment. The new code can be in the form of a UI rendering code executable by a client-side UI engine (e.g., a mobile application on a user's mobile phone), or a UI rendering code executable by a server-side UI engine (e.g., a network server that responds to HTTP requests with HTML code for rendering on a client-side UI engine).

[0050] Figure 2 is a block diagram of an example generative artificial intelligence system for automatically generating source code for a graphical user interface design according to some embodiments of the present invention.

[0051] exist Figure 2In the illustrated embodiment, the system 200 includes the architecture of a generative artificial intelligence system used in converting a graphical user interface design into executable mobile code. After receiving a graphical user interface page 102 provided by a user, the system 100 extracts the graphical user interface design specification from the graphical user interface page 102 and encodes it into a data structure 104 in JSON format. In some embodiments, when the designer uses an existing object or a variant thereof in an object library, the graphical user interface specification may include a pointer to a design markup data object. These existing user interface objects may be some objects that have been reviewed, and these objects may be regularly updated as the company's strategy for brand image management changes, and these objects are different from the related elements designed independently by the designer.

[0052] Such design markup data objects can have a limited set of configurable parameters, such as size, orientation, color, etc. In a later machine learning step, corresponding pointers can be included in the code. In this way, the generated code can also be automatically updated regularly as the related objects are updated.

[0053] The system 200 then inputs the JSON format data structure 104 into the keyword search module 300. The system 200 includes a keyword search module 300 for extracting the functional component type from the input JSON format data structure 104, and searching a limited number of similar code pair groups in a pre-established database 202. Preferably, in a further embodiment, the number of similar code pair groups to be searched can be preset, and the keyword search ends when a preset number of similar code pairs are identified to control the number of similar code pair groups and prevent waste of computing resources. The database 202 includes various code pairs. The database 202 first performs a keyword extraction operation on the code pair 204 to be added thereto to generate a keyword vector containing a keyword list for performing a similarity calculation operation, and then stores each code pair 204 together with its keyword list therein.

[0054] The reason for using the keyword search solution is that the technical solution provided by the present invention does not directly compare the code, but performs similarity calculation based on the keyword vector. For example, the keyword can be ['FRAME', 'navigation-bar-top', 'input-field-standard', 'TEXT', 'primary-button', 'home-indicator']. The code pairs 204 used to fill the database 202 are from the existing code base of the previous user interface design project and / or the source code generated and output by the system 200. These codes may have been approved and have been used in other usage scenarios in practice. The benefit of this method is that for large organizations (for example, corporate groups with a large number of user interface developers), those codes that have been written before and passed brand public relations audits and network security inspections can be considered as candidate baseline codes for subsequent code development, thereby at least saving the repetitive labor of repeated audits and inspections. This advantage is particularly evident compared to the case of non-customized or non-optimized codes generated by generative artificial intelligence engines without baseline codes.

[0055] The system 200 then executes a keyword search algorithm to find similar code pairs 206 for the JSON format data structure 104 (ie, the graphical user interface design) (the concept of code pairs is described in conjunction with Figure 5 ), each set of search results should cover all component types included in the graphical user interface page 102.

[0056] The database 202 can be a central data repository or a distributed data repository. When developers in different locations use related codes in a test or production design environment, the reviewed user interface code can be automatically called as part of the automatic migration process. In addition, the user interface code in the database 202 can be associated with corresponding metadata tags (e.g., device type, font size, accessibility options, day / night mode, etc.), and can be converted into a vector representation by a method similar to the vector conversion method described above. As described in some embodiments below, the database 202 also generates a natural language description for a specific code snippet through a large language model and generative artificial intelligence. Cosine similarity analysis is used to compare with the stored vectors, stored descriptions, or associated metadata tags to select candidate baseline codes (or slightly different candidate codes) from the code base.

[0057] The benefit of using the code pairs 204 is that experiments show that there are approximately 30-50 code pairs for each component, which is a relatively suitable search amount for determining relevant code snippets for comparison.

[0058] The system 200 uses a document object model (DOM) tree similarity module 400 to construct a document object model (DOM) tree structure corresponding to the JSON format data structure 104 and the selected similar code pair group 204, and then converts the tree structure into a vector. Since the hierarchical structure is included in the hierarchical data structure conversion, the vector converted by the system 200 includes the web page structure information. The logical node objects of the tree structure identify the various parts of the user interface. Therefore, the web page structure information retains the structural features of the design data object in the tree structure. Accordingly, the structural information (e.g., interactive function information, etc.) in the input graphical user interface page 102 is retained using the hierarchical structure (e.g., the DOM tree structure here) before the vectorization step, thereby increasing the accuracy of the final code presented.

[0059] In some embodiments, the JSON format data structure can be processed using a computation operation to reduce the token size and input load. This operation groups specific design tuples together into a single design node to reduce the overall token size. This is particularly useful when using a generative model with limited token size. TM When the Figma user interface tree is recursively traversed using the plug-in API, extracting all the data may result in a large amount of irrelevant information. Accordingly, if such a large amount of information is used directly for node conversion, a large amount of markup of the language model will be consumed and the quality of the final generated conversion will be negatively affected. In order to solve this problem, the technical solution of the present invention may include, in the development environment, identifying the corresponding user interface element type for each user interface node or component. In this way, the system can extract only the necessary information related to the specific element type. For example, if the system identifies a Figma user interface node as a text element, the system can be configured to extract only the size, text content, text color, and font information, while filtering out other unnecessary information.

[0060] In addition, designers sometimes often use several drawing nodes to create a combined graphic, such as using a line and a rectangle to create a specific graphic. At this time, the system identifies these drawing nodes as representing a single graphic node. Finally, during the initial JSON data structure generation process, the system output only includes a total of one graphic information entry in the JSON, rather than including one entry for each drawing node and generating multiple entries. This approach not only reduces the use of tags, but also improves the effect of the conversion process.

[0061] The reason why the DOM tree structure is taken as an important step is that, technically speaking, structural information is easier to be preserved in the process of generating nodes, which is conducive to maintaining the overall structure of the generated user interface code consistent with the initial input example structure.

[0062] In some embodiments, the DOM tree similarity module 400 then determines the similarity between the vectors by calculating the cosine similarity, and finds the best code pair 208 with the highest similarity score to the JSON format data structure. The best code pair 208 is then used as the context input of the large language model 210. The large language model 210 uses enhanced machine learning technology to generate the mobile application code 108 by integrating the context clues derived from the best code pair 208. This ensures a high degree of relevance and functional fidelity with the design of the graphical user interface page 102. The context clues derived from the code pair include information about the underlying rules of the code conversion. For example, if the code pair comes from an enterprise-level database, the context clues contained in the code pair can be the specific code implementation and requirements of the enterprise. Large enterprises often have a large number of rules that are difficult to list exhaustively, so using a machine learning architecture to learn relevant context clues and rules without explicitly defining certain rules is often a more efficient way. The inventor verified the effectiveness of this approach in initial experiments.

[0063] Since in actual applications, it is unlikely that there will be a complete match between the JSON format data structure 104 (i.e., the input original graphical user interface design) and the best code pair 208. Therefore, the system 200 can also be configured to generate an input data object based on the desired user interface design data object (original format or vectorized form, or a combination of a description generated by a large language model and generative artificial intelligence) and the best code pair 208. The input data object includes a computerized representation of the desired user interface design data object and the best code pair 208 based on the candidate baseline code fragment, as well as any relevant metadata tags and prompt parameters (e.g., the desired output device, device resolution, barrier-free options, etc.) or hyperparameters (e.g., noise / entropy level, etc.) and other information. The system then uses reinforcement learning technology and generative artificial intelligence to use the above input data as a prompt to generate a deployable user interface rendering code package including the mobile application code 108, which can be deployed to a test or production design environment in real time. Among them, the candidate baseline code fragment is automatically adapted and modified according to the target user interface design data object. The deployable user interface rendering code package can be upgraded through a package manager or pushed as an application update. For example, when a mobile application of an online system of a banking platform is updated, the updated user interface rendering code can be received and automatically upgraded on the client-side device.

[0064] This approach is particularly useful when a designer designs only for a test device, but the same code needs to be adapted and deployed to various other types of devices. For example, the test device has a 5*7 inch 1080p display, the latest operating system, and no accessibility options enabled, but the screens of other devices are 4.5*6 inch 720p displays, the operating system is not updated, and they are in night mode, etc. The same design object can be used as input to the system of the present invention, with different configuration parameters, so as to identify the best code pair 208 from the database 202, and then generate code that can be deployed immediately. In some embodiments, the generated code 108 can also be applied in the development environment of mobile or web applications, including supporting frameworks such as REACT.

[0065] In terms of constructing prompts for input to large language models and generative AI. Different forms of prompts can be used based on different programming languages. Furthermore, since there can be countless combinations between the components and other components of the user interface to present the final user interface page design, using large language models and generative AI to generate user interface page source code is far more efficient than manual methods.

[0066] Traditionally, the way of matching design components to design source code based on fixed rules is not flexible enough and requires a lot of time to build a mapping based on fixed rules. In contrast, the ability to generate code using contextual learning is more applicable. Generative artificial intelligence can generate source code by using code snippets that highly match code pairs as prompts, and then further edit the source code to generate the desired user interface design. In some embodiments, the output code can be further reviewed by human engineers as candidate code pairs that are input back into the database.

[0067] In some embodiments, prompts input to large language models and generative AI can be created using external automatic prompt generation tools that can generate different prompts for different scenarios and models.

[0068] Figure 3 is a flow chart of an example keyword search module for identifying similar code pairs according to some embodiments of the present invention.

[0069] The keyword search module 300 uses two steps to find all possible code pair sets in the database 202 for comparison in the DOM tree similarity module 400. First, the keyword search module 300 performs a keyword extraction step 302 to isolate the functional component types of the JSON format data structure 104 and reorganize them into a list format. In this step, the technical solution of the present invention includes building a DOM tree for the JSON format data structure 104, wherein each node includes a component type and its attributes, the former as a label, and the latter as data. Subsequently, a greedy algorithm 304 (Greedy Algorithm, for example, by solving a set covering problem) is used to find a limited number of code pair groups 206 from the database 202, which cover all component types in the functional component type list input extracted from the JSON format data structure 104.

[0070] In one embodiment, the keyword search module 300 can use a large language model and generative artificial intelligence to generate an intermediate natural language output based on the JSON format data structure 104 to describe all functional types of the JSON format data structure 104. At the same time, the large language model and generative artificial intelligence are also used to generate a natural language description for the JSON format data structure corresponding to each code pair group before each code pair group is added to the database 202 as a keyword extraction operation. The intermediate natural language description of the JSON format data structure 104 is then used to compare with the natural language description of each code pair in the database. The keyword search module 300 uses a greedy algorithm 304 to find a limited code pair group 206 from the database 202 as an output.

[0071] The output of the keyword search module 300 is used as the input of the subsequent DOM tree similarity module 400. The keyword search module 300 filters out components related to the original GUI page components from various components and can maintain consistency with the original GUI design, ensuring a comprehensive and accurate match.

[0072] Figure 4 is a flow chart of an example similarity comparison module for performing a similarity comparison between graphical user interface designs and similar code pairs according to some embodiments of the present invention.

[0073] The process of the DOM tree similar module 400 finding the best or optimal code pair group from the limited number of code pair groups 206 output by the keyword search module 300 is divided into three steps. First, the DOM tree similar module 400 respectively establishes a DOM tree data structure 402 based on the input original JSON format data structure 104 and the identified similar code pair group 206. The DOM tree data structure 402 contains many nodes, which include component types and their attributes, the former as labels and the latter as data. The DOM tree structure 402 retains the layout and structure information in the original graphical user interface and the similar code pair group 206. Among them, the step of establishing the DOM tree data structure for the similar code pair group 206 can be operated when establishing the database or in parallel with the establishment of the DOM tree data structure for the original JSON format data structure 104. Subsequently, the DOM tree similar module 400 uses a hash function to convert all DOM tree data structures into a multidimensional vector 404, so that the cosine similarity calculation between the target graphical user interface design and the identified code pair group 206 can be continued. The cosine similarity calculation of the multidimensional vector corresponding to each identified code pair and the vector corresponding to the data structure 104 in JSON format will obtain the final cosine similarity score 406. The best code pair 208 with the highest cosine similarity score with the target design is used as input and prompt for the large language model and generative artificial intelligence to learn and generate code. The DOM tree similar module 400 ensures that the nested structure and component attributes between the identified code pair and the target design have the greatest similarity, and is used to learn and train the large language model.

[0074] In some embodiments, some thresholds of cosine similarity scores may be preset, and different measures may be taken on the finally obtained application code 108 according to the different situations of the cosine similarity scores of the best code pair 208 actually calculated relative to the thresholds.

[0075] For example, a first threshold value may be set, and when the cosine similarity score of the best code pair 208 calculated actually reaches or exceeds the first threshold value, the generated application code 108 is transmitted to the development and testing environment for packaging and execution. When the cosine similarity score of the best code pair 208 does not reach the first threshold value, the system may prompt the user that the system is unable to generate source code for the input target design, and the user needs to write the source code by himself. Furthermore, a second threshold value higher than the first threshold value may be set, and if the cosine similarity score of the best code pair 208 calculated actually reaches or exceeds the second threshold value, the generated application code 108 may be directly deployed to a server accessed by one or more terminal users, and directly provided to the terminal customer for user interface rendering.

[0076] Figure 5 is a schematic diagram of example codes according to some embodiments of the present invention.

[0077] Figure 5 The structure of the code pair used in the systems 100 and 200 and how to use the code pair in the present invention are shown. Figure 5 As shown, each set of code pairs 502 ... 504, etc. corresponds a user interface page or a specific graphical user interface element 502a ... 504a therein (defined in the JSON format user interface specification) to a corresponding code 502b ... 504b prepared by a developer. These code pairs 502 ... 504 form a basic database 202, and the large language model 210 can access the basic database 202 to learn and apply the code encoding standards and rules in the development process. The maintainer expands the database by continuously adding new code pairs to the basic database 202. Through the expansion of the database and the learning of the large language model, the system 100 and 200 can continuously enhance its ability to generate accurate functional mobile code that can simultaneously meet the visual and functional requirements of the original target graphical user interface design 102. In some embodiments, the new code pairs can be reviewed by relevant professionals to determine which code pairs can be added to the database 202. In some embodiments, other improvements can be detected by performing regression testing.

[0078] Figure 6 600 is a schematic diagram of how a large language model (LLM) is trained based on code pairs and how new source code is generated according to some embodiments of the present invention. In the schematic diagram 600, existing user interface data is converted into vector expressions and stored in a vector database in a basic database.

[0079] This data can include user interface data from different code bases, including code in production and legacy code. This code has been previously approved for production and has been audited for security, usability, and brand consistency. Therefore, this base database can become a reliable source of reference information. Once prepared, the vector database can be used for downstream similarity comparisons.

[0080] exist Figure 6 In step 1, a specially configured plug-in is used to create a custom image from a user interface designer in, for example, FIGMA TMThe design object in the form of a JSON file is extracted from the user interface designed on the user interface design software such as jQuery, and then vectorized using the embedding model. In step 2, similarity matching is performed using methods such as keyword matching, and candidate code pairs are retrieved from the vector database to form a set of candidate code pairs. The code pair with the highest similarity score is selected from the set of candidate code pairs and provided as part of the prompt to the large language model data architecture (including generative artificial intelligence) in step 3 to generate the output code. In some embodiments, the prompt may also include other parameters, such as prompt instructions for interpolation / extrapolation from certain code snippets that have different parameters when designed, such as changing the resolution, changing the code syntax (for example, different mobile operating systems use different rendering codes).

[0081] The technical solution provided by the present invention utilizes part of the capabilities of the large language model (for example, contextual learning capabilities) to adjust code snippets based on the received instructions, and by providing a sufficiently close reference example, the code generation process is closer to the reference example than using the large language model itself for code generation. As an illustrative non-limiting example, the technical solution provided by the present invention can identify candidate codes with a 98.5% match degree produced by designers of another team in another field. For the remaining 1.5% of the code, by using the technical solution provided by the present invention, the best matching candidate code can be first identified, and then relevant prompts can be provided to the large language model, and the remaining 1.5% of the code can be extrapolated based on the best matching candidate code. Since the best matching candidate code has been approved and has passed relevant brand management audits and network security standards inspections, etc., the code generated by the extrapolation of the large language model is quite likely to meet these standards.

[0082] In practice, some large enterprises have a large database of user interface code lines, and some of the graphical user interface designs of these enterprises are only slightly different between branches due to the limitations of usage scenarios, for example, they are only locally adapted according to local characteristics and local consumer habits. In this case, the technical solution provided by the present invention is particularly useful in such enterprises.

[0083] The generated output can include rendering code that is automatically loaded into the test environment by the rendering pipeline for immediate review and approval by designers. And, upon approval, the rendering code can be automatically packaged into executable code that is executed by the front-end client or served by the back-end web server in production.

[0084] Figure 7 is a diagram of the example design input and the example intermediate JSON data structure output. Figure 7As shown in FIGMA, the design input 702 is extracted from, for example, FIGMA TM The input is first converted into a JSON data structure 704. In this example, the component types in the JSON data structure 704 are extracted and converted into a unique list form. For example, the processed output contains ['FRAME', 'navigation-bar-top', 'input-field-standard', 'TEXT', 'primary-button', 'home-indicator'].

[0085] Those skilled in the art will appreciate that the above description and examples are for exemplary purposes only.

[0086] Figure 8 is an example prompt according to some embodiments of the present invention, which has a portion where content can be inserted, and the inserted content is inserted by the generative user interface code system provided by the present invention for execution. Figure 8 As shown in , the provided hint 800 also includes other hint parameters and the type of target output rendering code.

[0087] By automating the process of translating a graphical user interface into functional mobile application code, the technical solution proposed by the present invention effectively connects the graphical user interface design with the mobile application code development, thereby improving production efficiency and product quality. In at least one aspect, the technical solution proposed by the present invention overcomes the complex challenges of computing resources caused by large-scale graphical interface development and improves the efficiency of related computing and resource utilization. By automatically reusing the existing interface design source code, the technical solution of the present invention can minimize the repetitive labor in the process of converting the user interface design into the source code, and still maintain accuracy and retain as much structural information as possible. In addition, for large enterprise organizations, these enterprises already have a large number of user interface nodes in use as a corpus, and these user interface codes have been reviewed by the corporate brand department and the security department. These user interface codes can be integrated to form a corpus and the code snippets used as references are used appropriately, so as to maintain the consistency of the code and achieve the reuse of the existing code base.

[0088] Furthermore, the present invention also proposes specific methods for overcoming other technical problems, such as maintaining document structure to improve automatic similarity analysis in the intermediate step of converting user interface design into intermediate data representation for machine learning and similarity analysis, and a node clustering method to reduce the total tag size.

[0089] In some non-limiting examples, the technical solution of the present invention can be applied to the development process of converting the user interface design of the online banking mobile application client launched by a bank or a large organization into source code. The technical solution of the present invention can use the existing code in the code library of the bank to build a code pair of the vector database as an example compared with the input graphical user interface design. In some embodiments, the generated output can include deployable code that can be loaded into the test environment library for verification and testing in real time. Or in some embodiments with a certain amount of fault tolerance, the generated output can be used in a production environment in real time. These generated codes can be used by front-end users (for example, user mobile devices with mobile applications) to render interfaces, or there are websites or web servers to serve for rendering.

[0090] Fig. 9 1 is an architecture diagram of an example system 900 for automatically generating source code for a graphical user interface design according to one aspect of the present invention. The architecture of the system corresponds one-to-one to the method for automatically generating source code for a graphical user interface design in the above-mentioned embodiment. The system 900 includes a data receiving module 901, a hierarchical data conversion module 902, a vector conversion module 903, a keyword retrieval module 904, a similarity calculation module 905, a prompt construction module 906, a large language model module 907, an output module 908, and optionally includes an operation module 909.

[0091] A data receiving module 901 is configured to receive an original data object representing a graphical user interface design;

[0092] A layered data conversion module 902, configured to convert the original data object into a target layered data object, wherein the layered data object retains the structural information of the original data object;

[0093] A vector conversion module 903, configured to use a hash function to convert the target hierarchical data object into a target multi-dimensional vector, wherein the target multi-dimensional vector retains the structural information of the hierarchical data object;

[0094] A keyword search module 904 is configured to use a keyword search method to identify one or more groups of similar code pairs in a database based on the target hierarchical data object, wherein the database includes a plurality of pre-input code pairs in the form of a multi-dimensional vector;

[0095] a similarity calculation module 905, configured to calculate the cosine similarity between the target multidimensional vector and each group of the similar code pairs to obtain a cosine similarity score of each group of the similar code pairs relative to the multidimensional vector, thereby determining an optimal similar code pair, wherein the optimal similar code pair has a highest cosine similarity score;

[0096] a prompt building module 906 configured to build a prompt based on the optimal similar code pair;

[0097] A large language model module 907, configured to input the prompt and the original data object into a large language model to perform context learning and generate a text instruction set based on multiple context clues of the original data object; and

[0098] The output module 908 is configured to output the generated text instruction set.

[0099] Optionally, the output module 908 is further configured to encapsulate the output text instruction set to generate a machine language instruction set, and output the machine language instruction set. It can also be further configured to link the machine language instruction set to an executable binary file, wherein the executable binary file is a collection of machine language instruction sets; and output the executable binary file.

[0100] The system 900 optionally further includes a running module configured to run the executable binary file to render a graphical user interface; and output the rendered graphical user interface.

[0101] Optionally, the similarity calculation module 905 is further configured to set a first threshold value of the highest cosine similarity score; determine whether the highest cosine similarity score is greater than or equal to the first threshold value; if the highest cosine similarity score is, transmit the generated text instruction set to the development test environment for packaging and execution. And the similarity calculation module 905 can be further configured to set a second threshold value of the highest cosine similarity score, the second threshold value is higher than the first threshold value; determine whether the highest cosine similarity score is greater than or equal to the second threshold value; if the highest cosine similarity score is greater than or equal to the second threshold value, deploy the generated instruction set to one or more servers accessible to end users.

[0102] Optionally, the keyword retrieval module 904 may be further configured to end the keyword search when a preset number of similar code pairs are identified.

[0103] Optionally, the output module 908 is further configured to display the generated text instruction set to a manual auditor for review, and determine whether to add the generated text instruction set to the database; after the manual auditor's review is passed, a code pair corresponding to the generated text instruction set is created, and the code pair is added to the database.

[0104] Fig.10 Schematic diagram of the structure of a computer device according to one aspect of the present invention. Fig.10 As shown, the computing device may include: at least one processor ( Fig.10 Only one is shown in the figure), a memory, a computer-readable storage medium, and a computer program stored on the computer-readable storage medium in the memory and executable on at least one processor, and when the processor executes the computer program, the steps in the above-mentioned method embodiment of automatically generating application code based on graphical user interface design are implemented. Those skilled in the art can understand that Fig.10 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0105] Those skilled in the art will appreciate that the above description and examples are for demonstration purposes only. Different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or a combination thereof.

[0106] The functional modules and units described in the present invention may include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. In addition, the technical features discussed herein may be implemented by dedicated processor circuits, or by executable instructions, and / or a combination thereof.

[0107] As understood by those of ordinary skill in the art, various terms are used only to describe specific embodiments and are not intended to limit the embodiments. For example, ordinal numbers used herein (such as "first", "second", "third", etc.) are used to modify an element, such as a structure, assembly, operation, etc., and do not themselves indicate the priority or order of the element relative to another element, but are only used to distinguish the difference from another element of the same name (without these ordinal numbers, the elements of the same name will not be distinguishable). The word "coupled" is defined as connected, although not necessarily directly connected or mechanically connected; the two items of "coupled" may be one. "One" is defined as one or more, unless the present disclosure explicitly requires other meanings. The word "substantially" means that in most cases the specified standard is met, but not completely consistent-it covers the specified standard; for example, "substantially 90 degrees" includes 90 degrees, and "substantially parallel" includes parallel. In any disclosed embodiment, "and / or" means parallel and or. For example, A, B and / or C include: A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B and C. In other words, "and / or" functions as an inclusive but not limited "or". In addition, "A, B, C, or a combination thereof" or "A, B, C, or any combination thereof" includes: A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C.

[0108] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element. The implementation of any device, system and method may consist of or be substantially composed of the steps, elements and (or) features described, rather than just containing / including / having these steps, elements and (or) features.

[0109] Furthermore, a device or system configured in a certain way is configured in at least that way, but it may also be configured in other ways not specifically described. Some aspects of one example may be applied to other examples even if not described or illustrated, unless this is expressly prohibited by the disclosure or the nature of the particular example.

[0110] Those skilled in the art will further understand that the various illustrative logic blocks, modules, circuits and algorithm steps associated with the present disclosure can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, the above-mentioned illustrative components, blocks, modules, circuits and steps are generally described by their functions. Whether this function is implemented as hardware or software depends on the design limitations imposed on the specific application and the overall system. Those skilled in the art may implement the described functions in different ways for each specific application, but these implementation decisions should not be understood as making the technical solution out of the scope of this article. Those skilled in the art will also easily recognize that the order or combination of components, methods or interactions described here are only examples, and the components, methods or interactions of various aspects of this article may be combined or executed in a manner different from that illustrated and described herein.

[0111] The processor referred to in the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0112] The various illustrative logical blocks, modules, and circuits mentioned herein may be implemented or executed by a processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination of these designs to implement the functions described in the present disclosure. The processor may be a microprocessor, but as an alternative, the processor may also be another form of processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration.

[0113] The steps of the method or algorithm described in the present invention may be directly embodied in hardware, or may be embodied by a software module executed by a processor, or may be embodied by a combination of the two. The software module may be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, or a CD-ROM. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be an integral part of the processor. The processor and the storage medium may be integrated in an application-specific integrated circuit (ASIC). The ASIC may be located in a user terminal. Alternatively, the processor and the storage medium may be located in a user terminal as separate components.

[0114] The functions described in one or more exemplary designs may be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, these functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include all media that facilitate the transfer of computer programs from one location to another, including both computer storage media and communication media. Computer-readable storage media may be any medium that can be accessed by a computer. By way of example and not limitation, these computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disk-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code means, in the form of instructions or data structures, and can be accessed by a computer or processor. In addition, a connection may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted via a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL), then the coaxial cable, fiber optic cable, twisted pair, or DSL is also included in the definition of the medium. As used herein, "disk" and "disc" include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), hard disk, solid state disk and Blu-ray disk, where disks generally reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0115] The above description and examples provide a complete description of the structure and use of illustrative embodiments. Although some of the above examples are described in a particular level of detail, or with reference to one or more separate examples, those skilled in the art may make many modifications to the disclosed embodiments without departing from the scope of the invention. Therefore, the various illustrative embodiments of the method and system are not limited to the specific forms disclosed. Instead, they include all modifications and alternatives that fall within the scope of the claims, and other examples may include some or all of the features of the illustrated examples. For example, some elements may be omitted, or they may be combined into a unit structure, and / or the connecting parts may be replaced. In addition, when appropriate, some aspects of any of the examples described above may be combined with some aspects of any of the other examples described to form further examples with similar or different properties and / or functions, and to solve the same or different problems. Similarly, it should be understood that the benefits and advantages described above may be related to one embodiment or to several embodiments.

[0116] Although various aspects of the present disclosure and their advantages have been described in detail, it should be understood that changes, substitutions and modifications may be made herein without departing from the spirit of the disclosure as defined by the appended claims. In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, means, methods and steps described in the specification. As will be readily understood by those of ordinary skill in the art from this disclosure, processes, machines, manufactures, material compositions, means, methods or steps currently existing or developed in the future that perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein may be utilized in accordance with the present disclosure. Therefore, the appended claims are intended to include such processes, machines, manufactures, material compositions, means, methods or steps within their scope.

Claims

1. A method for automatically generating application code based on a graphical user interface design, characterized in that: include: receiving a raw data object representing a graphical user interface design; converting the original data object into a target hierarchical data object, wherein the hierarchical data object retains structural information of the original data object; Using a hash function, converting the target hierarchical data object into a target multidimensional vector, wherein the target multidimensional vector retains structural information of the hierarchical data object; Using a keyword search method, identifying one or more groups of similar code pairs in a database based on the target hierarchical data object, the database comprising a plurality of pre-entered code pairs, the code pairs being in the form of multi-dimensional vectors; Calculating the cosine similarity between the target multidimensional vector and each group of the similar code pairs to obtain a cosine similarity score of each group of the similar code pairs relative to the multidimensional vector, thereby determining an optimal similar code pair, wherein the optimal similar code pair has a highest cosine similarity score; constructing a prompt based on the optimal pair of similar codes; Inputting the prompt and the original data object into a large language model to perform context learning and generate a text instruction set based on a plurality of context clues of the original data object; and Output the generated text instruction set.

2. The method according to claim 1, characterized in that The method further comprises: Encapsulating the output text instruction set to generate a machine language instruction set; and The machine language instruction set is output.

3. The method according to claim 2, characterized in that The method further comprises: linking the set of machine language instructions into an executable binary file, wherein the executable binary file is a collection of the set of machine language instructions; and Outputs said executable binary file.

4. The method according to claim 3, characterized in that The method further comprises: Running the executable binary file to render a graphical user interface; and Output the rendered GUI.

5. The method according to claim 1, characterized in that The method further comprises: Setting a first threshold of the highest cosine similarity score; determining whether the highest cosine similarity score is greater than or equal to the first threshold; If the highest cosine similarity score is greater than or equal to the first threshold, the generated text instruction set is transmitted to a development and testing environment for packaging and execution.

6. The method according to claim 5, characterized in that The method further comprises: setting a second threshold of the highest cosine similarity score, the second threshold being higher than the first threshold; determining whether the highest cosine similarity score is greater than or equal to a second threshold; If the highest cosine similarity score is greater than or equal to a second threshold, the generated instruction set is deployed to one or more servers accessible to end users.

7. The method according to claim 1, characterized in that The keyword search ends when a preset number of similar code pairs are identified.

8. The method according to claim 7, characterized in that The optimal similar code pair is manually selected from the one or more groups of similar code pairs.

9. The method according to claim 1, characterized in that The method further comprises: Displaying the generated text instruction set to a human auditor for review, and determining whether to add the generated text instruction set to the database; After the manual auditor passes the review, a code pair corresponding to the generated text instruction set is created and the code pair is added to the database.

10. The method according to claim 1, characterized in that The hierarchical data object is a text object model, which includes a tree structure composed of a plurality of nested nodes, wherein each nested node represents a data element of the original data object.

11. A system for automatically generating application code based on a graphical user interface design, characterized in that: include: a data receiving module configured to receive an original data object representing a graphical user interface design; a hierarchical data conversion module configured to convert the original data object into a target hierarchical data object, wherein the hierarchical data object retains the structural information of the original data object; A vector conversion module configured to convert the target hierarchical data object into a target multi-dimensional vector using a hash function, wherein the target multi-dimensional vector retains structural information of the hierarchical data object; a keyword retrieval module configured to use a keyword search method to identify one or more groups of similar code pairs in a database based on the target hierarchical data object, the database comprising a plurality of pre-input code pairs, the code pairs being in the form of multi-dimensional vectors; a similarity calculation module configured to calculate the cosine similarity between the target multidimensional vector and each group of the similar code pairs to obtain a cosine similarity score of each group of the similar code pairs relative to the multidimensional vector, thereby determining an optimal similar code pair, wherein the optimal similar code pair has a highest cosine similarity score; a prompt building module configured to build a prompt based on the optimal similar code pair; a large language model module configured to input the prompt and the original data object into a large language model for context learning and generate a text instruction set based on a plurality of context clues of the original data object; as well as An output module is configured to output the generated text instruction set.

12. The system according to claim 11, characterized in that The output module is further configured to: Encapsulating the output text instruction set to generate a machine language instruction set; and The machine language instruction set is output.

13. The system of claim 12, wherein: The output module is further configured to: linking the set of machine language instructions into an executable binary file, wherein the executable binary file is a collection of the set of machine language instructions; and Outputs said executable binary file.

14. The system of claim 13, wherein: The system further includes an operation module configured to: Running the executable binary file to render a graphical user interface; and Output the rendered GUI.

15. The system of claim 11, wherein: The similarity calculation module is further configured to: Setting a first threshold of the highest cosine similarity score; determining whether the highest cosine similarity score is greater than or equal to a first threshold; If the highest cosine similarity score is greater than or equal to a first threshold, the generated text instruction set is transmitted to a development and testing environment for packaging and execution.

16. The system of claim 15, wherein: The similarity calculation module is further configured to: setting a second threshold of the highest cosine similarity score, the second threshold being higher than the first threshold; determining whether the highest cosine similarity score is greater than or equal to a second threshold; If the highest cosine similarity score is greater than or equal to a second threshold, the generated instruction set is deployed to one or more servers accessible to end users.

17. The system of claim 11, wherein: The keyword retrieval module is further configured to end the keyword search when a preset number of similar code pairs are identified.

18. The system of claim 11, wherein: The output module is further configured to: Displaying the generated text instruction set to a human auditor for review, and determining whether to add the generated text instruction set to the database; After the manual auditor passes the review, a code pair corresponding to the generated text instruction set is created and the code pair is added to the database.

19. A computer device comprising a memory, a processor and a computer program stored therein and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for automatically generating application code based on a graphical user interface design as claimed in any one of claims 1 to 10 are implemented.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically generating application code based on a graphical user interface design as claimed in any one of claims 1 to 10 are implemented.

Images