Intelligent code generation system and method based on multi-modal interaction

Abstract

The invention discloses an intelligent code generation system and method based on multi-modal interaction. The method specifically comprises the following steps: (1) user demand description; (2) extracting data by the AI large model; (3) generating JSON data; (4) generating an interface document and front and rear end codes; (5) generating a database script, a menu script and the like; and (6) online environment edition issuing. According to the method, an AI large model and a low-code platform are integrated, structured data are generated through natural language, voice and image input, generation and online release of interface documents, front and rear end codes and database scripts are automatically completed, and full-process automatic development is achieved. And intelligent demand analysis, automatic code generation and visual development are coordinated, so that the input-output ratio of technical resources can be remarkably improved, and enterprises are assisted to quickly respond to digital transformation demands. The method can be used as a product prototype interface to directly enable customers to see the implementation effect while realizing quick delivery, so that the development efficiency is improved to a great extent, and powerful technical support is provided for promoting the combination of low codes and AI (Artificial Intelligence).

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence driven low code development, in particular to an intelligent code generation system and method based on multi-modal interaction. BACKGROUND

[0002] With the acceleration of enterprise digital transformation process, the software development field is facing the challenges of efficiency, cost and talent: the conversion efficiency of demand to code in traditional development process is less than 30%, and the cross-team communication consumes 20% of effective working hours; the cost of R&D manpower accounts for more than 50%, while the resource utilization rate and code reuse rate are less than 30%; the repeated development rate of similar business requirements reaches 30%, and the development efficiency of basic functions fluctuates by 3 times due to the difference in technical personnel's ability. In this context, the intelligent coding capability of AI code generation tool Fittencode is combined with the agile characteristics of low code platform to build an automatic closed loop of "demand-design-development-test", which will become a key technology path to break through the existing R&D performance bottleneck and realize cost reduction and efficiency improvement. SUMMARY

[0003] The present application provides an intelligent code generation system and method based on multi-modal interaction, which generates structured data through natural language, voice and image input, and automatically completes the generation and online release of interface documents, front-end and back-end codes, and database scripts, realizing full-process automatic development.

[0004] To achieve the above purpose, the present application adopts the following technical scheme: an intelligent code generation method based on multi-modal interaction, comprising the following steps: a. User requirement description: describe the desired product function through at least one multi-modal method of natural language text, voice input or image; b. AI large model data extraction: use AI large model to automatically extract data, and support extraction of related information of different types of user description content through multi-modal method; c. Generate JSON data: based on AI large model data extraction, assemble the required data source through code to provide strong data support for front-end and back-end code generation; d. Generate interface documents, front-end and back-end code: automatically parse JSON data into standard OpenAPI3.0 interface documents, and realize front-end and back-end code through code parsing based on the documents; e. Generate database scripts and menu scripts: generate related scripts required by the graphical interface; f. Online environment release: the generated front-end and back-end code is automatically merged into the release branch, and is automatically built and released through CI / CD.

[0005] Further, the multi-modal input in step (a) comprises: extracting text and layout information from the interface screenshot through OCR technology; converting the voice input into a text requirement description through voice recognition technology.

[0006] Further, the AI large model in step (b) extracts data as a pre-training language model based on a Transformer architecture, trains a code generation task, and supports syntax rules of Python, Vue, and C# languages.

[0007] Further, in step (d), when generating interface documents and front-end and back-end codes through Python, Vue, and C# languages according to JSON data, the front-end and back-end codes generate basic add, delete, query, and modify operation codes according to certain rules through JSON configuration, and the front-end and back-end are automatically bound to fields, which improves the front-end and back-end docking efficiency.

[0008] Further, the CI / CD pipeline in step (e) includes: automatically pushing the code to a specified branch of a Git version library; triggering an automated testing and building process; deploying to a production environment and generating an accessible prototype interface.

[0009] An intelligent code generation system implementing the above method, comprising: a multi-modal input module for receiving text, voice, or image input; an AI analysis engine for generating structured data based on a pre-training language model; a code generator for converting JSON intermediate representation into interface documents, front-end and back-end codes, and database scripts; an automated deployment module integrating a CI / CD tool chain to complete online publishing.

[0010] The present application deeply integrates an AI large model with a low-code platform, generates structured data through natural language, voice, and image input, and automatically generates interface documents, front-end and back-end codes, database scripts, and online publishing, realizing full-process automated development. Intelligent requirement analysis, automated code generation, and visual development collaboration can significantly improve the input-output ratio of technical resources and help enterprises quickly respond to digital transformation needs. While achieving the purpose of rapid delivery, it can also be used as a product prototype interface to directly show customers the implementation effect, greatly improving development efficiency and providing strong technical support for the combination of low-code and AI. Compared with the prior art, the present application has the following beneficial effects: 1. A pre-training language model FittenCodo is used for natural language understanding to analyze user input requirement descriptions; 2. Convert user requirements into structured data using semantic analysis techniques for code generation and interface design; 3. Support multi-round dialogue interaction and dynamically adjust the generated results to meet user requirements; 4. Use deep learning models with Transformer architecture to train code generation tasks, currently supporting programming languages such as Python, Vue, C#, etc. 5. Convert user requirements into executable code logic through Json strings as intermediate tables; 6. Combine code optimization techniques to ensure high readability and maintainability of the generated code; 7. Low-code platform architecture design: automatically generate interface layout and code through text recognition, screenshot, or voice recognition. BRIEF DESCRIPTION OF DRAWINGS

[0011] Fig. 1 Method flowchart of the present application; Fig. 2 Specific operation interface schematic diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be described in detail in combination with the accompanying Figs. 1-2 The specific embodiments of the intelligent code generation system and method based on multi-modal interaction of the present application will be further described in detail.

[0013] An intelligent code generation method based on multi-modal interaction, comprising the following steps: a. User requirement description: describe the desired product function through at least one of natural language text, voice input, or image in a multi-modal manner; b. AI large model data extraction: automatically extract data using an AI large model, and support extraction of related information for different types of user description content through multi-modal methods; c. Generate JSON data: based on AI large model data extraction, assemble the required data source through code, and provide strong data support for front-end and back-end code generation; d. Generate interface document, front-end and back-end code: automatically parse JSON data into standard OpenAPI3.0 interface document, and implement front-end and back-end code based on the document; e. Generate database script and menu script: generate related scripts required by the graphical interface; f. Online environment release: automatically merge the generated front-end and back-end code into the release branch, and automatically build and publish through CI / CD.

[0014] Further, the multi-modal input in step (a) includes: Extracting text and layout information from interface screenshots through OCR technology; Converting voice input into text requirement descriptions through voice recognition technology.

[0015] Further, the AI large model extracts data in step (b) as a pre-trained language model based on a Transformer architecture, trains the code generation task, and supports the syntax rules of Python, Vue, and C# languages.

[0016] Further, in step (d), when generating interface documents and front-end and back-end codes through Python, Vue, and C# languages based on JSON data, the front-end and back-end codes generate basic add, delete, query, and modify operation codes through JSON configuration according to certain rules, and the front-end and back-end are automatically bound to fields, improving the front-end and back-end integration efficiency.

[0017] Further, the CI / CD pipeline in step (e) includes: Automatically pushing the code to the specified branch of the Git version library; Triggering automated testing and building processes; Deploying to the production environment and generating an accessible prototype interface.

[0018] An intelligent code generation system implementing the above method, comprising: A multi-modal input module for receiving text, voice, or image input; An AI analysis engine based on a pre-trained language model to generate structured data; A code generator to convert JSON intermediate representation into interface documents, front-end and back-end codes, and database scripts; An automated deployment module integrating CI / CD tool chains to complete online publishing.

[0019] Specific embodiment: Settlement visa front-end and back-end code automatic generation and publishing; Inputting the screenshot as shown in Fig. 2 The function is called the settlement visa function.

[0020] Processing flow: 1) Call the API of Fittencode to extract relevant information to form JSON data; 2) Assemble OpenApi documents, Vue front-end code, C# back-end code, and SQL scripts based on JSON data; 3) Automatic publishing: push the code to the Git repository → trigger the CI / CD pipeline → deploy to the production environment.

[0021] Technical effects are as follows: ​

[0022] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A method for intelligent code generation based on multi-modal interaction, characterized in that, Includes the following steps: a. User Requirements Description: Describe the desired product features using at least one multimodal method, such as natural language text, voice input, or images; b. AI large model data extraction: Automatically extract data using AI large model, and support the extraction of relevant information from different types of user description content through multimodal methods; c. Generate JSON data: Extract data based on the AI ​​large model, assemble it into the data source we need through code, and provide strong data support for front-end and back-end code generation; d. Generate API documentation and front-end / back-end code: Automatically parse JSON data into standard OpenAPI 3.0 API documentation, and implement front-end and back-end code based on this documentation through code parsing; e. Generate database scripts and menu scripts: Generate the relevant scripts required for the interface based on the graphical interface; f. Online environment release: The generated front-end and back-end code is automatically merged into the release branch, and automatically built and released through CI / CD. 2.The intelligent code generation method based on multi-modal interaction according to claim 1, characterized in that: The multimodal inputs mentioned in step (a) include: Extract text and layout information from screenshots using OCR technology; Voice input is converted into a text description of requirements using speech recognition technology.

3. The intelligent code generation method based on multimodal interaction according to claim 1, characterized in that: In step (b), the AI ​​large model extracts data as a pre-trained language model based on the Transformer architecture to train code generation tasks, supporting the syntax rules of Python, Vue, and C# languages.

4. The intelligent code generation method based on multimodal interaction according to claim 1, characterized in that: In step (d), when generating interface documentation and front-end / back-end code using Python, Vue, and C# based on JSON data, the front-end and back-end code simultaneously generate basic add, delete, query, and modify operation code according to certain rules based on JSON configuration. At the same time, the front-end and back-end automatically bind fields, improving the efficiency of front-end and back-end integration.

5. The intelligent code generation method based on multimodal interaction according to claim 1, characterized in that: The CI / CD pipeline described in step (e) includes: Automatically push code to a specified branch of the Git repository; Trigger automated testing and build processes; Deploy to the production environment and generate an accessible prototype interface.

6. An intelligent code generation system for implementing the method of any one of claims 1-5, characterized in that, include: A multimodal input module for receiving text, voice, or image input; AI parsing engine, generating structured data based on pre-trained language models; A code generator that converts JSON intermediate representations into API documentation, front-end and back-end code, and database scripts. The automated deployment module integrates the CI / CD toolchain to complete online releases.

Images