Graphical code generation method, server, storage medium and program product

Abstract

The invention discloses a graphical code generation method, a server, a storage medium and a program product, relates to the technical field of artificial intelligence, and is used for improving the graphical code generation efficiency of a beginner. According to the method, when voice input containing function requirements of a user is received, a server converts the voice input into text information; the server converts the text information into a query vector based on a preset code embedding model; the server retrieves a preset number of visual programming components with the closest semantic distance to the query vector from a vector database as a candidate component set; the server splices the text information, the candidate component set and the identifier and the function description of each visual programming component in the candidate component set into a structured prompt text according to a preset format; and the server inputs the structured prompt text into a preset graphical code generation model to obtain a graphical code which corresponds to the text information and conforms to the standard of the target graphical programming platform.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a graphical code generation method, server, storage medium, and program product. Background Technology

[0002] In the fields of open-source hardware and programming education, embedded development platforms, represented by Arduino, are widely used in prototyping and electronic projects due to their powerful functionality and flexibility. However, the text-based programming method of the Arduino platform, based on C / C++ syntax, presents a high learning barrier for beginners, especially teenagers. To lower the learning barrier of programming education, graphical programming tools such as Blockly have emerged. These tools use a visual, block-based programming interface, allowing users to build program logic by dragging and dropping and combining graphical modules.

[0003] In related technologies, the conversion between Arduino text code and graphical code primarily employs a rule-based mapping approach. Specifically, this technology performs lexical and syntactic analysis on Arduino's C / C++ code using predefined parsing rules, identifying basic syntactic elements such as variable declarations, function calls, conditional statements, and loop structures. These syntactic elements are then mapped to corresponding graphical programming modules according to a predefined mapping table. For example, the `digitalWrite()` function is mapped to a digital output module, and a `for` loop is mapped to a repetitive execution module. For complex nested logic and specific hardware operations, this technology typically uses template matching, predefining graphical representations of common code patterns. When automatic conversion struggles to handle specific code snippets, the technology marks them as requiring manual intervention, allowing users to manually adjust and refine the organization of the graphical code.

[0004] However, when users manually adjust and improve the organization of graphical code, the relevant technologies still require operators to be familiar with both Arduino programming syntax and the proprietary component library of the graphical programming platform, resulting in low efficiency for beginners in generating graphical code. Summary of the Invention

[0005] This application provides a graphical code generation method, server, storage medium, and program product to improve the efficiency of beginners in generating graphical code.

[0006] Firstly, a graphical code generation method is provided, applied to a server. This method includes: when receiving voice input from a user containing functional requirements, the server converts the voice input into text information; the server converts the text information into a query vector based on a preset code embedding model; the server retrieves a preset number of visual programming components with the closest semantic distance to the query vector from a vector database as a candidate component set; the vector database is obtained by vectorizing all pre-collected visual programming components and functional descriptions of the target graphical programming platform based on the preset code embedding model; the server concatenates the text information, the candidate component set, and the identifiers and functional descriptions of each visual programming component in the candidate component set according to a preset format into structured prompt text; the server inputs the structured prompt text into a preset graphical code generation model to obtain graphical code corresponding to the text information and conforming to the specifications of the target graphical programming platform.

[0007] By adopting the above technical solutions, the server incorporates natural language understanding and AI code generation technologies, thereby simplifying the process of beginners generating graphical code and improving the efficiency of beginners in generating graphical code.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the training process of the preset graphical code generation model specifically includes: the server acquiring paired data of natural language descriptions and graphical code to obtain a training dataset; the server converting the graphical code into a structured text format containing node definitions and edge connections, wherein the structured text format describes the visual programming components through node objects and the connection relationships between the visual programming components through edge objects, wherein the node objects contain unique identifiers, component type identifiers, and parameter configuration dictionaries, and the edge objects define the output ports of the source nodes and the input ports of the target nodes; the server constructing a graphical code generation model based on an encoder-decoder architecture, wherein the encoder is responsible for understanding the natural language intent and the decoder is responsible for generating the structured text format; after pre-training the graphical code generation model based on a general code generation task, the server performs supervised fine-tuning training based on the training dataset to obtain the preset graphical code generation model.

[0009] By adopting the above technical solution, the server converts complex graphical code into a structured text format containing nodes and edges as the training target of the model, instead of directly generating XML. This allows the model to focus on generating more concise structured text that describes the core logic of the program, without having to learn and deal with the cumbersome syntax details and nested tags of XML, thus reducing the learning difficulty of the model.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of the server inputting structured prompt text into a preset graphical code generation model to obtain graphical code corresponding to the text information and conforming to the target graphical programming platform specification specifically includes: the server inputting structured prompt text into the preset graphical code generation model to obtain intermediate representation text conforming to the structured text format; the server extracting node sets, edge connection relationships, and parameter configuration information from the intermediate representation text; and the server converting the node sets and edge connection relationships into graphical code conforming to the Extensible Markup Language specification of the target graphical programming platform.

[0011] By adopting the above technical solution, the server introduces intermediate representation text as a bridge between the output of the graphical code generation model and the final graphical code. This allows the model to generate only simple, model-friendly intermediate representation text, while the task of converting the intermediate representation into strict, standardized XML code is handled by a rule-based compiler. This decouples the uncertain AI generation from the deterministic algorithmic conversion, thereby improving the accuracy of the generated graphical code.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, before the step of the server extracting the node set, edge connection relationship and parameter configuration information from the intermediate representation text, the method further includes: the server performing dependency analysis on the intermediate representation text to obtain a call dependency graph between components; the server detecting whether there are circular dependencies and deadlock problems based on the call dependency graph; if so, the server generating a warning message; if not, the server performing topological sorting optimization on the execution order of components in the intermediate representation text according to the execution timing specification of the target graphical programming platform.

[0013] By adopting the above technical solution, the server introduces a deterministic verification and optimization process before generating graphical code, which further improves the logical correctness and execution efficiency of the generated code.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, after the server converts the node set and edge connection relationships into graphical code conforming to the Extensible Markup Language specification of the target graphical programming platform, the method further includes: the server using the intermediate representation text of the graphical code as context information in the session state; when receiving a user's modification instruction, the server constructs an iterative modification prompt text containing the modification instruction and context information; the server inputs the iterative modification prompt text and a preset graphical code semantic patching language specification into a preset graphical code generation model to obtain a semantic patching instruction sequence describing the modification operation; after the semantic patching instruction sequence undergoes a validity check, the server applies the semantic patching instruction sequence line by line to the intermediate representation text of the graphical code before modification to obtain the modified intermediate representation text; the server generates graphical code for the modification instruction based on the modified intermediate representation text.

[0015] By adopting the above technical solution, when a user generates graphical code and then makes a modification suggestion, the graphical code generation model no longer directly regenerates new graphical code. Instead, it generates a semantic patch instruction sequence composed of atomic operations that describes how to modify the code. Subsequently, the graphical code generation model generates graphical code with modification instructions based on these instructions, thereby improving the efficiency of generating graphical code.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, a pre-defined graphical code semantic patching language specification is provided, specifically including: the graphical code semantic patching language specification defines atomic operation instructions, which are used to add, delete, or modify node objects or edge objects in the intermediate representation text.

[0017] By adopting the above technical solution, the server defines a set of graphical code semantic patching language specifications that include atomic operations such as adding, deleting, and modifying. This breaks down complex code structure editing tasks into a series of standardized operation instructions. As a result, the model does not need to understand how to directly manipulate complex intermediate representation text; it only needs to select and fill in these preset instruction templates according to the user's intent, thereby reducing the difficulty for the model to perform modification tasks.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, the server applies the semantic patch instruction sequence one by one to the intermediate representation text of the graphical code before modification to obtain the step of modifying the intermediate representation text. Specifically, this includes: the server parsing the operation type, target object, and operation parameters in each semantic patch instruction in the execution order of the semantic patch instruction sequence; when the operation type is an add operation, the server inserts a new node object or edge object at a specified position in the intermediate representation text; when the operation type is a delete operation, the server removes the specified node object or edge object and its related connection relationship from the intermediate representation text; when the operation type is a modify operation, the server updates the attribute parameters of the specified node object or edge object.

[0019] By adopting the above technical solution, the server performs addition, deletion and modification operations on the intermediate representation text according to predetermined rules by parsing and executing atomic operations in the patch sequence one by one, thereby improving the accuracy of modifying the intermediate representation text.

[0020] In a second aspect, embodiments of this application provide a server comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the server to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a server, cause the server to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a server, cause the server to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the server provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. Because the server incorporates natural language understanding and AI code generation technologies, it simplifies the process for beginners to generate graphical code, thereby improving the efficiency of beginners in generating graphical code.

[0025] 2. Because the server converts complex graphical code into a structured text format containing nodes and edges as the training target for the model, rather than directly generating XML, the model does not need to learn and deal with the cumbersome syntax details and tag nesting of XML. It can focus on generating more concise structured text that describes the core logic of the program, thereby reducing the learning difficulty of the model.

[0026] 3. When a user generates graphical code and then provides modification suggestions, the graphical code generation model no longer directly regenerates new graphical code. Instead, it generates a semantic patch instruction sequence consisting of atomic operations that describes how the code should be modified. The graphical code generation model then generates graphical code with modification instructions based on these instructions, thereby improving the efficiency of generating graphical code. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a graphical code generation method in an embodiment of this application.

[0028] Figure 2 This is another flowchart illustrating a graphical code generation method in an embodiment of this application.

[0029] Figure 3 This is a schematic diagram of the physical device structure of a server in an embodiment of this application. Detailed Implementation

[0030] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0031] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0032] This application provides a graphical code generation method, server, storage medium, and program product to improve the efficiency of beginners in generating graphical code.

[0033] To facilitate understanding, the application scenarios of the embodiments of this application are described below.

[0034] In current programming education and maker activities, educational tools such as the Future Board (an open-source hardware development board similar to Arduino) are widely used. These tools aim to inspire young people's interest in science, technology, engineering, and mathematics. To lower the barrier to entry for programming, graphical programming software (such as a development environment based on Blockly) is usually provided, allowing students to replace writing complex text code by dragging and dropping graphical blocks. However, even graphical programming still presents challenges for beginners in actual teaching and use.

[0035] In related technologies, students typically need to complete programming tasks manually. The following describes a scenario where graphical programming methods from these technologies are used.

[0036] In a related technological case, an elementary school student wanted to use a future-oriented programming board to complete a traffic light project, which involved making three LEDs (red, yellow, and green) light up in a sequential cycle. Using traditional graphical programming software, he first needed to understand the project logic and then manually search for the corresponding functional modules in the software's component library. He might need to repeatedly browse through multiple categories such as input / output, control, and logic to find the necessary building blocks for setting digital pins, delays, and cycles. This process was not only time-consuming but also required the student to have a certain level of prior knowledge of the platform's component library. If he didn't understand the technical concept that lighting up an LED corresponds to setting a digital pin to a high level, he might feel confused and frustrated while searching for the building blocks. After completing the project, if the teacher asked him to shorten the yellow light's duration, he would need to accurately locate the specific building block controlling the yellow light's on / off state and delay within the already assembled complex program and manually modify its parameters, which is equally prone to errors for beginners.

[0037] The graphical code generation method described in this application simplifies the process from idea to implementation by introducing natural language understanding and AI code generation technologies, thus improving the intuitiveness and efficiency of programming. The following describes scenarios where the graphical code generation method of this application is used.

[0038] After adopting this solution, a primary school student no longer needs to manually search for and drag blocks when working on a traffic light project. They can directly speak their ideas into the microphone: "Make a traffic light that turns red for 3 seconds, then green for 3 seconds, then yellow for 1 second, and repeats this process." Upon receiving this voice command, the server automatically converts it to text, understands its semantics through a preset code embedding model, and retrieves relevant components such as setting digital pins, delays, and loops from the Future Board's component library. This information is then integrated and input into a preset graphical code generation model, which directly generates complete graphical code conforming to the Future Board platform specifications. The student's screen immediately displays the assembled block program that matches their described traffic light logic. Later, when they want to modify the program, they simply need to speak the command again: "Change the yellow light time to 2 seconds." This solution understands this as a modification command based on the current code and generates a semantic patch command to update the parameters of the corresponding delay module; the graphical code on the interface is also updated in real time.

[0039] It is evident that the graphical code generation method described in this application improves the efficiency of beginners in generating graphical code.

[0040] To facilitate understanding, the method provided in the embodiments of this application will be described in process below, taking into account the above scenarios. Please refer to... Figure 1 This is a flowchart illustrating a graphical code generation method in an embodiment of this application.

[0041] S101. When the server receives voice input from the user containing functional requirements, it converts the voice input into text information.

[0042] Specifically, when a user, such as a student who wants to create something using the Future Board, presses the voice button on the Future Board, the server receives the audio stream or audio file uploaded by the Future Board. The server then processes the voice data and outputs text information corresponding to the voice content. For example, the user's voice command, "Create a traffic light, first turn red for 3 seconds," is accurately converted into the string "Create a traffic light, first turn red for 3 seconds."

[0043] S102. The server converts text information into query vectors based on a preset code embedding model.

[0044] The pre-defined code embedding model refers to an embedding model specifically designed for code retrieval. It maps natural language questions and code snippets to the same high-dimensional vector space, ensuring that semantically similar text and code are located close together within this space. In this embodiment, the model can specifically employ the OASIS-code-embedding-1.5B model. The query vector is a high-dimensional array of floating-point numbers, representing the text information mathematically within this high-dimensional vector space. Its direction and position encode the core semantics of the user instruction.

[0045] Specifically, after obtaining the text information generated in step S101, the server uses it as input to call the preset OASIS-code-embedding-1.5B model. This model, designed specifically for code tasks, performs deep semantic analysis of the text through its internal neural network structure, capturing intents related to programming logic and functional implementation. The output of the preset code embedding model is the query vector. Using a code-specific embedding model like OASIS-code-embedding-1.5B allows for a deeper understanding of the specific semantics of the programming domain compared to general text embedding models. For example, it can more accurately grasp the intrinsic connections between blinking and programming concepts such as loops, delays, and level flips, thereby generating a more targeted query vector.

[0046] In some embodiments, text information can be converted into query vectors in several ways: Optionally, a pre-trained OASIS-code-embedding-1.5B model can be used directly. Specific steps include: the server loads the pre-trained OASIS-code-embedding-1.5B model weights, segments and formats the user's text information to meet the model's input requirements, inputs the processed text into the model, and obtains the final output embedding vector as the query vector. Optionally, the code embedding model can also be domain-adaptive fine-tuned. Specific steps include: collecting functional description-component combination data pairs related to the target graphical programming platform, using this data to fine-tune the OASIS-code-embedding-1.5B model through contrastive learning or masked language modeling, etc., to better adapt it to the platform's proprietary terminology and expression habits. Then, the fine-tuned model is used to generate query vectors to achieve higher retrieval accuracy.

[0047] S103. The server retrieves a preset number of visual programming components that are semantically closest to the query vector from the vector database as a candidate component set.

[0048] The vector database refers to a data management system for storing and retrieving high-dimensional vectors. In this embodiment, the database can be the Chroma Database (ChromaDB), which is constructed by vectorizing all visual programming components and their functional descriptions of the target graphical programming platform that have been pre-collected, using the same OASIS-code-embedding-1.5B model as in step S102. Semantic distance refers to the distance between two vectors in the vector space, which can be measured using cosine similarity or Euclidean distance. The candidate component set refers to the set of visual programming components that are closest to the user's query intent after being sorted according to semantic distance.

[0049] Specifically, the server uses the query vector generated in step S102 as a query request and performs a nearest neighbor search in the pre-built Chroma vector database. This database stores vector representations of all available Blockly components on the target platform. These vectors are obtained by vectorizing the text of each component's name, functional description, and parameter information using the OASIS-code-embedding-1.5B model. The Chroma database uses its built-in vector similarity search algorithm to calculate the semantic distance between the query vector and all component vectors in the database, and returns a preset number of the closest components. For example, for a query to make a red light stay on for 3 seconds, the database can retrieve core functional components such as setting digital pins and delays from a knowledge base containing a large number of proprietary components. This process focuses the user's vague natural language requirements onto a specific list of components highly relevant to the platform, providing the necessary contextual knowledge for the accurate generation of the subsequent generative model.

[0050] In some embodiments, the construction and retrieval of the vector database can be implemented in several ways: Optionally, ChromaDB's in-memory mode can be used for rapid prototyping. Specific steps include: initializing a ChromaDB client when the server starts, then iterating through all Blockly component description texts of the target graphical programming platform, generating vectors for each component using the OASIS-code-embedding-1.5B model, and adding the text and vectors together to a ChromaDB collection. During retrieval, the `query` method of the collection is directly called, passing in the query vector and the `k` value to obtain the results. Optionally, ChromaDB's client / server mode can also be used to support larger scale and persistent storage. Specific steps include: independently deploying a ChromaDB server instance, using a batch processing task to vectorize all components and store them on the server, then using the application server as a client to connect to the ChromaDB server via the network to perform retrieval operations. This approach offers better scalability and stability.

[0051] S104. The server concatenates the text information, the candidate component set, and the identifiers and function descriptions of each visual programming component in the candidate component set into a structured prompt text according to a preset format.

[0052] Here, the identifier refers to the unique name or ID of each visual programming component. The functional description is natural language text explaining the component's purpose and parameters. The preset format refers to a well-structured text template designed for large language models. The structured prompt text refers to the final, complete text that includes the user's original requirements and relevant background knowledge, serving as input to the graphical code generation model.

[0053] Specifically, the server organizes text information, a set of candidate components, and the identifiers and functional descriptions of each visual programming component within the candidate component set into a format that is easy for the model to parse. For example, the prompt text contains the following structure: [User Requirement]: Make a traffic light, [Available Blockly Components]: Component ID: custom_set_pin, Description: Controls the level of a specific port on the future board, Parameters: PIN (number, selectable from 1-8), STATE (high / low). In this way, when generating code, the model is explicitly informed of the toolkit available for the current task (i.e., only platform-specific candidate components can be used) and their usage instructions (i.e., how to use each component). This greatly reduces the possibility of the model misinterpreting general components that are not applicable to this platform, or misusing proprietary component parameters.

[0054] S105. The server inputs the structured prompt text into the preset graphical code generation model to obtain graphical code that corresponds to the text information and conforms to the target graphical programming platform specifications.

[0055] The preset graphical code generation model refers to a finely tuned large language model trained to understand structured prompt text and generate code representations in a specific format. The target graphical programming platform specification refers to the code file format accepted by the target software (such as the Blockly environment in Future Board), typically an XML (Extensible Markup Language) format, which defines blocks, values, fields, and their nesting and connection relationships. Graphical code refers to the final generated, complete code file that can be directly loaded and displayed as visual building blocks on the target platform.

[0056] The training process for the pre-defined graphical code generation model is as follows: First, the server acquires a large amount of natural language description-graphical code pairing data as the training dataset. Next, to enable the model to learn and generate structured code, the server converts the graphical code (usually in XML format) in the training data into a structured text format. This format describes each visual programming component through node objects and the connections between components through edge objects. Specifically, a node object contains a unique identifier to distinguish different block instances, a component type identifier indicating its type, and a parameter configuration dictionary storing its configuration (such as pin numbers and delay times). An edge object defines the output port of the connected source node and the input port of the connected target node. Then, the server builds the graphical code generation model based on an encoder-decoder architecture (such as T5 or BART), where the encoder is responsible for understanding the natural language intent of the input, and the decoder is trained to generate the structured text format defined above. The model training is divided into two stages: first, pre-training is performed on a general code generation task to obtain basic code understanding and generation capabilities. Subsequently, supervised fine-tuning training is performed using the previously processed training dataset, enabling the model to specifically learn to generate structured text representations of graphical code for a specific platform from natural language descriptions. This process yields the pre-defined graphical code generation model for this step.

[0057] Specifically, the execution process of this step is as follows: The server inputs the structured prompt text constructed by S104 into a preset graphical code generation model. This model reads and understands all the information in the prompt text: what the user's goal is, and what building blocks are available. Based on this understanding, the model does not directly generate the final XML, but generates a more concise and easier-to-learn intermediate representation text. For example, the intermediate representation text can be a JSON format text, an example of which is: {"nodes":[{"id":"loop1","type":"controls_repeat_forever"},{"id":"led_on","type":"future_board_set_digital_pin","params":{"pin":9,"state":"HIGH"}}],"edges":[{"from":"loop1.DO","to":"led_on.PREV"}]}. Subsequently, a compiler built into the server begins to process this intermediate representation text. Before conversion, the server first performs dependency analysis on this intermediate representation text to construct a call dependency graph between components. Based on this graph, the server can detect logical problems such as circular dependencies and deadlocks. If a problem is detected, the server can generate a warning message to provide feedback to the user. If no problem is found, the server can also optimize the execution order of components by performing topological sorting according to the execution timing specifications of the target platform. Finally, the compiler extracts the node set, edge connection relationships, and parameter configuration information from this verified and optimized intermediate representation text, and converts it into graphical code that conforms to the Extensible Markup Language (XML) specification of the target graphical programming platform.

[0058] In some embodiments, the generation from prompts to final graphical code can be achieved in several ways: Optionally, an encoder-decoder model fine-tuning approach can be used. Specific steps include: First, preparing a large amount of structured prompt text-intermediate representation text pairing data to perform supervised fine-tuning of the model (e.g., T5). Then, during inference, the prompt text generated in S104 is input into the fine-tuned model to obtain the generated intermediate representation text. Finally, a compiler based on templates or DOM manipulation is written to convert the intermediate representation text into the final XML file. Optionally, a context learning approach based on a large generative model (e.g., GPT-4) can also be used. Specific steps include: Adding several additional prompt-intermediate representation examples to the prompt text constructed in S104. Then, this longer prompt text containing examples is input into a general large model, instructing it to output according to the example format. Finally, the intermediate representation output by the model is also converted into XML using a compiler.

[0059] The above embodiments describe how to generate graphical code from a single user's natural language input through a two-stage generation process involving speech recognition, vector retrieval, structured prompt construction, and an intermediate representation plus deterministic compilation. By introducing an intermediate representation text, the task of generating strictly structured data (such as XML), which large language models are not good at, is transformed into the task of generating concise structured text, which they are better at.

[0060] However, in programming education and project development, the programming process is often an iterative and progressively refined cycle. After seeing the initial generated code, users usually have new ideas and need to adjust, extend, or modify the existing code, such as making a light blink faster or adding a button as a switch. If only entirely new generation requests can be handled, then every minor modification by the user may require a complete re-description of the project logic, which is inefficient. Furthermore, allowing large language models to directly modify already generated complex XML code is extremely error-prone.

[0061] Please see Figure 2 This is another flowchart illustrating a graphical code generation method in an embodiment of this application.

[0062] S201. When the server receives voice input from the user containing functional requirements, it converts the voice input into text information.

[0063] S202. The server converts text information into query vectors based on a preset code embedding model.

[0064] S203. The server retrieves a preset number of visual programming components that are semantically closest to the query vector from the vector database as a candidate component set.

[0065] S204. The server concatenates the text information, the candidate component set, and the identifiers and function descriptions of each visual programming component in the candidate component set into a structured prompt text according to a preset format.

[0066] S205. The server inputs the structured prompt text into the preset graphical code generation model to obtain graphical code that corresponds to the text information and conforms to the target graphical programming platform specifications.

[0067] Step S201 is similar to step S101, step S202 is similar to step S102, step S203 is similar to step S103, step S204 is similar to step S104, and step S205 is similar to step S105, so they will not be repeated here.

[0068] S206. The server uses the intermediate representation text of the graphical code as context information in the session state.

[0069] In this context, session state refers to a segment of memory or storage space maintained by the server to preserve continuous interaction with a specific user, recording the user's current interaction history and program state. The intermediate representation text is the structured text generated in S205 that describes the logical structure of the code.

[0070] Specifically, after successfully generating the first version of the graphical code in step S205, the core output of this generation, namely the intermediate representation text, is not discarded. Instead, the server associates this intermediate representation text (e.g., a JSON structure describing the initial logic of a traffic light) with the current user's session ID and stores it in the session state manager. When the user subsequently submits modification suggestions, the server can know what the current program looks like, thus understanding that the modification instruction is for this existing program, rather than a completely new request.

[0071] S207. When a user's modification instruction is received, the server constructs an iterative modification prompt text containing the modification instruction and context information.

[0072] In this context, "modification instructions" refer to natural language input from the user after viewing the generated code, used to adjust, add, delete, or refactor the program logic. Examples include "It's too slow, make it twice as fast" or "Add a button that only starts flashing when pressed." Iterative modification prompts are specially designed structured prompts used to guide large language models to perform code editing rather than code generation tasks.

[0073] Specifically, this step occurs when a user makes a second or more modifications to the current program. For example, after generating the traffic light program, the user says: "Change the yellow light duration to 2 seconds." Upon receiving this instruction, the server first retrieves the intermediate representation text representing the current traffic light program, saved in S206, from the session state. Then, the server constructs a new prompt text, which differs structurally from the initial prompt text generated in S204. It contains two core parts: the current code and the modification request. For example, the prompt text might look like this: Task: Generate a modification patch for the following current code based on the user's modification request. Current code (intermediate representation format): {...complete intermediate representation text here...} User modification request: Change the yellow light duration to 2 seconds. Please generate a semantic patch instruction describing this modification.

[0074] In some embodiments, the construction of iterative modification prompt text can be implemented in several ways: optionally, differentiated highlighting can be used. Specific steps include: when constructing the prompt text, the server can first perform preliminary entity recognition on the user's modification instructions, attempting to locate components that may be modified (such as delay blocks related to the yellow light). Then, in the current code section, these potentially related code blocks are specially marked. Finally, the marked code is concatenated with the modification instructions to help the model locate the modification area more quickly.

[0075] S208. The server inputs the iterative modification prompt text and the preset graphical code semantic patch language specification into the preset graphical code generation model to obtain a sequence of semantic patch instructions describing the modification operation.

[0076] The pre-defined graphical code semantic patching language specification is a set of instructions defined in this application used to describe modifications to intermediate representation text. This specification defines a series of atomic operation instructions specifically for adding, deleting, and modifying node or edge objects in the intermediate representation text. For example, it includes instructions such as ADD_NODE (add node), REMOVE_NODE (delete node), and UPDATE_PARAM (modify node parameters). The semantic patching instruction sequence refers to a sequence, typically in JSON format, generated by the graphical code generation model based on the modification intent, consisting of one or more pre-defined graphical code semantic patching language specification instructions.

[0077] Specifically, the server inputs the iterative modification prompt text built by S207, along with the syntax description and examples of the preset graphical code semantic patching language specification, into the graphical code generation model. At this point, the model's task is no longer to generate complete intermediate representation text, but rather to generate a list of instructions describing how to change from the old version to the new version. For example, for the instruction to change the yellow light duration to 2 seconds, the model no longer regenerates the entire traffic light logic, but outputs a semantic patching instruction sequence, such as: [{"operation":"UPDATE_PARAM","payload":{"node_id":"delay_yellow_light","param_name":"duration","new_value":2000}}]. This breaks down a complex problem of modifying structured data into two sub-problems: the model generating an abstract description of the modification intent and the program executing that abstract description. The model only handles its strongest areas: semantic understanding and intent transformation.

[0078] In some embodiments, semantic patch generation can be achieved in several ways: Optionally, a detailed preset graphical code semantic patch language specification can be provided in the prompt text. Specific steps include: appending the definitions, parameter descriptions, and examples of all available instructions in the preset graphical code semantic patch language specification to the prompt text for each request, ensuring that even general-purpose large models can understand and follow the specification for output. Then, the model generates a JSON instruction sequence conforming to the specification based on the context and user instructions. Optionally, the model can also internalize the preset graphical code semantic patch language specification through fine-tuning. Specific steps include: constructing a training dataset containing a large number of current code + modification instructions -> semantic patches, and using this dataset to perform supervised fine-tuning on a base code model. After fine-tuning, the model learns the preset graphical code semantic patch language specification, and during inference, it no longer needs to provide the complete language specification in the prompt; simple instructions are sufficient to generate patches, thereby shortening the prompt text length and improving inference efficiency.

[0079] S209. After the semantic patch instruction sequence has passed the validity check, the server applies the semantic patch instruction sequence one by one to the intermediate representation text of the graphical code before modification, and obtains the modified intermediate representation text.

[0080] The validity check refers to a verification process used to ensure that each patch instruction generated by the model is valid and executable. This includes checking for the existence of node IDs and the correctness of parameter names within the instructions. It is applied to represent the actual addition, deletion, and modification operations performed on intermediate representation text objects in memory based on the instruction content. Modifying the intermediate representation text refers to the intermediate representation text obtained after applying all patch instructions, representing the new program logic.

[0081] Specifically, the server first receives the original intermediate representation text from S206 and the semantic patch instruction sequence generated in S208, and then performs a validity check on the instruction sequence. For example, for the UPDATE_PARAM instruction, it checks whether the node_id specified in the instruction exists in the original intermediate representation text, and whether the node actually has the param_name specified in the instruction. If the check fails, the server can refuse to modify and report an error to the user, or feed the error information back to the LLM for correction. After the check passes, the server parses the operation type, target object, and operation parameters of each instruction in the execution order of the instruction sequence. When the operation type is an add operation, the application inserts a new node object or edge object at the specified position in the intermediate representation text. When the operation type is a delete operation, the application removes the specified node object or edge object and its related connections from the intermediate representation text. When the operation type is a modify operation, the application updates the attribute parameters of the specified node object or edge object. After all instructions are executed, the original intermediate representation text is transformed into the new modified intermediate representation text.

[0082] S210. The server generates graphical code for modification instructions based on the modified intermediate representation text.

[0083] Specifically, the server takes the modified intermediate representation text generated in step S209 as input and passes it to the same compiler as in step S205. This compiler doesn't care whether the intermediate representation is newly generated or modified through a patch; it only converts any valid intermediate representation text into the standard XML format required by the target graphical programming platform. After compilation, the server sends the newly generated XML graphical code to the client. Upon receiving the new XML, the client software refreshes or replaces the current view, and the user can immediately see on the screen that the requested changes (e.g., the delay time next to the yellow light block changes from 1 second to 2 seconds) have taken effect. Through this step, the server completes a closed loop from natural language modification instructions to real-time updates of the visual code.

[0084] In the above embodiments, when a user generates graphical code and then provides modification suggestions, the graphical code generation model no longer directly regenerates new graphical code. Instead, it generates a semantic patch instruction sequence consisting of atomic operations that describes how the code should be modified. Subsequently, the graphical code generation model generates graphical code for the modification instructions based on these instructions, thereby improving the efficiency of generating graphical code.

[0085] The above describes a graphical code generation method in the embodiments of this application. The following describes an exemplary server 300 provided in the embodiments of this application.

[0086] Figure 3This is a schematic diagram of an exemplary hardware structure of a server 300 provided in an embodiment of this application. In some embodiments, the server 300 is a computer device. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers via a network connection. In some embodiments, the network interface can be a wired network interface; in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, it implements a graphical code generation method according to an embodiment of this application.

[0087] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0088] In some embodiments of this application, a computer-readable storage medium is also provided, including instructions that, when executed on the server 300, cause the server 300 to perform a graphical code generation method according to an embodiment of this application.

[0089] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0090] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0091] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A graphical code generation method, applied to a server, characterized in that, The method includes: When the server receives voice input from the user containing functional requirements, it converts the voice input into text information. The server converts the text information into a query vector based on a preset code embedding model. The server retrieves a preset number of visual programming components that are semantically closest to the query vector from the vector database as a candidate component set. The vector database is obtained by vectorizing all visual programming components and function descriptions of the target graphical programming platform collected in advance based on the preset code embedding model. The server concatenates the text information, the candidate component set, and the identifiers and function descriptions of each visual programming component in the candidate component set into a structured prompt text according to a preset format. The server inputs the structured prompt text into a preset graphical code generation model to obtain graphical code that corresponds to the text information and conforms to the target graphical programming platform specifications.

2. The method according to claim 1, characterized in that, The training process of the preset graphical code generation model specifically includes: The server acquires paired data of natural language descriptions and graphical codes to obtain a training dataset; The server converts the graphical code into a structured text format containing node definitions and edge connections. The structured text format describes the visual programming components through node objects and the connection relationships between the visual programming components through edge objects. The node objects contain unique identifiers, component type identifiers, and parameter configuration dictionaries. The edge objects define the output ports of the source nodes and the input ports of the target nodes. The server constructs a graphical code generation model based on an encoder-decoder architecture. The encoder is responsible for understanding natural language intent, and the decoder is responsible for generating the structured text format. After the graphical code generation model is pre-trained based on a general code generation task, the server performs supervised fine-tuning training based on the training dataset to obtain the preset graphical code generation model.

3. The method according to claim 2, characterized in that, The step of the server inputting the structured prompt text into a preset graphical code generation model to obtain graphical code that corresponds to the text information and conforms to the target graphical programming platform specification specifically includes: The server inputs the structured prompt text into the preset graphical code generation model to obtain intermediate representation text that conforms to the structured text format; The server extracts the node set, edge connection relationship, and parameter configuration information from the intermediate representation text; The server converts the set of nodes and the edge connection relationships into graphical code that conforms to the Extensible Markup Language specification of the target graphical programming platform.

4. The method according to claim 3, characterized in that, Prior to the step whereby the server extracts the node set, edge connection relationships, and parameter configuration information from the intermediate representation text, the method further includes: The server performs dependency analysis on the intermediate representation text to obtain a call dependency graph between components; The server detects whether there are circular dependencies and deadlock issues based on the call dependency graph; If so, the server generates a warning message; If not, the server optimizes the execution order of components in the intermediate representation text by topological sorting according to the execution timing specifications of the target graphical programming platform.

5. The method according to claim 3, characterized in that, After the step of the server converting the node set and the edge connection relationships into graphical code conforming to the Extensible Markup Language specification of the target graphical programming platform, the method further includes: The server uses the intermediate representation text of the graphical code as context information in the session state; When a user's modification instruction is received, the server constructs an iterative modification prompt text containing the modification instruction and the context information; The server inputs the iterative modification prompt text and the preset graphical code semantic patch language specification into the preset graphical code generation model to obtain a semantic patch instruction sequence describing the modification operation; After the semantic patch instruction sequence undergoes a validity check, the server applies the semantic patch instruction sequence line by line to the intermediate representation text of the original graphical code to obtain the modified intermediate representation text. The server generates graphical code for the modification instruction based on the modified intermediate representation text.

6. The method according to claim 5, characterized in that, The preset graphical code semantic patching language specification specifically includes: The graphical code semantic patching language specification defines atomic operation instructions, which are used to add, delete, or modify node objects or edge objects in the intermediate representation text.

7. The method according to claim 6, characterized in that, The server applies the semantic patch instruction sequence line by line to the intermediate representation text of the original graphical code to obtain the modified intermediate representation text. Specifically, this step includes: The server parses the operation type, target object, and operation parameters in each semantic patch instruction sequentially according to the execution order of the semantic patch instruction sequence. When the operation type is an add operation, the server inserts a new node object or edge object at a specified position in the middle representing the text; When the operation type is a deletion operation, the server removes the specified node object or edge object and its related connection relationship from the intermediate representation text; When the operation type is a modification operation, the server updates the attribute parameters of the specified node object or edge object.

8. A server, characterized in that, The server includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the server to perform the method as described in any one of claims 1-7.

9. A computer program product containing instructions, characterized in that, When the computer program product is run on the server, the server performs the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the server, the server causes the server to perform the method as described in any one of claims 1-7.

Images