Graphical programming system based on large language model and hardware constraint and use method

Abstract

The invention relates to the technical field of programming teaching, in particular to a graphical programming system based on a big language model and hardware constraints and a use method, and aims to realize automatic generation from a natural language to a graphical code by integrating the big language model and a Blockly graphical platform, and dynamically adapt in combination with hardware constraints of a master control development board (such as Raspberry Pi). The system comprises a client program, a knowledge base and an interactive programming interface, supports a user to control external hardware equipment through a natural language instruction, and simultaneously provides functions of code real-time mapping, dynamic debugging and teaching connection. The method reduces the programming threshold of software and hardware combination, is suitable for the fields of artificial intelligence education, Internet of Things development and robot control, and significantly improves the development efficiency and user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of programming teaching, specifically a graphical programming system based on large language models and hardware constraints and its usage method. Background Art

[0002] With the rapid development of technology, large language models (LLMs) have achieved remarkable results in the field of natural language processing and have become a research hotspot and key technology driver in the field of artificial intelligence. Large language models represented by GPT-4, DeepSeek, etc. demonstrate powerful language understanding and generation capabilities. Through pre-training on massive text data, they can handle diverse natural language tasks such as intelligent question answering, text creation, machine translation, etc. Their application scenarios are extensive, covering multiple industries such as healthcare, finance, education, and entertainment, significantly improving the information processing efficiency and intelligent service level.

[0003] In the field of software development and programming, large language models have also brought profound changes. The emergence of large language models provides new possibilities for the innovation of programming methods. It can understand the programming intent described in natural language and generate corresponding code snippets or even complete programs, greatly reducing the programming difficulty and improving the development efficiency, making programming more convenient and efficient. For example, developers only need to describe the required functions in natural language, and the large language model can generate a basic code framework. Developers can then fine-tune and optimize it on this basis to complete the program development, significantly shortening the development cycle. However, despite the great potential shown by large language models in programming assistance, there are still many challenges in practical applications. On the one hand, the code quality generated by large language models varies, with problems such as unreasonable code structures, syntax errors, and logical loopholes, making it difficult to be directly applied in complex business scenarios and key areas with extremely high requirements for code reliability.

[0004] At the same time, graphical programming, as an intuitive and visual programming method, plays an important role in lowering the programming threshold and improving programming efficiency. Through a graphical interface, graphical programming presents programming elements in the form of graphical modules. Developers only need to drag and connect these modules to complete the construction of the program, without having to write complex text code, greatly reducing the programming difficulty and making programming more intuitive and understandable.

[0005] In the existing technology, Scratch developed by the MIT Media Lab is a representative of current graphical programming systems. Scratchcopilot and creaticode combine with GPT-4 to achieve programming assistance functions such as analysis and evaluation, providing background, generating images, and answering questions, but still strictly follow the programming method of "manually dragging building blocks". The necessity and possibility of using GPT to automatically generate graphical programs are actively discussed on the Scratchmitedu forum.

[0006] In the existing technology, there is also a solution that combines graphical programming with hardware for programming and artificial intelligence teaching. Most of the software is derived and developed from blockly, and the hardware is basically developed based on single-chip microcomputers such as Arduino / microbit / control board. A relatively well-known one is the Mind+ software developed by DFROBOT, but it still strictly follows the programming method of "manually dragging building blocks". Even a product manager with programming experience posted a thread "Can graphical programming use large language models?" on the DFROBOT community forum, attempting to explore whether large language models can understand the graphical modules and logical relationships in graphical programming to achieve automatic generation of graphical code blocks.

[0007] In the existing technology, there is also a field that combines graphical hardware programming. The Chinese patent with the authorization announcement number CN110362299B proposes a programming learning system and method that combines software and hardware to solve the problems that in traditional solutions, virtual presentation makes the understanding of the programming system less intuitive and the creative expression less rich, and at the same time overcomes the problem that programming cannot be learned step by step without a graphical programming platform. However, through actual application, it is found that this solution has the following improvement requirements:

[0008] First, the code generation efficiency is limited by manual operations and cannot utilize the automation advantages of large language models.

[0009] Second, there is a lack of an adaptation mechanism for the physical constraints of the hardware, and the user's non-standard programming combinations need the development board to be restarted to return to normal. Summary of the Invention

[0010] The purpose of the present invention is to provide a graphical programming system based on large language models and hardware constraints to solve the problems raised in the above background technology.

[0011] The purpose of the present invention can be achieved through the following technical solutions:

[0012] A graphical programming system based on large language models and hardware constraints includes:

[0013] A main control development board, which is used to install a client program and connect to an external electronic device and a power supply module;

[0014] An external electronic device, including sensors, actuators, and input / output interfaces, is physically connected to the main control development board;

[0015] A large language model for parsing natural language prompts and generating graphical code;

[0016] A graphical programming platform knowledge base storing the working principle of the graphical programming platform, the working mechanism of specific graphical function modules, constraint descriptions, and a corpus of program cases;

[0017] A graphical programming platform based on Blockly, including a command area, an editing area, and a large language model interaction area, for converting prompts into graphical code and mapping it to Python code;

[0018] A client program deployed on the main control development board, including a communication management module, a code interpretation module, and a basic function library, for receiving and executing the Python code to control hardware devices.

[0019] Furthermore, the graphical programming platform knowledge base includes:

[0020] XML descriptions of graphical modules, Python code mapping rules, and hardware constraint conditions;

[0021] Implement similarity search between prompts and knowledge base corpus through a vector database to optimize the code generation logic of the large language model.

[0022] Furthermore, the client program also includes a code generation module, which is used to dynamically generate graphical code according to the feedback of the large language model and support the extension of user-defined function modules, including:

[0023] Binding of new graphical instructions and corpus entry;

[0024] Realize local knowledge update and large language model adaptation through a small knowledge base of function modules.

[0025] Furthermore, the large language model interaction area includes:

[0026] A prompt input box, which requires input of natural language instructions containing hardware connection interfaces, function descriptions, and constraint conditions;

[0027] A session record area, which displays the code generation results and error correction suggestions in real time.

[0028] Furthermore, the main control development board is a Raspberry Pi, a Banana Pi, or a Xingkong board, and the external electronic devices include an LED strip, a camera, a servo, and an IoT module, etc., and are connected to the main control development board through interfaces such as GPIO pins and USB.

[0029] Furthermore, all user-defined function blocks of the graphical programming platform are virtual modules, and their definitions and implementation methods are completely decoupled, including:

[0030] (a) Abstract interface description rules for function modules:

[0031] The function name is named semantically in English and is globally unique;

[0032] The input parameter is named in the form of param + serial number, and the parameter names within the same module are not repeated;

[0033] The generated executable code is uniformly mapped to the form of self.{function name}(param1,param2...);

[0034] (b) Natural language understanding adaptation mechanism, enabling the large language model to generate graphical code that meets requirements without knowing the underlying implementation through the following content stored in the knowledge base:

[0035] Natural language function description text and precautions for each function module;

[0036] Plain text rules for parameter value ranges and hardware constraints;

[0037] Virtual code example set generated based on the interface description rules

[0038] Another object of the present invention is to provide a usage method of a graphical programming system based on a large language model and hardware constraints, including the following steps:

[0039] S1: Initialize the programming environment, connect the main control development board to the hardware device, and start the client program and the graphical programming platform;

[0040] S2: Input natural language prompt words, generate graphical code through the large language model and map it to executable Python code;

[0041] S3: Send the Python code to the main control development board, dynamically parse and control the hardware device through the code interpretation module;

[0042] S4: Verify the execution result, and achieve function iteration and optimization by modifying the prompt words or manually adjusting the graphical code.

[0043] Furthermore, the natural language prompt words include hardware interface numbers, function logic, and the number of loops, and filter invalid or conflicting instructions through the constraint conditions of the knowledge base.

[0044] Furthermore, the code interpretation module supports dynamic code injection, interrupting and updating the code logic in real time during program operation to ensure the real-time performance and stability of hardware control.

[0045] Furthermore, it also includes the step of user-defined graphical instructions:

[0046] A1: Implement Python functional functions in the client program;

[0047] A2: Bind the functional functions to graphical instructions and enter them into the knowledge base;

[0048] A3: Verify the generation and execution of new instructions through interaction with the large language model.

[0049] Furthermore, the graphical code and Python code are mapped bidirectionally in real time, supporting seamless transition from graphical programming to text code in teaching scenarios, and automatically generating code defect repair solutions.

[0050] Advantages of the present invention:

[0051] The present invention solves the problem that traditional graphical programming software must strictly follow the "manual dragging of building blocks", achieving a breakthrough in graphical programming through natural language communication; at the same time, based on the understanding of hardware constraints, it can also display the hardware connection schematic diagram synchronized with the code, and real-time mapping of Python code. This not only enables beginners to start actual perception and experience more quickly, but also paves the way for teaching connection, greatly reducing the programming threshold and cost of software and hardware combination, making programming more attractive and more capable of stimulating the creativity and imagination of primary and secondary school students. Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings;

[0053] Figure 1 is the architecture diagram of the present invention;

[0054] Figure 2 is the diagram of the Raspberry Pi development board connected to the light strip;

[0055] Figure 3 is the corresponding diagram of the pin relationship of the Raspberry Pi development board;

[0056] Figure 4 is the connection schematic diagram of the light strip;

[0057] Figure 5 is the schematic diagram of the graphical structure and XML description corpus;

[0058] Figure 6It is a schematic diagram of the thinking process for the large language model to sequentially light up the LED beads numbered 0 to 59 on the light strip, with a red color and a time interval of 1 second;

[0059] Figure 7 It is a schematic diagram of the graphical program for the large language model to sequentially light up the LED beads numbered 0 to 59 on the light strip, with a red color and a time interval of 1 second;

[0060] Figure 8 It is a schematic diagram for the large language model to understand new graphical instructions and achieve autonomous programming; Specific implementation manners

[0061] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0062] Embodiment 1:

[0063] Please refer to Figures 1 to 8 , in the embodiment of the present invention, a graphical programming system based on a large language model and hardware constraints includes:

[0064] The main control development board is used to install the client program and connect to external electronic devices and a power module (the power module is a 5V power supply taking the Raspberry Pi as an example); such as the Raspberry Pi, Banana Pi, Xingkong Board, etc. Here, the Raspberry Pi, a microcomputer the size of a credit card, is taken as an example for illustration;

[0065] External electronic devices include input / output devices (such as light strips, weighing instruments, radio frequency cards) and sensors (such as temperature sensors, photoelectric sensors);

[0066] The large language model is used to parse natural language prompts and generate graphical code; it is an artificial intelligence model that can understand and generate human language. Currently, relatively mature ones include DeepSeek, ChatGPT-4o, Grok3. Here, deepseek-r1 is taken as an example for illustration;

[0067] The graphical programming platform knowledge base stores the working principle of the graphical programming platform, the working mechanism of specific graphical function modules, constraint descriptions, and a corpus of program cases;

[0068] The graphical programming platform based on Blockly includes a command area, an editing area, and a large language model interaction area, and is used to convert prompts into graphical code and map them to Python code;

[0069] The working principle of the graphical programming platform is an explanation of the combination of Blockly and the main control development board;

[0070] General description example: All custom function blocks are virtual modules. Their Python code representation is self.function(param1), where function is the English name of the function meaning (cannot be repeated), param1 is the parameter name, and all parameters of custom function blocks are represented in the form of param + serial number (cannot be repeated); Personalized supplementary description example: When using conditional judgment, only the bool type is supported.

[0071] Graphical modules include software function modules and hardware function modules. The corpus corresponding to the graphical instructions included includes the specific functions of the instructions, restrictions, JSON descriptions, graphical module style descriptions (XML), Python code, parameter and return value descriptions, etc. Since all custom function blocks are virtual modules and do not involve specific technical implementations, it is easy to clearly describe the functions and organizational forms of a function module in natural language and make the large model understand. Moreover, the large model can understand 1 function module and can also understand N function modules in the same way. For example, some of the corpus of the blocks for controlling the brightness and color of the light strip, as Figure 5 shown: The above XML description is a function module for controlling the color and brightness of the target small lights of the light strip. The Python code self.robot_ledstrip_setpixelcolor(param1, param2) corresponds to the value of type in json_cmd in the function library.

[0072] Among them, param1 represents which small light in the light strip is specifically set; param2 is to set the color and brightness of the small light, which is composed of a three-dimensional list to represent the RGB value, and the default is the self.robot_color(param3, param4, param5) module; robot_color is a module used to represent the three-dimensional list of colors; param3 is the first data of the three-dimensional list, controlling the red brightness, with a minimum of 0 and a maximum of 255; param4 is the second data of the three-dimensional list, controlling the green brightness, with a minimum of 0 and a maximum of 255; Param5 is the third data of the three-dimensional list, controlling the blue brightness, with a minimum of 0 and a maximum of 255.

[0073] The corpus of the program case set is the combined program and function description implemented by the client program and the function modules supported by the programming platform according to different application scenarios. The program is divided into two versions, XML and Python, and the two versions can be converted to each other, but both versions provide better understanding for the large model language;

[0074] The client program is deployed on the main control development board, including a communication management module, a code interpretation module and a basic function library, and is used to receive and execute the Python code to control the hardware device.

[0075] The basic function library consists of an electronic module driver package, an artificial intelligence function package, an Internet of Things function package, a robot motion control function package, and a general software function package, providing basic functions.

[0076] The communication management module is used to receive the Python code sent by the programming platform.

[0077] The basic function library uses Python language to encapsulate and implement many specific functions, such as sensor input operations, motion control, IoT communication, model loading and reasoning, etc.

[0078] The core of the code interpretation module is implemented as a Python class, whose instantiated objects have dynamic code injection and execution capabilities. Specifically, the external code snippets received by the communication management module (including method combinations called in the form of self.function(), where self refers to the current instance of the class) will be integrated into the instance context and then executed.

[0079] The code generation module refers to the client program itself that implements the programming platform's automatic programming function through a large language model and a small knowledge base of functional modules.

[0080] The functional module small knowledge base includes the small knowledge base that comes with the system image when it leaves the factory and the user-defined knowledge base.

[0081] The programming platform is a graphical programming platform, and its graphical programming function is implemented based on Blockly. The programming platform consists of a command area, a large language model interaction area, a programming platform knowledge base, an editing area, a debugging command output area, and a toolbar.

[0082] The command area consists of a basic logic module, a Python built-in function module, a software function module, and a hardware function module. Each module contains numerous graphical instructions, and all graphical instructions are entered and stored in the graphical command library. Each graphical instruction consists of a JSON data combination composed of a graphical state description and Python code. Among them, the graphical state description is transformed into a graphical module by the "graphical command parser based on Blockly" and presented on the programming platform. When forming a graphical program, graphical code and Python code combinations are generated simultaneously. The Python code corresponding to the graphical instructions contained in the basic logic module and the Python built-in function module is standard Python code. The Python code corresponding to the graphical instructions contained in the software function module and the hardware function module is an instantiated object of the "code interpretation module" plus functional functions. For example, self.ledOff(), where self is the instantiated object of the "code interpretation module" and ledOff() is the Python interface encapsulated and implemented by the hardware function module library. The Python code corresponding to the graphical instructions contained in these two modules is collectively referred to as custom function code.

[0083] The software function module is the general term for function modules supported by the main control development board but not involving hardware implementation, such as face recognition, speech recognition, text recognition, intelligent answering, Internet of Things, big data, etc. Specific functions are implemented locally by the client program or by calling the cloud server interface.

[0084] The hardware function module is the general term for all function modules related to physical hardware, such as LED lights, cameras, light strips, displays, printers, gesture control boards, various sensors, voice interfaces, analog-digital interfaces, robot dogs, robotic arms, and other types of physical entity devices. The graphical instructions corresponding to them are the drive commands for various devices in the client program. It is not only possible to read the data of input devices (sensors) through graphical instructions but also to control output devices (such as light strips) through graphical instructions.

[0085] The large language model interaction area consists of a user prompt input box and a session display record area. The session display record area will record the user's input requirements and display the feedback from the large language model and the analysis results of the programming platform. If the prompt is reasonable, the feedback from the large language model can be directly transformed into graphical code after analysis, and the graphical program will be displayed in the editing area. If the prompt is unreasonable or an error occurs during the transformation into graphical code, the possible reasons for the error and reference prompts will be feedback in the session display record area.

[0086] The prompt needs to have a functional description and hardware constraints, that is, the prompt needs to enable the large model to understand what specific hardware devices are used, which interfaces of the main control development board the hardware devices are specifically connected to, and what functions are achieved, etc. For example, the LED strip is connected to GPIO18, there are 60 lights in total, first the red lights are lit one by one, then the green lights are lit one by one in reverse, repeating 101 times.

[0087] The programming platform knowledge base includes the corpus of the graphical programming platform knowledge base and the vector database. The vector database has already saved the vectorized corpus of the graphical programming platform knowledge base.

[0088] The editing area is the area for programming by dragging and dropping graphical instructions. For the graphical code generated by the large language model, users can drag the graphical instructions from the command area to the editing area and combine and modify them in the form of building blocks according to the programming logic to form a new graphical code program.

[0089] The toolbar has functions such as running the program, sharing the code, saving the code, and the code library. Users can send the Python code corresponding to the graphical code program in the current editing area to the Raspberry Pi for execution by clicking the run program button. Users can share the graphical code program they have implemented with other users through the share code function. Users can save the graphical code program they have dragged and dropped in the editing area through the save code function. Users can view the code they have saved and the code shared by other users through the code library function.

[0090] Embodiment 2:

[0091] Please refer to Figures 1 to 8 , on the basis of Embodiment 1, this embodiment provides a usage method of a graphical programming system based on a large language model and hardware constraints, including the following steps:

[0092] S1: Initialize the programming environment, connect the main control development board to the hardware device, and start the client program and the graphical programming platform; specifically, first insert the microSD card pre-burned with the client program into the card slot of the Raspberry Pi, then connect the power module, display device, mouse, keyboard, and network cable to the Raspberry Pi development board, wait for the Raspberry Pi development board to start up normally and connect to the network normally; then open the programming platform and enter the programming interface; the client program runs automatically when the Raspberry Pi starts.

[0093] S2: Input a natural language prompt, generate graphical code through the large language model and map it to executable Python code;

[0094] S3: Send the Python code to the main control development board, and dynamically parse and control the hardware device through the code interpretation module;

[0095] S4: Verify the execution result and implement function iteration optimization by modifying the prompt or manually adjusting the graphical code.

[0096] Among them, the natural language prompt includes the hardware interface number, functional logic, and number of loops, and filters invalid or conflicting instructions through the constraint conditions of the knowledge base.

[0097] Among them, the code interpretation module supports dynamic code injection, interrupts and updates the code logic in real time during program operation, and ensures the real-time performance and stability of hardware control.

[0098] Among them, it also includes the steps of user-defined graphical instructions:

[0099] A1: Implement Python functional functions in the client program;

[0100] A2: Bind the functional function to the graphical instruction and enter it into the knowledge base;

[0101] A3: Verify the generation and execution of new instructions through interaction with the large language model.

[0102] Among them, the graphical code and Python code are mapped bidirectionally in real time, support seamless transition from graphical programming to text code in teaching scenarios, and automatically generate code defect repair solutions.

[0103] Refer to Figure 8 , to illustrate how to make the large language model understand new graphical instructions and implement autonomous programming:

[0104] 1. Implement specific functions with python code on the client program.

[0105] 2. Manually add the binding of the python functional function and the graphical command in the graphical command library, that is, a new graphical command is added to the graphical command library of the programming platform.

[0106] 3. After opening the programming platform, the newly added graphical command can be seen in the command area.

[0107] 4. Import the corpus such as the function description, constraint description, and cases of the graphical instruction into the knowledge base of the programming platform.

[0108] 5. After opening the programming platform, enter the prompt in the large language model interaction area to confirm that the feedback can understand the new corpus.

[0109] 6. Enter the prompt in the large language model interaction area to confirm that the graphical command can be automatically loaded into the editing area of the graphical programming platform.

[0110] 7. Enter complete functional prompts in different forms in the large language model interaction area to confirm that a complete graphical program can be automatically generated; if there is an error in the return, update the prompt according to the specific situation of the error, such as making the description more complete, adding precautions for special cases, using case programs and program descriptions, etc. Repeat steps 4 - 7 to automatically obtain the correct graphical program.

[0111] 8. Click to run the program.

[0112] 9. Send the Python code corresponding to the graphical block program to the Raspberry Pi client program.

[0113] After the communication management module instance of the client program receives the message (i.e., a section of Python code), it sends it to the code interpretation module instance for parsing. The parsed Python code, when running, calls the function implemented in the aforementioned 1 through the code interpretation module instance itself to achieve the function.

[0114] Compared with the prior art, the present invention does not require programming foundation. As long as one can understand the instruction manual, graphical programming case reproduction can be achieved through natural language communication. Students only need to master basic programming knowledge, and can organize software modules (such as artificial intelligence, Internet of Things, general software functions, etc.) and hardware modules (such as various input / output sensors, etc.) supported by the programming platform through natural language to organize prompts to achieve very complex (original) functions and solve practical problems.

[0115] The comparison between natural language logic and program execution results is directly visible. Through the prompts organized by natural language, it can be directly compared with the graphical code program automatically generated by the large language model, and the program execution results can be directly seen.

[0116] For example, if the light strip is connected to GPIO18, light up the second and third small lights of the light strip, first display red, and then display green after 0.2 seconds, and repeat 20 times.

[0117] Under the condition that the main task of the robot runs normally, the balance between generality and personalization requirements is achieved through graphical programming of the large language model and hardware constraints, meeting the rapid development of teaching and competition special themes.

[0118] For example, the quadruped robot dog has a patrol function and can run autonomously when powered on; add the function of raising the left foot to say hello after recognizing the user himself. Users only need to organize prompts through natural language to quickly complete the adjustment of personalized requirements. And the constraints of the programming platform and the addition of the development control hardware function to the background main program of the quadruped robot dog ensure stable operation.

[0119] The small knowledge base of the function module also supports users to establish a local small knowledge base and dynamically obtain feedback from the large language model to make personalization more intelligent.

[0120] The innovative teaching connection of the present invention includes graphical code, teaching demonstration cases, Python code implementation correspondence, and automatic generation of code defect repair solutions, thus truly achieving seamless connection and personalized guidance from graphical programming to Python programming learning.

[0121] The development mode is prominent: natural language description → large language model generates graphical code → sandbox verification → physical execution, replacing the traditional process: manual dragging → static compilation → direct burning.

[0122] It has cross-platform compatibility and cross-functional compatibility. Since all custom function blocks (graphical modules) are virtual modules, as long as the natural language description is reasonable, the large language model can understand and complete automated graphical programming according to the reasonable requirements of users.

[0123] Therefore, the present invention solves the problem that traditional graphical programming software must strictly follow the "manual dragging of building blocks", achieving a breakthrough in graphical programming through natural language communication; at the same time, based on the understanding of hardware constraints, it can also display the hardware connection schematic diagram synchronized with the code and map the Python code in real time. This not only enables beginners to start actual perception and experience more quickly but also paves the way for teaching connection, greatly reducing the programming threshold and cost of software-hardware combination, making programming more attractive and more capable of stimulating the creativity and imagination of primary and secondary school students.

[0124] The above describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification is only the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed.

Claims

1. A graphical programming system based on a large language model and hardware constraints, characterized by: include: Main control development board, used to install client programs and connect external electronic devices and power modules; External electronic devices, including sensors, actuators, and input and output interfaces, are physically connected to the main control development board; A large language model to parse natural language prompts and generate graphical code; The graphical programming platform knowledge base contains the working principle of the graphical programming platform, the working mechanism of specific graphical function modules, constraint descriptions, and program case collection corpus; A graphical programming platform based on Blockly, including a command area, an editing area, and a large language model interaction area, which is used to convert prompt words into graphical codes and map them into Python codes; The client program is deployed on the main control development board, including a communication management module, a code interpretation module and a basic function library, and is used to receive and execute the Python code to control the hardware device.

2. The graphical programming system based on a large language model and hardware constraints according to claim 1, characterized in that: The graphical programming platform knowledge base includes: XML description of graphical modules, Python code mapping rules, and hardware constraints; The similarity search between prompt words and knowledge base corpus is realized through the vector database, and the code generation logic of large language models is optimized.

3. The graphical programming system based on a large language model and hardware constraints according to claim 1, characterized in that: The client program also includes a code generation module, which is used to dynamically generate graphical codes according to the feedback of the large language model and supports the expansion of user-defined function modules, including: Added new graphical command binding and corpus entry; Localized knowledge update and large language model adaptation are achieved through the small knowledge base of functional modules.

4. The graphical programming system based on a large language model and hardware constraints according to claim 1, characterized in that: The large language model interaction area includes: The prompt word input box requires the input of natural language instructions including hardware connection interface, function description and constraint conditions; The session record area displays code generation results and error correction suggestions in real time.

5. The graphical programming system based on a large language model and hardware constraints according to claim 1, characterized in that: The main control development board is a Raspberry Pi, a Banana Pi or a Skyboard, and the external electronic devices include an LED light strip, a camera, a servo and an Internet of Things module, and are connected to the main control development board via GPIO pins.

6. The graphical programming system based on a large language model and hardware constraints according to claim 1, characterized in that: All user-defined functional building blocks of the graphical programming platform are virtual modules, and their definition and implementation are completely decoupled, including: (a) Abstract interface description rules for functional modules: Function names are named in semantic English and are globally unique; Input parameters are named in the form of param+serial number, and parameter names are not repeated in the same module; The generated executable code is uniformly mapped to the form of self.{function name}(param1,param2...); (b) Natural language understanding adaptation mechanism: Through the following contents stored in the knowledge base, the large language model can generate graphical code that meets the requirements without knowing the underlying implementation: Natural language functional description text and notes for each functional module; Explicit rules for parameter value ranges and hardware constraints; A set of virtual code examples generated based on the interface description rules.

7. A method for using the graphical programming system based on a large language model and hardware constraints as claimed in claim 1, characterized in that: The following steps are involved: S1: Initialize the programming environment, connect the main control development board and hardware devices, and start the client program and graphical programming platform; S2: Input natural language prompts, generate graphical codes through the large language model and map them into executable Python codes; S3: Send the Python code to the main control development board, and dynamically parse and control the hardware device through the code interpretation module; S4: Verify the execution results and implement iterative optimization of functions by modifying prompt words or manually adjusting graphical codes.

8. The method of use according to claim 7, characterized in that: The natural language prompts include hardware interface numbers, functional logic, and cycle times, and invalid or conflicting instructions are filtered through constraints in the knowledge base.

9. The method of use according to claim 7, characterized in that: The code interpretation module supports dynamic code injection, interrupts and updates code logic in real time during program execution, and ensures the real-time and stability of hardware control.

10. The method of use according to claim 7, characterized in that: It also includes steps for user-defined graphical instructions: A1: Implement Python functions in the client program; A2: Bind the function to the graphical instruction and enter it into the knowledge base; A3: Interactively verify the generation and execution of new instructions through a large language model.

11. The method of use according to claim 7, characterized in that: The graphical code and Python code are mapped in real time in both directions, supporting a seamless transition from graphical programming to text code in teaching scenarios, and automatically generating code defect repair solutions.

Images