Scratch virtual teaching demonstration method based on large language model

Abstract

A virtual demonstration and interactive teaching method for Scratch based on a large language model. The method includes: converting user-input learning intent into a predefined format using a publicly available large language model; capturing images of the visible area of ​​the Scratch application window, analyzing them using a computer vision processing framework and algorithms, and constructing a component state tree for the application window; predicting the next possible target component and interactive action based on the user intent and the text sequence of the component state tree using a large language model fine-tuned on a predefined dataset, and generating a textual explanation of the prediction results; simulating real interactive events and executing the predicted interactive actions; recapturing the application window content and calculating the differences in the component state tree; and repeating the above steps until the learning intent is satisfied. This invention can be applied to practical teaching scenarios based on the Scratch learning platform, providing personalized teaching guidance to students without time and space limitations by simulating teacher teaching activities.

Description

Technical Field

[0001] This invention belongs to the field of virtual teaching technology, specifically involving a Scratch virtual teaching demonstration method based on a large language model. Background Technology

[0002] As the education sector continues to develop, traditional teacher-led teaching methods are gradually revealing their shortcomings in terms of efficiency and personalization. Teachers have limited time and energy, making it difficult to provide individualized guidance based on each student's learning progress and comprehension ability. This is especially true on some programming learning platforms, where students often require personalized instruction when understanding complex concepts and operations. To address this challenge, educational technology is exploring automated teaching methods to replace the traditional teacher role.

[0003] Scratch, a graphical programming platform for children and beginners, has become an important tool for promoting programming education globally due to its intuitive interface and interactivity. However, traditional Scratch teaching models rely on in-person teacher instruction and student self-study, leading to uneven distribution of educational resources, low efficiency, and difficulty in dynamically responding to user needs. Therefore, combining virtual teaching methods with the Scratch platform to replace the teacher's role has become an important research direction for improving educational efficiency and quality. This automated teaching method can facilitate knowledge transfer. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and provide a Scratch virtual teaching demonstration method based on a large language model that integrates open source resources, reduces costs, and dynamically responds to user needs.

[0005] The technical solution adopted to solve the above technical problems is: a Scratch virtual teaching demonstration method based on a large language model, including the following steps:

[0006] Step 1. Train a large language model

[0007] Choose an open-source large language model and train it using a predefined dataset to enable it to format user learning intentions, predict interactive actions and goals, evaluate execution results and state tree logic code;

[0008] The formatted user learning intent is to process the user's input learning intent text into a specific format with clearly defined key data fields;

[0009] The interaction action and target prediction are based on the current component state tree, user intent, and interaction history to predict the next interaction action and target component, and generate a text explanation of the prediction results.

[0010] The execution result status evaluation is based on user feedback to determine whether the current interaction has achieved the final learning intention;

[0011] The component state tree logic code generation is to convert the state subtree of the component state tree related to the code editing area into a logic code text sequence;

[0012] Step 2. Determine the user's learning intent

[0013] The user's input text of learning intent is formatted using a large language model to ensure that it contains the necessary key data fields;

[0014] Step 3. Build the component state tree

[0015] The system obtains the window handle of the Scratch desktop application by calling the Win32 API interface, and then obtains the position and size of the visible content area of ​​the window to ensure that the window is in the foreground and in focus. The OpenCV package is used to capture the image of the visible content area of ​​the window and process the captured image to divide the functional area, and extract the size, position and text information of the components in each area to build the component state tree.

[0016] Step 4. Large Language Model Prediction

[0017] The trained large language model predicts the next interaction action and target component based on the formatted user learning intent, component state tree, and interaction history, and generates corresponding text explanations.

[0018] Step 5. Perform the interactive action

[0019] Based on the prediction results from step 4, the system calls the mouse and keyboard simulation software package to execute the predicted interactive actions;

[0020] Step 6. Determine the differences between component state trees

[0021] After the interaction is completed, step 3 is executed again to obtain a new component state tree. The difference between the old and new component state trees is calculated to evaluate the effect of the interaction.

[0022] Step 7. Result evaluation, iterative interaction, and state storage

[0023] Based on the evaluation results, determine whether the final learning intention has been achieved. If not, continue to execute steps 4, 5, and 6 in a loop, and store the result of each interaction as an interaction history in the form of a quadruple for subsequent analysis and adjustment of teaching strategies.

[0024] Preferably, the functional areas in step 3 include a block type switching area, a block selection area, a code editing area, a stage display area, a character editing area, and a stage switching area.

[0025] Preferably, the method for extracting the size, position, and text information of components in each region and constructing a component state tree is as follows:

[0026] The color image of each component is converted to a grayscale image, and then the grayscale image is converted to a binary image to facilitate the identification of the component shape. Noise in the binary image is removed to improve the clarity of the component outline. An edge detection algorithm is used to detect the edges in the image, and a contour detection algorithm is used to detect the contours in the image. The bounding rectangle of the component is extracted to obtain the position and size of the component. Filtering is performed based on the size of the component outline area to remove noise that does not meet the requirements. A shape matching algorithm is used to identify the shape of the component and confirm the type of the component.

[0027] By applying optical character recognition methods, text information is extracted from component images to obtain the component's type, function, and status value.

[0028] The extracted component information is organized into a tree structure to form a component state tree, where each node represents a component and contains the component's size, position, and text information.

[0029] Preferably, the method for calculating the difference between the old and new component state trees is one of the following: a recursive comparison method based on depth-first or breadth-first search, a hash value comparison method, or a tree edit distance method.

[0030] Preferably, the method for step 7 to determine whether the final learning intention has been achieved based on the evaluation results is as follows:

[0031] The system prompts the user to manually execute the code for the current block building. After the user executes the code, the system obtains the difference between the code execution result and the expected result through a feedback mechanism.

[0032] The prompting task of the large language model is set to "evaluation mode". The large language model judges whether the current interaction has achieved the final learning intention based on the user's feedback. If the user's feedback indicates that the code execution result is consistent with the expectation, the large language model outputs "yes" to indicate that the learning intention has been achieved; otherwise, it outputs "no" to indicate that the learning intention has not been achieved.

[0033] Preferably, the method for step 7 to determine whether the final learning intention has been achieved based on the evaluation results is as follows:

[0034] The prompting task of the large language model is set to "generation mode". The large language model converts the user's learning intention into corresponding block code, extracts the state subtree of the code editing area from the component state tree, and further obtains the block code.

[0035] Use code comparison methods to analyze whether the current interaction meets the preset learning objectives and final learning intentions;

[0036] The code comparison method is either a comparison method based on token sets, an embedding representation comparison method based on deep learning models, or a similarity comparison method based on syntax trees.

[0037] The beneficial effects of this invention are as follows:

[0038] This invention provides personalized teaching content and guidance based on each student's learning intentions and progress, meeting the diverse learning needs of students. By simulating realistic interactive actions, the system can interact with students, enhancing their learning interest and participation. This invention is not limited by time or space; students can learn anytime, anywhere, increasing learning flexibility. Through automated teaching processes, the system can quickly respond to students' learning needs, providing timely feedback and guidance, improving teaching efficiency. Simultaneously, schools can reduce the need for professional programming teachers, lowering teaching costs.

[0039] This invention, through scientific evaluation methods and rigorous teaching steps, enables the system to accurately assess students' learning outcomes, helping them better master programming knowledge and skills, lowering the threshold for programming learning, and promoting the popularization of programming education.

[0040] This invention promotes high-quality teaching resources to more regions through open-source large language models and the Scratch platform, helping schools and students in areas with scarce educational resources to obtain high-quality programming education, narrowing the educational gap between different regions, bringing new teaching models and methods to the field of education, and promoting educational innovation. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the Scratch virtual teaching demonstration method based on a large language model according to the present invention.

[0042] Figure 2 This is a schematic diagram illustrating the process of extracting components based on the OpenCV software package.

[0043] Figure 3 This is a schematic diagram of the completed component state tree.

[0044] Figure 4 This is a diagram illustrating the process of interactive actions being executed on a component. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0046] Example 1

[0047] This example illustrates the concept of "designing a chasing game".

[0048] exist Figure 1 This embodiment of a Scratch virtual teaching demonstration method based on a large language model includes the following steps:

[0049] Step 1. Train a large language model

[0050] Choose an open-source large language model for training, enabling it to format user learning intentions, predict interactive actions and goals, evaluate execution results and state tree logic code for components;

[0051] Formatting user learning intent: Processing the user's input learning intent text into a specific format with clearly defined key data fields, namely: project, role, and logic;

[0052] Interaction and target prediction: Based on the current component state tree, user intent, and interaction history, predict the next interaction action and target component, and generate a text explanation of the prediction results;

[0053] Execution result status evaluation: Determine whether the current interaction has achieved the final learning intention based on user feedback;

[0054] Component state tree logic code generation: Convert the state subtree related to the code editing area in the component state tree into a sequence of logic code text;

[0055] This embodiment selects the GLM-4-9B-Chat large language model from Zhipu AI, and trains it using the LoRA dataset in a multi-turn dialogue manner. The tasks are as follows: formatting user learning intent (Prompt task is set to Format); predicting interactive actions and targets (Prompt task is set to Predict); evaluating the execution result status (Prompt task is set to Evaluate); and generating the logic code for the component state tree (Prompt task is set to Generate).

[0056] Step 2. Determine the user's learning intent

[0057] The user's input learning intent text is formatted using a trained large language model to ensure that it contains the necessary key data fields;

[0058] For example, given the input text "Design a program that makes a kitten chase a basketball. The ball bounces off the wall and the kitten. The kitten meows when it hits the ball. Use the arrow keys to move the kitten." the large language model outputs the following specific formatted text: "Project: Chase game; Roles: kitten, basketball; Logic: The ball bounces off the wall and the kitten. Use the arrow keys to move the kitten. The kitten meows when it hits the basketball."

[0059] Step 3. Build the component state tree

[0060] The system obtains the window handle of the Scratch desktop application by calling the Win32 API interface, and then obtains the position and size (x,y,w,h) of the visible content area of ​​the window. (x,y) represents the coordinates of the upper left corner of the area, x=0,y=29, w and h represent the pixel width and height of the area, respectively, w=2560,h=1440, ensuring that the window is in the foreground and in focus.

[0061] The OpenCV package is used to capture the visible content area of ​​the window and process the captured image. An image mask is generated based on the background color of the Scratch application window. Based on this mask, a contour detection method is applied to obtain a list of all contours in the image, which is used to distinguish different functional areas.

[0062] Exclude areas smaller than 2×10 4 The outline, arranged from left to right and top to bottom, yields 6 functional areas: block type switching area, block selection area, code editing area, stage display area, character editing area, and stage switching area.

[0063] Extract the size, position, and text information of components in each functional area to construct a component state tree, such as... Figure 2 , 3 The details are as follows:

[0064] The color image of each component is converted to a grayscale image, and then the grayscale image is converted to a binary image to facilitate the identification of the component shape. Noise in the binary image is removed to improve the clarity of the component outline. An edge detection algorithm is used to detect the edges in the image, and a contour detection algorithm is used to detect the contours in the image. The bounding rectangle of the component is extracted to obtain the position and size of the component. Filtering is performed based on the size of the component outline area to remove noise that does not meet the requirements. A shape matching algorithm is used to identify the shape of the component and confirm the type of the component.

[0065] By applying optical character recognition methods, text information is extracted from component images to obtain the component's type, function, and status value.

[0066] The extracted component information is organized into a tree structure to form a component state tree T1. Each node represents a component and contains the component's size, position, and text information.

[0067] Step 4. Large Language Model Prediction

[0068] The trained large language model predicts the next interaction action and target component based on the formatted user learning intent, component state tree and interaction history, and generates corresponding text explanations. The interaction action is a finite set of mouse and keyboard events. In this embodiment, the elements in the set include: {click, drag, move, scroll wheel, input}.

[0069] In this embodiment, the interaction history at the time of the first prediction is empty. Taking the code logic "a cat will meow when it touches a basketball" as an example, the output of the large language model is "target component: block selection area - sound - play sound". <meow>Waiting for the playback to finish, in the code editing area - C3 - if < encounters <basketball>The interactive action is dragging; the text explanation is: Now that we've completed the conditional code for the kitten touching the basketball, let's write the specific content of the conditional code. Move the "Play Sound" button from the "Sound" block... <meow>Drag the "Wait for the playback to finish" block into the conditional statement code body, so that when the kitten touches the basketball, it will meow.

[0070] Step 5. Perform the interactive action

[0071] Based on the prediction results from step 4, the system calls the mouse and keyboard simulation software package to execute the predicted interactive actions;

[0072] In this embodiment, the target component types are: Block 1 and Block 2; their position information is (85,191,212,51) and (1020,729,264,109); the center positions of Block 1 and Block 2 are (191,217) and (1152,783) respectively; the specific effect of the interactive action "drag" is: move the mouse to position (191,217), press the left mouse button to trigger the Press event, move the mouse to position (1152,783), release the left mouse button to trigger the Release event. Figure 4 As shown.

[0073] Step 6. Determine the differences between component state trees

[0074] After the interaction is completed, step 3 is executed again to obtain a new component state tree T2. The difference between the old and new component state trees is calculated to evaluate the effect of the interaction.

[0075] The method for calculating the differences between the old and new component state trees is one of the following: a recursive comparison method based on depth-first or breadth-first search, a hash value comparison method, or a tree edit distance method. This embodiment uses the tree edit distance method. Component state tree T2 has one more block of code than component state tree T1, namely "play sound". <meow>"Waiting for the playback to finish", so the tree edit distance is 1. Component state tree T1 only needs to add one node to the tree to reach the state of component state tree T2;

[0076] Step 7. Result evaluation, iterative interaction, and state storage

[0077] Based on the evaluation results, determine whether the final learning intention has been achieved. If not, continue to execute steps 4, 5, and 6 in a loop, and store the result of each interaction as an interaction history in the form of a quadruple for subsequent analysis and adjustment of teaching strategies.

[0078] In this embodiment, the logic of "the kitten meows when it hits the ball and moves the kitten's position using the arrow keys" has been completed, but the logic of "the ball hits the wall and the kitten bounces away" has not yet been completed.

[0079] 7.1 Results Evaluation

[0080] To determine whether the current interaction has achieved the user's final learning intention, the system will prompt the user to manually execute the code currently built by the blocks. The system obtains the gap between the code execution result and the expected result through the feedback mechanism, sets the Prompt task of the large language model to Evaluate, inputs the user's feedback text, and the large language model determines whether the current interaction has achieved the final learning intention based on the feedback result.

[0081] Among them, the user feedback "the ball doesn't bounce off the wall and the cat" was answered by the large language model with "no", indicating that the user's final learning intention has not yet been achieved.

[0082] 7.2 Looping Interaction

[0083] The evaluation results showed that the final learning intent was not achieved. So we returned to step 3, recaptured the application window content, built a new component state tree, and re-executed steps 4, 5, 6, and 7 based on the new component state tree. This is a cyclical process until the user's final learning intent is met.

[0084] In the example, since the logic that the ball hits the wall and the kitten will not bounce off has not yet been completed, the system returns to step 3 to rebuild the component state tree and continue to perform the prediction task.

[0085] 7.3 State Storage

[0086] The output results of each step in this round are added to the global interaction history list in the form of a four-tuple. This helps the system analyze the user's operation sequence, learning progress, and difficulties, providing a reference for subsequent teaching and ensuring that students can gradually achieve the final learning goal. The four-tuple includes the component state tree, the target component, the interaction action, and the result.

[0087] Example 2

[0088] In this embodiment, the method for determining whether the final learning intention has been achieved based on the evaluation results in step 7 is as follows:

[0089] The Prompt task of the large language model is set to "Generate". The large language model converts the user's learning intention into corresponding block code, extracts the state subtree of the code editing area from the component state tree, and further obtains the block code.

[0090] In the example, the block code for the cat role generated by the large language model is as follows:

[0091] #When the green flag is clicked

[0092] #Repeat execution

[0093] #If you encounter (basketball)

[0094] #Play sound <meow>Waiting for the broadcast to finish

[0095] #When the → key is pressed#, the x-coordinate is increased by 10.

[0096] #When the up arrow key is pressed#, the y-coordinate is increased by 10.

[0097] #When you press the left arrow key#, the x-coordinate decreases by 10.

[0098] #When you press the ↓ key#, the y-coordinate decreases by 10.

[0099] #Finish

[0100] The code comparison method is used to analyze whether the current interaction meets the preset learning objectives and final learning intent. This comparison method can be a token set comparison, a deep learning model-based embedding representation comparison, or a syntax tree-based similarity comparison. In this embodiment, a deep learning model-based embedding representation comparison is used, and the output result is: {No}. The missing "Play sound" line is then added to the block code. <meow>The "Wait for the playback to finish" section is used to complete the function.

[0101] The other steps are the same as in Example 1.< / meow> < / meow> < / meow> < / meow> < / basketball> < / meow>

Claims

1. A virtual teaching demonstration method for Scratch based on a large language model, characterized in that, Includes the following steps: Step 1. Train a large language model Choose an open-source large language model and train it using a predefined dataset to enable it to format user learning intentions, predict interactive actions and goals, evaluate execution results and state tree logic code; The formatted user learning intent is to process the user's input learning intent text into a specific format with clearly defined key data fields; The interaction action and target prediction are based on the current component state tree, user intent, and interaction history to predict the next interaction action and target component, and generate a text explanation of the prediction results. The execution result status evaluation is based on user feedback to determine whether the current interaction has achieved the final learning intention; The component state tree logic code generation is to convert the state subtree of the component state tree related to the code editing area into a logic code text sequence; Step 2. Determine the user's learning intent The user's input text of learning intent is formatted using a large language model to ensure that it contains the necessary key data fields; Step 3. Build the component state tree The system obtains the window handle of the Scratch desktop application by calling the Win32 API interface, and then obtains the position and size of the visible content area of ​​the window to ensure that the window is in the foreground and in focus. The OpenCV package is used to capture the image of the visible content area of ​​the window and process the captured image to divide the functional area, and extract the size, position and text information of the components in each area to build the component state tree. Step 4. Large Language Model Prediction The trained large language model predicts the next interaction action and target component based on the formatted user learning intent, component state tree, and interaction history, and generates corresponding text explanations. Step 5. Perform the interactive action Based on the prediction results from step 4, the system calls the mouse and keyboard simulation software package to execute the predicted interactive actions; Step 6. Determine the differences between component state trees After the interaction is completed, step 3 is executed again to obtain a new component state tree. The difference between the old and new component state trees is calculated to evaluate the effect of the interaction. Step 7. Result evaluation, iterative interaction, and state storage Based on the evaluation results, determine whether the final learning intention has been achieved. If not, continue to execute steps 4, 5, and 6 in a loop, and store the result of each interaction as an interaction history in the form of a quadruple for subsequent analysis and adjustment of teaching strategies.

2. The Scratch virtual teaching demonstration method based on a large language model according to claim 1, characterized in that, The functional areas in step 3 include a block type switching area, a block selection area, a code editing area, a stage display area, a character editing area, and a stage switching area.

3. The Scratch virtual teaching demonstration method based on a large language model according to claim 1, characterized in that, The method for extracting the size, position, and text information of components in each region and constructing a component state tree is as follows: The color image of each component is converted to a grayscale image, and then the grayscale image is converted to a binary image to facilitate the identification of the component shape. Noise in the binary image is removed to improve the clarity of the component outline. An edge detection algorithm is used to detect the edges in the image, and a contour detection algorithm is used to detect the contours in the image. The bounding rectangle of the component is extracted to obtain the position and size of the component. Filtering is performed based on the size of the component outline area to remove noise that does not meet the requirements. A shape matching algorithm is used to identify the shape of the component and confirm the type of the component. By applying optical character recognition methods, text information is extracted from component images to obtain the component's type, function, and status value. The extracted component information is organized into a tree structure to form a component state tree, where each node represents a component and contains the component's size, position, and text information.

4. The Scratch virtual teaching demonstration method based on a large language model according to claim 1, characterized in that, The method for calculating the difference between the old and new component state trees is one of the following: a recursive comparison method based on depth-first or breadth-first search, a hash value comparison method, or a tree edit distance method.

5. The Scratch virtual teaching demonstration method based on a large language model according to claim 1, characterized in that, The method for determining whether the final learning intention has been achieved based on the evaluation results in step 7 is as follows: The system prompts the user to manually execute the code for the current block building. After the user executes the code, the system obtains the difference between the code execution result and the expected result through a feedback mechanism. The prompting task of the large language model is set to "evaluation mode". The large language model judges whether the current interaction has achieved the final learning intention based on the user's feedback. If the user's feedback indicates that the code execution result is consistent with the expectation, the large language model outputs "yes" to indicate that the learning intention has been achieved. Otherwise, output "No" to indicate that the learning intention has not been achieved.

6. The Scratch virtual teaching demonstration method based on a large language model according to claim 1, characterized in that, The method for determining whether the final learning intention has been achieved based on the evaluation results in step 7 is as follows: The prompting task of the large language model is set to "generation mode". The large language model converts the user's learning intention into corresponding block code, extracts the state subtree of the code editing area from the component state tree, and further obtains the block code. Use code comparison methods to analyze whether the current interaction meets the preset learning objectives and final learning intentions; The code comparison method is either a comparison method based on token sets, an embedding representation comparison method based on deep learning models, or a similarity comparison method based on syntax trees.

Images