Programming cognitive enhancement system and method based on augmented reality and generative AI

Abstract

The invention discloses a programming cognitive enhancement system and method based on augmented reality and generative AI, and relates to the field of programming education, the system comprises an augmented reality device layer, a data processing layer and an artificial intelligence generation layer; according to the method, code semantic information is extracted by using an abstract syntax tree, the code semantic information is mapped into a three-dimensional logic topological graph, user watching duration information is obtained through eye movement tracking, code annotations of corresponding levels are generated, and logic derivation animations of codes are generated and displayed in real time by adopting a generative artificial intelligence technology. The tactile feedback device is used for providing corresponding physical feedback signals for the user according to the code exception type, the problem that complex code logic is difficult to visually display in the prior art is effectively solved, and the method can be applied to multiple scenes such as programming education and program debugging.

Description

Technical Field

[0001] The present invention relates to the field of programming education, and more particularly, to a programming cognition enhancement system and method based on augmented reality and generative AI. Background Art

[0002] With the rapid development of computer science, the importance of programming education has become increasingly prominent. However, most traditional programming learning tools rely on text interfaces and lack intuitive representation in the spatial dimension, which poses many challenges for novice programmers during the learning process. Existing integrated development environments (IDEs) usually only provide two-dimensional views and cannot effectively display the execution process and memory state of programs. Especially during the debugging phase, it is difficult for programmers to intuitively understand the behavior and memory changes when the code runs.

[0003] At the same time, AI-assisted programming tools that have emerged in recent years (such as GitHub Copilot) have alleviated the difficulty of programming learning to a certain extent. However, the internal working mechanisms of these tools are often opaque, and users cannot effectively trace the logic and reasons for the generated code, resulting in the "black box" problem. In addition, the application of AR technology in programming education is still insufficient. Existing AR programming tools are mostly limited to simple visualization and have not been deeply integrated with the underlying compilation process.

[0004] For example, patent publication number CN113641346A discloses an intelligent auxiliary tool for standardized programming and its working method. This solution significantly reduces the ability requirements for developers through a component library module and an intelligent programming assistant module, and provides a standardized code generation function, with high practicality and efficiency.

[0005] However, this solution has certain limitations in programming learning:

[0006] This solution mainly relies on text-based code generation and component calls, lacks spatial visualization support for the program execution process and memory state, and is difficult to help developers intuitively understand the operating mechanism of the code.

[0007] Due to the lack of multimodal interaction feedback (such as tactile and visual reinforcement) in this solution, developers' memory of code logic mainly relies on repeated practice, and the learning curve is relatively gentle.

[0008] Although this solution provides standardized code, it does not deeply explain the logical basis for code generation, and it is difficult for developers to understand the decision-making process of AI suggestions, resulting in the "black box" problem. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a programming cognition enhancement system and method based on augmented reality and generative AI to solve the problems mentioned in the background art.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] A programming cognitive enhancement system based on augmented reality and generative AI includes an augmented reality device layer, a data processing layer, and an artificial intelligence generation layer;

[0012] The augmented reality device layer is used to realize the three-dimensional space mapping of the code structure and tactile feedback;

[0013] The data processing layer is used to realize code semantic parsing, logical topology graph generation, and user cognitive load evaluation;

[0014] The artificial intelligence generation layer is used to dynamically generate multi-level code annotations and logical derivation animations.

[0015] Optionally, the augmented reality device layer includes augmented reality glasses, a spatial positioning sensor, and a tactile feedback glove.

[0016] Optionally, the data processing layer includes a code semantic parsing module, a logical topology graph generation module, and a cognitive load evaluation module, where the code semantic parsing module is based on an abstract syntax tree.

[0017] Optionally, the logical topology graph generation module uses an improved Fruchterman-Reingold algorithm to implement three-dimensional topological layout, the node size is calculated according to the code complexity, and the edge strength between connected nodes is calculated by weighting according to the data coupling degree and the control coupling degree.

[0018] Optionally, the cognitive load evaluation module real-time monitors the user's gaze duration through eye tracking technology and triggers the generation of code annotations at corresponding levels.

[0019] Optionally, the multi-level code annotation module of the artificial intelligence generation layer uses a generative pre-trained Transformer model fine-tuned by a code corpus to generate real-time annotation content.

[0020] Optionally, the logical derivation animation module of the artificial intelligence generation layer presents the control flow and data flow animation effects of the code in real time based on a three-dimensional visualization engine.

[0021] Optionally, the tactile feedback glove provides corresponding physical feedback signals for different types of code exceptions, including fingertip pulse feedback for variable exceptions, continuous palm pressure feedback for memory leaks, and wrist vibration feedback for infinite loops.

[0022] Optionally, the augmented reality glasses accurately locate and display the code topology graph through spatial anchoring technology, and the spatial positioning sensor includes an inertial measurement unit and a depth camera.

[0023] The present invention also discloses a programming cognitive enhancement method based on augmented reality and generative AI, comprising the following steps:

[0024] Extract code semantic information using an abstract syntax tree;

[0025] Map the code semantic information into a three-dimensional logical topology graph;

[0026] Obtain the user's fixation duration information through eye tracking, and automatically generate corresponding-level code comments based on the fixation duration;

[0027] Adopt generative artificial intelligence technology to generate and display the logical derivation animation of the code in real time;

[0028] Use a haptic feedback device to provide corresponding physical feedback signals to the user according to the type of code anomaly.

[0029] The advantages of the present invention over the prior art are that the present invention proposes a programming assistance system based on augmented reality and generative artificial intelligence, realizes the three-dimensional space mapping of the code structure through an augmented reality device, intuitively displays the internal logical structure of the code and the data flow path, and reminds the user of the anomalies during the code running process in real time through haptic feedback, thereby effectively reducing the cognitive load of students and improving the code understanding efficiency and debugging accuracy.

[0030] Furthermore, the present invention realizes refined code semantic parsing and automatic generation of the logical topology graph through a data processing layer, and can accurately present the data dependency and control relationship between functions and modules in the code. In addition, the present invention also has a real-time user cognitive load evaluation function, dynamically adjusts the detail level of code comments through eye tracking technology, and significantly improves the user experience.

[0031] In the artificial intelligence generation layer, the multi-level code comment module proposed by the present invention adopts a generative pre-trained Transformer model to realize accurate and real-time code comment generation, effectively helping users quickly understand complex code structures and logics. At the same time, the present invention is also equipped with a logical derivation animation module to dynamically demonstrate the control flow and data flow of the code, making the code logic easier to understand and track.

[0032] In addition, the present invention provides a haptic feedback glove, which accurately generates physical feedback signals according to the type of code anomaly, enabling students to more intuitively feel the problem points during code running, helping to quickly troubleshoot problems, and significantly improving the debugging efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of the overall system architecture of the present invention;

[0034] Figure 2It is a schematic diagram of the layer structure of the augmented reality device of the present invention;

[0035] Figure 3 It is a schematic diagram of the data processing layer structure of the present invention;

[0036] Figure 4 It is a schematic diagram of the artificial intelligence generation layer and interactive feedback of the present invention. DETAILED DESCRIPTION

[0037] The specific implementation of the present invention will be described below in conjunction with the accompanying drawings.

[0038] The present invention relates to a programming assistance system based on augmented reality and generative artificial intelligence, which aims to solve the problems faced by students and children in the process of learning programming, such as difficulty in code comprehension and low debugging efficiency.

[0039] like Figure 1 As shown, the present invention includes three core parts: an augmented reality device layer, a data processing layer, and an artificial intelligence generation layer. The parts coordinate with each other to jointly provide intuitive programming support and interactive feedback.

[0040] like Figure 2 As shown in the figure, the augmented reality device layer includes augmented reality glasses, spatial positioning sensors and tactile feedback gloves. The augmented reality glasses can be selected from commercially available augmented reality head-mounted devices such as Microsoft HoloLens or Magic Leap, which mainly realize the spatial and intuitive display of the code logic structure, helping students to intuitively map the abstract code structure into the real environment.

[0041] Spatial positioning sensors include an inertial measurement unit and a depth camera. The inertial measurement unit is used to capture the user's head position and motion information, while the depth camera uses structured light or ToF technology to perform precise environmental scanning to ensure the accuracy and stability of code space mapping.

[0042] The tactile feedback gloves have built-in precision micro-vibration motors, pressure airbags or ultrasonic arrays, which can provide rich tactile prompts to achieve physical feedback of programming anomalies. When specific types of anomalies occur during code execution, such as abnormal changes in variable values, memory leaks or dead loops, the gloves generate corresponding tactile feedback signals: variable anomalies prompt students to pay attention to the problem points through slight pulse vibrations at the fingertips; memory leaks trigger continuous and gentle pressure in the palm area; and dead loops prompt students to intervene quickly through obvious vibration signals that gradually increase at the wrist.

[0043] like Figure 3As shown, the data processing layer is mainly responsible for in-depth analysis of code logic, optimization of topological layout, and real-time assessment of users' cognitive load. Specifically, it includes a code semantic parsing module, a logical topology graph generation module, and a cognitive load assessment module. The code semantic parsing module, based on the abstract syntax tree technology and with the support of the open-source ANTLR tool, supports multi-language environments (such as Java, Python, C++, C#, etc.) and conducts comprehensive and accurate structured analysis of the code.

[0044] The logical topology graph generation module uses an improved Fruchterman-Reingold layout algorithm. This algorithm first calculates the cyclomatic complexity of the code as the basis for determining the node size. Functions or modules with higher cyclomatic complexity are assigned larger node sizes to visually prompt students to pay attention to the risk areas of complex code. In addition, the algorithm also weights the edges between nodes, assigning a higher weight (such as 0.7) to data coupling degree and a lower weight (such as 0.3) to control coupling degree. In this way, the dependency relationship between code modules is clearly reflected, helping students quickly identify potential problems.

[0045] The cognitive load assessment module uses real-time eye movement tracking technology. Through the eye tracking sensors built into head-mounted eye tracking devices or augmented reality glasses, it continuously monitors the visual fixation time of students on different code segments. If the system detects that the user has fixed their gaze on a certain code segment for more than a preset threshold (such as 1 second, 3 seconds, etc.), it will automatically activate more in-depth code explanations or annotation content to ensure that the user does not generate excessive cognitive load due to being confused by complex code logic for a long time.

[0046] As Figure 4 shown, the artificial intelligence generation layer includes a multi-level code annotation module and a logical derivation animation module. The multi-level code annotation module, based on the generative pre-trained Transformer model (such as GPT-4 or later versions), can generate hierarchical annotations that highly match the code context in real time after large-scale fine-tuning with domain-specific code corpus data. These annotations cover basic syntax explanations, interpretations of data flow paths and logical relationships, suggestions at the design pattern and architecture levels, etc.

[0047] The logical derivation animation module uses the Unity 3D engine, combined with data-driven particle effects and animation generation technology, to achieve an intuitive and dynamic visualization of the code execution process. For example, in sorting algorithms or graph traversal algorithms, the data transmission path will be shown as dynamic particle flow or light band movement, clearly demonstrating the specific transmission route and conditional judgment path of the data in the algorithm, helping students quickly understand the specific execution details of the algorithm.

[0048] The tactile feedback device further enhances the intuitiveness of the user's interaction with the code, generating precise tactile feedback signals based on the specific situation of the real-time running code. The tactile feedback gloves process the abnormal information transmitted from the data processing layer in real time through the microprocessor according to the preset feedback rules, and automatically generate tactile feedback signals of corresponding intensity according to the predefined abnormal type and severity level.

[0049] The augmented reality glasses use spatial anchoring technology to achieve a stable fusion of the 3D code topology map and the real environment. The spatial anchoring technology uses the glasses' built-in spatial mapping API to detect key feature points in the physical environment (such as desktops, walls, floors, and other markers) in real time, and continuously tracks the spatial position changes of these feature points, thereby ensuring that the spatial position of the code visualization structure remains stable at all times, allowing students to interact with the code smoothly and naturally in a complex environment.

[0050] The specific implementation method of the present invention comprises the following steps:

[0051] First, the ANTLR parsing tool is used to extract the abstract syntax tree from the code to be analyzed and perform preliminary semantic annotation. Then, the annotated abstract syntax tree is mapped into a three-dimensional spatial topological structure through the logical topology graph generation algorithm. Subsequently, the augmented reality glasses display the structure in the user's physical environment in real time, and the user can control the annotation display level through eye tracking technology. At the same time, the artificial intelligence generation layer analyzes the user's gaze data in real time, and automatically generates detailed annotations and logical deduction animations that are adapted to the current focus of the code. Finally, the tactile feedback gloves monitor the code running status in real time, and generate physical feedback signals according to different abnormal types, so as to promptly remind students to pay attention to and correct specific problems.

[0052] The present invention is particularly valuable in the field of education. Programming beginners can quickly understand the internal logic of complex algorithms, such as recursive calls, loop structures, or data structure operations, through an intuitive three-dimensional visual interface, greatly shortening the learning curve. At the same time, the tactile feedback device corrects errors in real time during the coding process of beginners, intuitively enhancing the learning effect. In addition, in the field of enterprise software development and maintenance of large and complex systems, the present invention helps team members quickly identify potential problem points in the system and achieve clearer and more effective communication in remote collaboration.

[0053] Taking the recursive algorithm as an example, when students view the code through augmented reality glasses, the system will dynamically present the recursive call process in a three-dimensional tree structure in real time. Each recursive call will be clearly displayed in space and accompanied by corresponding code comments generated in real time. Students can also obtain corresponding physical feedback in a timely manner through the haptic feedback gloves when a logical error or performance bottleneck occurs during the code execution, such as a weak pulse reminder on the finger, so as to quickly find and fix the problem. In addition, students can actively operate and observe the dynamic structure of the algorithm through simple gestures, such as grasping and rotating, to deepen the intuitive understanding of the algorithm logic. This immersive and interactive teaching method effectively improves students' mastery and learning interest in complex algorithms and data structures, is significantly better than the traditional two-dimensional plane teaching mode, and greatly improves the teaching effect and learning efficiency.

[0054] As described above, it is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A programming cognitive enhancement system based on augmented reality and generative AI, characterized in that, It includes an augmented reality device layer, a data processing layer, and an artificial intelligence generation layer; The augmented reality device layer is used to achieve three-dimensional spatial mapping of code structures and haptic feedback; The data processing layer is used to achieve code semantic parsing, logical topology graph generation, and user cognitive load assessment; The artificial intelligence generation layer is used to dynamically generate multi-level code annotations and logical derivation animations.

2. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 1, wherein The augmented reality device layer includes augmented reality glasses, a spatial positioning sensor, and a haptic feedback glove.

3. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 1, wherein The data processing layer includes a code semantic parsing module, a logical topology graph generation module, and a cognitive load assessment module, where the code semantic parsing module is implemented based on an abstract syntax tree.

4. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 3, wherein The logical topology graph generation module uses an improved Fruchterman-Reingold algorithm to achieve three-dimensional topological layout. The node size is calculated according to the code complexity, and the edge strength between connected nodes is calculated by weighting according to the data coupling degree and the control coupling degree.

5. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 3, wherein The cognitive load assessment module real-time monitors the user's fixation duration through eye-tracking technology and triggers the generation of code annotations at corresponding levels.

6. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 1, wherein The multi-level code annotation module of the artificial intelligence generation layer uses a generative pre-trained Transformer model fine-tuned by a code corpus to generate real-time annotation content.

7. The programming cognition enhancement system based on augmented reality and generative AI according to claim 1, wherein The logical derivation animation module of the artificial intelligence generation layer presents the control flow and data flow animation effects of the code in real time based on a three-dimensional visualization engine.

8. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 2, wherein, The haptic feedback glove provides corresponding physical feedback signals for different types of code exceptions, including fingertip pulse feedback for variable exceptions, continuous palm pressure feedback for memory leaks, and wrist vibration feedback for infinite loops.

9. The programming cognitive enhancement system based on augmented reality and generative AI according to claim 2, wherein The augmented reality glasses accurately locate and display the code topology graph through spatial anchoring technology. The spatial positioning sensor includes an inertial measurement unit and a depth camera.

10. A programming cognitive enhancement method based on augmented reality and generative AI, characterized in that, It includes the following steps: Extract code semantic information using an abstract syntax tree; Map the code semantic information into a three-dimensional logical topology graph; Obtain the user's fixation duration information through eye-tracking and automatically generate code annotations at corresponding levels based on the fixation duration; Use generative artificial intelligence technology to generate and display the logical derivation animation of the code in real time; Use a haptic feedback device to provide corresponding physical feedback signals to the user according to the code exception type.

Images