Augmented reality and generative ai based programming cognitive enhancement system and method

Abstract

The application discloses a programming cognitive enhancement system and method based on augmented reality and generative AI, relates to the field of programming education, and comprises an augmented reality device layer, a data processing layer and an artificial intelligence generation layer; the application extracts code semantic information by using an abstract syntax tree, maps the code semantic information into a three-dimensional logical topology graph, acquires user gaze duration information by eye tracking and generates code annotations of corresponding levels, generates and displays logical deduction animations of the code in real time by using generative artificial intelligence technology, and provides corresponding physical feedback signals to the user according to code exception types by using a tactile feedback device, so that the problem that complex code logic cannot be intuitively displayed in the prior art is effectively solved, and the application can be applied to multiple scenes such as programming education and program debugging.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of programming education, more particularly, to a programming cognitive enhancement system and method based on augmented reality and generative AI. BACKGROUND

[0002] With the rapid development of computer science, the importance of programming education is increasingly prominent. However, traditional programming learning tools mostly rely on text interfaces, lacking intuitive representation in spatial dimensions, which makes novice programmers face many challenges in the learning process. Existing integrated development environments (IDEs) usually only provide two-dimensional views, which cannot effectively display the execution process and memory state of the program, especially in the debugging stage, programmers have difficulty in intuitively understanding the behavior of the code at runtime and the changes in memory.

[0003] At the same time, AI-assisted programming tools (such as GitHub Copilot) have emerged in recent years to some extent to alleviate the difficulty of programming learning, but the internal working mechanism of these tools is often opaque, and users cannot effectively track the logic and reasons of the generated code, resulting in a "black box" problem. In addition, the application of AR technology in programming education is still insufficient, and existing AR programming tools are mostly limited to simple visualization and have not been deeply combined with the underlying compilation process.

[0004] For example, patent publication CN113641346A discloses an intelligent auxiliary tool for standardized programming and its working method. This scheme significantly reduces the ability requirements of developers through the component library module and intelligent programming assistant module, and provides standardized code generation functions, with high practicality and efficiency.

[0005] However, this scheme has certain limitations in programming learning:

[0006] This scheme mainly relies on text-based code generation and component calling, lacking spatial visualization support for program execution process and memory state, making it difficult for developers to intuitively understand the running mechanism of the code.

[0007] This scheme lacks multi-modal interactive feedback (such as tactile, visual reinforcement), and developers' memory of code logic mainly relies on repeated practice, with a relatively flat learning curve.

[0008] Although this scheme provides standardized code, it does not deeply explain the logical basis of code generation, making it difficult for developers to understand the decision-making process of AI suggestions, resulting in a "black box" problem. SUMMARY

[0009] The technical problem to be solved by the present application is to provide a programming cognitive enhancement system and method based on augmented reality and generative AI to solve the problems mentioned in the background art.

[0010] To achieve the above object, the present application adopts the following technical solutions:

[0011] A programming cognitive enhancement system based on augmented reality and generative AI includes an augmented reality device layer, a data processing layer, and an artificial intelligence generation layer;

[0012] The augmented reality device layer is used to realize three-dimensional space mapping of code structure and tactile feedback;

[0013] The data processing layer is used to realize code semantic analysis, logical topology graph generation, and user cognitive load assessment;

[0014] The artificial intelligence generation layer is used to dynamically generate multi-level code annotations and logic derivation animations.

[0015] Optionally, the augmented reality device layer includes augmented reality glasses, a spatial positioning sensor, and a tactile feedback glove.

[0016] Optionally, the data processing layer includes a code semantic analysis module, a logical topology graph generation module, and a cognitive load assessment module, wherein the code semantic analysis module is based on an abstract syntax tree.

[0017] Optionally, the logical topology graph generation module uses an improved Fruchterman-Reingold algorithm to realize three-dimensional topology layout, the node size is determined according to the code complexity calculation, and the edge strength between connected nodes is determined according to the weighted calculation of data coupling degree and control coupling degree.

[0018] Optionally, the cognitive load assessment module monitors the user's gaze duration in real time through eye tracking technology, triggering the generation of code annotations of the corresponding level.

[0019] Optionally, the multi-level code annotation module of the artificial intelligence generation layer uses a generative pre-training Transformer model fine-tuned by a code corpus to generate real-time annotation content.

[0020] Optionally, the logic derivation animation module of the artificial intelligence generation layer is based on a three-dimensional visualization engine to present real-time control flow and data flow animation effects of the code.

[0021] Optionally, the tactile feedback glove provides corresponding physical feedback signals for different types of code exceptions, including fingertip pulse feedback for variable exceptions, palm sustained pressure feedback for memory leaks, and wrist vibration feedback for dead loops.

[0022] Optionally, the augmented reality glasses accurately position and display the code topology graph through spatial anchoring technology, and the spatial positioning sensor includes an inertial measurement unit and a depth camera.

[0023] The application also discloses a programming cognitive enhancement method based on augmented reality and generative AI, comprising the following steps:

[0024] code semantic information is extracted by using an abstract syntax tree;

[0025] the code semantic information is mapped into a three-dimensional logical topology graph;

[0026] user gaze duration information is obtained by eye tracking, and code comments of a corresponding level are automatically generated according to the gaze duration;

[0027] a logical derivation animation of the code is generated and displayed in real time by using generative artificial intelligence technology;

[0028] a corresponding physical feedback signal is provided to the user according to the code exception type by using a haptic feedback device.

[0029] The application has the advantages that the application provides a programming auxiliary system based on augmented reality and generative artificial intelligence, the three-dimensional space mapping of the code structure is realized by the augmented reality device, the internal logic structure and the data flow path of the code are intuitively displayed, and the abnormal situation in the code running process is reminded to the user in real time by haptic feedback, so that the cognitive load of the student is effectively reduced, and the understanding efficiency and the debugging accuracy of the code are improved.

[0030] Further, the application realizes fine code semantic analysis and automatic generation of a logical topology graph by a data processing layer, and can accurately present the data dependency and the control relationship between functions and modules in the code.

[0031] In the artificial intelligence generation layer, the multi-level code comment module provided by the application adopts a generative pre-training Transformer model to realize accurate and real-time comment generation of the code, effectively helping the user to quickly understand the complex code structure and logic.

[0032] In addition, the application provides a haptic feedback glove, accurately generates a physical feedback signal according to the code exception type, so that the student can more intuitively feel the problem points in the code running, which helps to quickly eliminate the problems and significantly improves the debugging efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a schematic diagram of the overall system architecture of the application;

[0034] Figure 2is a schematic diagram of an augmented reality device layer structure of the present application;

[0035] Figure 3 is a schematic diagram of a data processing layer structure of the present application;

[0036] Figure 4 is a schematic diagram of an artificial intelligence generation layer and interactive feedback of the present application. DETAILED DESCRIPTION

[0037] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0038] The present application relates to a programming assistance system based on augmented reality and generative artificial intelligence, aiming to solve the problems of code understanding difficulty and low debugging efficiency faced by students and children in the process of learning programming.

[0039] As shown in Figure 1 , the present application includes three core parts: augmented reality device layer, data processing layer and artificial intelligence generation layer, which coordinate with each other to provide intuitive programming support and interactive feedback.

[0040] As shown in Figure 2 , the augmented reality device layer includes augmented reality glasses, spatial positioning sensors and tactile feedback gloves. The augmented reality glasses can be selected from commercially available augmented reality head-mounted devices such as Microsoft HoloLens or Magic Leap, which mainly realize the spatialized intuitive display of code logic structure, helping students to intuitively map the abstract code structure to the real environment.

[0041] The spatial positioning sensor includes an inertial measurement unit and a depth camera, wherein the inertial measurement unit is used to capture the head position and action information of the user, and the depth camera performs accurate environment scanning through structured light or ToF technology, ensuring the accuracy and stability of code space mapping.

[0042] The tactile feedback gloves are equipped with precise miniature vibration motors, pressure airbags or ultrasonic arrays, which can provide rich tactile cues to realize physical feedback of programming exceptions. When specific types of exceptions occur during code execution, such as variable value abnormal change, memory leak or dead loop, the gloves produce corresponding tactile feedback signals: variable abnormality prompts students to pay attention to the problem point through slight pulse vibration of the fingertip position; memory leak triggers continuous and soft pressure in the palm area; and dead loop prompts students to intervene quickly through the gradually increasing obvious vibration signal in the wrist position.

[0043] As shown in Figure 3As shown, the data processing layer is mainly responsible for in-depth analysis of code logic, optimization of topology layout, and real-time evaluation of user cognitive load. Specifically, it includes code semantic analysis module, logical topology graph generation module, and cognitive load evaluation module. The code semantic analysis module is based on abstract syntax tree technology and supported by ANTLR open source tools in multi-language environment (such as Java, Python, C++, C# etc.), which conducts comprehensive and accurate structured analysis of the code.

[0044] The logical topology graph generation module uses the improved Fruchterman-Reingold layout algorithm. This algorithm first calculates the code circle complexity as the basis for determining the size of the node. Functions or modules with high circle complexity are assigned larger node sizes to intuitively prompt students to pay attention to the risk areas of complex code. In addition, the algorithm also performs weighted processing on the edges between nodes. Data coupling degree is given a higher weight (e.g. 0.7), and control coupling degree is given a lower weight (e.g. 0.3). In this way, the dependence relationship between code modules is clearly reflected, helping students quickly identify potential problems.

[0045] The cognitive load evaluation module uses real-time eye tracking technology. Through head-mounted eye tracking devices or eye tracking sensors built into augmented reality glasses, it continuously monitors the visual dwell time of students on different code segments. If the system detects that the user gazes at a certain code segment for more than a preset threshold (such as 1 second, 3 seconds, etc.), it automatically activates deeper levels of code explanation or comment content, ensuring that users do not have excessive cognitive load due to long-term confusion about complex code logic.

[0046] As shown in Figure 4 The artificial intelligence generation layer includes multi-level code annotation module and logical derivation animation module. The multi-level code annotation module is based on generative pre-training Transformer model (such as GPT-4 or later versions) and can generate hierarchical annotations that are highly consistent with the context of the code after large-scale fine-tuning of specific domain code corpus data. These annotations cover basic syntax explanation, data flow path and logical relationship explanation, design pattern and architecture level suggestions, etc.

[0047] The logical derivation animation module uses Unity three-dimensional engine, combined with data-driven particle effects and animation generation technology, to realize the intuitive and dynamic visualization of the code execution process. For example, in sorting algorithms or graph traversal algorithms, the data transmission path will be represented as dynamic particle flow or light strip movement, clearly showing the specific transmission route and condition judgment path of data in the algorithm, helping students quickly understand the specific execution details of the algorithm.

[0048] The tactile feedback device further enhances the intuitiveness of user interaction with the code, generating precise tactile feedback signals based on the specific situation of the real-time running code. The tactile feedback glove generates tactile feedback signals of corresponding intensity according to the pre-defined feedback rules, through real-time processing of abnormal information from the data processing layer by the microprocessor, and according to the pre-defined abnormal type and severity level.

[0049] The augmented reality glasses realize stable fusion of the three-dimensional code topology graph and the real environment through spatial anchoring technology. The spatial anchoring technology uses the built-in spatial mapping API of the glasses to detect key feature points (such as desktop, wall, floor, etc. marker points) in the physical environment in real time, and continuously tracks the spatial position changes of these feature points, so as to ensure that the spatial position of the code visualization structure always remains stable, so that students can still smoothly and naturally interact with the code in a complex environment.

[0050] The specific implementation method of the present application comprises the following steps:

[0051] Firstly, the ANTLR parsing tool is used to extract the abstract syntax tree from the code to be analyzed, and preliminary semantic annotation is performed; then, the annotated abstract syntax tree is mapped to a three-dimensional spatial topology structure through a logical topology graph generation algorithm; subsequently, the augmented reality glasses display the structure in the user's physical environment in real time, and the user can control the annotation display level through eye tracking technology; at the same time, the artificial intelligence generation layer analyzes the user's gaze data in real time, and automatically generates detailed annotations and logical deduction animations adapted to the current code focus point; finally, the tactile feedback glove monitors the code running state in real time, and generates physical feedback signals according to different abnormal types, so as to timely remind students to pay attention to and correct specific problems.

[0052] The present application has significant value in the field of education. Programming beginners can quickly understand the internal logic of complex algorithms such as recursive calls, loop structures or data structure operation processes through the intuitive three-dimensional visualization interface, greatly shortening the learning curve. At the same time, the tactile feedback device corrects errors in real time during the coding process of beginners, and intuitively enhances the learning effect. In addition, in the field of enterprise software development and maintenance of large and complex systems, the present application helps team members to quickly identify potential problem points in the system, and realizes clearer and more effective communication in remote collaboration.

[0053] For example, when a student views the code through augmented reality glasses, the system will dynamically present the recursive call process in a three-dimensional tree structure in real time. Each recursive call will be clearly displayed in space, accompanied by real-time generated code comments. Students can also use tactile feedback gloves to receive physical feedback, such as a weak pulse on the fingers, when the code execution encounters logical errors or performance bottlenecks, allowing them to quickly find and fix the problem. In addition, students can actively manipulate and observe the dynamic structure of the algorithm through simple gestures such as grabbing and rotating, deepening their intuitive understanding of the algorithm's logic. This immersive and interactive teaching method effectively improves students' mastery of complex algorithms and data structures and their learning interest, significantly outperforming traditional two-dimensional teaching methods and greatly improving teaching effectiveness and learning efficiency.

[0054] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical scope disclosed by the present application according to the technical solutions and inventive concepts of the present application, which should be covered within the protection scope of the present application.

Claims

1. An augmented reality and generative AI based programming cognitive augmentation system, characterized in that, The system comprises an augmented reality device layer, a data processing layer, and an artificial intelligence generation layer. The augmented reality device layer is used to realize three-dimensional space mapping of code structure and haptic feedback. The data processing layer is used to realize code semantic analysis, logical topology graph generation, and user cognitive load assessment. The artificial intelligence generation layer is used to dynamically generate multi-level code annotations and logical derivation animations. The data processing layer comprises a code semantic analysis module, a logical topology graph generation module, and a cognitive load assessment module, wherein the code semantic analysis module is based on abstract syntax tree. The cognitive load assessment module monitors user gaze duration in real time through eye tracking technology, triggering the generation of code annotations of corresponding levels.

2. The augmented reality and generative AI-based programming cognitive enhancement system of claim 1, wherein, The augmented reality device layer comprises augmented reality glasses, spatial positioning sensors, and haptic feedback gloves.

3. The augmented reality and generative AI-based programming cognitive enhancement system of claim 1, wherein, The logical topology graph generation module uses an improved Fruchterman-Reingold algorithm to realize three-dimensional topology layout, with node size determined according to code complexity and edge strength between connected nodes calculated according to data coupling degree and control coupling degree.

4. The augmented reality and generative AI-based programming cognitive enhancement system of claim 1, wherein, The multi-level code annotation module of the artificial intelligence generation layer uses a generative pre-training Transformer model fine-tuned on a code corpus to generate real-time annotation content.

5. The augmented reality and generative AI-based programming cognitive enhancement system of claim 1, wherein, The logical derivation animation module of the artificial intelligence generation layer presents control flow and data flow animation effects of the code in real time based on a three-dimensional visualization engine.

6. The augmented reality and generative AI-based programming cognitive enhancement system of claim 2, wherein, The haptic feedback gloves provide corresponding physical feedback signals for different types of code exceptions, including fingertip pulse feedback for variable exceptions, palm sustained pressure feedback for memory leaks, and wrist vibration feedback for dead loops.

7. The augmented reality and generative AI-based programming cognitive enhancement system of claim 2, wherein, The augmented reality glasses accurately position and display the code topology graph through spatial anchoring technology, and the spatial positioning sensors include an inertial measurement unit and a depth camera.

8. A programming cognitive enhancement method based on augmented reality and generative AI, characterized in that, The method comprises the following steps: Extracting code semantic information using abstract syntax tree; Mapping code semantic information to three-dimensional logical topology graph; Obtaining user gaze duration information through eye tracking and automatically generating code annotations of corresponding levels according to the gaze duration; Using generative artificial intelligence technology to generate and display logical derivation animations of the code in real time; Using haptic feedback devices to provide corresponding physical feedback signals to users according to code exception types.

Images