Multimodal companion education assistance method, system, and computer device

Abstract

The application discloses a multimodal accompanying education auxiliary method and system and a computer device. The method comprises the following steps: continuously collecting environmental image frames and audio data of a first visual angle of a user through a wearable collection terminal, and judging whether the user is in a silent state based on the collection result of the audio data; when the user is in the silent state, performing visual semantic analysis on the current environmental image frames, determining semantic information of the current environmental image frames, and mapping the semantic information to a target subject knowledge point according to the associated information corresponding to each subject knowledge point in a teaching research knowledge base; when the target subject knowledge point matches user historical learning data of the user, it is determined that the current scene has teaching value, active guidance content is generated based on the semantic information and the target subject knowledge point; and the active guidance content is pushed to the wearable collection terminal for output.

Description

Technical Field

[0001] This application relates to the field of intelligent education technology, and in particular to a multimodal accompanying education assistance method, system and computer device. Background Technology

[0002] With the rapid development of artificial intelligence and wearable device technology, the education sector is undergoing a transformation from digitalization to intelligentization. Traditional educational aids are increasingly unable to meet learners' needs for immediate, personalized, and contextualized learning. As an emerging technological application, accompanying educational aids can break through the limitations of time and space, extending educational services to everyday life scenarios outside the classroom, and have significant application value in improving learning efficiency and achieving individualized instruction.

[0003] Existing educational aids all employ a single interaction model of "user initiates a request - system passively responds," meaning the system cannot proactively perceive the user's learning status and changes in the environment. When the user is in a silent state, the system completely ceases to function, unable to proactively identify teaching opportunities or detect student confusion like a human teacher, resulting in the waste of numerous educationally valuable everyday scenarios. Summary of the Invention

[0004] In view of this, embodiments of this application provide a multimodal accompanying educational assistance method, system, and computer device.

[0005] According to one aspect of this application, a multimodal accompanying educational assistance method is provided, the method comprising: The system continuously collects environmental image frames and audio data from the user's first-person perspective using a wearable data acquisition terminal, and determines whether the user is in a silent state based on the audio data acquisition results. When the user is in a silent state, visual semantic analysis is performed on the current environmental image frame to determine the semantic information of the current environmental image frame, and the semantic information is mapped to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base. When the target subject knowledge point matches the user's historical learning data, it is determined that the current scenario has teaching value, and proactive guidance content is generated based on the semantic information and the target subject knowledge point. The proactive guidance content is pushed to the wearable data acquisition terminal for output.

[0006] According to another aspect of this application, a multimodal accompanying educational assistance system is provided, applied to the multimodal accompanying educational assistance method as described above, comprising: The wearable data acquisition terminal includes an external camera and a microphone. The external camera is used to acquire environmental image frames from the user's first-person perspective and upload them to the cloud. The microphone is used to acquire audio data and upload it to the cloud. A cloud-based multi-agent system, including gating agents and visual sentinel agents; The gated agent is used to determine whether the user is in a silent state based on the acquisition results of audio data. If the user is in a silent state, a silent state notification is sent to the visual sentinel agent. The visual sentinel agent is used to respond to a silent state notification, perform visual semantic analysis on the current environmental image frame, determine the semantic information of the current environmental image frame, and map the semantic information to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base; when the target subject knowledge point matches the user's historical learning data, it is determined that the current scene has teaching value, and based on the semantic information and the target subject knowledge point, it generates proactive guidance content; and pushes the proactive guidance content to the wearable acquisition terminal. The wearable data collection terminal is also used to output the active guidance content.

[0007] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described multimodal accompanying educational assistance method.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described multimodal accompanying educational assistance method.

[0009] By employing the above technical solutions, the multimodal accompanying educational assistance method, system, and computer device provided in this application, through continuous acquisition of user environment image and audio data and determination of silent state by a wearable acquisition terminal, can automatically identify appropriate times for educational intervention without interfering with the user's normal activities; through visual semantic analysis and knowledge mapping of the teaching and research knowledge base, it can automatically convert visual content in any environment into corresponding subject knowledge points; through matching and determining the target subject knowledge points with the user's historical learning data and generating proactive guidance content, it can ensure that the pushed educational content is aligned with the user's current learning needs, realizing accompanying personalized educational assistance. The entire process does not require the user to actively initiate an interaction request; the system can proactively provide educational services based on environmental perception and user profiles, improving the timeliness and relevance of educational assistance.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a multimodal accompanying educational assistance method provided in an embodiment of this application is shown. Figure 2 A flowchart illustrating another multimodal accompanying educational assistance method provided in an embodiment of this application is shown; Figure 3 This illustration shows a flowchart of yet another multimodal accompanying educational assistance method provided in an embodiment of this application; Figure 4 A flowchart illustrating another multimodal accompanying educational assistance method provided in an embodiment of this application is shown; Figure 5 A flowchart illustrating another multimodal accompanying educational assistance method provided in an embodiment of this application is shown; Figure 6 A flowchart illustrating another multimodal accompanying educational assistance method provided in an embodiment of this application is shown; Figure 7 This paper shows a schematic diagram of the structure of a multimodal accompanying educational assistance system provided in an embodiment of this application; Figure 8 A schematic diagram of the system structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides a multimodal accompanying educational assistance method, such as... Figure 1 As shown, the method includes: Step 101: Continuously collect environmental image frames and audio data from the user's first-person perspective using a wearable acquisition terminal, and determine whether the user is in a silent state based on the acquisition results of the audio data.

[0014] Wearable acquisition terminals refer to portable electronic devices capable of capturing first-person view images and audio. Specifically, they may include an external camera module and microphone array, and are worn on or near the user's head to record the scene from the user's perspective and capture sound. A silent state refers to a state where the user does not emit any active speech signals and the ambient audio does not contain any user voice components. This state is determined based on a comprehensive analysis of the short-time energy characteristics and zero-crossing rate characteristics of the audio signal.

[0015] Specifically, the wearable data acquisition terminal initiates a data acquisition process using an external camera and microphone in a continuous operation mode. The external camera continuously captures environmental image frames at a fixed frame rate, while the microphone continuously collects surrounding audio at a constant sampling rate. In one implementation, the acquired audio data is sent in real-time to a voice activity detection module to detect whether the user is currently in a silent state. This module first processes the audio stream into frames, with each frame lasting 20 to 30 milliseconds. Then, it calculates the short-time energy value and zero-crossing rate for each frame, comparing them with a preset dynamic threshold. This dynamic threshold is adaptively adjusted based on the background noise level within the last 1 to 2 seconds to handle noise fluctuations in different environments. When the short-time energy and zero-crossing rate of multiple consecutive frames are both below the dynamic threshold, the current time period is determined to be a silent segment; when a signal matching the spectral characteristics of human voice is detected, it is determined to be a user voice activity segment. If no valid voice activity is detected within a preset time window, the user is determined to be currently in a silent state. As an alternative implementation, a voice activity detection scheme based on bone conduction sensing can be used. This scheme detects the vibration signals generated when the user speaks using a bone conduction sensor, which is more robust in noisy environments. Through continuous data acquisition and silent state determination in this step, the system can identify the window of opportunity for proactive service even when the user has not actively issued a command, while avoiding the unnecessary computational overhead caused by misinterpreting environmental noise as user interaction requests.

[0016] Step 102: When the user is in a silent state, perform visual semantic analysis on the current environmental image frame to determine the semantic information of the current environmental image frame, and map the semantic information to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base.

[0017] Visual semantic analysis refers to the process of extracting features and understanding the semantics of visual content in image frames. Specifically, it can use deep learning models to convert pixel-level visual information into a sequence of semantic labels. The teaching and research knowledge base refers to a pre-constructed structured knowledge set containing knowledge points from multiple disciplines and their relationships, with each knowledge point corresponding to semantic association information.

[0018] Specifically, when the user is in a silent state, the active perception link function is executed. First, feature extraction and semantic understanding are performed on the current environmental image frame, and the semantic information of the current environmental image frame is output to form a semantic feature vector. The semantic feature vector is input into the mapping module of the teaching and research knowledge base. This module maintains a mapping relationship matrix from the semantic space to the knowledge point space. By calculating the similarity between the semantic feature vector and the associated information vector of each knowledge point in the teaching and research knowledge base, the knowledge points selected based on the similarity are used as the target subject knowledge points.

[0019] In an optional embodiment, the step of performing visual semantic analysis on the current environmental image frame to determine the semantic information of the current environmental image frame, and mapping the semantic information to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base, includes: identifying object instances in the current environmental image frame, extracting at least one attribute information corresponding to the object instance, using the object instance and the attribute information as the semantic information of the current environmental image frame, wherein the attribute information includes structural attributes, material attributes, and functional attributes; matching the semantic information with the association information corresponding to each subject knowledge point in the teaching and research knowledge base, and using the subject knowledge point corresponding to the successfully matched association information as the target subject knowledge point corresponding to the semantic information.

[0020] Among them, an object instance refers to a specific physical object or a categorical representation of an object detected in an image frame. It is located and identified from the image using a visual detection model, and the output can specifically include an object category label and its location region in the image. Attribute information refers to feature data that describes object instances in multiple dimensions. Structural attributes describe the geometric configuration or morphological features of the object (e.g., the arched structure of an arch bridge, the three sides of a triangle), material attributes describe the material composition of the object (e.g., stone, metal, wood, plastic), and functional attributes describe the purpose or mechanism of the object (e.g., a refrigerator for refrigeration and preservation, a bridge for carrying traffic). A teaching and research knowledge base refers to a collection of teaching resources organized in the form of a knowledge graph or structured database.

[0021] In this embodiment, after determining that the user is in a silent state, a three-layer semantic mapping process is performed on the current environmental image frame. The first layer is the visual feature layer, where the system first performs object detection on the current environmental image frame. For example... Figure 2As shown, the image frame is input into a pre-trained large-scale visual model. After the model outputs the detection results, the system performs non-maximum suppression on each detected object instance, removing redundant detection boxes with overlap exceeding a preset threshold, and retaining the detection result with the highest confidence score as the valid object instance. The second layer is the semantic attribute layer. Subsequently, the system extracts attribute information for each valid object instance. Specifically, the extracted structural attributes may include the object's geometric contour type, symmetry features, and spatial arrangement of its components; the extracted material attributes may include the object's surface texture features, color distribution, and material category based on visual reasoning; the extracted functional attributes may include a functional description based on the object category prior knowledge base and the role the object may play in the scene context. The identified object instances and their corresponding attribute information are used together as the semantic information of the current environment image frame.

[0022] The third layer is the pedagogical association layer. After semantic information extraction, the system inputs the semantic information into the teaching and research knowledge base for matching and retrieval. The teaching and research knowledge base stores knowledge points of various subjects and their associated information. This associated information consists of visual anchor descriptions corresponding to the knowledge points and attribute information descriptions corresponding to the visual anchor descriptions. For example, the knowledge point "parabola" is associated with visual anchors such as "arch bridge," "basketball trajectory," and "fountain water flow"; the knowledge point "buoyancy" is associated with visual anchors such as "boat," "floating wooden block," and "balloon rising"; and the knowledge point "density" is associated with visual anchors such as "different liquids stratify" and "objects sink and float." During the matching process, the system calculates the semantic similarity between each element in the semantic information (object instance name and attribute features) and the associated information of each knowledge point in the teaching and research knowledge base. When the object instance or attribute feature in the semantic information is the same as or semantically similar to the associated information description of a certain knowledge point, the two are considered to have matched successfully. Through object instance recognition, attribute information extraction, and matching with the teaching and research knowledge base in this embodiment, the system can automatically transform specific objects and their multi-dimensional attributes in the user's field of vision into structured subject knowledge points, providing semantically computable foundational data for subsequent teaching value determination. Object instances and attribute information jointly participate in matching as semantic information, expanding the association path between visual content and subject knowledge compared to schemes that only match based on object categories. For example, it identifies object instances in the image (such as "arch bridge," "refrigerator," and "butterfly"); extracts semantic features such as structural, material, and functional attributes of the objects (such as "arch bridge" → "catenary structure," "stress distribution," and "stone / steel reinforcement"); and maps semantic attributes to specific subject knowledge points based on the knowledge graph in the teaching and research knowledge base (such as "catenary structure" → mathematical "parabola equation" and physics "force equilibrium and decomposition"). This mapping process filters based on grade information in the user's profile database and learned knowledge points in the learning progress database, prioritizing knowledge points that match the user's current learning stage.

[0023] In an optional embodiment, after taking the subject knowledge points corresponding to the successfully matched association information as the target subject knowledge points corresponding to the semantic information, the method further includes: reading the grade information in the user's user profile database and the learned knowledge points in the learning progress database; filtering the target subject knowledge points according to the grade information and the learned knowledge points, and taking the filtered knowledge points that match the user's current learning stage as the final target knowledge points.

[0024] The user profile database stores users' static attribute information, which includes at least their current grade level and may also include basic data that does not frequently change with each learning session, such as grade level information and subject preference settings. The learning progress database stores users' dynamic learning process data, which may include a list of knowledge points the user has completed, a mark indicating the level of mastery for each knowledge point, and a timeline tag indicating the last time the user encountered that knowledge point.

[0025] In this embodiment, after obtaining the target subject knowledge points through matching with the teaching and research knowledge base, the system further reads the user's grade information from the user profile database and the user's list of learned knowledge points from the learning progress database. The grade information is used to determine the scope of subject knowledge the user should currently be learning, and the list of learned knowledge points is used to determine the set of knowledge the user has already encountered and has a cognitive foundation in. The system uses the target subject knowledge points determined through matching with the teaching and research knowledge base as candidate items to be filtered. For example, the following filtering logic can be executed: First, based on the grade information, determine the scope of knowledge points specified in the curriculum standards or teaching syllabus corresponding to the current grade level, and determine whether the target subject knowledge point belongs to this scope. If the teaching requirement grade corresponding to the target subject knowledge point is higher than the user's current grade, then the knowledge point is determined to exceed the user's current cognitive level and is removed from the candidate items; if the teaching requirement grade corresponding to the target subject knowledge point is lower than or equal to the user's current grade, then the knowledge point is retained for the next round of filtering. Subsequently, for candidate knowledge points filtered by grade level, further filtering is performed based on the list of learned knowledge points: if the target subject knowledge point already exists in the list, it is determined that the knowledge point is content that the user has already completed learning, and whether to retain it is decided according to the system's preset strategy. In one implementation, the knowledge point is retained for review and reinforcement guidance; in another implementation, the knowledge point is marked as completed and its priority is reduced, prioritizing knowledge points that are semantically related to learned knowledge points but not yet fully mastered as the final target. Combining the dual filtering results of grade level information and learned knowledge points, the knowledge points finally retained are the final target knowledge points that match the user's current learning stage.

[0026] Through the dual filtering mechanism of grade information and learned knowledge points in this embodiment, the determination result of the target subject knowledge points can be accurately aligned with the user's actual learning stage. This ensures that the proactive guidance content will not exceed the user's cognitive load due to the difficulty of the knowledge points, nor will it lose its freshness due to the outdated knowledge points. At the same time, the abnormal records of the filtering results accumulate dynamic data on the user's cognitive boundaries for the system.

[0027] In an optional embodiment, the step of filtering the target subject knowledge points based on the grade information and the learned knowledge points, and taking the filtered knowledge points that match the user's current learning stage as the final target knowledge points, includes: querying whether the target knowledge point matches any of the learned knowledge points; if it matches one of the learned knowledge points, only the included target knowledge points are retained; if it matches multiple of the learned knowledge points, the learned knowledge point with the most recent learning time among the matched learned knowledge points is taken as the final target knowledge point; if it does not match, querying whether the target knowledge point belongs to the required knowledge points corresponding to the grade information; if it does, the knowledge point ranked first among the required knowledge points is taken as the final target knowledge point; if it does not belong, the target subject knowledge point is ignored.

[0028] In this embodiment, the following filtering logic can also be executed: First, query whether the target knowledge point matches the list of learned knowledge points: compare the target knowledge point with each item in the list of learned knowledge points one by one, and determine whether it matches based on the unique identifier or semantic similarity of the knowledge point. If the query result is a match, then further determine the number of matches: if only one learned knowledge point matches, then directly retain the matched knowledge point as the final target knowledge point; if multiple learned knowledge points match, then the system reads the most recent learning timestamp of each matched knowledge point, and takes the knowledge point with the latest learning time (i.e., the most recently completed learning) as the final target knowledge point. This processing method is based on the "recency effect" principle in cognitive psychology, selecting the knowledge point that the user has recently encountered as the anchor point of the guiding content to enhance the smoothness of knowledge extraction.

[0029] If the query result is not found, meaning the target knowledge point is not in the list of learned knowledge points, the system further queries whether the target knowledge point belongs to the set of knowledge points to be learned corresponding to the user's current grade level. Specifically, based on the grade information in the user's profile, the system retrieves all knowledge points to be learned corresponding to the current grade from the grade-knowledge point mapping table to determine whether the target knowledge point exists in the set. If the target knowledge point belongs to the set of knowledge points to be learned, the knowledge point ranked first among all candidates belonging to the set of knowledge points to be learned is selected as the final target knowledge point. This ranking is based on the order in which knowledge points are taught as specified in the teaching syllabus. Selecting the knowledge point taught first as the guiding content conforms to the principle of gradual teaching and helps to lay the cognitive foundation for learning new knowledge. If the target knowledge point is neither found in the list of learned knowledge points nor belongs to the set of knowledge points to be learned for the current grade level, it is determined that the knowledge point is irrelevant to the user's current learning stage, and the system ignores the target subject knowledge point and does not trigger the subsequent active guidance process.

[0030] Through the filtering in this embodiment, the system can adaptively determine the final target knowledge point under different matching scenarios. When the list of learned knowledge points is matched, a single matched or recently learned knowledge point is selected as the target, ensuring that the guided content is based on the user's already mastered knowledge. When the list of learned knowledge points is not matched, the top-ranked knowledge point in the current grade's required knowledge points is selected as the target, transforming proactive guidance into a warm-up introduction of new knowledge, thus expanding the applicability of proactive guidance in educational scenarios.

[0031] Step 103: When the target subject knowledge point matches the user's historical learning data, it is determined that the current scenario has teaching value. Based on the semantic information and the target subject knowledge point, proactive guidance content is generated.

[0032] User historical learning data refers to a collection of data recording a user's past learning behaviors and outcomes. Proactive guided content refers to information actively generated by the system to guide users to focus on educationally valuable content within the current context. Educational value refers to the educational relevance between the environmental information in the current context and the knowledge points the user is about to learn; this relevance is measured by the degree of matching between the target subject knowledge points and the user's historical learning data.

[0033] Specifically, the system reads the user's historical learning data from the learning progress database, which may include a list of learned knowledge points, a set of memorized words, and annotations of knowledge points from past incorrect questions. The system then compares the identified target subject knowledge points with the user's historical learning data one by one. For example, if the target subject knowledge point exists in the list of learned knowledge points or the set of memorized words, or is the same as the knowledge point involved in a past incorrect question, then the system determines that the two are a successful match. After a successful match, the system further determines that the current scene has educational value and then enters the stage of generating proactive guided content. The generation of guided content is based on extracted semantic information and the matched target subject knowledge points: the system uses natural language to concatenate specific objects or scene descriptions in the visual scene with the target subject knowledge points to generate inspiring guided statements. For example, associating "the arch structure of an arched bridge" with "parabola" to form guided content such as "Do you see the arc of that arched bridge? Its shape is actually related to the parabola we have learned." Through the matching and guidance content generation in this step, the system can proactively initiate teaching interactions when it recognizes visual scenes related to the user's existing knowledge. This realizes the transformation from a passive mode of user questioning and system answering to an active mode of system perception and proactive guidance. At the same time, it ensures that the guidance content is related to the user's existing knowledge reserves rather than introducing completely unfamiliar concepts that would cause cognitive burden.

[0034] Step 104: Push the active guidance content to the wearable data collection terminal for output.

[0035] Specifically, proactive guidance content is transmitted from the cloud to the wearable data acquisition terminal via a wireless network in the form of data packets. Upon receiving the guidance content, the terminal outputs it, selecting the appropriate output channel based on the content format: for guidance statements presented in speech form, the text is sent to a speech synthesis service to be converted into natural speech audio, which is then played through a bone conduction audio playback unit. Bone conduction allows sound to be transmitted directly to the inner ear through skull vibrations without blocking the external auditory canal, enabling users to perceive ambient sounds while listening to the guidance; for guidance content containing key terms or formulas, the core text information is simultaneously rendered onto a miniature screen for display, achieving a dual-channel presentation of auditory and visual information. After the push is completed, the system records the trigger time, scenario information, and subsequent user interaction responses, writing this information into the learning progress database as updated material for the user's historical learning data. This ensures that proactive guidance content is effectively conveyed in a way that suits the user's current context, guaranteeing the accessibility of guidance information while avoiding excessive interference with the user's normal activities.

[0036] By applying the technical solution of this embodiment, through continuous acquisition of user environment image and audio data and determination of silent state via wearable acquisition terminals, the system can automatically identify appropriate times for educational intervention without interfering with the user's normal activities. Through visual semantic analysis and knowledge mapping with the teaching and research knowledge base, visual content in any environment can be automatically converted into corresponding subject knowledge points. By matching target subject knowledge points with the user's historical learning data and generating proactive guidance content, the system ensures that the pushed educational content aligns with the user's current learning needs, achieving personalized educational assistance. The entire process requires no user-initiated interaction requests; the system proactively provides educational services based on environmental perception and user profiles, improving the timeliness and relevance of educational assistance.

[0037] In an optional embodiment, the wearable acquisition terminal includes an outer camera and an inner camera, and the current environment image frame is acquired by the outer camera; the method further includes: The user's eye image frames are captured by the inner camera of the wearable acquisition terminal; When the user is in a silent state, extract the user's gaze point coordinate sequence and pupil diameter change data based on the eye image data; When the fixation point coordinate sequence determines that the duration of fixation in the target area exceeds a preset time threshold, and the pupil diameter change data shows a contraction trend, it is determined that the user is in a state of high cognitive load. Based on the gaze point coordinate sequence, the corresponding region in the environmental image frame captured by the outer camera is located, the text information in the corresponding region is extracted as the current confusion content, and the target answer is determined according to the current confusion content. The target answer is decomposed into flexible heuristic intervention content corresponding to multiple sub-questions. The flexible heuristic intervention content is pushed to the wearable data acquisition terminal for output.

[0038] The inner camera refers to the image acquisition component installed in the wearable acquisition terminal facing the user's eyes, with its optical axis pointing towards the user's eyeball area. It is used to capture eye image data including the pupil, iris, and corneal reflection points. An eye image frame refers to a sequential sequence of images continuously acquired by the inner camera, focusing on the user's eyes. It includes information such as pupil position coordinates, pupil diameter, and the position of the corneal reflection spot. The fixation point coordinate sequence is a data sequence calculated based on the pupil center position and corneal reflection point position in the eye image frame, showing the change of the user's gaze point position in space over time. It reflects the spatial distribution and dynamic migration trajectory of the user's visual attention. Pupil diameter change data refers to the continuous change in pupil diameter over time, calculated based on the number of pixels in the pupil area or the major and minor axes of a fitted ellipse in the eye image frame. It reflects the changing trend of the user's cognitive load level. When cognitive load increases, the pupil diameter constricts, a physiological response based on the activity mechanism of the autonomic nervous system. Fixation stagnation refers to a state where the spatial positional change of multiple consecutive sampling points in a fixation point coordinate sequence is less than a preset displacement threshold, indicating that the user's visual attention remains relatively fixed within a specific spatial region. The target region refers to the area in the environmental image frame where fixation points are concentrated, determined by spatial clustering of the fixation point coordinate sequence, specifically manifested as a dense distribution of fixation point coordinates within a local area. High cognitive load refers to a psychological state experienced by the user during cognitive processing due to an imbalance between task difficulty and cognitive resources. This state is determined by fixation stagnation exceeding the normal thinking time and the simultaneous occurrence of pupillary constriction. The corresponding region refers to a local sub-region in the environmental image frame corresponding to the spatial clustering position of the fixation point coordinate sequence, determined by mapping the fixation point coordinates from the eye image frame coordinate system to the environmental image frame coordinate system.

[0039] In this embodiment, as Figure 3As shown, when the user is in a silent state, a real-time monitoring process for the user's cognitive state is initiated simultaneously, triggering an image frame capture every 2 seconds, for example. The inner camera of the wearable acquisition terminal continuously captures image frames of the user's eyes. The eye image frames first undergo a preprocessing process, including grayscale conversion, noise reduction, image enhancement, and pupil region segmentation. Based on the preprocessed eye images, the eye-tracking model calculates the gaze coordinates using the pupil-corneal reflection method: detecting the relative offset between the pupil center position and the corneal reflection point, and converting the offset into the user's gaze spatial coordinates in the environmental image frame using a preset mapping function. The gaze coordinates change over time, forming a gaze coordinate sequence. The system performs sliding window analysis on this sequence, calculating the Euclidean distance between each sampling point within the window and determining whether it is below a preset displacement threshold. Simultaneously, the pupil diameter measurement model extracts the number of pixels in the pupil region from the eye image frames, maps the number of pixels to the pupil diameter value through a calibration curve, and forms a pupil diameter change data sequence.

[0040] The cognitive load calculation model jointly analyzes the fixation point coordinate sequence and the pupil diameter change data sequence. When the spatial displacement of multiple consecutive sampling points in the fixation point coordinate sequence is lower than a preset displacement threshold, the system determines that the user is currently in a fixation point stagnation state. While maintaining the fixation point stagnation state, the system simultaneously monitors the trend of pupil diameter change data and calculates the direction and magnitude of pupil diameter change within a continuous time window. When the fixation point stagnation duration exceeds a preset time threshold and the pupil diameter change trend shows a continuous contraction tendency, the system comprehensively determines that the user is currently in a high cognitive load state. A fixation point stagnation time exceeding the reference value corresponding to normal reading speed indicates that the user is deeply processing the same content, and pupil contraction further confirms that this processing is accompanied by high cognitive resource consumption rather than just slow reading speed. Specifically, after the fixation point stagnation time reaches the preset time threshold, the system extracts the pupil diameter sequence within the preset time window from that moment, calculates the linear regression slope of the sequence, and when the slope is negative and the absolute value exceeds the preset rate of change threshold, it determines that the user is in a high cognitive load state (i.e., cognitive deadlock).

[0041] After determining that the user is in a state of high cognitive load, the system locates the corresponding region in the environmental image frame currently being gazed upon based on the gaze point coordinate sequence. For example, this localization process is based on a pre-defined gaze point-scene coordinate mapping relationship, specifically including: extracting the coordinate values ​​of each sampling point during the stagnation phase from the gaze point coordinate sequence, calculating their mean coordinates as the central gaze point; determining a local region centered on this central gaze point with a preset expansion radius, the size of which is related to the object distance between the user and the scene, estimated by the focus status of the outer camera or depth sensor data; performing optical character recognition on the located corresponding region to extract the text information within that region; and using the extracted text information as the current confusion content.

[0042] Subsequently, the system retrieves the matching target answer from the teaching and research knowledge base based on the current confusion content. The teaching and research knowledge base stores question bank analyses and complete solution processes for various types of questions in the form of a vector database. The retrieval process is based on the similarity calculation between the text vector of the confusion content and the vectors of each entry in the knowledge base, selecting the entry with the highest similarity as the target answer. After obtaining the target answer, the system does not directly present it to the user. Instead, it performs a decomposition process: based on the question analysis structure or heuristic teaching rule base in the teaching and research knowledge base, the complete solution process corresponding to the target answer is broken down into multiple logically progressive sub-problems with gradually increasing cognitive load but without significant jumps. For example, the first sub-problem points to the extraction and understanding of known conditions, the second sub-problem points to the derivation of intermediate variables, and the third sub-problem points to the calculation path of the final goal, etc. The sub-problems are arranged in order of increasing complexity, forming a flexible heuristic intervention content sequence.

[0043] During implementation, if the duration of gaze stagnation exceeds a preset time threshold based on the gaze coordinate sequence but the pupil diameter change data does not show a contraction trend, the system will not enter the high cognitive load determination branch to avoid misjudging normal reading or deep thinking as a cognitive deadlock. When the system successfully determines a high cognitive load state but fails to extract effective text information from the corresponding area located based on the gaze coordinates (e.g., the user is looking at an image rather than text, or the gaze point falls on a formula graphic causing OCR to fail to recognize it), the system will instead extract the visual semantic information of the corresponding area (including object category, scene type, etc.) as alternative material for the current confusing content and combine this visual semantic information to generate general guided questions.

[0044] Through the collaborative monitoring of internal and external cameras and the cognitive deadlock determination mechanism in this embodiment, the system can automatically identify whether a user is in a state of silent stagnation based on the user's physiological feedback signals. Based on this, it automatically extracts the specific learning content the user is currently focusing on as the content causing confusion, and further provides flexible guidance by breaking down the answer into sub-questions rather than directly providing the answer. This approach provides precise intervention that meets the user's current cognitive needs while maintaining the user's initiative in self-learning. The flexible intervention strategy of breaking down the target answer into multiple sub-questions instead of directly presenting the complete answer provides cognitive support while preserving the user's independent reasoning space, avoiding the dependency and mental inertia that may result from direct answers. The dual-condition joint determination of the fixation point stagnation time threshold and pupil constriction trend effectively reduces the false trigger rate, avoiding unnecessary intervention interference during the user's normal thinking process.

[0045] In an optional embodiment, the step of breaking down the target answer into flexible heuristic intervention content corresponding to multiple sub-questions includes: Based on the current confusion and the target answer, generate empathetic opening content; The target answer is broken down into multiple sub-questions, and guided questions are generated based on these sub-questions. Retrieve historical scenes related to the current confusion and the target answer from the visual memory bank and learned knowledge points related to the current confusion and the target answer from the learning progress bank. Based on the retrieval results, modify the guided question content to associate the guided question content with the relevant historical scenes or learned knowledge points. Based on the empathetic opening and the final guiding questions, flexible heuristic intervention content is generated.

[0046] In this embodiment, the system first generates empathetic opening content based on the current confusing content and the target answer. Empathetic opening content refers to textual statements generated by the system after detecting that the user is in a state of high cognitive load, but before formally providing guidance information. These statements have an emotional soothing function and adopt a first-person perspective to express empathy, aiming to alleviate the user's anxiety or frustration caused by cognitive difficulties. Specifically, the system inputs the textual features of the current confusing content (question type, difficulty level, and knowledge domain involved) and the complexity features of the target answer into an opening statement generation template library. From the library, it selects an emotional expression template that matches the current situation. For example, for difficult questions, a soothing opening emphasizing "don't rush" is used to establish emotional resonance with the user by acknowledging the challenge of the current task. For more complicated steps, an encouraging opening emphasizing "let's take it one step at a time" is used.

[0047] Meanwhile, the system breaks down the target answer into multiple sub-questions according to its inherent logical structure. For example, the breakdown is based on the analytical structure templates for each question type in the teaching and research knowledge base, with different types of questions corresponding to different breakdown paths. For calculation-related questions, the breakdown path typically follows a three-stage structure: "extraction of known conditions—derivation of intermediate quantities—calculation of target quantity"; for proof-related questions, the breakdown path follows a structure: "expansion of known conditions—calling of key theorems—gradual approximation of the conclusion"; for concept comprehension-related questions, the breakdown path follows a structure: "decomposition of concept elements—explanation of the meaning of each element—comprehensive induction". Each breakdown path generates a set of sub-questions, and the sub-questions form a progressive dependency relationship by using the answer to the previous question as a condition for the next question. This allows the user to advance one reasoning step each time they answer a sub-question, forming guided question content.

[0048] After generating guided questions, the system performs a modification operation to enhance their personalization. This modification includes the following steps: First, it retrieves historical scene records from the visual memory bank that match the semantic tags of the current confusing content or target answer. Specifically, this retrieval can be done by calculating the cosine similarity between the semantic vector of the current confusing content or target answer and the scene tag vectors in the visual memory bank, using scene tags with similarity scores higher than a preset threshold as associated material. Second, it retrieves learned knowledge points from the learning progress database that are semantically similar to the current confusing content or target answer. It performs semantic matching between each knowledge point in the user's learned knowledge point list and the current confusing content or target answer, selecting one or more knowledge points with the highest matching degree as associated material. The related materials (historical scene tags, names of learned knowledge points) retrieved from the visual memory bank and learning progress bank are embedded into the corresponding sub-question statements according to their semantic relevance to each sub-question. The modified sub-question presentation is as follows: "Do you remember the [objects / phenomena] you saw in [scene tags] before? Think about this question in conjunction with the [learned knowledge points] we have learned" or "Recall the [core concepts] we talked about in [learned knowledge points] and use them to solve this step."

[0049] After generating empathic opening content and modifying guided questions, the system combines the two to form a complete sequence of flexible, heuristic intervention content. The opening content is at the beginning of the sequence, followed by the modified sub-questions arranged sequentially, with transitional statements maintaining coherence between them. This ensures that the generated flexible, heuristic intervention content gains user acceptance on an emotional level, conforms to the progressive learning pattern on a cognitive level, and reduces the unfamiliarity and cognitive burden of new questions by leveraging the user's own experiences and existing knowledge on a personal level.

[0050] In an optional embodiment, after generating proactive guidance content based on the semantic information and the target subject knowledge points, the method further includes: summarizing the semantic information into a text description and extracting semantic tags, and writing the text description and the semantic tags into the visual memory bank; Accordingly, the step of retrieving historical scenes related to the current confusion content and the target answer from the visual memory bank includes: retrieving the visual memory bank based on the current confusion content and the target answer to obtain semantic tags with a similarity higher than a first preset threshold and semantic tags with a similarity higher than a second preset threshold to the target answer, as target semantic tags, and obtaining the target text description corresponding to the target semantic tags to determine the relevant historical scenes.

[0051] The text description refers to the text data obtained by summarizing the visual semantic information (including object instances and their attribute information) in the image frame in the form of natural language sentences. It retains the key elements in the visual scene and their interrelationships, such as "The user is viewing a stone arch bridge outdoors, which has an arc structure." Semantic tags are index marks in the form of keywords or phrases extracted from semantic information, which are used for fast matching during subsequent retrieval, such as "arch bridge," "arc structure," and "stone." The visual memory bank is a historical database that stores text descriptions and semantic tags in the form of structured data tables or vector databases. Each record contains a text description, one or more semantic tags, and a corresponding storage timestamp and scene identifier, which are used to support subsequent cross-scene retrieval and association guidance.

[0052] In this embodiment, as Figure 4 As shown, after generating proactive guidance content based on semantic information and target subject knowledge points, the system further performs persistent storage operations for visual memory. The semantic information is summarized into text: the identified object instance name, extracted attribute information (structural attributes, material attributes, functional attributes), and target subject knowledge points matched from the teaching and research knowledge base are integrated to generate a structured natural language text description. This text description uses the sentence structure "The user is currently looking at [object instance], which has features such as [attribute 1], [attribute 2], etc., and these features are related to [target subject knowledge point]." Simultaneously, the system extracts core vocabulary from the semantic information as semantic tags, including the object instance name, key nouns in each attribute information, and the target subject knowledge point name. The system writes the generated text description and semantic tags into the visual memory bank as associated records. Each record can also include a current timestamp and scene type identifier (determined by the scene classification results of environmental image frames, such as "outdoor natural scene," "indoor learning scene," "home scene," etc.).

[0053] When generating content using flexible heuristic intervention, the system needs to retrieve historical scenarios related to the current confusing content and the target answer from the visual memory bank. For example, the retrieval process employs a two-way matching strategy: the system generates a first query vector based on the text features of the current confusing content and a second query vector based on the text features of the target answer. The first query vector is compared with the semantic tag vectors corresponding to each record in the visual memory bank, and semantic tags with similarity higher than a first preset threshold are selected as the first candidate set. Similarly, the second query vector is compared with the semantic tag vectors corresponding to each record in the visual memory bank, and semantic tags with similarity higher than a second preset threshold are selected as the second candidate set. The system performs a union or intersection operation on the first and second candidate sets. In one implementation, a union operation is used to expand the recall scope of historical scenarios; in another implementation, an intersection operation is used to ensure that the retrieved historical scenarios are relevant to both the confusing content and the target answer. The target text description corresponding to the final target semantic tag is read from the visual memory bank as relevant historical scenario data for use in the guided question content modification operation. By persistently storing the visual semantic information perceived during the proactive guidance process, a database of the user's cross-scenario visual experiences is formed, providing personalized related materials based on the user's own experiences for flexible intervention in subsequent cognitive deadlock states.

[0054] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another multimodal accompanying educational assistance method is provided, such as... Figure 5 As shown, the method includes: Step 201: When the user is in a non-silent state, the audio data is converted into text through speech recognition, and the gated intelligent agent judges the validity of the text.

[0055] In this embodiment, a real-time interaction link is executed when the user is in a non-silent state. When the user issues a voice command, the system determines that the user is currently in a non-silent state through a voice activity detection module. The wearable acquisition terminal uploads the collected audio data to the cloud in real time via a wireless network, such as... Figure 6As shown, after receiving audio data, the cloud-based speech recognition service converts the audio data into text. The recognized text is then sent to the gate agent. Upon receiving the recognized text, the gate agent performs validity checks. The checks are as follows: First, the recognized text is length-checked. If the text length is less than a preset minimum valid character count threshold, the text is considered invalid input. This threshold is set based on the minimum number of characters required for a complete sentence in natural language and is used to filter out incomplete text fragments caused by coughing, breathing sounds, or short noises. For text that passes the length check, the gate agent further performs semantic validity checks: the text is input into a lightweight semantic classifier, which is trained using pre-labeled valid and invalid request samples. The classifier outputs the probability value that the text belongs to a valid teaching request. When this probability value is higher than a preset validity threshold, the text is considered a valid educational assistance request; when the probability value is lower than the threshold, the text is considered invalid input, and the system does not perform further processing.

[0056] Step 202: When the text is determined to be valid, it is synchronously distributed to the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent for parallel processing. The query processing sub-agent is used to rewrite the text into a structured query. The retrieval enhancement sub-agent is used to use the structured query or the text as retrieval input to perform knowledge retrieval and enhance the retrieval results. The tool invocation sub-agent is used to invoke the corresponding tool based on the text.

[0057] In this embodiment, the gated agent distributes the text that is determined to be valid to three independently running sub-agents in parallel: the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent.

[0058] Specifically, the process by which the query processing sub-agent rewrites the text into a structured query includes: identifying and removing meaningless text fragments from the text based on a predefined stop word list; determining the referential information corresponding to pronouns in the removed text sequence based on the historical dialogue context of the current session in which the text is located, and replacing the pronouns in the removed text sequence with the corresponding referential information; and performing subject and object completion, context background supplementation, and normalization conversion from colloquial expressions to standard teaching terms on the text sequence after referential information replacement to generate the structured query.

[0059] The query processing sub-agent receives valid text and performs a structured query rewriting operation. The rewriting process includes the following sub-steps: First, it removes meaningless text fragments that hit the stop word list using a predefined stop word list; second, it reads the historical dialogue context from the current session's dialogue record library, extracts key entities and concepts from the most recent rounds of question and answer, and performs referential resolution on the valid text, that is, it identifies pronouns (such as "it," "that," "this question") and omitted components in the text, and replaces or completes them with explicit entity references in combination with the historical dialogue context. For example, when a user asks "What is the Pythagorean theorem?" and then enters "What is its proof?", the query processing sub-agent rewrites "What is its proof?" as "What is the proof of the Pythagorean theorem?"; finally, it performs structured rewriting of the complete expression after referential resolution according to a predefined query template to generate a structured query statement containing query intent tags, core keywords, and limiting conditions.

[0060] Specifically, the process by which the retrieval enhancement agent uses the structured query or the text as retrieval input to perform knowledge retrieval and enhance the retrieval results includes: converting the structured query or the text into a semantic vector using a vector encoder; performing an approximate nearest neighbor retrieval in the vector index of the teaching and research knowledge base to obtain a preset number of teaching and research content fragments with the highest semantic similarity; for each content fragment in the teaching and research content fragments, labeling each sentence as a definition sentence, background sentence, principle sentence, example sentence, or detail sentence using a sentence-level semantic classifier, retaining the principle sentence and example sentence unchanged, and removing background sentences, definition sentences with a semantic similarity to the query focus in the semantic vector that is lower than a third preset threshold, and detail sentences with a semantic similarity to the semantic vector that is lower than a fourth preset threshold, so as to achieve the de-encyclopediaization of the teaching and research content fragments; rewriting the conclusion statements in the de-encyclopediaized teaching and research content fragments into a guided question sequence, breaking down the complete problem-solving steps into step-by-step prompts, and inserting transitional guiding statements between knowledge points to obtain the refined teaching and research knowledge.

[0061] The retrieval enhancement agent uses the structured query as input, which is converted into a semantic vector by a vector encoder. The retrieval enhancement agent then uses this semantic vector as the basis for retrieval, performing an approximate nearest neighbor search within the vector index of the teaching and research knowledge base. After the retrieval is complete, the retrieval enhancement agent retrieves a preset number of teaching and research content fragments with the highest semantic similarity as preliminary retrieval results. After retrieving the teaching and research content fragments, the retrieval enhancement agent performs sentence-level semantic classification on each sentence within each content fragment. For example, the sentence-level semantic classifier uses a general large language model and designs specific classification prompts to annotate sentences. The general large language model outputs classification results sentence by sentence according to the category criteria defined in the prompts. After classification, the retrieval enhancement agent performs de-encyclopedia processing based on the classification results: for sentences labeled as principle sentences and example sentences, the retrieval enhancement agent retains them completely without any modification, because principle sentences carry the core logic of knowledge, and example sentences provide application references for knowledge; both are core materials for teaching responses. For sentences labeled as background sentences, the retrieval enhancer agent removes them directly. While the background knowledge and academic citations provided by background sentences aid in complete understanding, they are not essential for the current teaching interaction. Retaining such information increases the length of the reasoning context and distracts the main agent from the core knowledge. For sentences labeled as definition sentences, the retrieval enhancer agent calculates the semantic similarity between the sentence's semantic vector and the query focus. If the similarity is below a first preset threshold, the definition sentence is removed; if the similarity is not below the first preset threshold, it is retained. This is because definitions highly relevant to the current query form the basic framework for answering user questions, while definitions deviating from the query focus introduce conceptual interpretations irrelevant to user needs. For sentences labeled as detail sentences, the retrieval enhancer agent calculates the semantic similarity between the sentence's semantic vector and the overall semantic vector. If the similarity is below a second preset threshold, the detail sentence is removed; if the similarity is not below the second preset threshold, it is retained. This is because detail sentences with high similarity provide valuable supplementary information to principle and example sentences, while detail sentences with low similarity are trivial details irrelevant to the current teaching needs. After removing the encyclopedia-like elements, the remaining teaching and research content undergoes heuristic refinement. For example, heuristic refinement is achieved by calling a general-purpose large language model and designing specific rewriting prompts that require the model to transform the input teaching and research content according to pedagogical principles.Heuristic refinement involves three aspects: First, conclusive statements in the de-encyclopedic content are rewritten into a sequence of guided questions. Sentences directly stating conclusions are transformed into multiple questions of increasing difficulty, shifting the knowledge presentation from direct instruction to guided discovery. Second, complete problem-solving steps are broken down into step-by-step hints. Problem-solving processes with sequential logical relationships are divided into several independent steps, each providing only a direction for further thought or operational clues rather than a complete execution result. Third, transitional guiding statements are inserted between knowledge points. For adjacent knowledge points with cognitive leaps, transitional guiding statements are inserted to ensure the knowledge transition conforms to the cognitive pattern of moving from the known to the unknown and from the concrete to the abstract. After all the above processing is completed, the retrieval enhancement agent uses the de-encyclopedic and heuristically refined teaching and research knowledge fragments as the enhancement result and sends them to the multimodal main agent through a structured message interface, providing high-quality knowledge input for the multimodal main agent to generate educational auxiliary information with teaching guidance.

[0062] Specifically, the process of the tool-calling sub-agent calling the corresponding tool based on the text includes: determining whether the valid request requires calling an external tool, wherein the external tool is uniformly managed through the model context protocol, and the external tool includes at least one of a calculator, a weather query interface, a search engine interface, and a translation service; if it is determined that an external tool needs to be called, then the corresponding external tool is called through the model context protocol to obtain auxiliary data.

[0063] The tool invocation sub-agent receives valid text and then performs a tool invocation determination. This sub-agent first classifies the valid text by intent, determining whether the current request requires external tools to obtain auxiliary information. External tools are managed uniformly through the model context protocol. Available tools include calculator tools (for evaluating and validating mathematical expressions), weather query tools (for obtaining real-time meteorological data), search engine tools (for retrieving information in real-time or outside the teaching and research knowledge base), and translation service tools (for multilingual translation). Based on the intent classification result, the tool invocation sub-agent determines the type of tool required, constructs a call request conforming to the tool's interface specification (including necessary input parameters), sends the call request to the corresponding external tool through the model context protocol, and receives the return result.

[0064] Step 203: After aggregating the processing results of the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent, input them into the multimodal main agent for fusion reasoning to generate response content.

[0065] Specifically, the step of aggregating the processing results of the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent and inputting them into the multimodal main agent (multimodal main agent) for fusion reasoning includes: assembling the structured query from the query processing sub-agent, the refined teaching and research knowledge from the retrieval enhancement sub-agent, the auxiliary data from the tool invocation sub-agent, the current environment image frames uploaded in real time by the wearable acquisition terminal, the visual textualized historical memory read from the visual memory bank, and the user's historical learning data read from the learning progress bank into a reasoning input context and performing fusion reasoning.

[0066] In this embodiment, before performing fusion reasoning, the multimodal main agent first assembles the reasoning input context. This assembly process organizes the following six types of data into a unified input sequence according to a preset template: The first type of data is structured queries from the query processing sub-agent, located at the beginning of the input context. It serves as the core task declaration for this reasoning, clearly defining the current user intent and the problem objective to be solved. The second type of data is refined educational knowledge from the retrieval enhancement sub-agent, immediately following the structured query and labeled with the semantic tag "related knowledge material." This type of data contains core knowledge point descriptions after de-encyclopedic processing and heuristic refinement, serving as the primary source of educational professionalism during the reasoning process. The third type of data is auxiliary data from the tool invocation sub-agent, included only in the input context when the tool is actually invoked. This auxiliary data is labeled with the semantic tag "external reference data," along with a data source identifier and acquisition timestamp, enabling the reasoning model to accurately reference external data sources in its response. The fourth type of data is a semantic description of the current environment image frames uploaded in real-time by the external camera of the wearable acquisition terminal. The first type of data consists of scene text descriptions generated by the visual large model after inferring from the current image frame. These are labeled with the semantic tag "current visual scene," enabling the inference model to perceive the user's current environmental context when generating responses. The fifth type of data is visual textualized historical memory retrieved from the visual memory bank. This type of data contains past scene records that are semantically related to the current structured query. Before the assembly operation, the multimodal agent uses the keyword vector of the structured query as retrieval input to retrieve historical scene text descriptions and semantic tags from the visual memory bank with semantic similarity exceeding a preset threshold. The retrieval results are labeled with the semantic tag "historically related scenes," used to establish cross-scene associations between the current problem and historical experiences during the inference process. The sixth type of data is user historical learning data retrieved from the learning progress database, including lists of learned knowledge points and recently completed learning content. This type of data is labeled with the semantic tag "user learning progress," enabling the inference model to adjust the depth and complexity of the expression based on the user's current cognitive foundation when generating responses.

[0067] After organizing the six types of data, the multimodal agent feeds the assembled reasoning input context into the generative model for fusion reasoning. During the reasoning process, the generative model determines the direction of the answer based on structured queries, ensures the educational professionalism of the answer based on refined teaching and research knowledge, provides external factual evidence based on auxiliary data, associates the answer with the user's environment based on the current visual scene description, enables cross-scene analogy and review based on visual textual historical memory, and adjusts the degree of personalization of the answer based on the user's historical learning data.

[0068] By applying the technical solution of this embodiment, the system can accurately capture users' proactive questions and filter invalid inputs by performing speech recognition on audio data when the user is in a non-silent state and then performing validity judgment through a gating agent. By distributing valid text in parallel to the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent for independent processing, the semantic processing of spoken text, the retrieval enhancement of the teaching and research knowledge base, and the acquisition of auxiliary data from external tools are completed synchronously within the same time window, significantly reducing the overall latency of real-time response compared to serial processing schemes. By integrating the processing results of the three sub-agents with real-time environmental image frames and user learning progress data into a multimodal main agent for fusion reasoning, the generated response content simultaneously possesses multiple guarantees of intent accuracy, educational professionalism, and data accuracy, providing a complete real-time interactive technology implementation solution for accompanying educational assistance in scenarios where users proactively ask questions.

[0069] In an optional embodiment, the method further includes: storing the conversation in a conversation record library after each conversation ends, and extracting the complete dialogue data of the current conversation from the conversation record library upon detecting the end of a conversation, wherein a conversation includes at least one conversation; performing semantic clustering analysis on the complete dialogue data, wherein the semantic clustering analysis includes at least one of the following: statistically analyzing the frequency and depth of questions asked by the user in different knowledge domains to identify points of interest, identifying knowledge points where the user repeatedly asks questions, exhibits logical confusion, or displays cognitive deadlock to mark weaknesses, and combining emotional vocabulary in the conversation text of the complete dialogue data to depict the trend of emotional changes; and generating a growth analysis report based on the analysis results.

[0070] In this embodiment, as Figure 7As shown, the system executes the asynchronous cognitive link and triggers the asynchronous cognitive analysis process upon detecting the end of a session. The detection of session end is based on the following rules: when the system does not detect new voice input within a preset silent timeout window, and the active perception link does not push new proactive guidance content within that window, while the real-time interaction link has no ongoing question-and-answer requests, the system determines that the current session has ended. After the session ends, the system writes all interaction records of this session into the session record database, including the user's original text or speech-to-text transcription, the system-generated response text, the timestamp corresponding to this round of interaction, and additional information such as the scene semantic tags detected by the active perception link when this round of interaction occurred. After writing, the system triggers the asynchronous analysis task. This asynchronous task runs independently of the real-time interaction process, does not consume the computing resources of real-time interaction, and does not affect the processing of subsequent new sessions initiated by the user. The asynchronous task extracts the complete dialogue data of this session from the session record database, based on the condition that the session identifier stored in the record matches the end event of the current session. The extracted complete dialogue data is arranged in ascending chronological order, preserving the temporal context of the dialogue.

[0071] The system performs semantic clustering analysis on the complete dialogue corpus. The specific processing flow is as follows: First, each user question text and its corresponding system response text in the dialogue corpus are concatenated into an interaction record text. After word segmentation and stop word filtering, each interaction record text is converted into a semantic vector using an embedding model. Second, a clustering algorithm is performed on the semantic vectors of all interaction record texts, grouping interaction records with similar semantic distances into the same category. Finally, each category is labeled with a topic by extracting high-frequency keywords from each text in that category or matching them with knowledge point tags in the teaching and research knowledge base to determine the knowledge domain or knowledge point corresponding to that category. The clustering results serve as the foundational data for subsequent analyses.

[0072] Based on this, the system performs specific analysis in the following three dimensions. The first dimension is interest point identification: the system counts the number of user question texts in each category after clustering, marking the categories with the highest question frequency as user interest points; for each category marked as an interest point, the system further counts the number of deep follow-up questions in consecutive question-answer pairs within that category. When the number of deep follow-up questions exceeds a preset threshold, the interest point is marked as a "high-depth interest point" to reflect the user's proactive exploration tendency in that area. The second dimension is weakness point identification: the system selects the following types of knowledge points from the clustering results and marks them as candidate weaknesses: users repeatedly ask similar questions on the same knowledge point (repeated inquiries), users' statements on related questions on the same knowledge point are inconsistent or they cannot complete logical reasoning (logical confusion), and the proactive perception link during the learning process of that knowledge point records a cognitive deadlock trigger event (cognitive deadlock trigger record, i.e., high cognitive load state). The third dimension is emotion trend analysis: the system extracts emotional words from each user question text in the complete dialogue corpus, matches each emotional word with a predefined emotional polarity weight, and plots the change sequence of emotional polarity according to the time sequence of the questions, i.e., the emotion trend. A gradual increase in the proportion of positive emotion words indicates a positive shift in user experience, while a gradual increase in the proportion of negative emotion words indicates a negative shift. Furthermore, following a pre-set report template, the analysis results are mapped to various chapters in the report, generating a growth analysis report. Parents and teachers can view the growth report via a web interface, and also view the user's interaction with the system in real time, as well as the recorded playback of that interaction.

[0073] Furthermore, as Figure 1 To specifically implement the method, this application provides a multimodal accompanying educational assistance system, such as... Figure 8 As shown, the system includes: The wearable data acquisition terminal includes an external camera and a microphone. The external camera is used to acquire environmental image frames from the user's first-person perspective and upload them to the cloud. The microphone is used to acquire audio data and upload it to the cloud. A cloud-based multi-agent system, including gating agents and visual sentinel agents; The gated agent is used to determine whether the user is in a silent state based on the acquisition results of audio data. If the user is in a silent state, a silent state notification is sent to the visual sentinel agent. The visual sentinel agent is used to respond to a silent state notification, perform visual semantic analysis on the current environmental image frame, determine the semantic information of the current environmental image frame, and map the semantic information to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base; when the target subject knowledge point matches the user's historical learning data, it is determined that the current scene has teaching value, and based on the semantic information and the target subject knowledge point, it generates proactive guidance content; and pushes the proactive guidance content to the wearable acquisition terminal. The wearable data collection terminal is also used to output the active guidance content.

[0074] In an optional embodiment, the wearable acquisition terminal further includes an inner camera, which is used to acquire image frames of the user's eyes and upload them to the cloud; The visual sentinel agent is further configured to extract the user's gaze point coordinate sequence and pupil diameter change data from the eye image data when the user is in a silent state; when the gaze point coordinate sequence determines that the duration of the gaze point in the target area exceeds a preset time threshold, and the pupil diameter change data shows a contraction trend, the agent determines that the user is in a state of high cognitive load; based on the gaze point coordinate sequence, the agent locates the corresponding area in the environmental image frame captured by the outer camera, extracts the text information in the corresponding area as the current confusion content, determines the target answer based on the current confusion content, decomposes the target answer into flexible heuristic intervention content corresponding to multiple sub-questions, and pushes the flexible heuristic intervention content to the wearable acquisition terminal; The wearable data acquisition terminal is also used to output the flexible heuristic intervention content.

[0075] In an optional embodiment, it further includes: The system comprises a query processing sub-agent, a retrieval enhancement sub-agent, a tool invocation sub-agent, and a multimodal master agent. The gated intelligent agent is also used to convert the audio data into text through speech recognition, to determine the validity of the text, and when it is determined to be valid, to synchronously distribute the text to the query processing sub-intelligent agent, the retrieval enhancement sub-intelligent agent and the tool invocation sub-intelligent agent. The query processing sub-agent is used to rewrite the text into a structured query; The retrieval enhancement sub-agent is used to take the structured query or the text as retrieval input to perform knowledge retrieval and enhance the retrieval results; The tool invokes a sub-agent, which is used to invoke the corresponding tool based on the text; The multimodal master agent is used to aggregate the processing results of the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent for fusion reasoning to generate response content.

[0076] In an optional embodiment, it further includes: A cognitive growth analysis agent is used to store each conversation in a conversation record library after the end of each conversation, and to extract the complete dialogue corpus of the current conversation from the conversation record library upon detecting the end of a conversation, wherein each conversation includes at least one conversation; to perform semantic clustering analysis on the complete dialogue corpus, wherein the semantic clustering analysis includes at least one of the following: counting the frequency and depth of the user's questions in different knowledge domains to identify points of interest, identifying knowledge points where the user has repeated questions, logical confusion or cognitive deadlock to mark weaknesses, and combining emotional words in the conversation text of the complete dialogue corpus to depict the trend of emotional changes; and to generate a growth analysis report based on the analysis results.

[0077] It should be noted that other corresponding descriptions of the functional units involved in the multimodal accompanying education assistance system provided in this application embodiment can be found in the following references. Figures 1 to 6 The corresponding descriptions in the method will not be repeated here.

[0078] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 8 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0079] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0081] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0082] It should be noted that the user personal information involved in the embodiments of this application is all authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals. It should be noted that if any software tools or components other than those of this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.

[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal accompanying educational assistance method, characterized in that, The method includes: The system continuously collects environmental image frames and audio data from the user's first-person perspective using a wearable data acquisition terminal, and determines whether the user is in a silent state based on the audio data acquisition results. When the user is in a silent state, visual semantic analysis is performed on the current environmental image frame to determine the semantic information of the current environmental image frame, and the semantic information is mapped to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base. When the target subject knowledge point matches the user's historical learning data, it is determined that the current scenario has teaching value, and proactive guidance content is generated based on the semantic information and the target subject knowledge point. The proactive guidance content is pushed to the wearable data acquisition terminal for output.

2. The method according to claim 1, characterized in that, The step of performing visual semantic analysis on the current environmental image frame to determine the semantic information of the current environmental image frame, and mapping the semantic information to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base, includes: Identify object instances in the current environmental image frame, extract at least one attribute information corresponding to the object instance, and use the object instance and the attribute information as semantic information of the current environmental image frame, wherein the attribute information includes structural attributes, material attributes and functional attributes; The semantic information is matched with the associated information corresponding to each subject knowledge point in the teaching and research knowledge base, and the subject knowledge point corresponding to the successfully matched associated information is taken as the target subject knowledge point corresponding to the semantic information.

3. The method according to claim 2, characterized in that, After taking the subject knowledge points corresponding to the successfully matched association information as the target subject knowledge points corresponding to the semantic information, the method further includes: Read the grade information from the user's user profile database and the learned knowledge points from the learning progress database; The target subject knowledge points are filtered based on the grade information and the learned knowledge points, and the filtered knowledge points that match the user's current learning stage are taken as the final target knowledge points.

4. The method according to claim 3, characterized in that, The step of filtering the target subject knowledge points based on the grade information and the learned knowledge points, and selecting the filtered knowledge points that match the user's current learning stage as the final target knowledge points, includes: Check whether the target knowledge point matches any of the learned knowledge points; if it matches one of them, only the target knowledge point is retained; if it matches multiple of them, the learned knowledge point with the most recent learning time among the matched learned knowledge points is taken as the final target knowledge point. If no match is found, then check whether the target knowledge point belongs to the required knowledge point corresponding to the grade information; if it belongs, then the knowledge point ranked first among the required knowledge points is taken as the final target knowledge point; if it does not belong, then the target subject knowledge point is ignored.

5. The method according to claim 1, characterized in that, The wearable acquisition terminal includes an outer camera and an inner camera, and the current environment image frame is acquired by the outer camera; the method further includes: The user's eye image frames are captured by the inner camera of the wearable acquisition terminal; When the user is in a silent state, extract the user's gaze point coordinate sequence and pupil diameter change data based on the eye image data; When the fixation point coordinate sequence determines that the duration of fixation in the target area exceeds a preset time threshold, and the pupil diameter change data shows a contraction trend, it is determined that the user is in a state of high cognitive load. Based on the gaze point coordinate sequence, the corresponding region in the environmental image frame captured by the outer camera is located, the text information in the corresponding region is extracted as the current confusion content, and the target answer is determined according to the current confusion content. The target answer is decomposed into flexible heuristic intervention content corresponding to multiple sub-questions. The flexible heuristic intervention content is pushed to the wearable data acquisition terminal for output.

6. The method according to claim 5, characterized in that, The process of breaking down the target answer into multiple sub-questions and corresponding flexible heuristic interventions includes: Based on the current confusion and the target answer, generate empathetic opening content; The target answer is broken down into multiple sub-questions, and guided questions are generated based on these sub-questions. Retrieve historical scenes related to the current confusion and the target answer from the visual memory bank and learned knowledge points related to the current confusion and the target answer from the learning progress bank. Based on the retrieval results, modify the guided question content to associate the guided question content with the relevant historical scenes or learned knowledge points. Based on the empathetic opening and the final guiding questions, flexible heuristic intervention content is generated.

7. The method according to claim 6, characterized in that, After generating proactive guidance content based on the semantic information and the target subject knowledge points, the method further includes: The semantic information is summarized into a text description and semantic tags are extracted, and the text description and semantic tags are written into the visual memory bank; Accordingly, the retrieval of historical scenes related to the current confusing content and the target answer from the visual memory bank includes: Based on the current confusion content and the target answer, the visual memory database is retrieved to obtain semantic tags with a similarity higher than a first preset threshold and semantic tags with a similarity higher than a second preset threshold to the target answer, which are used as target semantic tags. The target text description corresponding to the target semantic tags is then obtained to determine the relevant historical scene.

8. The method according to claim 1, characterized in that, After determining whether the user is in a silent state based on the audio data collection results, the method further includes: When the user is in a non-silent state, the audio data is converted into text through speech recognition, and the gated intelligent agent judges the validity of the text; When a text is deemed valid, it is synchronously distributed to the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent for parallel processing. The query processing sub-agent rewrites the text into a structured query, the retrieval enhancement sub-agent uses the structured query or the text as retrieval input to perform knowledge retrieval and enhance the retrieval results, and the tool invocation sub-agent invokes the corresponding tool based on the text. The processing results of the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent are aggregated and input into the multimodal main agent for fusion reasoning to generate response content.

9. The method according to claim 8, characterized in that, The process by which the query processing sub-agent rewrites the text into a structured query includes: Based on a predefined stop word list, meaningless text fragments in the text are identified and removed; Based on the historical dialogue context of the current session in which the text is located, determine the referential information corresponding to the pronouns in the removed text sequence, and replace the pronouns in the removed text sequence with the corresponding referential information; The text sequence after the referential information has been replaced is subjected to subject and object completion, contextual background supplementation, and normalization transformation from colloquial expressions to standard teaching terminology to generate the structured query.

10. The method according to claim 8, characterized in that, The process by which the retrieval enhancement sub-agent takes the structured query or the text as retrieval input to perform knowledge retrieval and enhances the retrieval results includes: The structured query or the text is converted into a semantic vector by a vector encoder, and an approximate nearest neighbor retrieval is performed in the vector index of the teaching and research knowledge base to obtain a preset number of teaching and research content fragments with the highest semantic similarity. For each content segment in the teaching and research content segment, each sentence is labeled as a definition sentence, background sentence, principle sentence, example sentence, or detail sentence by a sentence-level semantic classifier. Principle sentences and example sentences are retained unchanged, while background sentences, definition sentences with a semantic similarity to the query focus in the semantic vector that is lower than a third preset threshold, and detail sentences with a semantic similarity to the semantic vector that is lower than a fourth preset threshold are removed, so as to achieve the de-encyclopediaization of the teaching and research content segment. The conclusion-style statements in the de-encyclopedic teaching and research content segments are rewritten into a sequence of guided questions. The complete problem-solving steps are broken down into step-by-step prompts, and transitional guiding statements are inserted between knowledge points to obtain the refined teaching and research knowledge.

11. The method according to claim 8, characterized in that, The process by which the tool-calling sub-agent invokes the corresponding tool based on the text includes: Determine whether the valid request requires calling an external tool, wherein the external tool is uniformly managed through the model context protocol, and the external tool includes at least one of a calculator, a weather query interface, a search engine interface, and a translation service; If it is determined that an external tool needs to be invoked, the corresponding external tool is invoked through the model context protocol to obtain auxiliary data.

12. The method according to claim 8, characterized in that, The step of aggregating the processing results of the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent and then inputting them into the multimodal main agent for fusion reasoning includes: The structured query from the query processing sub-agent, the refined teaching and research knowledge from the retrieval enhancement sub-agent, the auxiliary data from the tool calling sub-agent, the current environment image frames uploaded in real time by the wearable acquisition terminal, the visual textualized historical memory read from the visual memory bank, and the user's historical learning data read from the learning progress bank are assembled into a reasoning input context and fused reasoning is performed.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: After each session ends, the session is stored in the session record library, and upon detecting the end of a session, the complete dialogue data of this session is extracted from the session record library, wherein a session includes at least one conversation; Semantic clustering analysis is performed on the complete dialogue corpus. The semantic clustering analysis includes at least one of the following: statistically analyzing the frequency and depth of questions asked by the user in different knowledge domains to identify points of interest; identifying knowledge points where the user repeatedly asks questions, has logical confusion or cognitive deadlock to mark weaknesses; and combining emotional words in the conversation text of the complete dialogue corpus to depict the trend of emotional changes. A growth analysis report is generated based on the analysis results.

14. A multimodal accompanying educational assistance system, characterized in that, The multimodal accompanying educational assistance method applied to any one of claims 1 to 13 includes: The wearable data acquisition terminal includes an external camera and a microphone. The external camera is used to acquire environmental image frames from the user's first-person perspective and upload them to the cloud. The microphone is used to acquire audio data and upload it to the cloud. A cloud-based multi-agent system, including gating agents and visual sentinel agents; The gated agent is used to determine whether the user is in a silent state based on the acquisition results of audio data. If the user is in a silent state, a silent state notification is sent to the visual sentinel agent. The visual sentinel agent is used to respond to a silent state notification, perform visual semantic analysis on the current environmental image frame, determine the semantic information of the current environmental image frame, and map the semantic information to the target subject knowledge point according to the association information corresponding to each subject knowledge point in the teaching and research knowledge base; when the target subject knowledge point matches the user's historical learning data, it is determined that the current scene has teaching value, and based on the semantic information and the target subject knowledge point, it generates proactive guidance content; and pushes the proactive guidance content to the wearable acquisition terminal. The wearable data collection terminal is also used to output the active guidance content.

15. The system according to claim 14, characterized in that, The wearable acquisition terminal also includes an inner camera, which is used to capture the user's eye image frames and upload them to the cloud; The visual sentinel agent is further configured to extract the user's gaze point coordinate sequence and pupil diameter change data from the eye image data when the user is in a silent state; when the gaze point coordinate sequence determines that the duration of the gaze point in the target area exceeds a preset time threshold, and the pupil diameter change data shows a contraction trend, the agent determines that the user is in a state of high cognitive load; based on the gaze point coordinate sequence, the agent locates the corresponding area in the environmental image frame captured by the outer camera, extracts the text information in the corresponding area as the current confusion content, determines the target answer based on the current confusion content, decomposes the target answer into flexible heuristic intervention content corresponding to multiple sub-questions, and pushes the flexible heuristic intervention content to the wearable acquisition terminal; The wearable data acquisition terminal is also used to output the flexible heuristic intervention content.

16. The system according to claim 15, characterized in that, Also includes: The system comprises a query processing sub-agent, a retrieval enhancement sub-agent, a tool invocation sub-agent, and a multimodal master agent. The gated intelligent agent is also used to convert the audio data into text through speech recognition, to determine the validity of the text, and when it is determined to be valid, to synchronously distribute the text to the query processing sub-intelligent agent, the retrieval enhancement sub-intelligent agent and the tool invocation sub-intelligent agent. The query processing sub-agent is used to rewrite the text into a structured query; The retrieval enhancement sub-agent is used to take the structured query or the text as retrieval input to perform knowledge retrieval and enhance the retrieval results; The tool invokes a sub-agent, which is used to invoke the corresponding tool based on the text; The multimodal master agent is used to aggregate the processing results of the query processing sub-agent, the retrieval enhancement sub-agent, and the tool invocation sub-agent for fusion reasoning to generate response content.

17. The system according to any one of claims 14 to 16, characterized in that, Also includes: A cognitive growth analysis agent is used to store each conversation in a conversation record library after the end of each conversation, and to extract the complete dialogue corpus of the current conversation from the conversation record library upon detecting the end of a conversation, wherein each conversation includes at least one conversation; to perform semantic clustering analysis on the complete dialogue corpus, wherein the semantic clustering analysis includes at least one of the following: counting the frequency and depth of the user's questions in different knowledge domains to identify points of interest, identifying knowledge points where the user has repeated questions, logical confusion or cognitive deadlock to mark weaknesses, and combining emotional words in the conversation text of the complete dialogue corpus to depict the trend of emotional changes; and to generate a growth analysis report based on the analysis results.

18. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 13.

Images