Companion content generation method and system based on multi-modal intent perception

Abstract

This application discloses a method and system for generating learning content based on multimodal intent perception. The method includes: caching audio and video data from an audio / video stream to construct a dynamic context window; acquiring multimodal interaction signals; spatiotemporally aligning the multimodal interaction signals with the audio and video data within the dynamic context window to extract a target interaction object; combining the multimodal interaction signals with the target interaction object to determine cognitive breakpoints and corresponding question intent features; in response to detecting the cognitive breakpoint, extracting corresponding semantic segments from the dynamic context window; generating learning content using a generative model based on the question intent features and the semantic segments; and outputting the learning content as feedback. This application can accurately capture cognitive states and provide pixel-level precise scene-based tutoring without disrupting the learning flow.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method and system for generating learning content based on multimodal intent perception. Background Technology

[0002] With the widespread adoption of online education and remote work, audio and video have become the primary mediums for knowledge dissemination and information acquisition. Currently, mainstream media playback systems mainly provide receivers with basic playback control and auxiliary functions such as progress bar control, playback speed switching, and discussion forum comments. However, existing general-purpose technical solutions have significant limitations in application scenarios targeting deep learning and immersive reading.

[0003] Current interactive mechanisms suffer from thought-chain disruptions. In traditional video learning, when encountering complex knowledge points that are difficult to understand, learners typically need to go through a cumbersome sequence of operations, including manually pausing playback, exiting full-screen immersive mode, entering keywords into an external search engine, searching for answers among a vast amount of results, and finally returning to the video playback interface. This cumbersome process disrupts the thought process, making learners more susceptible to frustration in their flow state and reducing cognitive efficiency and immersion.

[0004] Existing systems lack the ability to proactively perceive multimodal interaction intentions. Mainstream media players typically operate in a passive mode, waiting for explicit textual commands, and cannot automatically identify and understand cognitive confusion that arises during viewing. When micro-expressions such as frowning or head shaking are captured on the screen, or when users frequently make irregular circles on the screen using a stylus, the system fails to capture and transform the implicit feedback signals representing internal cognitive impairment into effective interaction triggers, thus missing the opportunity for intervention and guidance.

[0005] Existing tutoring generation solutions based on natural language processing technology suffer from low content matching accuracy and a lack of spatiotemporal scene awareness. The system cannot accurately know where the current video timeline is or what specific visual objects are displayed in the frame. As a result, the generated tutoring content cannot be aligned with the spatiotemporal coordinates of the current audio-visual context, making it difficult to resolve specific questions related to the current micro-scene. Summary of the Invention

[0006] This application provides a method and system for generating learning content based on multimodal intent perception, which is used to overcome the technical defects of existing technologies, such as interaction gaps, sluggish intent perception, and lack of scenario matching in tutoring content.

[0007] This application provides a method for generating audio-visual learning content based on multimodal interactive intent awareness. The method includes:

[0008] Cache audio and video data in the audio and video streams to build a dynamic context window;

[0009] Acquire multimodal interaction signals;

[0010] The multimodal interaction signals are spatiotemporally aligned with the audio and video data within the dynamic context window to extract the target interaction object;

[0011] By combining the multimodal interaction signals with the target interaction object, a cognitive breakpoint and the corresponding question intent features are obtained.

[0012] In response to the detection of the cognitive breakpoint, the corresponding semantic slice is extracted from the dynamic context window;

[0013] Based on the question intent features and the semantic slices, a generative model is used to generate learning companion content;

[0014] Output the learning support content as feedback.

[0015] This application also provides an audio-visual learning content generation system based on multimodal interactive intent awareness, including:

[0016] The window building module is configured to cache audio and video data from audio and video streams to build dynamic context windows.

[0017] The signal acquisition module is configured to acquire multimodal interactive signals;

[0018] The alignment detection module is configured to perform spatiotemporal alignment of the multimodal interaction signals with the audio and video data in the dynamic context window, extract the target interaction object, and combine the multimodal interaction signals with the target interaction object to make a judgment, thereby obtaining cognitive breakpoints and corresponding question intent features.

[0019] The slice extraction module is configured to extract the corresponding semantic slice fragment from the dynamic context window in response to the detection of the cognitive breakpoint.

[0020] The learning companion generation module is configured to generate learning companion content using a generative model based on the question intent features and the semantic slice fragments.

[0021] The adaptive feedback module is configured to output the learning support content as feedback.

[0022] This application also provides a computer device, including a processor and a memory communicatively connected to the processor; the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0023] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0024] By adopting the above technical solution, this application performs spatiotemporal alignment and weighted fusion detection of implicit signals and explicit operations, accurately capturing the true cognitive breakpoint state and avoiding false triggers caused by single-modal judgment. Combining semantic backtracking boundaries and pre-trained model technology, the system can extract precise objects in the current audio-visual spatiotemporal coordinates to construct prompt words, and adopts a bypass preview mechanism that does not forcibly interrupt the learning flow, providing pixel-level fine-grained and highly scene-aware tutoring content while ensuring immersion. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the audio-visual learning content generation method based on multimodal interactive intent perception provided in an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating the dynamic context window construction and feature extraction method provided in this embodiment of the invention.

[0027] Figure 3 This is a flowchart illustrating the multimodal intent recognition and visual object segmentation detection provided in an embodiment of the present invention.

[0028] Figure 4 This is a logical structure block diagram of the audio and video learning content generation system based on multimodal interactive intent perception provided in the embodiments of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] To support the closed-loop and underlying operation of the technical solution, the hardware implementation environment of this application includes an audio and video decoding and rendering engine that performs underlying streaming media unpacking and decoding tasks, used for playing and delivering digital video images and audio streams. The system is equipped with a sensor data preprocessing and feature extraction algorithm unit, used for pre-filtering and feature extraction of high-frequency level signals input from the underlying camera, microphone, and touchpad. To connect local devices with large-scale pre-trained models in the cloud, the system is also equipped with a network communication protocol module, used for performing network communication operations such as packet encryption and bidirectional full-duplex socket connections. It should be noted that the aforementioned underlying hardware or protocol units are not limited to the conventional components listed above; those skilled in the art can also implement the basic supporting functions of this application using hardware devices with equivalent performance or upgradeable features.

[0031] Combination Figures 1 to 3 This application provides a main embodiment of a method for generating audio-visual learning content based on multimodal interactive intent awareness. For example... Figure 1 As shown, the method mainly consists of a series of steps to achieve a closed loop of immersive tutoring feedback with zero interruptions.

[0032] In the construction of the dynamic context window S101, the system monitors and caches audio and video data within a preset time period before and after the current playback time node in real time during the audio and video stream playback, and constructs the dynamic context window based on this.

[0033] Since subsequent large-scale model inference relies on data support from the preceding time window, the system is configured with a sliding space that scrolls with the progress bar. Based on the specific application scenario of this application, the system is configured to continuously cache data N seconds before and after the current time point T. This dynamic context window is precisely limited to the interval from the playback progress bar T-30s to T, and all subtitle text and keyframe images within this interval are loaded into memory.

[0034] Combination Figure 2 The detailed processing flow shown above, specifically includes time-segmentation S201 to S204, of the dynamic construction mechanism.

[0035] In the time segmentation S201, the system divides the continuous audio and video stream into multiple closely connected time segments according to a preset duration threshold.

[0036] In the sliding window buffer S202, the processor uses a double-ended queue data structure to buffer these multiple time slices in a sliding window. As the playback progress bar moves forward, the audio and video data in the dynamic context window is continuously updated by dequeuing the head pointer and enqueuing the tail pointer. It should be noted that the double-ended queue structure here is only an example. In other embodiments, the buffer structure can also be a circular buffer or a linked list structure.

[0037] In the keyframe extraction and mapping process S203, the system performs high-frequency keyframe extraction operations on multiple time slices within the dynamic context window for the cached data to obtain video keyframes containing the main information and audio spectrum sequences synchronized with those frames. The system calls the underlying visual feature extraction network and audio feature extraction network to map the extracted video keyframes and audio spectrum sequences into multi-dimensional visual feature tensors and audio feature tensors, respectively.

[0038] In the cross-modal concatenation matrix S204, in order to solve the problem of cross-modal information isolation, the system performs time axis alignment and depth concatenation on the acquired visual feature tensor and audio feature tensor according to the corresponding timestamps, and finally forms a cross-modal context matrix.

[0039] To prevent long-term caching from exhausting system memory, a gradient-based adaptive tensor dimensionality reduction method is designed. When the dimension of the cross-modal context matrix exceeds the memory allocation limit threshold M, the system iterates through the matrix and calculates the content change gradient value of the audio and video frames within each time slice. The system performs downsampling pooling on time slices with content change gradient values ​​below a preset gradient threshold T, while maintaining the original sampling rate of the remaining time slices with drastic content changes, and reassembles them to form the dimensionality-reduced target context tensor sequence. Through this cross-modal concatenation and adaptive pooling mathematical logic, the system preserves temporal information within computational constraints.

[0040] In the synchronous acquisition of interactive signals S102, the system synchronously activates multi-channel sensors to acquire multimodal interactive signals. These multimodal interactive signals are explicitly divided into two categories. Explicit interactive features mainly include directly input touch trajectory coordinates and audio segments of voice commands. Implicit behavioral features mainly capture involuntary physiological behaviors caused by cognitive blockage, such as facial expression parameters and head posture angles.

[0041] In this data acquisition process, a globally unified high-precision timestamp clock was established to unify the time baseline of the various heterogeneous sensors. The system allocates asymmetric sampling quotas. Rapidly changing touch trajectory coordinates and voice command audio segments are acquired at a first sampling frequency. Relatively slower facial expression parameters and head posture angles are acquired at a second sampling frequency. The first sampling frequency must be significantly higher than the second sampling frequency to ensure the smoothness of high-frequency handwriting sampling.

[0042] After acquiring continuous facial expression parameters and minute displacement signals such as head posture angles, the system introduces a filtering and denoising algorithm to counteract background noise caused by natural body swaying. The system uses a Kalman filter algorithm to perform state estimation and smoothing denoising on the continuously acquired data. During this process, the system calculates the L2 norm of the measurement residual vector at the current time t, which serves as the basis for dynamically adjusting the filter gain. In a preferred embodiment, the above determination logic is implemented using the following formula:

[0043] The system performs the calculation for determining the L2 norm condition of the measurement residual:

[0044]

[0045] Among them, variables Let z represent the L2 norm of the measurement residual vector at the current time t. t This represents the vector of actual implicit signal measurements. H represents the system's prior predicted state vector derived from the state equation, and H is the measurement matrix. Variable E represents the preset abnormal jitter threshold used to determine if abnormal, drastic displacement has occurred. When the L2 norm exceeds the preset abnormal jitter threshold E, the system dynamically increases the system noise covariance weight coefficient corresponding to the current state covariance matrix, thereby reducing the contamination of the implicit behavioral feature sequence stability by sudden displacement. The system extracts the rate of change of key point displacements from the smoothed facial expression parameters and encapsulates it as an implicit tension index reflecting confusion, outputting a denoised implicit feature vector. Through this adaptive filtering feedback mechanism, the system eliminates motion interference caused by non-cognitive behaviors.

[0046] In the extraction of visual objects from the screen S103, the system uses spatiotemporal alignment technology to perform three-dimensional fusion mapping between the captured multimodal interaction signals and the audio and video data cached in the dynamic context window, and accurately extracts the audio and video visual objects at the corresponding positions in the interaction space coordinates.

[0047] Combination Figure 3 The method flow shown breaks down the detailed logic of alignment and segmentation into matching time-lapse video frames S301 to S303.

[0048] In the video frame S301 at the matching time, the system extracts the precise timestamp of the multimodal interaction signal and uses this timestamp as the retrieval benchmark to reverse match and lock the video frame at the moment when the interaction occurs in the dynamic context window sequence.

[0049] In the point-to-point spatial mapping S302, the system performs spatial mapping, which translates and maps the two-dimensional touch points captured in the explicit interaction features to the global spatial coordinate system of the video frame at the corresponding moment.

[0050] In generating the Region of Interest (ROI) and segmenting S303, the system uses the mapped 2D touch point as the physical center and expands outwards with a preset radius to generate a rectangular ROI. This ROI is input into the underlying pre-trained instance segmentation network, outputting multiple candidate object masks and their respective target classification confidence scores. To avoid segmentation misalignment risks caused by relying solely on deep network confidence scores, the system introduces an intersection-union (IU) parameter for strong spatial constraints. The system calculates the IU parameter between the bounding box of each candidate object mask and the polygonal region corresponding to the actual input 2D touch point. Based on this, the system calculates the optimal target object by constructing a weighted scoring function. In a preferred embodiment, this calculation logic is implemented using the following formula:

[0051] The system performs the final weighted score calculation for the target object:

[0052]

[0053] In this system, the variable `Score` represents the final weighted score of the mask image, `Conf` represents the target classification confidence score output by the instance segmentation network, and `IoU` represents the spatial region intersection-union ratio parameter calculated above. Variables `k1` and `k2` represent adjustable weight coefficients used to adjust the weight of each variable. After multiple rounds of comparison, the system selects the candidate object mask image with the highest weighted score, extracts independent pixel blocks containing complete connected components, and uses these as the final visual objects for the audio and video images. Through this dual-track segmentation logic that integrates semantic confidence and physical spatial intersection, the system achieves granular alignment.

[0054] In determining the cognitive breakpoint S104, the system places the extracted implicit behavioral features and the previously determined audio and video visual objects into a joint detection architecture for weighted fusion operation, so as to comprehensively determine whether a cognitive breakpoint has been encountered and abstract the exact question intent features.

[0055] The fusion process mainly consists of cross-attention and probabilistic classification steps, such as Figure 3As shown in S304 and S305. In S304, the system uses a multilayer perceptron to classify and compress implicit behavioral features to obtain implicit intent vectors. Simultaneously, it performs semantic parsing, including optical character recognition, on visual objects in the audio / video feed to obtain characteristic visual content vectors. The system feeds these two vectors into a cross-feature fusion layer containing an attention mechanism. Using the visual content vector as the query matrix and the implicit intent vector as the key and value matrices, the system calculates the activation response values ​​of visual objects in the audio / video feed to specific implicit state dimensions through matrix multiplication and the Softmax function, generating a joint intent feature representation that integrates spatiotemporal semantic elements and biological implicit representations.

[0056] At the output probability decision breakpoint S305, the system feeds the joint intent feature representation into the final multilayer perceptron classification head to output the probability distribution. The system is configured to configure the classification head to output independent probability values ​​for three dimensions corresponding to the understood state, the ununderstood state, and the doubtful state. In a preferred embodiment, the probability value distribution of these three dimensions follows the following normalization formula:

[0057] The system performs probability-normalized constraint calculations:

[0058]

[0059] Here, variable p1 represents the probability of being judged as "understanding", variable p2 represents the probability of being judged as "not understanding", and variable p3 represents the probability of being judged as "questionable".

[0060] To achieve precise threshold segmentation, the system sets a rigid threshold T for determining the unknown state. a And the threshold for determining the state of doubt T b The system uses a combined detection algorithm based on combined effects to determine the state. When p2 is detected to be greater than the undetermined state determination threshold T... a At that time, it was determined that a cognitive breakpoint for the concept missing category was triggered. When p3 is greater than the threshold T for determining the state of doubt. b Furthermore, when related touch behaviors such as selection are detected synchronously within its temporal neighborhood, the system determines a cognitive breakpoint that triggers a logical contradiction. This mechanism, which uses a three-dimensional cross-validation method—such as frequently rewinding a progress bar and superimposing a visual frown or head shaking and touch swiping—avoids false triggers caused by actions like head shaking simply due to neck pain. Based on the determined cognitive breakpoint category, the system extracts the corresponding joint intent feature representation to generate a structured question intent feature message.

[0061] In the semantic slice segment extraction S105, in response to the precisely detected cognitive breakpoint, the system extracts the corresponding semantic slice segment containing rich association information from the dynamic context window based on the spatiotemporal alignment result obtained in the preceding sequence.

[0062] To avoid abrupt time truncation that could lead to incomplete sentences, the system employs intelligent backtracking technology. The system extracts audio data that is strictly synchronized with the audio and video streams, performs low-level automatic speech recognition to obtain a text timeline sequence containing start and end time tags. The system determines the exact timestamp of the cognitive breakpoint as the central anchor point, matches the target sentence whose central anchor point precisely falls within the text timeline sequence, and extracts the start and end timestamps of the target sentence.

[0063] At this node, the system performs supplementary backtracking on the text. The system invokes a natural language parser to perform dependency parsing on the target sentence, extracting the deep dependency tree structure of subject, verb, and object. The system determines whether there are pronouns or omitted components pointing to the external context of the sentence in the dependency tree structure. When pronouns or omitted components are present, the system triggers an anti-segmentation expansion mechanism. The system expands and scans backward along the timeline with a preset step size of N sentences, until the newly expanded preceding text paragraph and the pronoun in the target sentence form a logical closed loop relationship in terms of referential resolution. The system uses the final time interval after completing the expansion and closed-loop verification as the truncation boundary of the semantic slice fragment, extracting it as a whole from the audio and video data. Through this mechanism, the system ensures that the fed knowledge raw materials possess self-sufficient integrity.

[0064] In generating learning content S106, the system integrates the obtained question intent features with the extracted and purified semantic segments to construct a set of prompt word sequences with scene constraints. This sequence is then input into a pre-trained language model deployed locally or in the cloud to generate differentiated learning content that is strongly associated with the current playback scene.

[0065] The system retrieves internally preset role task instruction templates, for example, setting the role as a professional tutor, and then assembles them according to slots. The system fills the task instruction template with the specific question direction parsed from the question intent features, the underlying subtitle text content contained in the semantic slices, and the text tags contained in the audio-visual visual objects obtained through pre-analysis. Assuming the system is in a specific knowledge point slice scenario, it receives a selection formula...

[0066]

[0067] The system will extract features containing the specific variable m, along with the fragment information of that timestamp, and encapsulate them into the prompt words, adding strong constraints for formatted output to generate the final prompt word sequence.

[0068] To address the illusion problem in large models, the system incorporates an external knowledge mounting step based on retrieval enhancement during the process. The system uses the constructed prompt word sequence as the core query condition and performs entity similarity vector retrieval in an external graph database built locally or deployed on a local area network. During this process, the system utilizes a text embedding model to transform the prompt word sequence into a high-dimensional, multi-dimensional query vector V. q And calculate the multidimensional query vector V. q Embedded vectors V of each knowledge node in the graph database i The cosine similarity value S between them i The system initially selected the cosine similarity value S. i Greater than the benchmark retrieval threshold T s The system generates a preliminary set of nodes. To ensure the systematic connection of knowledge, the system performs a secondary scoring calculation on the preliminary set of nodes based on graph relationships to obtain the final comprehensive score. In a preferred embodiment, this secondary scoring logic is implemented using the following formula for the sum of linear products:

[0069] The system performs a comprehensive scoring calculation for secondary ranking:

[0070]

[0071] Among them, the variable Score final This represents the comprehensive score obtained by secondary evaluation for each candidate knowledge node. Variable S i This refers to the cosine similarity value between the node and the query vector obtained from the above steps. Variable W c This represents the connectivity weight calculated based on the topological links between the node and its adjacent entity nodes in the graph. Variables a and b represent the weight coefficients assigned to similarity and connectivity, used to adjust their proportions. After sorting, the system extracts the text descriptions of the top K knowledge nodes from high to low according to the comprehensive score, forming triplet knowledge paragraphs. The system directly concatenates these relevant entity and relation triplet knowledge paragraphs as core additional background information to the beginning of the prompt word sequence. This prompt word stream, enriched with context and enhanced with graph knowledge, passes through the inference layer of the pre-trained language model, outputting a customized answer structure group. This intermediate structured text is typically forced to output a preset hierarchy containing definition paragraphs, derivation paragraphs, and analysis paragraphs, and is parsed locally into a tree-like node structure graph-text component. Through this scene-aware generation pipeline, the system can generate targeted tutoring content for different video contexts of the same selected object.

[0072] In the adaptive feedback S107, the system executes the final output presentation. The adaptive feedback module will provide feedback on the generated differentiated learning content through the bypass interactive interface, based on the question type attribute contained in the question intent feature, either through non-interrupted playback or by dynamically adjusting the playback state intelligently.

[0073] The system has established a set of anti-interference playback logic mechanisms to deal with different types of cognitive impairment.

[0074] If the classification result indicates a lack of shallow concepts, the system will adopt minimal intervention logic. The system will maintain the normal playback of the audio and video streams without sending intervention commands. In this state, the system will only gradually display the glossary cards in the form of a sidebar through a bypass interactive interface at the edge of the screen that does not obstruct the core image.

[0075] If the judgment result escalates to a deep logical fault, the system will take proactive intervention measures. The system will send a hard pause command to the kernel of the playback process, suspending the current playback rendering thread and preventing the input of subsequent information. The system displays differentiated learning content containing a logic chain diagram in a semi-transparent floating window in the bypass interactive interface in the center of the screen. To prevent this semi-transparent floating window from obscuring the key content of the video being analyzed, the system performs spatial collision calculations before rendering. The system obtains the bounding box coordinates of the pixel positions of the visual objects in the audio and video images currently displayed on the screen, calculates the area covered by the bounding box and the two-dimensional available space distribution of the remaining blank area of ​​the screen. The system dynamically solves the anti-occlusion offset vector, constraining the display coordinate center point of the semi-transparent floating window to the largest continuous two-dimensional available space. The system uses the graphics processor to read the overall average brightness value of the background video frame, reverse calculates the background blur parameters of the semi-transparent floating window and the high contrast parameters of the foreground font, and performs interface layer rendering operations.

[0076] If the judgment falls under the category of questioning or reflection, the system will preload system-generated extended analysis content in the form of a bypass floating window or bubble, without forcibly pausing playback. The system remains hidden and waits for further explicit interaction to confirm the need for review before formally launching the full analysis interface.

[0077] At the end of the tutoring session, the system generates additional targeted practice tasks. Based on the currently focused time segment and knowledge points, the system dynamically generates multiple-choice questions that float on the side. Only after the correct option is submitted via touch does the system automatically release the pause command and smoothly resume playback at the previous speed. By employing this multi-layered intelligent feedback mechanism, the system achieves an immersive learning loop.

[0078] In another preferred variant, considering the compatibility of this application across different hardware carriers and application scenarios, this application also provides a series of alternative solutions for multimodal input signal capture and feedback output formats. For the input modality capture stage of visual and behavioral signals, in addition to the stylus touch control combined with a camera scheme described in the main embodiment, the system also supports eye-tracking sensor arrays in other embodiments. The system sets an eye-tracking time threshold such that once it detects that the eye is continuously focused on a certain point on the screen for more than 3 seconds, and the camera simultaneously captures a frowning micro-expression feature, the system also considers it as a valid spatial selection and doubt intention composite feature, thus eliminating the hardware dependence on a physical stylus. In PC applications, the system can also degenerate to support mouse word selection, while on mobile devices it supports long-pressing a hot area on the screen as an explicit input source. To obtain high-dimensional implicit physiological data, the system provides a data interface with wearable devices such as smartwatches, receiving abnormal heart rate acceleration signals or skin conductance response changes transmitted via Bluetooth as cognitive stress indicators for auxiliary judgment, or the system supports activating a pure voice wake-up path, directly using the built-in wake word to trigger background voice semantic analysis. In resource-constrained environments, the system supports downgrading to delayed or batch processing modes. During playback, the system silently records various behavioral markers. After the video stream finishes, batch processing generates a comprehensive question list and review summary covering the entire timeline. Alternatively, if the system has a built-in map of common historical difficulties extracted from bullet screen density or a large database, it can implement big data preprocessing methods. This preloads computationally intensive explanations into video memory before the timeline reaches a difficult point, enabling zero-latency page rendering upon detecting even a minor head-shaking motion. As for auxiliary measures, the system is not limited to generating new graphic cards. It can perform time-based automatic rewind actions, such as commanding the video to rewind 15 seconds and simultaneously slowing down the playback speed. The system can also directly utilize external interfaces to recommend relevant short video links from renowned instructors for supplementary learning. In the event of multiple failed intelligent interaction judgments, the system's underlying layer will activate an emergency call subsystem, interrupting the large model's response output and automatically performing a direct connection to a live teaching assistant for complex questions, ensuring that the points of failure are properly resolved. By introducing the aforementioned functional variations, the system expands its technical resilience to cope with complex interactive ecosystems and the usage habits of different groups.

[0079] Based on an inventive concept that is completely consistent with the above method, combined with Figure 4 This application also provides a logical structure for an audio-visual learning content generation system based on multimodal interactive intent perception. For example... Figure 4As shown, the system includes several interconnected virtual functional components. The window construction module 401 is configured to reside in memory for an extended period during audio / video stream playback, listening to and intercepting the underlying audio / video data streams within a preset time frame before and after the current playback time node, and using it to construct a dynamic context window driven by a dual-end queue. The signal acquisition module 402 synchronously acquires multimodal interaction signals in real time by calling the underlying driver interface. These multimodal interaction signals include explicit interaction features provided by the touchscreen and microphone, and implicit behavioral features recognized and converted by the front-facing camera matrix. The alignment detection module 403, embedded with a high-precision clock synchronizer and an instance segmentation network, is responsible for performing spatiotemporal alignment fusion operations on the multimodal interaction signals reported by the preceding modules and the structured audio / video data within the dynamic context window. It uses a spatial mask to extract the audio / video visual objects corresponding to the interaction space coordinates, and then inputs them and the extracted implicit behavioral features into a multilayer perceptron for attention-weighted joint detection operations, thereby outputting a Boolean judgment value that explicitly indicates a cognitive breakpoint and a corresponding question intent feature message with structured semantics. The slice extraction module 404 is awakened by the Boolean decision value response. It uses a speech-text axis sequence and dependency parsing algorithm to backtrack from the dynamic context window and extract a semantic slice corresponding to a semantic loop. The companion learning generation module 405 receives the aforementioned message and slice data through a shared memory area. It automatically assembles a prompt word sequence by fusing the question intent features with the semantic slice fragments using template slots. This sequence is then delivered and input into the locally loaded pre-trained language model computation graph for forward inference, thereby generating differentiated companion learning content that contains graph information and is strongly correlated with the current playback scene. The adaptive feedback module 406 parses the companion learning content, extracts the question type attribute contained in the question intent features, and drives the front-end rendering engine to dynamically adjust the playback state in a way that intelligently adapts to the playback state—either by not interrupting normal video playback or by sending an interruption suspension command to the operating system. The differentiated companion learning content is then encapsulated and presented through a floating window or other bypass interactive interface mechanism.

[0080] It should be noted that the above division of modules is solely for the convenience of logical function interpretation. At the physical level, all of the above virtual functional modules can be integrated into a single system main control chip firmware or distributed within different processing clusters of the learning machine device. Through the data packet flow and functional coordination between the various modules within the above system, the hardware implementation of the core method described in this application is effectively supported.

[0081] This application also provides a computer device, which mainly consists of two physical hardware components: a processor (the core of the computing system) and a memory responsible for storing program code and runtime data. The memory and processor are connected via a high-bandwidth communication connection through a system bus. The memory is configured to store a large number of computer program instructions that can be directly read and invoked by the processor. When the device is powered on and its main process starts, the processor begins to read and execute the stored computer program instructions. Driven by the instruction stream, the device can implement the algorithmic steps and functional logic of an audio-visual learning content generation method based on multimodal interactive intent perception.

[0082] This application also provides a non-volatile or volatile computer-readable storage medium, such as, but not limited to, flash memory chips, solid-state drives, optical discs, or even high-speed cache storage arrays configured on a cloud server side. A packaged binary computer program is inscribed within this computer-readable storage medium. When this computer program is loaded into the processor of an external computing device with a compatible architecture and scheduled for execution, it can reproduce and implement the technical solution system of the audio-visual learning content generation method based on multimodal interactive intent perception disclosed in the foregoing embodiments.

[0083] It should be noted that many concepts mentioned in the embodiments of this application should be interpreted broadly. The multimodal interaction signals are not limited to handwriting trajectories and facial expressions, but can also be eye-tracking focus coordinate signals, skin conductance response change signals, or voice wake-up sound wave signals. The generative model is not limited to a specific large-scale language model, but can also be any pre-trained inference model capable of sequence-to-sequence generation, such as convolutional neural networks, recurrent neural networks, or variants based on the Transformer architecture. The accompanying learning content includes not only image and text parsing cards, but can also be automatically extracted summary outlines, related exam question bank indexes, or redirection URIs linking to other high-quality short videos. The bypass interactive interface is not limited to screen sidebars or semi-transparent floating windows, but can also be haptic feedback on a smartwatch combined with small screen prompts, or directional voice narration played directly through bone conduction headphones. Through the substitution and combination of the above multiple technical means, the solution of this application is portable.

[0084] This application utilizes end-to-end data acquisition to include not only explicit operation commands such as handwriting and voice, but also implicit physiological representation signals such as head shaking and frowning. These signals are then incorporated into a three-dimensional multimodal spatiotemporal alignment and fusion verification framework. This allows the system to overcome decision-making illusions caused by missing information or noise interference in a single interaction dimension, achieving higher precision in breakpoint trigger capture. Supported by semantic backtracking truncation technology and a large-model external generation architecture, the system directly targets specific timelines and pixel-level image regions to generate closed-loop knowledge feeds with local scene relevance and contextual coherence. Combined with on-demand, hierarchical, non-interrupted feedback interface scheduling logic, this application achieves an evolved interactive experience and improves the efficiency of intelligent teaching aids while avoiding disruption to normal learning flow.

[0085] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating audio-visual learning content based on multimodal interactive intent perception, characterized in that, include: Cache audio and video data in the audio and video streams to build a dynamic context window; Acquire multimodal interaction signals; The multimodal interaction signals are spatiotemporally aligned with the audio and video data within the dynamic context window to extract the target interaction object; By combining the multimodal interaction signals with the target interaction object, a cognitive breakpoint and the corresponding question intent features are obtained. In response to the detection of the cognitive breakpoint, the corresponding semantic slice is extracted from the dynamic context window; Based on the question intent features and the semantic slices, a generative model is used to generate learning companion content; Output the accompanying learning content as feedback.

2. The method as described in claim 1, characterized in that, The method of caching audio and video data in the audio and video streams to construct a dynamic context window includes: During the playback of the audio and video stream, audio and video data within a preset time period before and after the current playback time node are monitored and cached to construct the dynamic context window; The multimodal interaction signals include explicit interaction features and implicit behavioral features.

3. The method as described in claim 1, characterized in that, The extraction of the target interaction object includes: Extract the audio and video visual objects corresponding to the interaction space coordinates as the target interaction object; The step of combining the multimodal interaction signals with the target interaction object to determine and obtain cognitive breakpoints and corresponding question intent features includes: By combining the implicit behavioral features with the visual objects in the audio and video images for weighted joint detection, the cognitive breakpoint and the corresponding questioning intent feature are determined.

4. The method as described in claim 1, characterized in that, The step of extracting the corresponding semantic slice from the dynamic context window includes: Based on the spatiotemporal alignment result, the corresponding semantic slice containing the complete statement is extracted from the dynamic context window; The step of generating learning companion content using a generative model based on the question intent features and the semantic slices includes: The question intent features are fused with the corresponding semantic slices to construct a prompt word sequence, which is then input into the pre-trained language model of the generation model to generate differentiated learning content associated with the current playback scene as the learning content.

5. The method as described in claim 1, characterized in that, The output of the accompanying learning content provides feedback, including: Based on the question type contained in the question intent features, the differentiated learning content is fed back through the bypass interactive interface in a way that is either non-interrupted or adaptively adjusted in playback state.

6. The method as described in claim 2, characterized in that, The step of monitoring and caching audio and video data within a preset time period before and after the current playback time node during the audio and video stream playback to construct the dynamic context window includes: The continuous audio and video stream is divided into multiple time segments based on a preset duration threshold; A double-ended queue data structure is used to perform sliding window caching on the multiple time slices, and the audio and video data in the dynamic context window is updated.

7. The method as described in claim 6, characterized in that, After updating the audio and video data within the dynamic context window, the method further includes: Keyframes are extracted from the multiple time slices within the dynamic context window to obtain video keyframes and corresponding audio spectrum sequences. Visual feature extraction networks and audio feature extraction networks are used to map the video keyframes and the audio spectrum sequence into visual feature tensors and audio feature tensors, respectively.

8. The method as described in claim 7, characterized in that, After obtaining the visual feature tensor and the audio feature tensor, the method is configured as follows: The visual feature tensor and the audio feature tensor are concatenated along the time axis according to the corresponding timestamps to form a cross-modal context matrix; When the dimension of the cross-modal context matrix exceeds the memory allocation upper limit threshold M, calculate the content change gradient value of the audio and video frames within each time slice; The time slices whose content change gradient values ​​are lower than a preset gradient threshold T are downsampled and pooled, while maintaining the original sampling rate of the remaining time slices. They are then reassembled to form the dimensionality-reduced target context tensor sequence.

9. The method as described in claim 2, characterized in that, The explicit interactive features include touch trajectory coordinates and voice command audio segments; The implicit behavioral features include facial expression parameters and head posture angles.

10. The method as described in claim 9, characterized in that, The synchronous acquisition of multimodal interaction signals includes: Establish a globally unified high-precision timestamp clock; The touch trajectory coordinates and the audio segment of the voice command are obtained at a first sampling frequency; The facial expression parameters and the head posture angle are obtained at a second sampling frequency, wherein the first sampling frequency is higher than the second sampling frequency.

11. The method as described in claim 10, characterized in that, After acquiring the facial expression parameters and the head posture angle at a second sampling frequency, the method is configured as follows: The continuously acquired facial expression parameters and head posture angles are subjected to state estimation and smoothing denoising using a filtering algorithm. Calculate the measurement residual vector at the current time t. When the L2 norm of the measurement residual vector is greater than the preset abnormal jitter threshold E, increase the system noise covariance weight coefficient corresponding to the state covariance matrix at the current time to reduce the impact of sudden displacement on the stability of the implicit behavior feature sequence. The displacement change rate of key points in the smoothed facial expression parameters is extracted as an implicit tension index, and the noise-reduced implicit feature vector is output.

12. The method as described in claim 2, characterized in that, The step of aligning the multimodal interaction signal with the audio and video data in time and space includes: Extract the timestamps of the occurrence of the multimodal interaction signals; Based on the occurrence timestamp, match the video frame at the corresponding time in the dynamic context window; The two-dimensional touch points in the explicit interactive features are mapped to the spatial coordinate system of the video frame at the corresponding time.

13. The method as described in claim 12, characterized in that, The step of extracting the audio and video visual objects corresponding to the interaction space coordinates as the target interaction object includes: Using the mapped two-dimensional touch points as the center, the region of interest is generated by expanding outwards with a preset radius. Edge detection and contour closure calculations are performed on the image pixels within the region of interest to segment independent visual instance objects containing connected components.

14. The method as described in claim 13, characterized in that, The step of performing edge detection and contour closure calculation on image pixels within the region of interest to segment independent visual instance objects containing connected components includes: The region of interest is input into a pre-trained instance segmentation network, which outputs multiple candidate object mask images and their corresponding target classification confidence scores. Calculate the intersection-union ratio (IUGR) parameter between the bounding box of each candidate object mask and the polygonal region corresponding to the actual input two-dimensional touch point; Define a weighted scoring function, and perform linear weighted fusion of the target classification confidence score and the intersection-over-union (IoU) parameter with coefficients k1 and k2. Select the candidate object mask with the largest weighted score as the final visual object of the audio and video image.

15. The method as described in claim 2, characterized in that, The step of combining the implicit behavioral features with the visual objects in the audio and video footage for weighted joint detection to determine the cognitive breakpoint and the corresponding questioning intent features includes: The implicit behavioral features are classified to obtain the implicit intent vector; Semantic parsing is performed on the visual objects of the audio and video images to obtain visual content vectors; The implicit intent vector and the visual content vector are fused at the feature level and then input into a multilayer perceptron to output the state distribution probability.

16. The method as described in claim 15, characterized in that, The step of performing feature-level fusion of the implicit intent vector and the visual content vector includes: An attention mechanism is employed, using the visual content vector as the query matrix and the implicit intent vector as the key and value matrices, to calculate the activation response value of the visual object in the audio-visual image to a specific implicit state. Generate a joint intent feature representation that integrates spatiotemporal semantics and implicit representation.

17. The method as described in claim 16, characterized in that, The input to the multilayer perceptron outputs a state distribution probability, which is used to determine the cognitive breakpoint and the corresponding questioning intent feature, including: Configure the multilayer perceptron to output probability values ​​p1, p2, and p3 for three dimensions corresponding to the understood state, the ununderstood state, and the questionable state, wherein the sum of the probability values ​​of the three dimensions is equal to 1; Set the threshold T for determining the unknown state. a And the threshold for determining the state of doubt T b ; When p2 is greater than the unknown state determination threshold T a At that time, determine the cognitive breakpoint that triggers the category of missing concept; When p3 is greater than the threshold T for determining the suspicious state b Furthermore, when a related touch behavior is detected, a cognitive breakpoint is determined to trigger a logical contradiction category. Based on the determined cognitive breakpoint categories, the joint intent feature representation is extracted to generate the corresponding structured question intent feature.

18. The method as described in claim 2, characterized in that, The spatiotemporal alignment result extracts the corresponding semantic slice fragment containing the complete sentence from the dynamic context window, including: The exact timestamp of the cognitive breakpoint is determined as the central anchor point; Based on the subtitle text content of the current playback scene, sentence boundaries are searched in the forward and backward directions using the central anchor point; The audio and video data segments containing complete sentences are extracted as the corresponding semantic slices.

19. The method as described in claim 18, characterized in that, The step of searching for sentence boundaries in the forward and backward directions based on the subtitle text content of the current playback scene, using the center anchor point, includes: Extract the audio data synchronized with the audio and video stream, and perform speech recognition to obtain a text timeline sequence containing time tags; Match the target sentence whose central anchor point falls within the text timeline sequence; Extract the start and end timestamps of the target sentence.

20. The method as described in claim 19, characterized in that, After extracting the start and end timestamps of the target sentence, the process also includes: Natural language dependency parsing was performed on the target sentence to extract the subject-verb-object dependency tree structure; Determine whether there are pronouns or omitted elements pointing to the external context of the sentence in the dependency tree structure; When the pronoun or the omitted component is present, the truncation range is expanded sequentially in the backtracking direction at a granularity of N sentences with a preset step size, until the newly expanded text paragraph forms a closed loop relationship with the target sentence, and the final time interval after expansion is used as the truncation boundary of the corresponding semantic slice.

21. The method as described in claim 2, characterized in that, The process of fusing the question intent features with the corresponding semantic segments to construct a prompt word sequence includes: Get the preset task instruction template; The question direction parsed from the question intent feature, the text content corresponding to the corresponding semantic slice fragment, and the text tags of the audio and video visual objects are filled into the task instruction template; Add formatted output constraints to generate the prompt word sequence.

22. The method as described in claim 21, characterized in that, After generating the prompt word sequence, the input is used as the pre-trained language model of the generation model to generate differentiated learning content associated with the current playback scene as the learning content, including: Using the aforementioned prompt word sequence as query conditions, a similarity vector retrieval is performed in a locally constructed external graph database; Extract the top-ranking related entity and relation triplet knowledge segments; The relevant entity and relation triplet knowledge paragraphs are appended as additional background information to the beginning of the prompt word sequence.

23. The method as described in claim 22, characterized in that, The process of performing similarity vector retrieval in a locally constructed external graph database to extract top-ranking related entities and relation triplet knowledge segments includes: The prompt word sequence is transformed into a multi-dimensional query vector V using a text embedding model. q ; Calculate the multidimensional query vector V q Embedded vectors V of each knowledge node in the graph database i The cosine similarity value S between them i ; Filter out the cosine similarity value S i Greater than the benchmark retrieval threshold T s The initial screening node set; The initial set of nodes is then scored a second time, taking into account the connectivity weights W between nodes. c Calculate the overall score, which depends on coefficients a and b, and S. i and W c The sum of linear products; The content of the top K knowledge nodes, ranked from highest to lowest according to the comprehensive score, is used to form the triplet knowledge paragraph.

24. The method as described in claim 2, characterized in that, The step of providing feedback on the differentiated learning content through a bypass interactive interface, based on the question type included in the question intent features, in a manner that allows for uninterrupted playback or adaptive adjustment of the playback state, includes: When the question type is a lack of shallow concepts, the playback state of the audio and video stream is maintained, and the differentiated learning content is gradually displayed in the form of a sidebar in the bypass interactive interface at the edge of the screen. When the question type is a deep logical gap, a pause command is sent to the playback process, suspending the current playback thread. The differentiated learning content containing the logic map is displayed in a semi-transparent floating window in the bypass interactive interface in the center of the screen. When the question type is a questioning or reflection, the differentiated learning content containing extended analysis is pre-loaded in the form of a bypass floating window or bubble, and the complete interface is opened in response to another explicit interaction. At the end of the tutoring session, a floating multiple-choice practice task is dynamically generated, and the pause command is automatically released and playback resumes in response to receiving the correct option.

25. The method as described in claim 24, characterized in that, The differentiated learning content, including a logic map, is displayed in a semi-transparent floating window in the bypass interactive interface at the center of the screen, including: Obtain the bounding box coordinates of the pixel position of the visual object of the audio and video image currently displayed on the screen; Calculate the area covered by the bounding box coordinates and the two-dimensional available space distribution of the remaining blank area; The display coordinate center point of the semi-transparent floating window is constrained to the largest continuous two-dimensional available space, and the anti-obstruction offset vector is calculated to adjust the edge position of the floating window; The background blur parameter and foreground font contrast parameter of the semi-transparent floating window are calculated based on the overall average brightness value of the background video frame, and the interface rendering operation is performed.

26. The method as described in claim 2, characterized in that, After generating the prompt word sequence, the input is used as the pre-trained language model of the generation model to generate differentiated learning content associated with the current playback scene as the learning content, including: Appendix the sequence of prompt words with reference sample data of the desired output format; The initial output of the pre-trained language model is parsed to extract the intermediate structured text containing the definition section, the derivation section, and the discrimination section; Based on the hierarchical logical structure of knowledge points, the intermediate structured text is mapped and rendered into a tree-like node structure of graphic components.

27. A system for generating audio-visual learning content based on multimodal interactive intent perception, characterized in that, include: The window building module is configured to cache audio and video data from audio and video streams to build dynamic context windows. The signal acquisition module is configured to acquire multimodal interactive signals; The alignment detection module is configured to perform spatiotemporal alignment of the multimodal interaction signals with the audio and video data in the dynamic context window, extract the target interaction object, and combine the multimodal interaction signals with the target interaction object to make a judgment, thereby obtaining cognitive breakpoints and corresponding question intent features. The slice extraction module is configured to extract the corresponding semantic slice fragment from the dynamic context window in response to the detection of the cognitive breakpoint. The learning companion generation module is configured to generate learning companion content using a generative model based on the question intent features and the semantic slice fragments. The adaptive feedback module is configured to output the learning support content as feedback.

28. A computer device, characterized in that, Includes a processor and a memory communicatively connected to the processor; The memory stores a computer program that can be executed by the processor, which, when executing the computer program, implements the method as described in any one of claims 1 to 23.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 23.

Images