Multi-modal data real-time analysis and feedback method and system

Abstract

The invention provides a multi-modal data real-time analysis and feedback method and system, and the method comprises the steps: collecting a video frame and an audio frame, reading a count value of the same monotonic timer when the collection of the video frame and the audio frame is completed, and generating an audio and video sequence; calculating the behavior popularity of each region in each time slice based on the sequence, and generating a region set for multi-modal event analysis in combination with a preset threshold and a quantity upper limit; scheduling the corresponding video sub-blocks to a visual computing power unit for target positioning and action classification, and executing voice activity detection and keyword category judgment on the time-aligned audio clips at the same time; visual and audio results are fused, and a structured event sequence is generated according to time slice and region compression; and constructing a classroom teaching chain through the sequence, identifying a key event, and finally generating a teaching event description. According to the invention, under the condition of domestic chip combination, cost, time delay, multi-modal consistency and data security controllability are considered, and real-time perception and feedback of teaching behaviors are effectively supported.

Description

Technical Field

[0001] This invention belongs to the field of data processing, and in particular relates to a method and system for real-time analysis and feedback of multimodal data. Background Technology

[0002] With the advancement of intelligent education and information-based teaching, the types and scale of data available in classroom scenarios are rapidly increasing. Typical examples include multiple high-definition or ultra-high-definition video streams, multi-channel audio streams, and related teaching process information. Analyzing this data during the classroom process provides a basis for controlling classroom pace, reviewing interactions, and evaluating the teaching process, thus creating a real demand for edge-side real-time analysis systems. However, current common solutions often employ a centralized architecture based on general-purpose servers and foreign GPUs, uploading audio and video to the cloud or a central node for unified decoding and processing. This results in high deployment costs and power consumption, making large-scale deployment in primary and secondary education scenarios difficult. Furthermore, due to network limitations and centralized computing queuing, the analysis process often suffers from significant latency, hindering timely output during the classroom. More importantly, multimodal analysis is typically performed by different hardware units or software modules, lacking a unified time benchmark and collaborative scheduling mechanism. This can easily lead to inaccurate alignment between video behavior and audio semantics at the same moment, resulting in inconsistent event descriptions and difficulties in cross-modal fusion. On the other hand, classroom data is sensitive, and solutions that rely on external cloud platforms and foreign chips raise concerns about data security and supply chain controllability, making it difficult to meet the requirements of domestic production and security and reliability in the context of information technology innovation. Summary of the Invention

[0003] The purpose of this invention is to design a method and system for real-time analysis and feedback of multimodal data, which completes the entire process from data acquisition synchronization, computing power scheduling, event structuring to teaching chain modeling and result presentation on the edge, so as to adapt to the requirements of classroom scenarios for real-time performance and stability.

[0004] To achieve the above objectives, a method for real-time analysis and feedback of multimodal data is provided in a first aspect of the present invention, the method comprising: Capture video and audio frames; When the video frame and the audio frame are acquired, the count value of the same monotonic timer is read to generate a video frame sequence and an audio frame sequence with a unified time stamp. Based on the video frame sequence and the audio frame sequence, the behavioral heat of each region in each time slice is calculated, and a region set for multimodal event analysis is generated according to a preset threshold and a maximum number limit. Based on the set of regions, the video sub-blocks of the corresponding regions are delivered to the visual computing unit to perform target localization and action classification. At the same time, speech activity detection and keyword category determination are performed on the audio segments of the aligned time slices to obtain visual and audio results. The visual and audio results are compressed by time slice and region to generate structured event entries with time stamps, region numbers, event types and comprehensive confidence levels, and then aggregated into a structured event sequence. The structured event sequence is input into the semantic processing unit, and through event embedding, temporal coding and relation reading processing, a classroom teaching chain is constructed and key events are identified; Based on the aforementioned classroom teaching chain and key events, a teaching event description is generated and output locally in a visual format.

[0005] Furthermore, the video frames are received via the MIPI-CSI interface, and the audio frames are received via the I2S interface.

[0006] Furthermore, when calculating the behavioral heat, a regional weight corresponding to the classroom seating area and the camera viewpoint is introduced. The regional weight is pre-configured and fixed in the configuration file during the system deployment phase.

[0007] Furthermore, when the action classification performs cropping and resampling of candidate boxes, it expands the original area by a fixed boundary before inputting it into the classifier to preserve local contextual information.

[0008] Furthermore, the keyword list used for keyword category determination is pre-set according to teaching scenarios during the deployment phase, including question-type, roll call-type, and discussion-type keywords.

[0009] Furthermore, during the event compression process, the event confidence is calculated based on the visual action confidence, audio keyword confidence, and regional active background intensity, and suppression is applied in the low-activity background.

[0010] Furthermore, the semantic processing unit processes the structured event sequence using a sliding time window. When the time interval between events within the window exceeds a preset expected time interval, a time interval penalty is applied to the chain consistency score.

[0011] Furthermore, the teaching event descriptions are generated using predefined templates, with the template type determined based on the event combination pattern and key event types of the classroom teaching chain.

[0012] Furthermore, the visualization output is completed locally, displaying teaching events in the form of a timeline. The significance of the events is reflected by the color or brightness of the color blocks, without relying on external servers or network communication.

[0013] A second aspect of the present invention provides a multimodal data real-time analysis and feedback system, the system comprising: The time synchronization acquisition module is used to acquire video frames and audio frames; when the video frames and audio frames are acquired, the module reads the count value of the same monotonic timer and generates a video frame sequence and an audio frame sequence with a unified time stamp. The regional heat screening module is used to calculate the behavioral heat of each region in each time slice based on the video frame sequence and the audio frame sequence, and generate a set of regions for multimodal event analysis according to a preset threshold and a maximum number of regions. The multimodal collaborative analysis module is used to deliver video sub-blocks of the corresponding regions to the visual computing unit for target localization and action classification based on the region set, and simultaneously perform speech activity detection and keyword category determination on the audio segments of the aligned time slices to obtain visual and audio results. The structured event generation module is used to compress the visual results and the audio results by time slice and region, generate structured event entries with time stamps, region numbers, event types and comprehensive confidence levels, and aggregate them into a structured event sequence. The teaching semantic understanding module is used to input the structured event sequence into the semantic processing unit, and construct a classroom teaching chain and identify key events through event embedding, temporal encoding and relation reading processing; based on the classroom teaching chain and the key events, it generates teaching event descriptions and performs local visualization output.

[0014] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides a method and system for real-time analysis and feedback of multimodal data. It uses a unified time reference as the foundation for multimodal collaboration, establishing a consistent time reference for data from video and audio channels at the data acquisition source, thus providing usable temporal anchors for subsequent cross-modal fusion. Furthermore, considering the naturally sparse spatiotemporal distribution of classroom behavior, a dynamic computing power allocation mechanism oriented towards regions and time slices is introduced. This concentrates high-performance inference resources on regions and time periods with higher interaction probabilities, avoiding the real-time performance degradation caused by high-load inference across the entire screen and all time periods. Further, heterogeneous inference outputs are uniformly compressed into structured event representations with temporal and regional contexts, eliminating fusion difficulties caused by inconsistencies in output formats from different chips and modalities. The chain structure and key nodes of the teaching process are recovered based on event sequences on the semantic chip, enabling the system to transition from discrete events to a structured expression at the teaching process level. Finally, the chain results are organized into an event-based presentation format suitable for classroom use, achieving edge-side output without relying on the cloud. Through the above design, this invention takes into account cost, latency, multimodal consistency and data security control under the condition of domestic chip combination, and makes the system output closer to the actual organization of classroom teaching process, thereby overcoming the key problem that existing solutions are difficult to stably achieve high-quality classroom process understanding and real-time presentation at the edge. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a flowchart of a multimodal data real-time analysis and feedback method according to the present invention.

[0017] Figure 2 This is a framework diagram of a multimodal data real-time analysis and feedback system according to the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] In one or more embodiments, such as Figure 1 As shown, a method for real-time analysis and feedback of multimodal data is disclosed, the method comprising the following: S1: Acquire video frames and audio frames; when the video frames and audio frames are acquired, read the count value of the same monotonic timer to generate a video frame sequence and an audio frame sequence with a unified time stamp; Specifically, in this step, the system addresses the issue of inconsistencies in the temporal dimension of multimodal data in classroom scenarios by constructing a unified time reference mechanism completed at the data acquisition source. The classroom environment contains both video and audio acquisition devices, which differ in their driving paths, buffering strategies, and data generation rhythms. If alignment is based solely on data arrival order or user-state time, modal misalignment can easily occur, affecting subsequent judgments of classroom behavior and teaching status. Therefore, the core objective of this step is to establish a consistent and stable time reference for all modal data through driver-level operations on a domestically produced main control chip, ensuring that subsequent analysis is always based on the same timeline. The input for this step comes directly from the physical acquisition devices within the classroom. Video data is acquired by a fixed ultra-wide-angle camera in the classroom and connected to the image processing channel of the domestically produced main control chip via a MIPI-CSI interface; audio data is acquired by a microphone array, converted from analog to digital, and input to the audio acquisition channel in I2S format. The two types of data are managed by the video acquisition driver and audio acquisition driver in the kernel, respectively, and an interrupt is triggered when the data buffer is filled. In the specific implementation, when a frame of video data is written to the DMA buffer, the interrupt service routine of the video acquisition driver immediately reads the high-precision monotonic timer inside the main control chip, obtains the current count value, and uses it as the time stamp for that frame. Similarly, when audio data completes filling a fixed-length buffer block, the current value of the same timer is read in the interrupt handling and written to the metadata area of ​​the audio data. Since both video and audio use the same hardware timing source, this time stamp has natural consistency across different modalities.

[0020] Each piece of collected data is encapsulated into a unified data structure: ; in, This indicates a single-frame data packet. This represents the corresponding original media data content; in the case of video, it is an image frame matrix, and in the case of audio, it is an audio sample array. This represents the system time value read the instant data collection is completed. This structure is only used to describe the correspondence between data and time, and does not contain any identity identifiers, account information, or characteristic fields that can be used to identify a specific natural person.

[0021] To ensure the time stability of the acquisition link during operation, the system performs statistical analysis on the time intervals of consecutive acquisition frames. It determines the presence of abnormal jitter by calculating the deviation between adjacent time markers. The calculation method is as follows: ; in, and Indicates the time stamp of two adjacent frames of data. This represents the expected frame interval for the corresponding mode under normal operating conditions. This indicates the number of samples within the statistical window. This calculation result is used only internally to determine whether the data collection rhythm is stable and does not participate in any individual behavior analysis. Regarding privacy protection, this step clearly defines the scope of data use and processing methods. All video and audio frames, after being time-stamped and entering subsequent processing, are only briefly cached in a circular buffer for real-time analysis; they are not stored long-term, nor are continuous index relationships established across time periods. (The encapsulation structure...) This only represents the perceived signal itself and does not contain name, student ID, biometric template, or other information that can be used for identification; time stamp. This data is used solely for multimodal alignment and event timing determination, and not for constructing individual behavioral trajectories or identity association. All subsequent processing by the system focuses on the overall classroom state or regional behavioral characteristics, rather than identifying, authenticating, or continuously tracking specific individuals, thus strictly limiting the data processing purpose to the technical scope of classroom teaching state analysis. This step ultimately outputs two types of data streams with a unified time reference: one is a time-stamped video frame sequence. Another type is audio frame sequences with time stamps. .

[0022] S2: Based on the video frame sequence and the audio frame sequence, calculate the behavioral heat of each region in each time slice, and generate a region set for multimodal event analysis according to a preset threshold and a maximum number of regions. Specifically, this step addresses a key fact of classroom scenarios: classroom behavior is naturally sparse in both space and time, and the areas and time periods truly worthy of high-performance, high-computing-power analysis often constitute only a small proportion. The bottleneck of domestically produced heterogeneous edge computing platforms lies precisely in their inability to treat every frame and every region as equally important. Therefore, this step does not involve re-synchronizing or preprocessing data, but rather strictly adheres to the unified time benchmark established in Step One, integrating data from the same timeline... and This is transformed into a scheduling basis for "where and when high-precision analysis is needed," and outputs a scheduling set that can directly drive the subsequent chip division of labor. This allows the subsequent step three to only perform structured event extraction on the selected area, thereby achieving an engineering-applicable balance between classroom real-time requirements and computing power constraints.

[0023] The input for this step comes entirely from the output of step one: a sequence of video frames with uniform time stamps. With audio frame sequence In terms of implementation, Corresponding to the continuous output sequence of data packets from the video capture driver, each data packet contains one frame of image content and a time stamp associated with it; Corresponding to the audio segment sequence output by the audio acquisition driver, each segment also carries a time stamp that can be aligned with the video timeline. Since both have already been annotated using the same hardware timing source in step one, this step can directly retrieve video frames and audio segments at the "same moment" according to the time stamp alignment, without needing to guess the correspondence through queue order.

[0024] In practice, the system executes two lightweight computing paths on the domestically produced main control chip and outputs region-level behavioral intensity at fixed time slices. The first path targets... To avoid the computational burden of full-resolution calculations, the system performs only rapid downsampling and gridding (e.g., dividing the image into several fixed grid regions, each retaining a sparsely sampled set of pixels) on the main control end for each video frame. Then, it calculates a statistical measure of pixel variation within the same region at adjacent time points to approximate whether significant motion or pose changes exist in that region. This statistical measure is normalized and used as... ,in This is the grid area number. The second path targets... The system calculates the overall amplitude change statistics for audio segments aligned with that time point, and then normalizes them to obtain... Its purpose is to quickly determine whether there are any speaking, discussion, or sudden sound events within the current time slice. Neither path introduces additional data categories, nor does it rely on online learning or parameter self-updating, ensuring long-term stable operation at the edge.

[0025] Another common but easily overlooked engineering problem in classroom scenarios is that students in the back rows occupy a smaller portion of the frame and their behavioral signals are weaker. Relying solely on frame differences or audio energy can easily result in "back row actions being drowned out," failing to meet the need for focusing on the back row area in classroom analysis. Therefore, this step introduces a regional weight related to the classroom layout when merging heat signatures. This data is not additionally collected; rather, it's pre-generated lookup table parameters based on camera installation positions and seating area divisions during deployment. Areas closer to the far end of the frame and more easily weakened correspond to higher weights, allowing the system to remain sensitive to key areas even with low-cost computation. Simultaneously, to avoid frequent fluctuations in the scheduling set due to occasional classroom noise, this step uses a smooth update for the heat index, making scheduling decisions more "continuous" and better reflecting the actual rhythm of classroom behavior changes. Considering all these factors, this step calculates behavioral heat index for each time slice and region: ; in, For time slices ,area The popularity of the behavior; The heat value of the same area in the previous time slice is stored in memory by the system on the main control terminal and updated over time. It is a smoothing coefficient used to suppress instantaneous fluctuations and enhance the temporal continuity of heat. For area-weighted lookup items, the data is pre-calculated and embedded into the configuration file based on classroom seating areas and camera angles during the deployment phase. At runtime, only the following steps are required: Index reading; From the Adjacent frames in the region The pixel changes within the range are statistically analyzed and normalized to obtain the result; From the In time slice The amplitude variations of aligned audio segments are statistically analyzed and normalized. and To fix the fusion weights and balance the contributions of visual and audio activities in the classroom, the configuration is determined empirically during deployment. All the above items have been normalized before fusion to avoid bias caused by directly adding statistics from different sources.

[0026] In obtaining Then, the system needs to generate a scheduling set that can directly drive heterogeneous computing power. In terms of engineering implementation, The goal is to ensure that truly high-value regions are sent to high-performance vision units while avoiding congestion in the downstream queue caused by selecting too many regions at once. Therefore, the system employs a combined strategy of "threshold gating + quantity limit": first, regions with popularity exceeding the threshold are selected, and then, when there are too many candidates, only the most popular regions are retained. This selection process can be expressed in the following form: ; in, For time slices The set of regions that need to enter the high-computing-power analysis link; A fixed threshold is used to eliminate low-value areas; This is an upper limit parameter used to control the number of regions that simultaneously enter deep analysis within a single time slice, thus preventing sudden congestion in downstream high-performance units; Indicates selecting from the candidate set by Sort by high to low and take the first one Each region. In actual implementation, These are directly converted into region cropping instructions for video frames and task delivery instructions for heterogeneous chips: [belonging to] The region, after coordinate clipping and data encapsulation at the main control terminal, is sent to the visual computing unit for high-precision analysis, and does not enter... By keeping certain areas low-frequency or not analyzing them, the computing budget can be concentrated on the local areas in the classroom where key behaviors are most likely to occur.

[0027] S3: Based on the set of regions, the video sub-blocks of the corresponding regions are delivered to the visual computing unit to perform target localization and action classification. At the same time, voice activity detection and keyword category determination are performed on the audio segments of the aligned time slices to obtain visual and audio results. Specifically, this step applies the "time-consistent data stream" and "executable schedule set" from the first two steps to real heterogeneous inference execution, producing a structured event sequence that can be directly consumed by subsequent semantic modeling. The classroom scenario is unique in that: on the one hand, multiple behaviors may occur simultaneously in the frame (e.g., a hand is raised in one area while discussion takes place in another); on the other hand, the outputs of heterogeneous chips are inherently different (visual side: target boxes and action categories; audio side: speech activity and keyword categories). If unified encapsulation is not achieved at this layer, subsequent chain modeling will be forced to process a large number of inconsistent "fragmented results." Therefore, this step uses the data from step one... As the data entry point, step two To address the constraints of "only processing where and how to filter," task orchestration and event compression are performed at the main control end, unifying heterogeneous outputs into time-slice-based organization. The input for this step comes from two parts: one is the output of step one. and Each of their data packets carries a timestamp from the same timeline, allowing them to be processed by time slice at the master control end. The first step is to accurately retrieve the corresponding video frames and audio segments; the second step is the output of step two. and ,in Give in time slice The set of regions that need to be entered into high-precision analysis The intensity of the active background in these regions in the time slice is given, which is used to distinguish "the same action / speech" as "occurring in a high-interaction background" or "occurring in a low-interaction background" in the future, so as to avoid the distortion of events in the classroom context.

[0028] The main control chip in each time slice Trigger a task orchestration: First press Retrieve the region index ,from The image sub-blocks of the region are cropped from the middle, and the sub-blocks are combined with time slice markers. and area code The data is packaged into a visual task descriptor and delivered to the visual computing unit for inference. The visual task is executed in a fixed pipeline on a high-performance vision chip: The first segment is the target localization network, which takes a region sub-block as input and outputs several candidate bounding boxes and their corresponding confidence levels. This network uses a convolutional backbone and a multi-scale detector head. The inference results are post-processed to perform non-maximum suppression and obtain a small number of stable candidate bounding boxes. The second segment is the action / pose classification network, which crops and resamples each candidate bounding box before inputting it into the classifier. The classifier consists of several depthwise separable convolutional layers and global pooling layers, and outputs the confidence distribution of action categories required for the classroom (e.g., raising a hand, standing up, bowing, turning the head, and continuously gazing at the podium). In classroom deployment, to ensure that the back row area does not lose details due to the small scale of the subject, cropping and resampling expands the candidate bounding box around the original area by a fixed boundary before feeding it into the classifier. This incorporates the local context into the action determination, reducing the probability of "small target actions in the back row" being misclassified as static.

[0029] In parallel with the vision task, the main control chip processes the same time slice. Aligned The lightweight audio path is implemented as follows: The first step is speech activity detection, which uses a joint criterion of short-window energy and zero-crossing rate to output the confidence of "whether a speech segment exists". When the speech activity confidence exceeds a threshold, the second step is to perform keyword category determination. The keyword table is pre-set according to the teaching scenario during the deployment phase (e.g., question, roll call, discussion). This is achieved by extracting Mel features from the audio segment and inputting them into a small one-dimensional convolutional network, which outputs the confidence of each keyword category. No long text or individual identity identifiers are generated on the audio side; only a structured result of "speech activity / keyword category" is generated.

[0030] Subsequently, the main control chip processed the same time slice. , the same area The core of event compression for visual and audio results is merging "multi-frame jitter" and "multi-source fragmentation" into a single stable event. Compression is constrained by two types of consistency: temporal consistency (if similar results occur repeatedly within adjacent time slices, they are merged into a continuous event) and contextual consistency (event confidence needs to be more cautious in low-activity contexts, while greater sensitivity is allowed in high-activity contexts). Therefore, for each time slice... ,area Construct event entries Its event type The event confidence is obtained by combining visual action categories and audio keyword categories according to preset combination rules (e.g., "raising hand + question-type keywords" are combined into "question action event"). Calculate as follows: ; in, Comprehensive confidence level for event entries; A compression function of fixed shape, used to map linear synthesis results to a stable confidence scale; The action classification network output from the visual computing unit is taken as follows: Confidence value for the corresponding category; The keyword category determination or speech activity detection output from the audio side is taken and... If the matching category confidence is not triggered in the time slice, then the voice activity confidence is used instead. The regional active background intensity obtained in step two has been normalized and can be directly used in the calculation. A fixed scaling factor is used to coordinate the synthesis amplitude of visual and audio confidence; , which is the background suppression coefficient, is used to apply stronger suppression to event confidence in a low-activity background, reducing the miscompression of incidental noise into events in “overall silent” classroom segments.

[0031] After obtaining each candidate event entry Then, the main control chip presses... The entries are then encapsulated, and the events are aggregated into a structured event sequence by time slice. To prevent a region from being triggered by a short-term spike and then immediately disappearing, which could make it difficult for downstream chains to form a stable structure, a "threshold + continuous confirmation" approach is used during encapsulation: when a region is triggered by a spike in the current time slice... If the threshold is reached, the event is written to the event set; if the threshold is not reached but a similar event has already occurred in the previous time slice and the current time slice still maintains a certain level of confidence, then the event's persistence state is continued, and its time span is updated. This aggregation form is represented as: ; in, For time slices A structured collection of events; For time slices ,area The generated event entries must contain at least the following: , , , and the corresponding As a contextual field; This is the set of scheduling regions for step two, ensuring that only the selected regions enter the event encapsulation chain; Thresholds are set for events, determined during the deployment phase based on classroom size and expected event density, and remain stable during operation. Event sets are stored in a short-term window buffer on the main control unit for subsequent semantic chain modeling and window retrieval; no individual association indexes spanning long time periods are established.

[0032] This step outputs a structured event sequence. Each Composed of several event entries The components and entries are determined by the time-aligned input from step one, the scheduling and contextual strength constraints from step two, and the heterogeneous reasoning results from this step.

[0033] S4: Compress the visual results and audio results by time slice and region to generate structured event entries with time stamps, region numbers, event types and comprehensive confidence levels, and aggregate them into a structured event sequence; Specifically, this step transforms the structured event sequence generated in step three into a "teaching chain," enabling the system to extract the logical structure of the classroom teaching process from the temporal sequence and combination relationships of events. Step one ensures consistency of cross-modal time stamps, step two concentrates computing power on high-value areas and time periods, and step three compresses the output of heterogeneous chips into unified event entries and attaches active background intensity to the events. This step directly utilizes these existing fields to perform temporal semantic modeling of the event sequence on a domestically produced semantic chip, outputting a time-expanded chain structure and key nodes in the chain for subsequent teaching event generation and presentation.

[0034] The input for this step is the structured event sequence output from step three. . It is a collection of events organized by time slices, where each event entry can be considered as... and includes time slice markers Area code Event Type Comprehensive confidence and active background intensity These fields have already been formed and maintain a consistent meaning in step three. This step does not introduce new original data, nor does it rerun the visual or audio inference.

[0035] In actual execution, the semantic chip reads data using a sliding time window. Each window covers several consecutive time slices, and the set of events within the window is denoted as . ,in This indicates the window's position on the timeline.

[0036] Events within a window first enter the event sequence modeling network. This network is deployed on a semantic chip with a fixed structure, containing three consecutive computational segments: an event embedding segment, a temporal encoding segment, and a relation readout segment. The event embedding segment maps the discrete and continuous fields of each event into a vector: the discrete fields (event type and region number) are embedded into a dense vector through a lookup table; the continuous fields (confidence and active background strength) are first linearly mapped to the same dimension as the embedding vector before being fused with it. This approach is targeted at specific classroom scenarios: the same event type has different "teaching semantic roles" in different regions and under different active backgrounds. For example, short-duration speech events occurring in a high-activity background are more likely to be related to interaction, while similar events occurring in a low-activity background are more likely to be scattered noise. The temporal encoding segment consists of several self-attention encoding layers connected in series. Each layer calculates attention allocation based on the temporal order of events within the window, enabling the network to capture dependencies such as "which events are more likely to occur after a certain type of event." The relation readout segment reads the directed association strength between event pairs from the encoded event vector, used to describe... Initiating or Undertaking The likelihood of this association strength can be derived directly from the aggregation of attention weights, or from a lightweight mapping superimposed on the attention output and then normalized. Since the input event already carries... and When calculating attention, the encoding layer naturally favors events with high confidence and active backgrounds, thus making the chain structure closer to the actual rhythm of the classroom.

[0037] The core focus of chain modeling is to score the consistency of event relationships within a window, and then use this score for chain segment selection and anomaly localization. A key engineering constraint for classroom chains is "time interval reasonableness": for example, question and response events usually occur within a relatively close time frame and are not considered part of the same chain even if they span a long time segment. Therefore, the scoring introduces an "expected time interval constraint" while considering the strength of event pair relationships. The expected time interval is configured as a type-pair lookup table entry based on the classroom rhythm during the deployment phase, denoted as... During runtime, only table lookups are performed; no online updates are involved. Window consistency score is calculated using the following formula: ; in, Display window Chain consistency score; A collection of events within the window; and The comprehensive confidence field is taken from the event entry (corresponding to step three). ); The directed association strength obtained from the relation readout segment reflects "the degree to which the earlier event supports the chain of events that follow"; The table entries are weighted for event types to emphasize more critical combinations in the classroom (e.g., combinations of explanation-related, question-related, and response-related have a higher weight than background action combinations). Indicates an event and The time interval within the window is obtained by differentiating the time slice markers in the event entries; For each type, look up the table entry for the expected time interval; This indicates that penalties will only be applied to the portion exceeding the expected time interval; This is a penalty strength coefficient used to suppress unreasonable cross-temporal connections. The score incorporates both "relationship strength" and "classroom rhythm constraints" to prevent chains from being mistakenly linked by chance or distant events.

[0038] Get each window After that, the chain The construction is based on the principle of "continuous concatenation of highly consistent windows": when adjacent windows... While maintaining a high level, the event fragments covered by the corresponding window are spliced ​​into a chain fragment in chronological order; when When a significant drop occurs, the chain segment naturally breaks at that point, triggering the location of critical nodes. Critical node location does not rely on additional models but directly utilizes "window consistency" to attribute events: events falling within a low-consistency window and possessing high event confidence are more likely to correspond to breakpoints or abnormal trigger points in the teaching chain, and are therefore included in the critical event set. Its form is: ; in, Indicates time slice A collection of key events in the vicinity; This is the set of events for that time slice; Indicates overlay event A collection of windows This represents the aggregated consistency score results for these windows; This is the chain consistency threshold, used to distinguish between normal chain segments and abnormal segments; For event entry confidence fields; A threshold for screening critical events is used to avoid mislabeling low-confidence noise events as critical nodes. This judgment combines "window-level chain quality" and "event-level reliability," enabling relatively stable location of critical points in the classroom, such as unanswered questions, sudden interruptions in interaction, and abnormal concentration.

[0039] This step outputs two results: the classroom teaching chain. With key event set . It consists of chain segments organized in chronological order, with each segment originating from a combination of highly consistent events within a consecutive window; This is a set of key events corresponding to a break or anomaly in the chain on the timeline; its elements come directly from... Event entries.

[0040] S5: Input the structured event sequence into the semantic processing unit, and construct a classroom teaching chain and identify key events through event embedding, temporal encoding and relation reading processing; based on the classroom teaching chain and the key events, generate teaching event descriptions and perform local visualization output; Specifically, after step four, the system has obtained the teaching semantic chain on the domestically produced semantic chip. and the corresponding set of key teaching events These results are highly structured internally, but if they remain in the form of chains or lists of nodes, teachers will find it nearly impossible to understand and use them in real-time in the classroom. Therefore, the goal of this step is not to introduce new judgment logic, but rather to address the classroom usage scenario. and The information is reorganized into a clear expression of "what time period, what area, and what type of teaching event occurred," and then visualized locally through the main control chip, ensuring that teachers can perceive key information without interrupting the teaching rhythm.

[0041] In its implementation, the main control chip first uses the time axis as the main thread to... The process involves windowing. Here, "window" doesn't refer to simple fixed-length divisions, but rather to grouping segments based on their natural boundaries: when multiple segments are temporally consecutive and semantically consistent, they are merged into a single teaching time slot; when a segment breaks or exhibits a significant anomaly at a certain point, that point is used as the dividing point for the time slot. This is because real classroom teaching activities often use "a knowledge point" or "an interaction" as natural units, rather than strict time slices, directly using... The chain structure can better reflect the actual teaching pace.

[0042] After obtaining each teaching time segment, the system needs to determine whether that time segment is worth presenting to the teacher. To this end, the main control chip, combining the chain consistency results and key event distribution already calculated and cached in step four, calculates the salience of teaching events for each time segment. Salience reflects both the continuity of the teaching process within that time segment and whether key nodes are concentrated within it; its calculation form is as follows: ; in, Indicates teaching time period The significance score; This represents the aggregated result of the consistency score of the corresponding chain within this time period; Indicates falling within this time period A subset of key events; This indicates the number of chain segments contained within that time period; The weights configured for the deployment phase are used to balance "overall teaching coherence" and "abnormal event density". For example, if the overall consistency of the chain is high and there are almost no key events within a certain period of time, the salience score of that segment will be low, and the system believes that there is no need to disturb the teacher; conversely, if the consistency of the chain drops significantly and key events occur in a concentrated manner, the score of that segment will increase significantly and it will become the priority for presentation.

[0043] After completing the saliency calculation, the system generates teaching event descriptions based on the structural information of each time period. The generation process uses predefined teaching event templates, rather than complex natural language generation models. The template selection is directly based on... Chain type and The template defines key event types, such as: when the chain is "question - no response" and this time period is highly significant, the template corresponds to "interaction interrupted"; when the chain is "question - response - continued explanation" and key events are scarce, the template corresponds to "effective interaction formed". The generated teaching events are uniformly encapsulated into structured entries and written into the teaching event output set. Each entry must include at least the time range, the corresponding region, and an event type identifier.

[0044] To facilitate teachers' quick understanding, the main control chip will be located locally. This is mapped to visual interface elements. The interface is based on a horizontal timeline, with class time unfolding from left to right; each teaching event is marked with a color block or icon on its corresponding time interval, with color or brightness varying accordingly. A positive correlation is established, making significant events visually stand out naturally. For example, in a lesson, if there are consecutive "no response to questions" events in the back row within a five-minute period, that time period will be marked with a prominent color on the timeline; while during periods of smooth teaching flow, it will only be displayed as background color, without interfering with the teacher's attention. The entire rendering process is completed locally on the main control chip, without relying on external servers or network communication.

[0045] In one or more embodiments, such as Figure 2 As shown, a multimodal data real-time analysis and feedback system is disclosed, the system comprising: The time synchronization acquisition module is used to acquire video frames and audio frames; when the video frames and audio frames are acquired, the module reads the count value of the same monotonic timer and generates a video frame sequence and an audio frame sequence with a unified time stamp. The regional heat screening module is used to calculate the behavioral heat of each region in each time slice based on the video frame sequence and the audio frame sequence, and generate a set of regions for multimodal event analysis according to a preset threshold and a maximum number of regions. The multimodal collaborative analysis module is used to deliver video sub-blocks of the corresponding regions to the visual computing unit for target localization and action classification based on the region set, and simultaneously perform speech activity detection and keyword category determination on the audio segments of the aligned time slices to obtain visual and audio results. The structured event generation module is used to compress the visual results and the audio results by time slice and region, generate structured event entries with time stamps, region numbers, event types and comprehensive confidence levels, and aggregate them into a structured event sequence. The teaching semantic understanding module is used to input the structured event sequence into the semantic processing unit, and construct a classroom teaching chain and identify key events through event embedding, temporal encoding and relation reading processing; based on the classroom teaching chain and the key events, it generates teaching event descriptions and performs local visualization output.

[0046] It is worth noting that the specific workflow of the multimodal data real-time analysis and feedback system provided in this embodiment of the invention is the same as that of the multimodal data real-time analysis and feedback method described in the above embodiment, and will not be repeated here.

[0047] This invention also provides a multimodal data real-time analysis and feedback device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of a multimodal data real-time analysis and feedback method, for example... Figure 1 The steps S1 to S5 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0048] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the multimodal data real-time analysis and feedback device.

[0049] The aforementioned multimodal data real-time analysis and feedback device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the device may also include input / output devices, network access devices, buses, etc.

[0050] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASACs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the multimodal data real-time analysis and feedback device, connecting all parts of the device via various interfaces and lines.

[0051] The memory can be used to store the computer program and / or modules. The processor realizes various functions of the multimodal data real-time analysis and feedback device by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the operation of the air conditioner controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0052] The module integrated into the multimodal data real-time analysis and feedback device, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0053] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0054] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for real-time analysis and feedback of multimodal data, characterized in that, The method includes: Capture video and audio frames; When the video frame and the audio frame are acquired, the count value of the same monotonic timer is read to generate a video frame sequence and an audio frame sequence with a unified time stamp. Based on the video frame sequence and the audio frame sequence, the behavioral heat of each region in each time slice is calculated, and a region set for multimodal event analysis is generated according to a preset threshold and a maximum number limit. Based on the set of regions, the video sub-blocks of the corresponding regions are delivered to the visual computing unit to perform target localization and action classification. At the same time, speech activity detection and keyword category determination are performed on the audio segments of the aligned time slices to obtain visual and audio results. The visual and audio results are compressed by time slice and region to generate structured event entries with time stamps, region numbers, event types and comprehensive confidence levels, and then aggregated into a structured event sequence. The structured event sequence is input into the semantic processing unit, and through event embedding, temporal coding and relation reading processing, a classroom teaching chain is constructed and key events are identified; Based on the aforementioned classroom teaching chain and key events, a teaching event description is generated and output locally in a visual format.

2. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, The video frames are received via the MIPI-CSI interface, and the audio frames are received via the I2S interface.

3. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, When calculating the heat of the behavior, a regional weight corresponding to the classroom seating area and the camera viewpoint is introduced. The regional weight is pre-configured and fixed in the configuration file during the system deployment phase.

4. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, When the action classification performs cropping and resampling of candidate boxes, it expands the original area by a fixed boundary before inputting it into the classifier to preserve local contextual information.

5. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, The keyword list used for keyword category determination is pre-set according to teaching scenarios during the deployment phase, including question-type, roll call-type, and discussion-type keywords.

6. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, During the event compression process, the event confidence is calculated by weighting the visual action confidence, audio keyword confidence, and regional active background intensity, and suppression is applied in the low-activity background.

7. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, The semantic processing unit processes the structured event sequence using a sliding time window. When the time interval between events within the window exceeds a preset expected time interval, a time interval penalty is applied to the chain consistency score.

8. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, The descriptions of the teaching events are generated using predefined templates, and the template type is determined based on the event combination pattern and key event types of the classroom teaching chain.

9. The method for real-time analysis and feedback of multimodal data according to claim 1, characterized in that, The visualization output is completed locally, displaying teaching events in the form of a timeline. The significance of the events is reflected by the color or brightness of the color blocks, without relying on external servers or network communication.

10. A multimodal data real-time analysis and feedback system, characterized in that, The system includes: The time synchronization acquisition module is used to acquire video frames and audio frames; when the video frames and audio frames are acquired, the module reads the count value of the same monotonic timer and generates a video frame sequence and an audio frame sequence with a unified time stamp. The regional heat screening module is used to calculate the behavioral heat of each region in each time slice based on the video frame sequence and the audio frame sequence, and generate a set of regions for multimodal event analysis according to a preset threshold and a maximum number of regions. The multimodal collaborative analysis module is used to deliver video sub-blocks of the corresponding regions to the visual computing unit for target localization and action classification based on the region set, and simultaneously perform speech activity detection and keyword category determination on the audio segments of the aligned time slices to obtain visual and audio results. The structured event generation module is used to compress the visual results and the audio results by time slice and region, generate structured event entries with time stamps, region numbers, event types and comprehensive confidence levels, and aggregate them into a structured event sequence. The teaching semantic understanding module is used to input the structured event sequence into the semantic processing unit, and construct a classroom teaching chain and identify key events through event embedding, temporal encoding and relation reading processing; based on the classroom teaching chain and the key events, it generates teaching event descriptions and performs local visualization output.

Images