A data fusion method for dynamic missing of multi-source time series data

Abstract

The application discloses a data fusion method for dynamic missing of multi-source time sequence data, comprising the following steps: acquiring multi-source time sequence original data; adding observation mark processing to construct a time sequence data sequence; loading an analysis model to input the time sequence data sequence into the analysis model; converting the time sequence data sequence into vector features and mapping to a unified feature space; performing time sequence state modeling in the unified feature space to acquire time sequence representation corresponding to the vector features; adopting state evolution control when performing the time sequence state modeling; capturing time steps of data missing or unavailability in the state evolution control process, and not triggering hidden state updating of placeholder information corresponding to the time steps; fusing the time sequence representation to generate a fusion representation; and acquiring an analysis result through the analysis model. According to the above technical scheme, when dynamic missing exists in multi-source time sequence data, the interference of unreliable data sources on behavior analysis results can be reduced, and the robustness of multi-source data management and behavior analysis is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence time series prediction and data fusion, and more specifically, to a data fusion method for dynamically missing multi-source time series data. Background Technology

[0002] In smart education and digital teaching scenarios, the interactive teaching environment of "subject-platform-resource" is gradually becoming an important carrier for organizing teaching activities, recording learning processes, and evaluating teaching quality. This teaching environment continuously generates multi-source heterogeneous teaching data, including text, audio, visual data, and platform behavior logs. Legally acquiring this multi-source heterogeneous teaching data, with consent, and then managing, integrating, and analyzing its behavior, can identify resource utilization and interaction quality, providing a basis for personalized teaching and optimization of teaching resources.

[0003] Existing methods for fusing and analyzing heterogeneous teaching data from multiple sources typically assume that text, audio, and visual modalities are all complete and available, or primarily focus on feature recovery, representation alignment, or text-dominated compensation under conditions of modality loss. These approaches generally first extract features from each modality separately, then obtain a fused representation through attention mechanisms, tensor fusion, gated fusion, cross-modal alignment, or text-based compensation networks. Finally, a classification or regression head outputs the sentiment category or continuous sentiment intensity. In real-world applications, factors such as dropped video frames, camera occlusion, audio noise, speech recognition errors, transcription errors, and short-term sensor anomalies can all cause modality observations to exhibit dynamic missing features. Dynamic missing refers to the fact that missing locations may be distributed across different samples, different modalities, and different time steps; valid observations and missing placeholders may intersperse within the same modality sequence, rather than the entire modality being completely missing.

[0004] Therefore, current technologies for behavioral analysis based on multi-source heterogeneous teaching data have the following shortcomings: 1) Dynamic missing data is usually simplified to whole-mode missing data, fixed patterns, or uniform noise, making it difficult to characterize the fine-grained state of effective observations and interleaved missing positions within the same sequence; at the same time, the missing disturbance mechanisms of different modes (such as continuous signals and discrete text) are different, and using a uniform processing method weakens the specificity of missing data modeling; 2) If the zero vector, unknown marker, and other occupant information of the missing position are directly input into the time series model, unreliable information will participate in the hidden state update, destroying the continuity of the time series; in addition, the features of explicit reconstruction or completion are not necessarily equivalent to the true observations, if 3) Directly treating these as reliable evidence for fusion may introduce uncertain pseudo-semantics and lead to incorrect predictions; 4) If time observation masks, missing density, and modality-specific perturbation cues are not fully utilized to explicitly estimate the reliability of each modality, it is difficult to dynamically suppress the negative impact of highly missing or highly perturbation modalities during the fusion stage, resulting in the fusion results being interfered with by unreliable data; 5) In interactive teaching environments, multi-source data of expressive behaviors such as logs, voice, video, and resource access records are often collected abnormally, each with dynamic missing data problems; if the reliability differences of data from different sources are not modeled, and they are directly spliced ​​or uniformly processed, abnormal data and unreliable observations are likely to interfere with the judgment of learning status.

[0005] Therefore, a technical solution is needed that does not primarily aim to explicitly complete all missing data fragments, but rather to process and integrate dynamically missing content from multi-source time-series data, and combine it with an analysis model to achieve effective behavioral analysis. Summary of the Invention

[0006] To achieve the above objectives, this application provides a data fusion method for dynamically missing multi-source time-series data, comprising the following steps: Acquire raw time-series data from multiple sources; the modalities of the raw time-series data from multiple sources include text modality, audio modality, and visual modality; The multi-source time series raw data is preprocessed and time-aligned to construct a time series data sequence. When constructing the time series data sequence, observation labeling is added to generate observation masks and data source level missing density. Load the analysis model and input the time series data sequence into the analysis model; Transform time-series data sequences into vector features and map them to a unified feature space; Temporal state modeling is performed in a unified feature space to obtain the temporal representation corresponding to the vector features. When performing temporal state modeling, state evolution control is adopted. During the state evolution control process, time steps with missing or unavailable data are captured, and the missing placeholder information corresponding to the time step does not trigger hidden state updates. The temporal representations are fused to generate a fused representation. When fusing temporal representations, based on the reliable decoupling and semantic prototype calibration of the preceding sequence, the shared representation and private cue information are used to perform modal reliability estimation and weight adjustment reliability residual fusion. The analysis results are obtained by analyzing the model.

[0007] The observation mask is used to mark whether each observation unit in the corresponding data source in the time series data has a valid observation: if a certain observation unit has valid data, its mask value is 1; if the observation unit is unavailable, its mask value is 0. Data source level missing density is used to characterize the proportion of unavailable observation units in the corresponding data source, thereby quantifying the overall missing degree of time series data.

[0008] Before loading the analysis model, the analysis model is trained and optimized, including establishing a complete view path with shared parameters and a damaged view path; The complete view path receives complete multi-source input, sets the observation mask of each data source or modality to all 1s and the missing density to 0, in order to provide semantic reference; The damaged view path receiving part can observe multimodal inputs to learn robust prediction capabilities under dynamic mode missing conditions.

[0009] Furthermore, the goal of training the analysis model is achieved through a loss function; The loss function is expressed as: ,in, to Assign weights to each loss term. For behavioral analysis to predict losses, For the loss of consistency of features of the complete view, For prototype supervision loss, For the loss of consistency of private clues, Loss due to the separation of shared and private ownership.

[0010] Furthermore, state evolution control includes: For the m-th data source or mode, the definition is in the state evolution sequence of the m-th... The potential input at each time step is represented as: The corresponding observation mask is The previous hidden state was ; The candidate hidden states are generated by the state update function and are represented as follows: ; State evolution control is performed based on the observation mask, as follows: ,in, This represents the state update function for the corresponding data source or modality; This indicates that there are valid observations at the current time step, and the candidate hidden state is used to update the current hidden state; This indicates that the observation corresponding to the current time step is missing or unavailable, and the current hidden state remains the same as the previous hidden state.

[0011] Furthermore, reliable decoupling is used to achieve shareable behavioral semantic representations. and proprietary reliability clues used for reliability estimation Cross-source integration; Among them, shareable behavioral semantic representation Used to carry cross-source fusionable behavioral semantics; Private reliability clues Used to record residual information that is specific to data source disturbances, lacks induced instability, or is unsuitable for direct fusion.

[0012] For reliable decoupling, for audio and visual modalities, the state-evolution-controlled audio and visual sequence representations are transformed to the frequency domain. An adaptive soft filter is generated by combining the overall embedding of the observation mask and the frequency domain amplitude response, and relatively stable frequency components are selected to form a shared behavioral representation. The residual frequency components then form a private reliability clue. ; For the text modality, the semantic response distribution and corresponding entropy value are calculated based on the text hidden state, and the entropy value is used to generate text gating; the text gating is applied to the text hidden state to obtain the text shared behavior representation, and its complementary gating obtains text private reliability clues.

[0013] Furthermore, semantic prototype calibration includes the following steps: Sequence aggregation and attention convergence are performed on the shared behavior representations to obtain modality-level shared vectors. , represented as: ,in, This indicates sequence aggregation or global pooling; Calculate modal level shared vectors With each semantic prototype in the semantic prototype library The similarity is used to obtain the prototype assignment weight, which is expressed as: ,in, For the first A learnable semantic prototype; Assign weights to the prototype; Weights are assigned to semantic prototypes based on their prototypes. The semantic anchors are obtained by weighted combination, and are represented as follows: ,in, As a semantic anchor point; It is a residual mapping network; By performing residual mapping and normalization between modal-level shared vectors and semantic anchors, a calibrated shared behavior representation is obtained. , represented as: ,in, For normalization.

[0014] Furthermore, reliability estimation includes the following steps: Private reliability clues Perform global pooling or sequence aggregation to obtain private vectors at the data source level. , represented as: ; Data source level private vectors Corresponding data source level missing density After concatenation, the data is input into a reliability estimation network to obtain the reliability score of the data source or mode, which is represented as: ,in, Represents a reliability estimation network. Indicates the first Reliability score for each data source or modality; The reliability scores of each data source or modality are normalized to obtain the reliability weights, which are expressed as follows: ,in, This represents the normalized reliability weight.

[0015] Furthermore, reliability residual fusion includes the following steps: Using reliability weights Representation of behavior We perform weighted aggregation to obtain the initial fusion basis representation, which is: ,in, This represents the initial fusion basis. Computational behavior representation Relative to the initial fusion basis representation Residual update; According to reliability weight Weighted injection is applied to the residual update to obtain the final fused representation, namely: ,in, This represents the final fusion.

[0016] Furthermore, the multi-source time-series raw data is a data set composed of data collected from the teaching subject, teaching platform and teaching resource sides after the teaching activities in the interactive teaching environment of "subject-platform-resource" are taken in the order of time. It is divided into several time segments or time steps according to the continuous teaching process. The analysis results include participation in interactive teaching involving the subject, platform, and resources, resource usage preferences, interaction quality assessment, and risk warnings.

[0017] According to the present invention, behavioral analysis of interactive teaching environments can reduce the interference of unreliable data sources on the judgment of learning status and behavioral analysis results when there are dynamic missing data, asynchronous collection, or local anomalies in the teaching subject behavior, platform interaction logs, and resource usage records that constitute multi-source time-series data. This improves the robustness of multi-source data management and behavioral analysis, and can more stably output results such as learning participation, learning status, resource usage preferences, interaction quality, and learning risk warnings, providing data support for personalized teaching intervention, learning early warning, and optimization of teaching resources. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of a data fusion method for dynamically missing multi-source time-series data provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the construction process of a time-series data sequence according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a reliable decoupling mechanism provided according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a semantic prototype calibration mechanism provided according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a reliability gating fusion and behavior prediction mechanism provided in an embodiment of the present invention. Detailed Implementation

[0019] In an interactive teaching environment based on "subject-platform-resources," a basic framework for a multi-source time-series data prediction and behavior analysis model is constructed. Based on time-series data such as teaching subject behavior, platform interaction records, and resource usage records, and combined with an analysis model for multi-source time-series data prediction and behavior analysis, behavior analysis results are generated. The data fusion method for dynamically missing multi-source time-series data proposed in this invention embeds a data processing mechanism for dynamically missing data into key components of the basic framework of the multi-source time-series data prediction and behavior analysis model. This improves the stability and reliability of behavior analysis results, addressing situations where multi-source data is incomplete, unreliable, or acquired asynchronously.

[0020] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] The data fusion method for dynamically missing multi-source time-series data provided by this invention is as follows: Figure 1 As shown, it includes the following steps: Step S100: Obtain raw time series data from multiple sources; In this step, multi-source interaction samples are acquired within the "subject-platform-resource" interactive teaching environment to form multi-source time-series raw data. These multi-source interaction samples include subject behavior data, platform interaction data, and resource usage data. Subject behavior data may further include classroom speaking text, voice interaction features, facial expression and posture features, answering behavior, and discussion behavior. Platform interaction data may include login records, click records, submission records, dwell time, interaction frequency, and learning paths. Resource usage data may include course video viewing records, courseware browsing records, test answering records, homework submission records, discussion text, and knowledge point resource access records.

[0022] Multi-source time-series raw data includes various modalities such as text modality, audio modality, and visual modality. It is a data set that is collected from multiple sources, including the teaching subject, teaching platform, and teaching resources, according to the time sequence of teaching activities, and then divided into several time segments or time steps according to the continuous teaching process.

[0023] Step S110: Preprocess and time-align the multi-source time-series raw data to construct a time-series data sequence; The content of multi-source time series raw data has the characteristics of different sources, different sampling frequencies and different data forms. In the preprocessing process, the source time series raw data is cleaned and regularized. When performing time alignment, matching and serialization are performed based on timestamps, and a time series data sequence that can be input into the model is constructed.

[0024] When constructing time series data sequences, observation labeling is added; observation labeling refers to labeling the observable state of each observation unit in each data source and quantifying the overall missingness of different data sources.

[0025] Step S111: During observation labeling, generate an observation mask and data source-level missing density for any data source or modal sequence; 1) The observation mask is used to mark whether each observation unit in the time series data has valid observations: if a certain observation unit has valid data, its mask value is 1; if the observation unit is unavailable due to reasons such as missing logs, video occlusion, voice noise, missing text, incomplete resource records or platform delay, its mask value is 0.

[0026] The observation mask for the m-th data source or modality sequence is represented as: Where m represents the data source sequence number, Indicates data source The sequence length, This indicates whether there is a valid observation in the i-th observation unit of the m-th data source.

[0027] 2) Data source level missing density is used to characterize the proportion of unavailable observation units in the corresponding data source, thereby quantifying the overall missing degree of time series data.

[0028] The missing density of a data source is defined as: ,in, The larger the value, the higher the proportion of unavailable observation units in the corresponding data source, and the lower its overall reliability.

[0029] In this step, the construction process of the time series data sequence is as follows: Figure 2 As shown: 1) For text modal data, obtain word / tag sequences. Then, missing markers are used to place the missing positions, while maintaining necessary beginning and end markers to indicate observability, to further generate the observation mask for the text modality. and data source level missing density ; 2) For continuous behavioral data of audio modalities, obtain frame-level acoustic features. Then, zero vectors are used to place the missing frames, and observation masks for the audio modalities are further generated. and corresponding data source level missing density ; 3) For continuous behavioral data of visual modalities, obtain frame-level facial or pose features. Then, zero vectors are used to fill in the missing positions, and observation masks for the video modalities are further generated. and corresponding data source level missing density .

[0030] The placeholder data in the constructed time series data sequence is only used to form model input and is not used as evidence for reliable behavioral analysis.

[0031] Step S120: Load the analysis model, input the time series data sequence into the analysis model to achieve analysis and prediction; Before loading the analysis model, the analysis model is trained and built; the analysis model is built by defining time series modeling based on state space model (such as Mamba); and the analysis model is optimized for dynamic missing conditions: a complete view path and a damaged view path with shared parameters are established. The complete view provides semantic reference, and the damaged view learns the robust analysis capability under dynamic missing conditions, thereby improving the applicability of the model in real interactive teaching environments.

[0032] The specifics are as follows: 1) The complete view path receives complete multi-source input and sets the observation mask of each data source or modality to all 1s and the missing density to 0 to provide semantic reference; 2) The observable multimodal input of the damaged view path receiving part is used to learn the robust prediction capability under the condition of dynamic mode missing.

[0033] 3) Train the analysis model using historical teaching data and annotation results, and optimize the analysis model based on prediction errors or analysis task objectives to enable it to adapt to dynamically changing teaching scenarios.

[0034] The training objective of the model is achieved through a loss function, including behavioral analysis and prediction loss. Complete view feature consistency loss Prototype supervision loss Private clue consistency loss And shared-private separation loss Behavioral analysis predicts losses. Used to optimize path prediction results for damaged views; Complete view feature consistency loss. Used to constrain the fusion representation of damaged views to approximate the semantic reference of the complete view; prototype-supervised loss. Used to enable the semantic prototype library to reflect the coarse-grained structure of instructional behavior; private thread consistency loss. Used to stabilize private reliability clues when observations are sufficient; shared-private separation loss Used to reduce information aliasing between shared behavioral representations and private reliability clues.

[0035] The loss function is expressed as: ,in, to The weights for each loss term can be adjusted based on the dataset, task type, and validation set performance. During the inference phase, the complete view path is removed, and only the damaged view path is retained. Therefore, the inference phase does not introduce complete input dependencies, nor does it add additional complexity to the complete view model structure.

[0036] Based on the analysis model built after training, the processing of time series data sequences in the following steps can be achieved.

[0037] Step S130: Convert the time series data sequence into vector features and map them to a unified feature space so that data from different sources can be modeled in the same representation space.

[0038] Step S140: Perform temporal state modeling in the unified feature space to obtain the temporal representation corresponding to the vector features; Temporal state modeling can obtain hidden state representations that reflect behavioral changes based on the dependencies between different time steps, and generate temporal representations corresponding to vector features.

[0039] When performing the time-series state modeling, state evolution control is adopted for dynamic missing cases. If the data at a certain time step is missing or unavailable, the placeholder information corresponding to that time step will not trigger the hidden state update, thereby reducing the risk of missing noise propagating along the time-series link.

[0040] Step S141: Perform state evolution control based on the observation mask; State evolution control refers to capturing time steps where data is missing or unavailable, and the placeholder information corresponding to the time step does not trigger hidden state updates.

[0041] In the state evolution control stage, the observation mask is expanded into time-step control signals according to the time sequence, which are used to control whether the hidden state is updated.

[0042] For the m-th data source or mode, let its position in the state evolution sequence be... The potential input at each time step is represented as: The corresponding observation mask is The previous hidden state was The specific process of state evolution control is as follows: First, the candidate hidden states are generated by the state update function, represented as follows: ; Then, state evolution control is performed based on the observation mask, as follows: ,in, This represents the state update function corresponding to the data source or modality, which can be implemented by a state-space model, a recurrent network, a temporal convolution, or other temporal coding units. This indicates that there are valid observations at the current time step, and the candidate hidden state is used to update the current hidden state; This indicates that the observation corresponding to the current time step is missing or unavailable, and the current hidden state remains the same as the previous hidden state.

[0043] By controlling the state evolution, missing placeholder information will not trigger hidden state updates, thereby reducing the propagation of missing noise along the state evolution link.

[0044] Step S150: Fuse the timing representations to generate a fused representation. ; The temporal representation of the processing in this step is either a global sequence representation or an aggregate representation after state evolution.

[0045] This step generates a fusion representation that comprehensively reflects the behavioral process. Through the output layer of the analysis model, behavioral analysis results such as participation level, participation status, resource usage preferences, interaction quality assessment, or risk warnings can be output.

[0046] Step S151: When fusing temporal representations, based on the reliable decoupling of the preceding sequence and semantic prototype calibration, shared representation and private cue information are obtained, and reliability residual fusion of modality reliability estimation and weight adjustment is performed.

[0047] 1) Reliable decoupling is used to implement shareable behavioral semantic representations. and proprietary reliability clues used for reliability estimation Cross-source integration; shareable behavioral semantic representations Used to carry cross-source fusionable behavioral semantics, such as instructional behavioral semantics; private reliability cues Used to record residual information that is specific to data source disturbances, lacks induced instability, or is unsuitable for direct fusion.

[0048] The process of reliable decoupling is as follows: Figure 3 As shown.

[0049] Different methods are used for different modes: Audio and visual modalities are typically continuous signals; missing frames or zero-vector segments can cause local breaks and unstable frequency responses. In this step, the state-evolution-controlled audio and visual sequence representations are transformed to the frequency domain, and an adaptive soft filter is generated by combining the overall embedding of the observation mask and the frequency domain amplitude response. Relatively stable frequency components are selected to form a shared behavioral representation. The residual frequency components then form a private reliability clue. .

[0050] The process takes the overall sequence of states after evolution as input, and is represented as follows: , , in, , , This represents the overall sequence representation of the m-th data source or mode after state evolution control; and These represent frequency domain transform and inverse transform, respectively. Indicates the frequency domain amplitude response; This indicates the overall embedding of the observation mask; It is an adaptive soft filter with values ​​between 0 and 1; This indicates element-wise multiplication.

[0051] The text modality is a discrete semantic sequence. Missing text locations typically manifest as unknown markers or semantic fragment breaks, and its unreliability is mainly reflected in the uncertainty of semantic responses. Therefore, in this step, the semantic response distribution and corresponding entropy values ​​are calculated based on the text hidden state, and then the entropy values ​​are used to generate text gating, expressed as: Where i represents the i-th observation unit, The semantic response distribution representing the text location, This represents the semantic response entropy; the larger the entropy value, the more unstable the semantic response. This indicates text gating.

[0052] Text gating is applied to the hidden state of text to obtain a representation of text-shared behavior, and its complementary gating provides clues to the reliability of text-private information.

[0053] 2) Semantic prototype calibration is used to constrain the semantic direction of the damaged shared representation, thereby reducing the offset of the shared semantic representation caused by dynamic missing values; Achieving semantic prototype calibration, such as Figure 4 As shown, it includes the following steps: First, sequence aggregation and attention convergence are performed on the shared behavioral representations of various data sources or modalities to obtain modality-level shared vectors. , represented as: ,in, This indicates sequence aggregation or global pooling; Then calculate the modal level shared vector. With each semantic prototype in the semantic prototype library The similarity is used to obtain the prototype assignment weight, which is expressed as: ,in, For the first A learnable semantic prototype; Assign weights to the prototype; Then, weights are assigned to semantic prototypes. The semantic anchors are obtained by weighted combination, and are represented as follows: ,in, As a semantic anchor point; It is a residual mapping network; Finally, residual mapping and normalization are performed on the modal-level shared vectors and semantic anchors to obtain the calibrated shared behavior representation. , represented as: ,in, For normalization.

[0054] By calibrating semantic prototypes, damaged shared representations can be reconstrained to a more stable behavioral semantic space, reducing the behavioral semantic drift induced by missing features.

[0055] 3) Reliability estimation is used to determine the credibility of different data sources based on missing density and private clues; In this phase, private reliability clues for each data source or modality are established. Perform global pooling or sequence aggregation to obtain private vectors at the data source level. , represented as: ; Furthermore, private vectors at the data source level Corresponding data source level missing density After concatenation, the data is input into a reliability estimation network to obtain the reliability score of the data source or mode, which is represented as: ,in, Represents a reliability estimation network. Indicates the first Reliability score for each data source or modality; Next, the reliability scores of each data source or modality are normalized to obtain the reliability weights, which are expressed as follows: ,in, This represents the normalized reliability weight.

[0056] Data sources with high missing density or strong private perturbation will have a lower reliability score.

[0057] 4) Reliability residual fusion is used to adjust the contribution of different data sources to the final fused representation according to reliability weights.

[0058] This stage first utilizes reliability weights. Representation of the calibrated shared behavior of each mode We perform weighted aggregation to obtain the initial fusion basis representation, which is: ,in, This represents the initial fusion basis. Then, the residual update of each modality's calibrated shared behavioral representation relative to the initial fused basis representation is calculated, expressed as: or ,in, For the first Residual updates of each data source or modality relative to the fusion basis representation; For learnable residual mapping networks; Finally, the residual updates are weighted according to reliability weights to obtain the final fused representation, namely: ,in, This represents the final fusion.

[0059] By using reliability residual fusion, reliable modes can provide effective behavioral evidence for the fused representation, while limiting the impact of unreliable modes on the final prediction.

[0060] Step S160: Obtain the analysis results by analyzing the model; like Figure 5 As shown in the diagram, based on the shared behavioral representation obtained from reliable decoupling and the private reliability clue information, reliability weights are obtained through modal reliability estimation using a reliability gating unit, the reliability weights are adjusted by dual-source collaborative gating, and reliability residuals are fused using a fusion gating system, resulting in the final fused representation. Input the prediction header and output the behavioral analysis results of the multi-source time-series raw data in the interactive teaching environment of "subject-platform-resource" to be analyzed.

[0061] The prediction head can be a multilayer perceptron, a linear regression head, a multi-label classification head, or a sequence prediction head.

[0062] When the task is engagement, interaction quality, or risk assessment, the prediction head can output a continuous score or risk probability. When the task is state recognition, resource usage preference recognition, or behavior type judgment, the prediction head can output category probability or category label.

[0063] Behavioral analysis results include, but are not limited to, engagement levels, resource usage preferences, interaction quality assessments, and risk alerts.

[0064] This invention provides a specific example of a dynamic missing multi-source teaching data processing scenario, focusing on data management and behavior analysis within a "subject-platform-resource" interactive teaching environment. The input consists of multi-source interactive samples containing text, audio, and visual modalities: text modalities can originate from automatic speech recognition or manually transcribed text; audio modalities can originate from audio frame-level acoustic features; and visual modalities can originate from facial or pose features in video frames. For each modality, a temporal observation mask and missing density are constructed. Missing text locations are marked with unknown markers, while missing audio and visual frames are marked with zero vectors. Unified projection, mask state evolution control, modal reliable decoupling, prototype semantic calibration, reliability residual fusion, and prediction output are executed sequentially. An analysis model is constructed based on a state-space model (such as Mamba) for temporal modeling. During model training, complete and damaged view paths are constructed. The complete view path provides semantic reference under complete input, while the damaged view path learns and predicts under dynamic missing conditions. Multi-loss joint constraints improve the model's robustness under partially observable inputs. During inference, the complete view path is removed, retaining only the damaged view path. When the camera is not turned on, the audio signal is interfered with by noise, the platform log is delayed, resource access records are missing, or the discussion text is incomplete, the observation mask at the corresponding time step is set to a missing state, and the state evolution control module suppresses the update of the hidden state by the missing placeholder information. Subsequently, the system performs reliable decomposition on continuous audio, visual, or behavioral time series, performs gated decomposition on text or discrete interaction sequences, and uses a semantic prototype library to calibrate the damaged shared representation. Finally, the system estimates the reliability weights of different data sources based on the missing density and private reliability cues, obtains the teaching behavior analysis representation through reliability residual fusion, and outputs results such as participation, state, resource usage preferences, interaction quality, or risk warnings.

[0065] The system that enables data management and behavior analysis in an interactive teaching environment of “subject-platform-resource” can be deployed on servers, edge computing devices, intelligent interactive terminals or video analysis platforms. It can perform reliable fusion on inputs with missing text, audio noise, visual occlusion or multimodal local missing data, and reduce the interference of unreliable modal fragments on the final prediction.

[0066] The data fusion method for dynamically missing data in multi-source time series provided by this invention does not rely on explicit completion of all missing segments. Instead, it directly controls the propagation of unreliable observations through reliability modeling, reducing the risk of uncertain reconstructed features being mistakenly used as reliable evidence. In implementation, hidden state updates are controlled by a time observation mask, preventing missing placeholders from triggering state updates and reducing their contamination of subsequent time step representations, thus suppressing the cumulative propagation of missing noise in time series modeling. For audio and visual modal data, stable frequency components and residual perturbations can be separated through frequency domain soft filtering. Text is gated and decomposed using semantic response entropy, making it more consistent with the data forms of different modalities and improving the targeting of missing perturbation modeling for different modalities. Modal-level missing density and private reliability cues are used together to estimate reliability weights, thereby adjusting modal weighted aggregation and residual injection. For model training optimization, the training phase utilizes a complete view to provide semantic reference, while the inference phase does not rely on complete input, improving robustness under dynamic missing conditions and avoiding the need for complete modal input during inference.

[0067] In particular, for behavioral analysis in interactive teaching environments, when there are dynamic gaps, asynchronous collections, or local anomalies in the behavior of teaching subjects, platform interaction logs, and resource usage records, this invention can reduce the interference of unreliable data sources on the judgment of learning status and behavioral analysis results, improve the robustness of multi-source data management and behavioral analysis, and more stably output results such as learning participation, learning status, resource usage preferences, interaction quality, and learning risk warnings, providing data support for personalized teaching intervention, learning early warning, and optimization of teaching resources.

[0068] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A data fusion method for dynamically missing multi-source time-series data, characterized in that, Includes the following steps: Acquire multi-source time-series raw data; the modalities of the multi-source time-series raw data include text modality, audio modality, and visual modality; The multi-source time-series raw data are preprocessed and time-aligned to construct a time-series data sequence; when constructing the time-series data sequence, observation labeling is added to generate an observation mask and data source-level missing density. Load the analysis model by inputting the time series data sequence into the analysis model; The time-series data sequence is converted into vector features and mapped to a unified feature space; Temporal state modeling is performed in the unified feature space to obtain the temporal representation corresponding to the vector features; during the temporal state modeling, state evolution control is adopted; during the state evolution control process, time steps with missing or unavailable data are captured, and the missing placeholder information corresponding to the time step does not trigger hidden state updates. The time series representations are fused to generate a fused representation. When fusing the temporal representation, based on the reliable decoupling of the preceding sequence and semantic prototype calibration, shared representation and private cue information are obtained, and reliability residual fusion of modality reliability estimation and weight adjustment is performed. The analysis results are obtained through the aforementioned analysis model.

2. The data fusion method for dynamically missing multi-source time-series data according to claim 1, characterized in that, The observation mask is used to mark whether each observation unit in the corresponding data source in the time series data has a valid observation: if a certain observation unit has valid data, its mask value is 1; if the observation unit is unavailable, its mask value is 0. The data source level missing density is used to characterize the proportion of unavailable observation units in the corresponding data source, thereby quantifying the overall missing degree of time series data.

3. The data fusion method for dynamically missing multi-source time-series data according to claim 1, characterized in that, Before loading the analysis model, the analysis model is trained and optimized, including establishing a complete view path and a damaged view path with shared parameters; The complete view path receives complete multi-source input, sets the observation mask of each data source or modality to all 1s and the missing density to 0, in order to provide semantic reference; The damaged view path receiving section can observe multimodal inputs to learn robust prediction capabilities under dynamic mode loss conditions.

4. The data fusion method for dynamically missing multi-source time-series data according to claim 3, characterized in that, The goal of training the analytical model is achieved through a loss function; The loss function is expressed as: ,in, to Assign weights to each loss term. For behavioral analysis to predict losses, For the loss of consistency of features of the complete view, For prototype supervision loss, For the loss of consistency of private clues, Loss due to the separation of shared and private ownership.

5. The data fusion method for dynamically missing multi-source time-series data according to claim 1, characterized in that, The state evolution control includes: For the m-th data source or mode, the definition is in the state evolution sequence of the m-th... The potential input at each time step is represented as: The corresponding observation mask is The previous hidden state was ; The candidate hidden states are generated by the state update function and are represented as follows: ; State evolution control is performed based on the observation mask, as follows: ,in, This represents the state update function for the corresponding data source or modality; This indicates that there are valid observations at the current time step, and the candidate hidden state is used to update the current hidden state; This indicates that the observation corresponding to the current time step is missing or unavailable, and the current hidden state remains the same as the previous hidden state.

6. The data fusion method for dynamically missing multi-source time-series data according to claim 1, characterized in that, The reliable decoupling is used to achieve shareable behavioral semantic representation. and proprietary reliability clues used for reliability estimation Cross-source integration; Wherein, the shareable behavioral semantic representation Used to carry cross-source fusionable behavioral semantics; The private reliability clue Used to record residual information that is specific to data source disturbances, lacks induced instability, or is not suitable for direct fusion; During the reliable decoupling, for both audio and visual modalities, the state-evolution-controlled audio and visual sequence representations are transformed to the frequency domain. An adaptive soft filter is generated by combining the overall embedding of the observation mask and the frequency domain amplitude response, and relatively stable frequency components are selected to form a shared behavioral representation. The residual frequency components then form a private reliability clue. ; For the text modality, the semantic response distribution and corresponding entropy value are calculated based on the text hidden state, and the entropy value is used to generate text gating; the text gating is applied to the text hidden state to obtain the text shared behavior representation, and its complementary gating obtains text private reliability clues.

7. The data fusion method for dynamically missing multi-source time-series data according to claim 6, characterized in that, The semantic prototype calibration includes the following steps: Sequence aggregation and attention convergence are performed on the shared behavior representation to obtain the modality-level shared vector. , represented as: ,in, This indicates sequence aggregation or global pooling; Calculate modal level shared vectors With each semantic prototype in the semantic prototype library The similarity is used to obtain the prototype assignment weight, which is expressed as: ,in, For the first A learnable semantic prototype; Assign weights to the prototype; Weights are assigned to semantic prototypes based on their prototypes. The semantic anchors are obtained by weighted combination, and are represented as follows: ,in, As a semantic anchor point; It is a residual mapping network; By performing residual mapping and normalization between modal-level shared vectors and semantic anchors, a calibrated shared behavior representation is obtained. , represented as: ,in, For normalization.

8. The data fusion method for dynamically missing multi-source time-series data according to claim 7, characterized in that, The reliability estimation includes the following steps: Private reliability clues Perform global pooling or sequence aggregation to obtain private vectors at the data source level. , represented as: ; Data source level private vectors Corresponding data source level missing density After concatenation, the data is input into a reliability estimation network to obtain the reliability score of the data source or mode, which is represented as: ,in, Represents a reliability estimation network. Indicates the first Reliability score for each data source or modality; The reliability scores of each data source or modality are normalized to obtain the reliability weights, which are expressed as follows: ,in, This represents the normalized reliability weight.

9. The data fusion method for dynamically missing multi-source time-series data according to claim 8, characterized in that, The reliability residual fusion includes the following steps: Using reliability weights Representation of behavior We perform weighted aggregation to obtain the initial fusion basis representation, which is: ,in, This represents the initial fusion basis. Calculate the behavior representation Relative to the initial fusion basis representation Residual update; According to reliability weight Weighted injection is applied to the residual update to obtain the final fused representation, namely: ,in, This represents the final fusion.

10. The data fusion method for dynamically missing multi-source time-series data according to claim 1, characterized in that, The multi-source time-series raw data is a data set composed of data collected from the teaching subject, teaching platform and teaching resource side according to the time sequence of teaching activities in the "subject-platform-resource" interactive teaching environment, and divided into several time segments or time steps according to the continuous teaching process. The analysis results include participation in interactive teaching involving the "subject-platform-resource" model, resource usage preferences, interaction quality assessment, and risk warnings.

Images