A video conference multi-modal data alignment method and device based on a causal mask, equipment and medium

Abstract

The application discloses a video conference multi-modal data alignment method and device based on a causal mask, equipment and a medium, and relates to the technical field of computers. The method comprises the following steps: performing feature extraction and fusion on original audio, original video stream and original documents in an online video conference, performing time sequence division on obtained multi-modal fusion features to obtain a triple time sequence window, determining initial weight values corresponding to the triple time sequence window, normalizing and adjusting the initial weight values by using a preset constraint condition to obtain adjusted weights, indexing a preset time sequence offset matrix by using a speaking identifier of a speaker, correcting the original time sequence of the triple time sequence window based on the indexing result, determining target attention results corresponding to the triple time sequence window by using a preset causal mask mechanism, and performing multi-level alignment and fusion on the multi-modal fusion features based on the target attention results to obtain a multi-modal alignment result. The precision of the multi-modal alignment technology is improved, and future information leakage is avoided.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for aligning multimodal data in video conferencing based on causal masking. Background Technology

[0002] With the widespread adoption of remote collaboration and online video conferencing, intelligent processing of multimodal information has become a crucial technological requirement. Existing multimodal alignment technologies have significant limitations when handling complex temporal relationships. Traditional cross-modal attention mechanisms employ a bidirectional static alignment strategy, which cannot effectively capture the complex temporal behavioral patterns of speakers in video conferences, such as dynamic interaction patterns like showing before explaining or referring back to previous content.

[0003] The core problem with existing technologies lies in the lack of deep modeling of temporal causal relationships. Traditional methods typically ignore the temporal dependencies between multimodal information, leading to future information leakage problems and severely impacting real-time processing performance and logical consistency. Furthermore, existing technologies do not adequately consider the individualized temporal habits of different speakers, such as habitual forward-looking expressions and delayed interpretations, resulting in insufficient alignment accuracy.

[0004] As can be seen from the above, how to improve the accuracy of multimodal alignment technology and avoid future information leakage is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for multimodal data alignment in video conferencing based on causal masking, which can improve the accuracy of multimodal alignment technology and prevent future information leakage. The specific solution is as follows:

[0006] Firstly, this application provides a method for aligning multimodal data in video conferencing based on causal masks, including:

[0007] A preset encoder is used to extract and fuse features from the original audio, original video stream and original document in the online video conference to obtain multimodal fusion features. Based on the multimodal fusion features, temporal division is performed to obtain a triple temporal window including a look-ahead window, a current window and a backtracking window.

[0008] Based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, the initial weight values ​​corresponding to the triple time window are determined. The initial weight values ​​are then normalized and adjusted using preset constraints to obtain the adjusted weights.

[0009] The preset timing offset matrix is ​​indexed using the speaker's speaking identifier, and the original timing corresponding to the triple timing window is corrected based on the obtained indexing result to obtain the corrected timing.

[0010] Based on the corrected temporal sequence and the adjusted weights, and using a preset causal masking mechanism, the target attention result corresponding to the triple temporal window is determined, and the multimodal fusion feature is fused in a multi-level alignment based on the target attention result to obtain the multimodal alignment result.

[0011] Optionally, the step of using a preset encoder to extract and fuse features from the original audio, original video stream, and original document in the online video conference to obtain multimodal fusion features includes:

[0012] Acoustic features are extracted from the raw audio of an online video conference using an audio encoder to obtain the first feature;

[0013] Visual features are extracted from the raw video stream in online video conferencing based on the video encoder to obtain the second feature;

[0014] A text embedding encoder is used to extract semantic features from the original documents in an online video conference to obtain a third feature;

[0015] Feature fusion is performed based on the first feature, the second feature, and the third feature to obtain multimodal fusion features.

[0016] Optionally, the temporal segmentation based on the multimodal fusion features to obtain a triple temporal window including a look-ahead window, a current window, and a backtracking window includes:

[0017] The current window is determined based on the multimodal fusion features corresponding to the current moment, and the backtracking window is determined using the multimodal fusion features prior to the current moment and the backtracking length parameter.

[0018] The look-ahead window is determined based on the multimodal fusion features and look-ahead length parameters after the current time, and a triple time-series window is determined using the current window, the backtracking window, and the look-ahead window.

[0019] Optionally, the step of determining the initial weight values ​​corresponding to the triple temporal window based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, and then normalizing and adjusting the initial weight values ​​using preset constraints to obtain the adjusted weights, includes:

[0020] Based on the multimodal features corresponding to the current moment, a lightweight neural network is used to learn the mapping relationship between features and causal weights;

[0021] Based on the mapping relationship, weights are assigned to the look-ahead window, the current window, and the backtracking window respectively to obtain initial weight values ​​corresponding to the look-ahead window, the current window, and the backtracking window respectively, and the weight sum is determined using each initial weight value;

[0022] Based on the weights, it is determined whether the initial weight values ​​meet the preset constraints; the preset constraints are that the weights are consistent with the first target value, and each of the initial weight values ​​is not less than the second target value.

[0023] If the preset constraint conditions are met, the initial weight value will be determined as the adjusted weight.

[0024] If the preset constraints are not met, the initial weight value is normalized to obtain a new initial weight value and a weight sum, and the process jumps to the step of determining whether the initial weight value meets the preset constraints based on the weight sum, until the preset constraints are met to obtain the adjusted weight.

[0025] Optionally, the step of indexing the preset timing offset matrix using the speaker's speech identifier and correcting the original timing corresponding to the triple timing window based on the obtained indexing result to obtain the corrected timing includes:

[0026] Based on the original audio, the original video stream, and the original document, each modality pair is constructed. The modality pair and the speaker's speech identifier are used to index the preset temporal offset matrix to obtain the temporal offset mode of the speaker under different modality pairs.

[0027] The timing offset value is determined using the timing offset mode. Based on the timing offset value and using a preset correction formula, the original timing corresponding to the triple timing window is corrected to obtain the corrected timing.

[0028] Optionally, determining the target attention result corresponding to the triple temporal window based on the corrected temporal sequence and the adjusted weights using a preset causal masking mechanism includes:

[0029] Based on the adjusted weights and the corrected time series, the first attention result corresponding to the look-ahead window, the second attention result corresponding to the current window, and the third attention result corresponding to the backtracking window are determined respectively.

[0030] The first attention result is masked based on a preset causal masking mechanism to obtain the masked first attention result.

[0031] The target attention result is determined using the first attention result, the second attention result, and the third attention result after masking.

[0032] Optionally, the step of performing multi-level alignment fusion on the multimodal fusion features based on the target attention result to obtain the multimodal alignment result includes:

[0033] Based on the target attention result, a global layer alignment is performed between the overall structure of the online video conference and the temporal relationship between the presentation chapters of the original document to obtain a first alignment result;

[0034] The semantics between the demo page corresponding to the original document and the speech segment corresponding to the original audio are aligned at the paragraph level using the target attention result to obtain a second alignment result;

[0035] Based on the target attention result, the target document elements in the original document and the target speech elements in the original audio are aligned to obtain a third alignment result;

[0036] The multimodal alignment result is determined using the first alignment result, the second alignment result, and the third alignment result.

[0037] Secondly, this application provides a video conferencing multimodal data alignment device based on causal masking, comprising:

[0038] The feature fusion module is used to extract and fuse features from the original audio, original video stream and original document in the online video conference using a preset encoder to obtain multimodal fusion features. Based on the multimodal fusion features, temporal division is performed to obtain a triple temporal window including a look-ahead window, a current window and a backtracking window.

[0039] The weight adjustment module is used to determine the initial weight value corresponding to the triple time window based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, and to normalize and adjust the initial weight value using preset constraints to obtain the adjusted weight.

[0040] The timing correction module is used to index the preset timing offset matrix using the speaker's speech identifier, and to correct the original timing corresponding to the triple timing window based on the obtained index result, so as to obtain the corrected timing.

[0041] The feature alignment module is used to determine the target attention result corresponding to the triple temporal window based on the corrected temporal sequence and the adjusted weight and using a preset causal masking mechanism, and to perform multi-level alignment and fusion on the multimodal fusion features based on the target attention result to obtain the multimodal alignment result.

[0042] Thirdly, this application provides an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] A processor is configured to execute the computer program to implement the aforementioned causal mask-based video conferencing multimodal data alignment method.

[0045] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned causal mask-based video conferencing multimodal data alignment method.

[0046] This application utilizes a preset encoder to extract and fuse features from the original audio, original video stream, and original document in an online video conference to obtain multimodal fusion features. Based on these multimodal fusion features, temporal segmentation is performed to obtain a triple temporal window including a look-ahead window, a current window, and a backtracking window. Based on the multimodal fusion features corresponding to the current moment, a lightweight neural network is used to determine initial weight values ​​corresponding to the triple temporal window. These initial weight values ​​are then normalized and adjusted using preset constraints to obtain adjusted weights. The speaker's speaking identifier is used to index a preset temporal offset matrix, and the original temporal sequence corresponding to the triple temporal window is corrected based on the obtained indexing results to obtain corrected temporal sequences. Based on the corrected temporal sequences and the adjusted weights, a preset causal masking mechanism is used to determine the target attention result corresponding to the triple temporal window. Finally, multi-level alignment fusion is performed on the multimodal fusion features based on the target attention result to obtain a multimodal alignment result.

[0047] As can be seen from the above, this application utilizes a preset encoder to extract modal features from the original audio, original video stream, and original document in online video conferencing. These modal features are then fused to obtain multimodal fusion features. Based on these multimodal fusion features, a triple temporal window is obtained, comprising a look-ahead window, a current window, and a backtracking window. This triple temporal window covers the temporal relationships of the past, present, and future. A lightweight neural network is used, and the weights of each temporal window are dynamically allocated according to the content at the current moment. Next, a preset temporal offset matrix is ​​indexed using speaker identifiers, and the original temporal sequence is corrected based on the indexing results. This eliminates the impact of individual temporal differences among speakers on the multimodal alignment effect. A preset causal masking mechanism is used, combined with the corrected temporal sequence and adjusted weights, to calculate the target attention result. In this way, multi-level alignment fusion of the multimodal fusion features using the target attention result supports different levels of application requirements, reduces alignment errors caused by individual differences, and effectively improves the robustness and overall consistency of the alignment. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This application discloses a flowchart of a video conferencing multimodal data alignment method based on causal masking.

[0050] Figure 2 This is a schematic diagram of a video conferencing multimodal data alignment device based on causal masking disclosed in this application;

[0051] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Currently, existing multimodal alignment techniques lack in-depth modeling of temporal causal relationships and ignore the temporal dependencies between multimodal information, leading to future information leakage problems and severely affecting real-time processing performance and logical consistency. Furthermore, existing technologies do not fully consider the individualized temporal habits of different speakers, such as habitual look-ahead expressions and delayed interpretations, resulting in insufficient alignment accuracy. Therefore, this application provides a video conferencing multimodal data alignment method based on causal masks. This method utilizes target attention results to perform multi-level alignment and fusion of the multimodal fusion features, supporting different levels of application requirements, reducing alignment errors caused by individual differences, and effectively improving the robustness and overall consistency of the alignment.

[0054] See Figure 1 As shown, this embodiment of the invention discloses a video conferencing multimodal data alignment method based on causal masking, comprising:

[0055] Step S11: Use a preset encoder to extract and fuse features from the original audio, original video stream and original document in the online video conference to obtain multimodal fusion features. Based on the multimodal fusion features, perform temporal division to obtain a triple temporal window including a look-ahead window, a current window and a backtracking window.

[0056] In this embodiment, an audio encoder is used to extract acoustic features from the original audio in the online video conference to obtain a first feature; the audio encoder can be an encoder determined based on CNN (Convolutional Neural Network) and Transformer (deep learning model); a video encoder is used to extract visual features from the original video stream in the online video conference to obtain a second feature; the video encoder can be an encoder determined based on ViT (VisionTransformer, image recognition model) and TCN (Temporal Convolutional Network); a text embedding encoder is used to extract semantic features from the original document in the online video conference to obtain a third feature; the text embedding encoder can be an encoder determined based on a pre-trained language model; then, the first feature, the second feature, and the third feature are fused through a temporal dimension to obtain a multimodal fusion feature.

[0057] Specifically, the step of using a preset encoder to extract and fuse features from the original audio, original video stream, and original document in an online video conference to obtain multimodal fusion features includes: using an audio encoder to extract acoustic features from the original audio in the online video conference to obtain a first feature; using a video encoder to extract visual features from the original video stream in the online video conference to obtain a second feature; using a text embedding encoder to extract semantic features from the original document in the online video conference to obtain a third feature; and fusing features based on the first feature, the second feature, and the third feature to obtain multimodal fusion features.

[0058] It is understandable that, after obtaining the multimodal fusion features... Then, the current window is determined based on the multimodal fusion features corresponding to the current time. The current window is... The current window ensures that the alignment process is always based on the current moment, maintaining the continuity and consistency of the temporal sequence. The backtracking window is determined using the multimodal fusion features prior to the current moment and the backtracking length parameter; the backtracking window is... ,in, The backtracking length parameter is defined; the backtracking window models the relationship between the speaker's anaphoric behavior, summarizing expressions, and historical content. The lookahead window is determined based on the multimodal fusion features after the current moment and the lookahead length parameter. ,in, The look-ahead length parameter is used; the look-ahead window captures the temporal pattern of the speaker's prepared or predictive language, and can identify the relationship between forward-looking expressions such as "what will be introduced next" and subsequent content. A triple temporal window is determined using the current window, the backtracking window, and the look-ahead window.

[0059] Specifically, the step of performing temporal segmentation based on the multimodal fusion features to obtain a triple temporal window including a look-ahead window, a current window, and a backtracking window includes: determining the current window based on the multimodal fusion features corresponding to the current time, and determining the backtracking window using the multimodal fusion features before the current time and the backtracking length parameter; determining the look-ahead window based on the multimodal fusion features after the current time and the look-ahead length parameter, and determining the triple temporal window using the current window, the backtracking window, and the look-ahead window.

[0060] Step S12: Based on the multimodal fusion features corresponding to the current time, determine the initial weight value corresponding to the triple temporal window using a lightweight neural network, and normalize the initial weight value using preset constraints to obtain the adjusted weight.

[0061] In this embodiment, weights are assigned to the look-ahead window, the current window, and the backtracking window based on a preset causal weight function to obtain initial weight values ​​for each of the look-ahead window, the current window, and the backtracking window. The formula for the preset causal weight function is as follows:

[0062] ;

[0063] in, The first initial weight value corresponds to the look-ahead window; The second initial weight value corresponding to the current window; The third initial weight value corresponding to the backtracking window; It is a lightweight neural network; For the multimodal fusion features The lightweight learning network is used to learn the mapping relationship between the multimodal fusion features and causal weights at the current moment. Then, the first initial weight value, the second initial weight value and the third initial weight value are added together to obtain the weight sum. If the weight sum is equal to 1 and the first initial weight value, the second initial weight value and the third initial weight value are all not less than 0, it indicates that the preset constraint condition is met, and the initial weight value is determined as the adjusted weight.

[0064] Specifically, the step of determining the initial weight values ​​corresponding to the triple temporal windows based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, and normalizing and adjusting the initial weight values ​​using preset constraints to obtain adjusted weights, includes: learning the mapping relationship from the multimodal features corresponding to the current time to causal weights using a lightweight neural network; assigning weights to the look-ahead window, the current window, and the backtracking window based on the mapping relationship to obtain initial weight values ​​corresponding to the look-ahead window, the current window, and the backtracking window respectively; and using each initial weight... The weight sum is determined by the weight sum; based on the weight sum, it is determined whether the initial weight value meets the preset constraint condition; the preset constraint condition is that the weight sum is consistent with the first target value, and each of the initial weight values ​​is not less than the second target value; if the preset constraint condition is met, the initial weight value is determined as the adjusted weight; if the preset constraint condition is not met, the initial weight value is normalized to obtain a new initial weight value and weight sum, and the process jumps to the step of determining whether the initial weight value meets the preset constraint condition based on the weight sum, until the preset constraint condition is met to obtain the adjusted weight.

[0065] Step S13: Index the preset timing offset matrix using the speaker's speaking identifier, and correct the original timing corresponding to the triple timing window based on the obtained index result to obtain the corrected timing.

[0066] In this embodiment, the original audio, the original video stream, and the original document are arbitrarily paired to construct modal pairs. Each modal pair includes original audio and original video stream, original video stream and original document, and original audio and original document. The definition of the preset temporal offset matrix is ​​as follows:

[0067] ;

[0068] in, The preset timing offset matrix; This indicates that the preset timing offset matrix is ​​a learnable matrix, which can learn the individual timing habits of each speaker; It serves as the unique identifier for the speaker. The modality pair is defined as follows: The preset timing offset matrix is ​​indexed using the modality pair and the speaker's speech identifier to obtain a quantified representation of the speaker's timing habits, i.e., the timing offset value. Based on the timing offset value, the original timing corresponding to the triple timing window is corrected using a preset correction formula, as follows:

[0069] ;

[0070] in, For the corrected timing; The original time series corresponding to any multimodal fusion feature within the triple time series window; The time offset value is the stated value.

[0071] Specifically, the step of indexing a preset timing offset matrix using the speaker's speech identifier and correcting the original timing corresponding to the triple timing window based on the obtained indexing result to obtain the corrected timing includes: constructing modal pairs based on the original audio, the original video stream, and the original document; indexing the preset timing offset matrix using the modal pairs and the speaker's speech identifier to obtain the speaker's timing offset pattern under different modal pairs; determining the timing offset value using the timing offset pattern; and correcting the original timing corresponding to the triple timing window based on the timing offset value and using a preset correction formula to obtain the corrected timing.

[0072] Step S14: Based on the corrected temporal sequence and the adjusted weights, and using a preset causal masking mechanism, determine the target attention result corresponding to the triple temporal window, and perform multi-level alignment fusion on the multimodal fusion features based on the target attention result to obtain the multimodal alignment result.

[0073] In this embodiment, after obtaining the corrected time series, the first attention result corresponding to the look-ahead window, the second attention result corresponding to the current window, and the third attention result corresponding to the backtracking window are determined based on the adjusted weights and the triple time series window corresponding to the corrected time series. To balance real-time performance and accuracy, a controllable causal masking mechanism is introduced into the look-ahead window. This causal masking mechanism ensures that in real-time processing scenarios, there is no excessive reliance on future information, maintaining the real-time performance of the processing. The formula corresponding to the causal masking mechanism is as follows:

[0074] ;

[0075] in, is the mask value; i is the index of the current time; j is the index of the future time. A target look-ahead step size threshold is defined. Based on the current time and the target look-ahead step size threshold, a target look-ahead step size range is determined. When a future time j exceeds the target look-ahead step size range, the mask value is 0, indicating that the future information is masked; when it does not exceed the target look-ahead step size range, the mask value is 1, indicating that the future information is retained. A corresponding mask matrix is ​​determined based on the mask value, and the first attention result is adjusted using the mask matrix to obtain a masked first attention result. The target attention result is determined based on the adjusted weights and using the masked first attention result, the second attention result, and the third attention result. It is worth noting that the target look-ahead step size threshold can be adjusted according to actual conditions, and is not specifically limited here.

[0076] Specifically, determining the target attention result corresponding to the triple time series window based on the corrected time series and the adjusted weights using a preset causal masking mechanism includes: determining the first attention result corresponding to the look-ahead window, the second attention result corresponding to the current window, and the third attention result corresponding to the backtracking window based on the adjusted weights and the corrected time series; masking the first attention result based on the preset causal masking mechanism to obtain the masked first attention result; and determining the target attention result using the masked first attention result, the second attention result, and the third attention result.

[0077] Understandably, comprehensive alignment from macro to micro is achieved through global, paragraph, and element layers. Based on the target attention results, global alignment is performed on the temporal relationship between the overall structure of the online video conference and the presentation chapters of the original document to achieve macro-level temporal alignment. Then, the target attention results are used to perform paragraph-level semantic alignment between the presentation page corresponding to the original document and the speech segments corresponding to the original audio to achieve meso-level semantic alignment. Finally, based on the target attention results, element alignment is performed on the target document elements in the original document and the target speech elements in the original audio to achieve fine-grained alignment, thus obtaining a multimodal alignment result. Specifically, the step of performing multi-level alignment and fusion on the multimodal fusion features based on the target attention result to obtain a multimodal alignment result includes: performing global-level alignment on the temporal relationship between the overall structure of the online video conference and the presentation chapters of the original document based on the target attention result to obtain a first alignment result; performing paragraph-level alignment on the semantics between the presentation page corresponding to the original document and the speech segment corresponding to the original audio using the target attention result to obtain a second alignment result; performing element alignment on the target document elements in the original document and the target speech elements in the original audio based on the target attention result to obtain a third alignment result; and determining the multimodal alignment result using the first alignment result, the second alignment result, and the third alignment result.

[0078] It is worth mentioning that this embodiment employs a bidirectional information transmission mechanism at the global layer, paragraph layer, and element layer. In one specific implementation, the global layer transmits a message to the paragraph layer: "Switch to the product design page in 10 minutes." Based on this message, the paragraph layer is constrained to switch to the product design page after 10 minutes to avoid matching other pages. The paragraph layer operates similarly to the element layer. If the element layer detects a mismatch between the current audio and document elements, it feeds back the detected error to the paragraph layer so that the paragraph layer can recalculate the semantic similarity, adjust the matching relationship between the current audio and document elements, and feed the adjustment result back to the global layer. Furthermore, a corresponding multimodal alignment model can be constructed based on the above steps, and a comprehensive loss function corresponding to each stage can be designed. The formula for the comprehensive loss function is as follows:

[0079] ;

[0080] in, The comprehensive loss function is... Cross-modal alignment loss is a loss value determined based on the similarity between audio features and text features in the aligned multimodal fusion features. Loss due to causal constraints; This is the temporal continuity loss, which measures the difference in the multimodal alignment results between adjacent time steps; This refers to the multi-level consistency loss, which measures the deviation of the multimodal alignment results at different levels. The corresponding weight parameters are used; the multimodal alignment model is optimized based on the obtained total loss value to obtain the target multimodal alignment model.

[0081] As can be seen from the above, this application utilizes a preset encoder to extract modal features from the original audio, original video stream, and original document in online video conferencing. These modal features are then fused to obtain multimodal fusion features. Based on these multimodal fusion features, a triple temporal window is obtained, comprising a look-ahead window, a current window, and a backtracking window. This triple temporal window covers the temporal relationships of the past, present, and future. A lightweight neural network is used, and the weights of each temporal window are dynamically allocated according to the content at the current moment. Next, a preset temporal offset matrix is ​​indexed using speaker identifiers, and the original temporal sequence is corrected based on the indexing results. This eliminates the impact of individual temporal differences among speakers on the multimodal alignment effect. A preset causal masking mechanism is used, combined with the corrected temporal sequence and adjusted weights, to calculate the target attention result. In this way, multi-level alignment fusion of the multimodal fusion features using the target attention result supports different levels of application requirements, reduces alignment errors caused by individual differences, and effectively improves the robustness and overall consistency of the alignment.

[0082] Accordingly, see Figure 2 As shown, this application also provides a video conferencing multimodal data alignment device based on causal masking, comprising:

[0083] The feature fusion module is used to extract and fuse features from the original audio, original video stream and original document in the online video conference using a preset encoder to obtain multimodal fusion features. Based on the multimodal fusion features, temporal division is performed to obtain a triple temporal window including a look-ahead window, a current window and a backtracking window.

[0084] The weight adjustment module is used to determine the initial weight value corresponding to the triple time window based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, and to normalize and adjust the initial weight value using preset constraints to obtain the adjusted weight.

[0085] The timing correction module is used to index the preset timing offset matrix using the speaker's speech identifier, and to correct the original timing corresponding to the triple timing window based on the obtained index result, so as to obtain the corrected timing.

[0086] The feature alignment module is used to determine the target attention result corresponding to the triple temporal window based on the corrected temporal sequence and the adjusted weight and using a preset causal masking mechanism, and to perform multi-level alignment and fusion on the multimodal fusion features based on the target attention result to obtain the multimodal alignment result.

[0087] In some specific embodiments, the feature fusion module 11 may specifically include:

[0088] The first feature extraction unit is used to extract acoustic features from the raw audio in the online video conference using an audio encoder to obtain the first feature;

[0089] The second feature extraction unit is used to extract visual features from the original video stream in the online video conference based on the video encoder to obtain the second feature;

[0090] The third feature extraction unit is used to extract semantic features from the original document in the online video conference using a text embedding encoder to obtain the third feature;

[0091] The feature fusion unit is used to perform feature fusion based on the first feature, the second feature and the third feature to obtain multimodal fused features.

[0092] In some specific embodiments, the feature fusion module 11 may specifically include:

[0093] The backtracking window determination unit is used to determine the current window based on the multimodal fusion features corresponding to the current time, and to determine the backtracking window using the multimodal fusion features before the current time and the backtracking length parameter;

[0094] The triple window determination unit is used to determine the look-ahead window based on the multimodal fusion features and look-ahead length parameters after the current time, and to determine the triple time-series window using the current window, the backtracking window and the look-ahead window.

[0095] In some specific embodiments, the weight adjustment module 12 may specifically include:

[0096] The mapping relationship learning unit is used to learn the mapping relationship between features and causal weights based on the multimodal features corresponding to the current time and using a lightweight neural network.

[0097] The weight value determination unit is used to assign weights to the look-ahead window, the current window, and the backtracking window based on the mapping relationship, so as to obtain the initial weight values ​​corresponding to the look-ahead window, the current window, and the backtracking window respectively, and to determine the weight sum using the initial weight values.

[0098] The weight sum judgment unit is used to judge whether the initial weight value satisfies the preset constraint condition based on the weight sum; the preset constraint condition is that the weight sum is consistent with the first target value, and each of the initial weight values ​​is not less than the second target value;

[0099] The first weight determination unit is used to determine the initial weight value as the adjusted weight if the preset constraint condition is met.

[0100] The second weight determination unit is used to normalize and adjust the initial weight value if the preset constraint condition is not met, so as to obtain a new initial weight value and a weight sum, and then jump to the step of determining whether the initial weight value meets the preset constraint condition based on the weight sum, until the preset constraint condition is met, so as to obtain the adjusted weight.

[0101] In some specific embodiments, the timing correction module 13 may specifically include:

[0102] The matrix indexing unit is used to construct each modality pair based on the original audio, the original video stream, and the original document, and to index the preset temporal offset matrix using the modality pair and the speaker's speech identifier to obtain the temporal offset mode of the speaker under different modality pairs.

[0103] The timing correction unit is used to determine the timing offset value using the timing offset mode, and to correct the original timing corresponding to the triple timing window based on the timing offset value and using a preset correction formula to obtain the corrected timing.

[0104] In some specific embodiments, the feature alignment module 14 may specifically include:

[0105] The attention result determination unit is used to determine the first attention result corresponding to the look-ahead window, the second attention result corresponding to the current window, and the third attention result corresponding to the backtracking window based on the adjusted weights and the corrected time sequence, respectively.

[0106] Attention result masking unit, used to mask the first attention result based on a preset causal masking mechanism to obtain the masked first attention result;

[0107] The target result determination unit is used to determine the target attention result using the first attention result after masking, the second attention result, and the third attention result.

[0108] In some specific embodiments, the feature alignment module 14 may specifically include:

[0109] The temporal relationship alignment unit is used to perform global-level alignment of the temporal relationship between the overall structure of the online video conference and the presentation chapters of the original document based on the target attention result, so as to obtain a first alignment result;

[0110] The semantic alignment unit is used to perform paragraph-level alignment of the semantics between the presentation page corresponding to the original document and the speech segment corresponding to the original audio using the target attention result, so as to obtain a second alignment result;

[0111] An element alignment unit is used to perform element alignment between target document elements in the original document and target speech elements in the original audio based on the target attention result, so as to obtain a third alignment result.

[0112] The alignment result determination unit is used to determine the multimodal alignment result using the first alignment result, the second alignment result, and the third alignment result.

[0113] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the causal mask-based multimodal data alignment method for video conferencing disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0114] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0115] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0116] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the causal mask-based video conferencing multimodal data alignment method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0117] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for causal masking-based multimodal data alignment in video conferencing. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0119] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0121] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for aligning multimodal data in video conferencing based on causal masking, characterized in that, include: A preset encoder is used to extract and fuse features from the original audio, original video stream and original document in the online video conference to obtain multimodal fusion features. Based on the multimodal fusion features, temporal division is performed to obtain a triple temporal window including a look-ahead window, a current window and a backtracking window. Based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, the initial weight values ​​corresponding to the triple time window are determined. The initial weight values ​​are then normalized and adjusted using preset constraints to obtain the adjusted weights. The preset timing offset matrix is ​​indexed using the speaker's speaking identifier, and the original timing corresponding to the triple timing window is corrected based on the obtained indexing result to obtain the corrected timing. Based on the corrected temporal sequence and the adjusted weights, and using a preset causal masking mechanism, the target attention result corresponding to the triple temporal window is determined, and the multimodal fusion feature is fused in a multi-level alignment based on the target attention result to obtain the multimodal alignment result.

2. The video conferencing multimodal data alignment method based on causal masking according to claim 1, characterized in that, The process of extracting and fusing features from the original audio, video stream, and document in an online video conference using a preset encoder to obtain multimodal fusion features includes: Acoustic features are extracted from the raw audio of an online video conference using an audio encoder to obtain the first feature; Visual features are extracted from the raw video stream in online video conferencing based on the video encoder to obtain the second feature; A text embedding encoder is used to extract semantic features from the original documents in an online video conference to obtain a third feature; Feature fusion is performed based on the first feature, the second feature, and the third feature to obtain multimodal fusion features.

3. The video conferencing multimodal data alignment method based on causal masking according to claim 1, characterized in that, The temporal segmentation based on the multimodal fusion features yields a triple temporal window comprising a look-ahead window, a current window, and a backtracking window, including: The current window is determined based on the multimodal fusion features corresponding to the current moment, and the backtracking window is determined using the multimodal fusion features prior to the current moment and the backtracking length parameter. The look-ahead window is determined based on the multimodal fusion features and look-ahead length parameters after the current time, and a triple time-series window is determined using the current window, the backtracking window, and the look-ahead window.

4. The video conferencing multimodal data alignment method based on causal masking according to claim 1, characterized in that, The process involves determining initial weight values ​​corresponding to the triple temporal window based on the multimodal fusion features corresponding to the current time moment and utilizing a lightweight neural network. Then, the initial weight values ​​are normalized and adjusted using preset constraints to obtain the adjusted weights, including: Based on the multimodal features corresponding to the current moment, a lightweight neural network is used to learn the mapping relationship between features and causal weights; Based on the mapping relationship, weights are assigned to the look-ahead window, the current window, and the backtracking window respectively to obtain initial weight values ​​corresponding to the look-ahead window, the current window, and the backtracking window respectively, and the weight sum is determined using each initial weight value; Based on the weights, it is determined whether the initial weight values ​​meet the preset constraints; the preset constraints are that the weights are consistent with the first target value, and each of the initial weight values ​​is not less than the second target value. If the preset constraint conditions are met, the initial weight value will be determined as the adjusted weight. If the preset constraints are not met, the initial weight value is normalized to obtain a new initial weight value and a weight sum, and the process jumps to the step of determining whether the initial weight value meets the preset constraints based on the weight sum, until the preset constraints are met to obtain the adjusted weight.

5. The video conferencing multimodal data alignment method based on causal masking according to claim 1, characterized in that, The step of indexing a preset timing offset matrix using the speaker's speech identifier and correcting the original timing corresponding to the triple timing window based on the obtained indexing result to obtain the corrected timing includes: Based on the original audio, the original video stream, and the original document, each modality pair is constructed. The modality pair and the speaker's speech identifier are used to index the preset temporal offset matrix to obtain the temporal offset mode of the speaker under different modality pairs. The timing offset value is determined using the timing offset mode. Based on the timing offset value and using a preset correction formula, the original timing corresponding to the triple timing window is corrected to obtain the corrected timing.

6. The video conferencing multimodal data alignment method based on causal masking according to claim 1, characterized in that, The step of determining the target attention result corresponding to the triple temporal window based on the corrected temporal sequence and the adjusted weights, and using a preset causal masking mechanism, includes: Based on the adjusted weights and the corrected time series, the first attention result corresponding to the look-ahead window, the second attention result corresponding to the current window, and the third attention result corresponding to the backtracking window are determined respectively. The first attention result is masked based on a preset causal masking mechanism to obtain the masked first attention result. The target attention result is determined using the first attention result, the second attention result, and the third attention result after masking.

7. The video conferencing multimodal data alignment method based on causal masking according to any one of claims 1 to 6, characterized in that, The step of performing multi-level alignment fusion on the multimodal fusion features based on the target attention result to obtain the multimodal alignment result includes: Based on the target attention result, a global layer alignment is performed between the overall structure of the online video conference and the temporal relationship between the presentation chapters of the original document to obtain a first alignment result; The semantics between the demo page corresponding to the original document and the speech segment corresponding to the original audio are aligned at the paragraph level using the target attention result to obtain a second alignment result; Based on the target attention result, the target document elements in the original document and the target speech elements in the original audio are aligned to obtain a third alignment result; The multimodal alignment result is determined using the first alignment result, the second alignment result, and the third alignment result.

8. A video conferencing multimodal data alignment device based on causal masking, characterized in that, include: The feature fusion module is used to extract and fuse features from the original audio, original video stream and original document in the online video conference using a preset encoder to obtain multimodal fusion features. Based on the multimodal fusion features, temporal division is performed to obtain a triple temporal window including a look-ahead window, a current window and a backtracking window. The weight adjustment module is used to determine the initial weight value corresponding to the triple time window based on the multimodal fusion features corresponding to the current time and using a lightweight neural network, and to normalize and adjust the initial weight value using preset constraints to obtain the adjusted weight. The timing correction module is used to index the preset timing offset matrix using the speaker's speech identifier, and to correct the original timing corresponding to the triple timing window based on the obtained index result, so as to obtain the corrected timing. The feature alignment module is used to determine the target attention result corresponding to the triple temporal window based on the corrected temporal sequence and the adjusted weight and using a preset causal masking mechanism, and to perform multi-level alignment and fusion on the multimodal fusion features based on the target attention result to obtain the multimodal alignment result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the causal mask-based video conferencing multimodal data alignment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the causal mask-based video conferencing multimodal data alignment method as described in any one of claims 1 to 7.

Images