Time sequence multi-mode large model event understanding method for high-speed heavy fog scene

Abstract

The invention relates to the technical field of intelligent traffic, and discloses a high-speed heavy fog scene-oriented time sequence multi-mode large model event understanding method, which comprises the following steps of: 1, splitting a video stream of a high-speed scene into independent frame sequences according to a time sequence, and carrying out data preprocessing, 2, inputting video time sequence data into a defogging network for processing, and 3, carrying out defogging processing on the video time sequence data. The method comprises the following steps of 1, defogging video data, 2, combining the defogged video data with related text data to form time sequence + text multi-modal data input, 4, describing a time sequence-language hierarchical structure by using a dual-scale node to realize structure alignment, 5, guiding a logic relationship between time sequence data and language prompt by using a directed edge to realize logic alignment, and 6, carrying out structure alignment. And inputting the processed multi-modal data into an event understanding module, identifying and classifying various events in the scene, and realizing semantic understanding.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a time-series multimodal large model event understanding method for high-speed foggy scenes. Background Art

[0002] Dense fog poses a significant threat to traffic safety in high-speed driving scenarios, such as on highways. Due to the drastic reduction in visibility caused by fog, drivers struggle to detect vehicles ahead, obstacles, or road signs, significantly increasing the risk of rear-end collisions, same-direction scrapes, and vehicles running off the road. Existing strategies primarily rely on passive measures such as speed limits and road closures. While these strategies can provide some assurance of safety, they significantly reduce road efficiency and fail to meet the technical balance between safety and efficiency required by modern transportation systems.

[0003] On a technical level, traditional event understanding methods rely heavily on single-modal data, such as video or sensor information. However, in complex environments (such as foggy weather), single-modal data can be severely interfered with, making it difficult to provide reliable information, significantly reducing the accuracy of environmental perception and event judgment. For example, video data may lose target details due to haze, and radar signals may produce false detections due to multipath effects, making it difficult to guarantee the reliability of single-modal systems.

[0004] To overcome these limitations, multimodal data fusion technology has become a research hotspot. By integrating heterogeneous data from multiple sources, such as video, audio, and text, the system can capture subtle features that are difficult to discern using a single modality (such as the correlation between a car horn and a flashing taillight in fog), thereby improving the multidimensional understanding of complex scenes.

[0005] Therefore, image defogging technology has become a key means to improve video quality in foggy environments. Traditional image defogging methods are mainly divided into two categories:

[0006] Image enhancement-based methods and image restoration-based methods. Image enhancement-based methods improve visual effects by adjusting the contrast and brightness of images, but they are difficult to eliminate the damage caused by haze to scene details;

[0007] The image restoration-based method uses the atmospheric scattering model to try to reconstruct the fog-free image, which has achieved better results, but it is highly sensitive to parameters and has high computational complexity.

[0008] In recent years, the development of deep learning technology has brought new opportunities for image dehazing. Through models such as convolutional neural networks (CNN), researchers can learn complex dehazing mapping relationships from large amounts of data to achieve more efficient dehazing processing. However, existing deep learning methods mostly focus on single-frame static image processing and do not fully consider the temporal characteristics of videos (such as motion continuity and inter-frame redundancy). This leads to problems such as flickering and color distortion in the dehazed video, affecting the stability of the subsequent event understanding module. Summary of the Invention

[0009] The purpose of the present invention is to provide a temporal multimodal large-model event understanding method for high-speed foggy scenes. By defogging video time series data and fusing multimodal features, the event perception capability under complex weather conditions is enhanced. At the same time, the temporal-language hierarchy is described by dual-scale nodes to achieve structural alignment of temporal information and language prompts. Directed edges are used to guide the logical association between multimodal data, thereby improving the accuracy of event understanding. This provides a solid theoretical basis and technical support for intelligent traffic safety management in high-speed foggy environments.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A method for understanding time-series multimodal large-scale events in high-speed foggy scenes includes the following steps:

[0012] S1, splits the video stream of the high-speed scene into independent frame sequences according to the time sequence and performs data preprocessing;

[0013] S2, inputs the video time series data into the dehazing network for processing;

[0014] S3, combines the dehazed video data with the relevant text data to form a time series + text multimodal data input;

[0015] S4, using dual-scale nodes to describe the temporal-linguistic hierarchy and achieve structural alignment;

[0016] S5, uses directed edges to guide the logical relationship between time series data and language cues to achieve logical alignment;

[0017] S6, inputs the processed multimodal data into the event understanding module to identify and classify various events in the scene and achieve semantic understanding.

[0018] Preferably, the step S1 specifically includes:

[0019] S101, splitting the continuous high-speed fog video stream into independent frame sequences in chronological order and adding an accurate timestamp to each video frame to maintain the integrity of the inter-frame timing information;

[0020] S102 , preprocessing the frame sequence, eliminating pathological features caused by atmospheric scattering and lighting conditions by calculating projection loss and reflection consistency loss, and optimizing the decomposition quality of the illumination component and the reflectance component based on Retinex loss to improve image contrast and correct color distortion.

[0021] Preferably, the step S2 specifically includes:

[0022] S201, automatically learning reflectivity information and illumination information from an input video frame sequence using a shared prior estimator;

[0023] S202, integrating features through a dynamic wavelet separable convolution module;

[0024] S203, using an adaptive color corrector to restore the color information of the image and solve the problem of unclear colors.

[0025] Preferably, the step S3 specifically includes:

[0026] S301, performing text information processing and encoding text data using a pre-trained language model;

[0027] S302, the dehazed video features and the encoded text features are jointly modeled through a multimodal fusion mechanism. Combining the temporal features of the video data with the semantic description of the text information, the model can understand the temporal changes of events in the video and classify, predict, or generate supplementary content for the events based on the text information.

[0028] Preferably, the step S4 specifically includes:

[0029] The features after multimodal fusion are input into the dual-scale contextual alignment graph neural network, where structural alignment is first performed and dual-scale nodes are used to integrate modal information.

[0030] Preferably, the step S5 specifically includes:

[0031] S501, constructing directed edges to establish semantic associations between time series data and language prompts, where the time series data is used as an upstream information source and all directed edge weights are set to fixed values;

[0032] S502: Construct a directed edge set to achieve cross-modal information transfer and update the node embedding matrix through the graph neural network to preserve the timing details and macro-logical information;

[0033] S503 introduces a learnable interaction mechanism to aggregate macro-logical information into time series nodes, ensuring that key task details are not lost;

[0034] S504, finally generates an embedding representation with clear structure and semantic coherence to support large language models to understand time sequence tasks.

[0035] Preferably, the step S6 specifically includes:

[0036] S601: Input the processed multimodal features into a deep neural network for hierarchical feature extraction, and generate high-level semantic feature representation through multi-layer nonlinear transformation;

[0037] S602: Classify and identify the semantic features using a fully connected layer and a Softmax function.

[0038] Preferably, the step S6 further includes:

[0039] S603 introduces the attention mechanism to capture the long-distance dependencies between features, dynamically calculates the attention weights through trainable parameters, and integrates contextual information to generate enhanced semantic representations to achieve deep semantic analysis and reasoning of events.

[0040] Preferably, in step S102, the reflection consistency loss is calculated based on the physical property constraint, and the physical consistency of the decomposition result is optimized by comparing the reflectivity map output by the reflectivity estimator with the reflectivity map of the coarse dehazed image frame.

[0041] Preferably, the step S4 specifically includes the following steps:

[0042] S401, using fine-grained graph neural networks to treat each independent tag in the multimodal fusion feature as a node, constructing fine-grained time-series-language associations;

[0043] S402, aggregate continuous tags under the same modality into a single node through a coarse-grained graph neural network to form a macro feature representation within the modality.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention enhances event perception capabilities under complex weather conditions through defogging of video time series data and fusion of multimodal features. It also uses dual-scale nodes to describe the time series-language hierarchy, achieving structural alignment of time series information and language cues. It also guides the logical associations between multimodal data through directed edges, improving the accuracy of event understanding. This provides a solid theoretical foundation and technical support for intelligent traffic safety management in high-speed foggy environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a method for understanding time-series multimodal large-model events for high-speed foggy scenes proposed by the present invention. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0048] The present invention proposes to use a dual-scale context alignment method, aiming to activate and enhance the capabilities of large language models (LLMs) on time series data by aligning time series data with the language components in the language environment. This method uses multimodal fusion to enable the model to better understand the contextual information in the time series data, thereby improving the understanding and prediction capabilities of events; on the other hand, to address the problem of video quality degradation in foggy environments, this patent uses a new unpaired image defogging network, which significantly enhances the adaptability and generalization capabilities of the defogging method by automatically learning rich physical priors (such as reflectivity and illumination) from foggy images, making it possible to obtain clear video data in foggy environments, providing a reliable data foundation for subsequent event understanding. The present invention integrates advanced defogging technology and multimodal data fusion strategies, which can achieve accurate perception and understanding of traffic events in complex environments, thereby improving road safety.

[0049] Reference Figure 1 A method for understanding time-series multimodal large-scale events in high-speed foggy scenes includes the following steps:

[0050] (1) Split the high-speed scene video stream into independent frame sequences in chronological order and perform data preprocessing;

[0051] (2) Input the video time series data into the defogging network for processing;

[0052] (3) Combining the dehazed video data with relevant text data to form a multimodal data input of time series + text;

[0053] (4) Using dual-scale nodes to describe the temporal-linguistic hierarchy and achieve structural alignment;

[0054] (5) Use directed edges to guide the logical relationship between time series data and language prompts to achieve logical alignment;

[0055] (6) The processed multimodal data is input into the event understanding module to identify and classify various events in the scene to achieve semantic understanding.

[0056] The step (1) specifically refers to: first, splitting the continuous high-speed fog video stream into independent frame sequences V = {v1, v2, ..., v t}, where v iIt is the image information of the i-th frame, and an accurate timestamp is added to each video frame. The timestamp can be expressed as T = {t1, t2, ..., t N}, where t i is the timestamp of the i-th frame to maintain the integrity of the timing information; then, the video frame sequence is preprocessed to remove pathological features caused by atmospheric scattering and lighting conditions, which usually include uneven lighting, low contrast and color distortion. Specifically, the projection loss is calculated Among them, I hazy represents the original foggy image frame, I project Represents the image frame after removing redundant features; calculate redundant features Where δ represents the error, L represents the illumination component, and R represents the reflectivity component; the reflection consistency loss is calculated based on the physical properties of the object: L reflect =||R1-R2|| 2 , where R1 and R2 represent the reflectivity maps obtained by the reflectivity estimator and the coarse dehazed image frame, respectively; the Retinex loss used to measure the reflectivity and illumination decomposition quality is calculated: Among them, L init Represents the initially estimated illumination information, L represents the illumination component, and stopgrad(L) refers to preventing the gradient propagation of the illumination component L during the training process, which is used to guide the decomposition so that I project More precisely, R represents the reflectivity component.

[0057] The step (2) specifically refers to: first, using a shared prior estimator to automatically learn reflectivity and illumination information from the input video frame sequence. Specifically, a data-driven method is first used to automatically and iteratively learn adaptive priors from the high-speed fog and restored video frame sequences, and the image with high-speed fog is expressed as an equation through Retinex decomposition:

[0058]

[0059] Among them, I hazy and I clear They represent the high-speed fog image frame and the restored image frame in the iterative process, L hazy and L clear Represents the corresponding illuminated image frame. The two share the same reflectivity component R, ensuring the consistency of the surface features of the object in the image before and after defogging, thereby avoiding scene distortion caused by unreasonable illumination restoration. Retinex decomposition can effectively handle the illumination changes and reflection problems of the image, thereby obtaining a more realistic image.

[0060] Next, dynamic wavelet separable convolution is applied to effectively integrate low-frequency and high-frequency features. Specifically, the module first uses wavelet decomposition to separate the input feature map into high-frequency and low-frequency components. The low-frequency component usually represents the macroscopic structure of the image, while the high-frequency component focuses on the details of the image, such as edges and textures. This separation enables subsequent convolution operations to process different image features more accurately, which helps to improve the model's understanding of details and overall structure; small kernel convolution is applied to these two frequency components respectively, and the processed features are fused back to the original space through wavelet reconstruction. High-frequency features correspond to details, such as the edges or local features of small objects, while low-frequency features are the global background information of the image. Processing these two pieces of information separately can make the model perform better at different scales; in addition, a scaling component is introduced, which uses learnable weight parameters to dynamically scale the feature map after the wavelet transform. This learnable scaling mechanism can help the model respond to different image patterns more flexibly. , so that it can adapt to different input data more accurately, such as when dealing with different types of noise, illumination changes or motion blur; then, using a depth-wise separable convolution structure (compared to traditional convolution, depth-wise separable convolution greatly reduces the computational complexity by processing the spatial convolution and channel integration of each input channel separately. It reduces the number of parameters while maintaining the effectiveness of the model in local feature extraction), the traditional 3×3 convolution kernel is replaced with wavelet transform convolution, and point-by-point convolution (1×1 convolution) is used to integrate channel information, allowing deep convolution to process each input channel independently in the wavelet domain, thereby effectively capturing local features, and point-by-point convolution fuses these features into the final output. By integrating channel information through point-by-point convolution (1×1 convolution), the feature map can be more accurately fused in the channel dimension, which helps to capture the correlation between channels and improve the representation ability of the model. The specific formula is as follows: in, represents the input feature map of the i-th layer, Represents the feature map of the i-th layer after convolution and inverse wavelet transform, W represents the filter or weight parameter of the wavelet transform operation, which is used to extract low-frequency and high-frequency information from the image or feature map, WT represents the wavelet transform operation, and IWT represents the inverse wavelet transform, which is used to combine the low-frequency and high-frequency features and restore them to the original image space.

[0061]

[0062] in, and Represents the return value after applying wavelet transform to the low-frequency features of the previous layer. Specifically, represents all high-frequency features of the current layer, and Represents the low-frequency features of the current layer, and Represents the features after convolution.

[0063] Finally, the adaptive color corrector is used to restore the color information of the image and solve the problem of unclear colors. Specifically, the color mapping table C is defined F , and use the hazy feature image prediction F as pseudo ground truth to train the color estimation network Φ, the color estimation color map From the dehazed feature image F1, define the training loss: Among them C F Represents the RGB three-channel color estimation of the hazy image, Represents the RGB three-channel color estimation of the dehazed image during training.

[0064] To sum up, the total loss function can be expressed as: Among them, λ i Represents different loss weights.

[0065] The step (3) specifically refers to: firstly, performing text information processing, using a pre-trained language model to encode the text data, especially the description, instruction or event label related to the video content, to obtain the semantic representation of the text T sem , which provides high-quality text representation for subsequent multimodal fusion, ensuring that video and text information can be efficiently integrated together; then, multimodal fusion is performed to obtain the joint representation Z: Z = f(V seq ,T sem ), where f represents the multimodal fusion function, V seq To represent the features of the video sequence, the visual features and text features obtained in the previous steps are jointly modeled through a multimodal fusion mechanism. By combining the temporal features of the video data with the semantic description of the text information, the model can understand the temporal changes of events in the video and classify, predict or generate supplementary content for the events based on the text information.

[0066] The step (4) specifically refers to: inputting the features after multimodal fusion into the dual-scale context alignment graph neural network, first performing structural alignment, and using dual-scale nodes to integrate modal information. Fine-grained neural network G F Treating each token as a node enables the model to capture fine-grained semantic information, while the coarse-grained neural network G C Treating consecutive tags belonging to the same modality as a single node helps the model capture macroscopic semantic structures and long-range dependencies. C Using two learnable linear layers f ε and To embed time series tokens and language tokens into M-dimensional space, the network can better understand the relationship between time series data and language data, and combine the information of these two modalities in the subsequent processing process, which can be formalized as ε = f ε (e1,e2,…,e n ); Then, we convert it into a coarse-grained representation as follows: It is a coarse-grained representation of time series tokens. It is a coarse-grained representation of the language token, and each element in the formula is regarded as G C By converting the input into a coarse-grained representation, the model can reduce information redundancy and focus on higher-level semantic structures rather than the details of each individual token.

[0067] The step (5) specifically refers to: using directed edges to ensure the correct semantic association between different components during the logical alignment process. In the prompt “Use previous data to predict future sequences:”, “previous data:” corresponds to time series (TS) data, and information needs to be extracted from the time series data. Therefore, for G C , construct directed edges Indicates that time series data is the upstream information source of the prompt, and the weights of all directed edges are set to 1. F , construct a directed edge set: {E ij :e i →z j |i∈{1,…,n},j∈{1,…,m}} to pass the information of each time series token to the prompt token, where e i Represents the i-th token of time series data, z j Represents the jth token of the language data; in addition, the constraint is imposed where w ij For E ij edge weights to emphasize the time series as a whole, It is proportional to the cosine similarity of the corresponding node. The updated G F and G C The node embedding matrices are expressed as and The update strategy based on graph neural network is calculated as follows: in, is the transpose of the node embedding matrix before updating, W k ∈R M×M is the learnable matrix of the graph neural network, σ is the ReLU activation function, A′k =A k +I is the adjacency matrix weighted by the identity matrix, D k is a diagonal matrix satisfying D k,ii =∑ j A′ k,ij .G F The structure preserves the time series details, while G C Provides macro-logical information. However, G C The key details required for time series tasks may be lost, so a learnable interaction mechanism ΔN is introduced to transform G C The macro information is transmitted to G F , the calculation formula is as follows: ΔN=W C→F N C Γ C→F ,in, Assign a 0-1 matrix, Γ ij =1 means G F The jth node in G is aggregated to C The i-th node in W C→F ∈R M×M is a learnable weight matrix. Finally, G F The embed is updated to:

[0068] After the structure and logic are aligned, Maintaining clear structure and coherent semantic information helps LLM understand time series tasks and activate potential capabilities. and is input into the pre-trained LLM, the dual-scale contextual aligned graph neural network can be flexibly integrated into various layers of the pre-trained LLM and only requires f when applied for the first time. ε and Get a coarse-grained graph neural network and output G F The branch is used to calculate the mean squared error loss relative to the ground truth.

[0069] The step (6) specifically refers to: firstly, feature extraction is performed, the aligned multimodal feature representation is input into the deep neural network, and high-level semantic features are extracted. The feature representation after passing through the L-layer neural network is H=f L (f L-1 (…f1(F)…)), where f iRepresents the nonlinear transformation function of the i-th layer. This feature extraction mechanism enables the model to effectively understand the complex relationships in the input data and provide strong feature support for subsequent event classification and recognition. Then, using the extracted high-level features H, the fully connected layer and the Softmax function are used to classify and recognize events. Assuming the number of event categories is C, the classification result P = Softmax(WH+b), where W is the weight matrix and b is the bias vector. Finally, in order to deeply understand the semantic information of the event, the attention mechanism is introduced to capture the long-distance dependency between features. The attention weight α i The calculation of is as follows: in, u is the context vector, W h and W s is a trainable parameter, h i is the hidden state of the i-th time step, and s is the context information. The final semantic representation r is:

[0070] As can be seen from the above technical solutions, the beneficial effects of the present invention are as follows: (1) A dehazing network is used to enhance video clarity, improve contrast and color reproduction capabilities, and ensure the availability of visual information in inclement weather. (2) Combining video and text information, a dual-scale graph neural network and a directed edge logic alignment mechanism are used to achieve deep fusion of multimodal data, improving the logic and coherence of event prediction. (3) By optimizing computational efficiency and reducing computational costs through wavelet transform and depthwise separable convolution, the method can process videos and understand events in real time in a dynamic environment with high-speed driving.

[0071] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A temporal multimodal large-model event understanding method for high-speed foggy scenes, characterized by: The following steps are involved: S1, splits the video stream of the high-speed scene into independent frame sequences according to the time sequence and performs data preprocessing; S2, inputs the video time series data into the dehazing network for processing; S3, combines the dehazed video data with the relevant text data to form a time series + text multimodal data input; S4, using dual-scale nodes to describe the temporal-linguistic hierarchy and achieve structural alignment; S5, uses directed edges to guide the logical relationship between time series data and language cues to achieve logical alignment; S6, inputs the processed multimodal data into the event understanding module to identify and classify various events in the scene and achieve semantic understanding.

2. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 1 is characterized by: The step S1 specifically includes: S101, splitting the continuous high-speed fog video stream into independent frame sequences in chronological order and adding an accurate timestamp to each video frame to maintain the integrity of the inter-frame timing information; S102 , preprocessing the frame sequence, eliminating pathological features caused by atmospheric scattering and lighting conditions by calculating projection loss and reflection consistency loss, and optimizing the decomposition quality of the illumination component and the reflectance component based on Retinex loss to improve image contrast and correct color distortion.

3. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 1 is characterized by: The step S2 specifically includes: S201, automatically learning reflectivity information and illumination information from an input video frame sequence using a shared prior estimator; S202, integrating features through a dynamic wavelet separable convolution module; S203: Using an adaptive color corrector to restore the color information of the image, thereby solving the problem of unclear colors.

4. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 1 is characterized by: The step S3 specifically includes: S301, performing text information processing and encoding text data using a pre-trained language model; S302, the dehazed video features and the encoded text features are jointly modeled through a multimodal fusion mechanism. Combining the temporal features of the video data with the semantic description of the text information, the model can understand the temporal changes of events in the video and classify, predict, or generate supplementary content for the events based on the text information.

5. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 1 is characterized by: The step S4 specifically includes: The features after multimodal fusion are input into the dual-scale contextual alignment graph neural network, where structural alignment is first performed and dual-scale nodes are used to integrate modal information.

6. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 1 is characterized by: The step S5 specifically includes: S501, constructing directed edges to establish semantic associations between time series data and language prompts, where the time series data is used as an upstream information source and all directed edge weights are set to fixed values; S502: Construct a directed edge set to achieve cross-modal information transfer and update the node embedding matrix through the graph neural network to preserve the timing details and macro-logical information; S503 introduces a learnable interaction mechanism to aggregate macro-logical information into time series nodes, ensuring that key task details are not lost; S504, finally generates an embedding representation with clear structure and semantic coherence to support large language models to understand time sequence tasks.

7. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 1 is characterized by: The step S6 specifically includes: S601: Input the processed multimodal features into a deep neural network for hierarchical feature extraction, and generate high-level semantic feature representation through multi-layer nonlinear transformation; S602: Classify and identify the semantic features using a fully connected layer and a Softmax function.

8. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 6 is characterized by: The step S6 further includes: S603 introduces the attention mechanism to capture the long-distance dependencies between features, dynamically calculates the attention weights through trainable parameters, and integrates contextual information to generate enhanced semantic representations to achieve deep semantic analysis and reasoning of events.

9. The method for understanding time-series multimodal large-model events in high-speed foggy scenarios according to claim 2 is characterized by: In step S102, the reflection consistency loss is calculated based on the physical property constraint, and the physical consistency of the decomposition result is optimized by comparing the reflectivity map output by the reflectivity estimator with the reflectivity map of the coarse dehazed image frame.

10. The method for understanding time-series multimodal large-model events in high-speed foggy scenes according to claim 5 is characterized by: The step S4 specifically includes the following steps: S401, using fine-grained graph neural networks to treat each independent tag in the multimodal fusion feature as a node, constructing fine-grained time-series-language associations; S402, aggregate continuous tags under the same modality into a single node through a coarse-grained graph neural network to form a macro feature representation within the modality.

Images