A multimodal large model context understanding method for high-speed abnormal exposure scenarios

Abstract

The present invention relates to the field of smart transportation technology and discloses a multimodal large-model context understanding method for high-speed abnormal exposure scenarios. In the first step, a rotational position embedding (RoPE) variant is used to encode the complex spatiotemporal structure of video data to obtain temporal features. In the second step, the temporal features are combined with text features to form a multimodal data input of temporal + text. In the third step, the weights of the Laplacian pyramid decomposition kernel are dynamically predicted based on the input multimodal data features. In the fourth step, the features are refined and transformed at each layer of the feature pyramid, and reconstructed to the original resolution layer by layer to generate a corrected image. In the fifth step, during the training process, the model parameters are optimized by combining pixel reconstruction loss, adversarial loss, and kernel loss. In the sixth step, the corrected image is input into an event understanding module to identify and classify various events in the scene to achieve semantic understanding.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a multimodal large-model context understanding method for high-speed abnormal exposure scenarios. Background Art

[0002] With the rapid development of smart transportation, real-time processing of image and video data in complex and dynamic environments with abnormal exposure has become a pressing challenge. This is especially true in high-speed scenarios, where significant lighting changes, motion blur, and exposure distortion exist, posing significant challenges to traditional image processing and understanding technologies.

[0003] Currently, image processing methods for abnormal exposure scenes mainly focus on dehazing, exposure correction, and noise suppression. However, these technologies often face problems of insufficient accuracy and poor real-time performance when processing high-speed dynamic scenes. Although traditional methods based on convolutional neural networks (CNNs) have achieved remarkable results in static image processing, they still face significant processing bottlenecks in dynamic video scenes, especially those with high-speed motion or large-scale lighting changes. In addition, existing multimodal data fusion technologies also have limitations in modeling spatiotemporal features and information transmission. In particular, in complex contextual understanding tasks, how to effectively integrate information from different modalities (such as video and text) remains an unresolved problem.

[0004] The multimodal large-model contextual understanding method proposed in this paper for high-speed abnormal exposure scenarios can more accurately capture the spatiotemporal features in video data, especially in scenes with high speed and large-scale illumination changes, by innovatively combining a variant of rotational position embedding (RoPE) and the technology of dynamically predicted Laplacian pyramid decomposition kernel weights. At the same time, through the multi-level feature extraction and fusion in the feature pyramid, the correction accuracy and stability of images and videos are improved, ensuring high-quality image reconstruction. In addition, the use of a comprehensive multi-component loss function for optimization effectively overcomes the accuracy and real-time bottlenecks of traditional methods when dealing with complex dynamic scenes, thereby improving the collaborative understanding ability of multimodal information, which is particularly suitable for complex environments with high speed, dynamics and abnormal exposure. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and propose a multimodal large-model context understanding method for high-speed abnormal exposure scenarios. By generating a dynamic kernel, the multimodal features of abnormal exposure scene images are personalized, effectively improving the processing accuracy and event understanding capabilities of image and video data.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A multimodal large-model context understanding method for high-speed abnormal exposure scenarios includes the following steps:

[0008] S1, encodes the complex spatiotemporal structure of video data and extracts temporal features;

[0009] S2, fusing the temporal features with the text features to form multimodal input data;

[0010] S3, dynamically predicts the weights of the Laplacian pyramid decomposition kernel based on multimodal input data;

[0011] S4, generates the corrected image through multi-level feature pyramid decomposition and reconstruction;

[0012] S5, optimize model parameters by combining multi-component loss function;

[0013] S6, based on the corrected image, recognizes and classifies scene events to achieve semantic understanding.

[0014] Preferably, the step S1 includes:

[0015] S101, obtaining a spatiotemporal sequence of video data, including continuous video frames on a time axis and their spatial pixel distribution;

[0016] S102, constructing a rotation position embedding variant encoder, and performing asymmetric dimension mapping of spatiotemporal positions through a low-frequency time allocation strategy;

[0017] S103, generating a spatiotemporal position encoding matrix using a diagonal layout, so that the relative position relationship between the visual markers and the rotational position encoding of the text markers maintain symmetry constraints;

[0018] S104: extracting temporal features to obtain visual feature vectors.

[0019] Preferably, the step S102 specifically includes:

[0020] The time dimension is assigned to the high-order part of the embedding vector, and the number of dimensions is positively correlated with the length of the time series;

[0021] The spatial dimensions are assigned to the low-order part of the embedding vector, and the horizontal and vertical positions occupy adjacent dimensions in an interleaved pattern.

[0022] Preferably, step S2 includes:

[0023] S201, performing video text information processing, encoding the text data using a pre-trained language model, and generating a text feature vector;

[0024] S202: Jointly modeling the visual feature vector and the text feature vector through a multimodal fusion mechanism to form multimodal input data.

[0025] Preferably, step S3 includes:

[0026] S301, receiving multimodal input data and extracting its joint feature representation;

[0027] S302, constructing a super network, the super network including an asymmetric convolutional layer group with gradually increasing number of channels;

[0028] S303, performing multi-level downsampling on the multimodal features through the hypernetwork to generate a dynamic weight matrix related to the input features, wherein the weight matrix is ​​used to parameterize the Laplacian pyramid decomposition kernel;

[0029] S304: Process the input data using a dynamically weighted Laplacian pyramid decomposition kernel and output a multi-scale decomposition result.

[0030] Preferably, the super network comprises alternating asymmetric convolutional layer groups, each group comprising at least one horizontal convolution kernel (3×1) and a vertical convolution kernel (1×3).

[0031] Preferably, step S4 includes:

[0032] S401, inputting an original image, constructing a feature pyramid comprising multiple resolution levels, wherein the resolution of each level decreases according to a preset ratio;

[0033] S402, perform iterative processing on each level:

[0034] Dynamically generate convolution kernels based on the current level features, and perform low-pass filtering on the image to generate a predicted image;

[0035] Upsample the predicted image to the next higher resolution and calculate the high-frequency residual with the original image;

[0036] S403 , aggregating high-frequency residuals of all levels and low-frequency predictions of the final level to reconstruct a corrected output image.

[0037] Preferably, step S5 includes:

[0038] S501, constructing a generator network, wherein the generator network is used to output a corrected image;

[0039] The generator network is trained using the following joint loss function:

[0040] Pixel reconstruction loss function: measures the pixel-level difference between the generated image and the target image;

[0041] Adversarial loss function: measures the ability of generated images to deceive the discriminator;

[0042] Kernel constraint loss function: constrains the similarity between the kernel generated by the hypernetwork and the preset target kernel;

[0043] S502. Back-propagate the gradient according to the joint loss function to synchronously optimize the parameters of the generator and the hypernetwork.

[0044] Preferably, step S6 includes:

[0045] S601, receiving the corrected multimodal image data, and extracting high-level semantic features layer by layer through a deep neural network;

[0046] S602, inputting the high-level semantic features into a fully connected layer and a Softmax function for event classification and recognition;

[0047] S603, dynamically aggregates the contextual dependencies of semantic features based on the attention mechanism, captures the long-range dependencies between features, and generates the final representation vector of the event semantics;

[0048] S604: Execute event recognition and classification tasks according to the final representation vector.

[0049] Preferably, the deep neural network comprises alternately stacked nonlinear transformation layers and feature compression layers.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The multimodal large-model contextual understanding method proposed in this paper for high-speed abnormal exposure scenarios can more accurately capture the spatiotemporal features in video data, especially in scenes with high speed and large-scale illumination changes, by innovatively combining a variant of rotational position embedding (RoPE) and the technology of dynamically predicted Laplacian pyramid decomposition kernel weights. At the same time, through the multi-level feature extraction and fusion in the feature pyramid, the correction accuracy and stability of images and videos are improved, ensuring high-quality image reconstruction. In addition, the use of a comprehensive multi-component loss function for optimization effectively overcomes the accuracy and real-time bottlenecks of traditional methods when dealing with complex dynamic scenes, thereby improving the collaborative understanding ability of multimodal information, which is particularly suitable for complex environments with high speed, dynamics and abnormal exposure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a multimodal large-model context understanding method for high-speed abnormal exposure scenarios proposed by the present invention. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present application.

[0055] like Figure 1 As shown, a multimodal large model context understanding method for high-speed abnormal exposure scenarios includes the following steps in sequence:

[0056] (1) In view of the complex spatiotemporal structure of video data, a variant of Rotational Position Embedding (RoPE) is used to encode temporal features. RoPE is a method that improves the model's ability to capture temporal dependencies of sequences and can enhance the temporal understanding of video sequences, which is crucial for high-speed anomaly detection.

[0057] (2) Combining temporal features with textual features to form a multimodal data input of temporal + text. This integration enables the model to understand not only the visual context in the video, but also any textual information, thereby gaining a more comprehensive understanding of the scene, which is crucial for complex anomaly detection in high-speed environments.

[0058] (3) Dynamically predicting the weights of the Laplacian pyramid decomposition kernel based on the input multimodal data features, enabling the model to adaptively adjust its feature extraction process to better handle different levels of resolution in image and video data, thereby optimizing the processing of multi-resolution inputs;

[0059] (4) At each level of the feature pyramid, features are refined and transformed, and layer-by-layer reconstruction is performed to the original resolution to generate a corrected image. This multi-scale approach enables the model to retain detailed spatial information while focusing on higher-level abstractions, thereby improving the accuracy of anomaly detection at different scales.

[0060] (5) During training, pixel reconstruction loss, adversarial loss, and kernel loss are combined to optimize model parameters. This combined loss function (pixel reconstruction, adversarial, and kernel loss) during training is a powerful method to improve the model's capabilities in feature extraction and anomaly detection. Adversarial loss ensures that the generated features are realistic, while pixel reconstruction and kernel loss fine-tune the model to retain key details, especially for high-speed and high-risk scenarios.

[0061] (6) The corrected image is fed into the event understanding module to identify and classify various events in the scene and achieve semantic understanding. This step bridges the gap between raw visual data and actionable insights, enabling the model to understand and interpret complex events, which is a key aspect of real-time monitoring and anomaly detection in applications.

[0062] The step (1) specifically refers to: first, low-frequency time allocation (LTA) reduces the oscillation phenomenon in temporal modeling by assigning higher dimensions to the time axis, thereby better capturing long-range dependencies. The frame index in video and the token index in text are essentially different. This method scales the time index, using higher dimensions for temporal features in a longer context and lower dimensions for spatial features, which are resolution-limited and have a fixed range. At the same time, the dimensions of spatial features are staggered to avoid the gap between horizontal and vertical positions. Specifically, the horizontal position and vertical position Interlacing occupies the lower dimensions, while time Occupies higher dimensions, and for fair comparison, its values ​​are 48, 48, and 32 respectively; then, a diagonal layout is adopted to ensure spatial symmetry and maintain the relative position relationship between visual and textual labels. Specifically, the entire input is arranged along the diagonal line, and the 3D position of the central patch of each video frame is , other patches are offset in all directions; finally, the adjustable time interval (ATS) introduces a scaling factor , to align the temporal information between visual and textual tokens and adapt to contexts of different lengths. Specifically, let Represents the tag index, The total number of tokens representing the text, Indicates the total number of video frames, The total number of tokens that indicate the end of text. For text tokens , the time, horizontal and vertical indices are set directly to the original marker indices . For video input , Indicates the index of the current frame relative to the beginning of the video, and is pressed Controls the space in the time dimension, for the end text , the time, horizontal and vertical indices remain consistent, thus creating a linear progression. For a multimodal input, the input consists of Marked text, each frame with Patch Frames of subsequent video and with The end text of the tag is composed of or visual markers VideoRoPE location index Defined as:

[0063]

[0064] where w and h represent the horizontal and vertical indices of the visual patch within a frame, respectively.

[0065] The step (2) specifically refers to: firstly, processing the video text information, using the pre-trained language model to encode the text data, especially the description, instruction or event label related to the video content, to obtain the text information Next, the visual features and text features obtained in the previous steps are jointly modeled through a multimodal fusion mechanism (the design of the multimodal fusion function allows video and text information to be processed in the same context rather than separately. This helps the model establish associations between different modalities, making the processing results more consistent and accurate. Especially in video understanding, visual and text data are often complementary, and joint modeling can make full use of their complementary information to improve the performance of the model.) to obtain a joint representation ,in, represents the multimodal fusion function, Represents the temporal feature representation obtained in the previous step.

[0066] The step (3) specifically refers to: using HyperNetwork to process the multimodal features of the input and generate personalized kernels customized according to the features. Compared with fixed convolution kernels, dynamically generated kernels can better capture the complex relationships in the input data, especially in the case of multimodal data fusion (video and text features). The architecture consists of six convolutional layers forming a hypernetwork, and the number of channels in each convolutional layer gradually increases. By gradually increasing the number of channels in the convolutional layer, the network can gradually extract simple features from the lower layer to the complex feature representation of the higher layer, which helps to improve the model's learning ability for multi-scale and multi-dimensional data. These convolutional layers use irregular convolution kernels (such as 3×1 and 1×3) and pooling operations, which can better capture details and changes in different directions, enhance the network's perception of diverse spatial information, and achieve significant downsampling. The architectural details are as follows: the initial convolutional layer uses a convolution kernel of size 3×1, followed by ReLU activation and maximum pooling operations; the second convolutional layer uses a 1×3 convolution kernel, followed by ReLU activation and maximum pooling to continue the downsampling process; the third and fourth convolutional layers use 3×1 and 1×3 convolution kernels, each followed by a ReLU activation layer and a pooling layer; the fifth convolutional layer uses a 3×3 convolution kernel, followed by maximum pooling and a final 1×1 convolution, which reduces the dimension precisely to 5×5. Compared with large convolution kernels, 1×1 convolution kernels only adjust the number of channels. The precise control of feature dimensions helps avoid excessive computational burden.

[0067] The step (4) specifically refers to: input image ,in Indicates the image height, Represents the image width, first calculate the low-pass prediction image , by applying a dynamically generated kernel Perform convolution operation, the formula is as follows:

[0068] ,in, Represents pixels The surrounding neighborhood, is the kernel weight; then, to ensure reversible reconstruction, the high-frequency residual is calculated as follows: ,in, The function uses bicubic interpolation to Upsample back to the original size; after that, this process will be iteratively applied to subsequent levels of images, further decomposing the image into multiple low-frequency and high-frequency components. By processing at different scales, the model can analyze the image content from both global and local levels, avoiding the information loss that may be caused by processing the image at only a single scale. This method is particularly suitable for processing complex image data, such as scenes with high-speed motion and large changes in lighting. The image is decomposed through multiple pyramid levels, which include three levels in this architecture, corresponding to resolution , ,and , the specific mathematical expression is as follows:

[0069]

[0070]

[0071]

[0072]

[0073] in, is the downsampling function, and the final pyramid is composed of Composed of multiple levels, these pyramid-level features help express low-frequency and high-frequency features of images at different scales.

[0074] The step (5) specifically refers to: combining a multi-component loss function of pixel reconstruction loss, adversarial loss and kernel loss to effectively train the generator. The pixel reconstruction loss measures the difference between the generated image and the target highlight image, ensuring the consistency of the generated image with the target image at the pixel level and ensuring the restoration of details. It is usually calculated using the mean square error (MSE): , Indicates the generated image, represents the target image; the adversarial loss measures the ability of the generated image to deceive the discriminator, and its specific form depends on the Type; kernel loss is used to constrain the kernel generated by the hypernetwork to be close to the target kernel (such as Gaussian kernel), and is also calculated using mean square error: , the target kernel here Refers to the Gaussian kernel used in the LPTN framework. The loss is used to update the model so that the kernel predicted by the network is close to the Gaussian kernel, but is also dynamic to the input. Ultimately, the total loss function of the generator is: ,in, and The contributions of pixel reconstruction loss and kernel loss to the total loss are controlled separately. By minimizing the above loss functions, the parameters of the generator and hypernetwork (if applicable) are optimized to improve the quality and realism of the generated images.

[0075] The step (6) specifically refers to: firstly, feature extraction is performed, the modified multimodal feature representation is input into the deep neural network, and high-level semantic features are extracted layer by layer. After the layer neural network, the feature representation is ,in Indicates the The nonlinear transformation function of the layer, through the multi-layer neural network, the model can gradually abstract from low-level features to high-level semantic features, so that the final feature representation Ability to accurately capture complex patterns in input data, which is particularly important for processing multimodal data (such as a combination of video and text) because the relationships between these data are often complex and dynamic; then, using the extracted high-level features , event classification and recognition are performed through the fully connected layer and Softmax function. Let the number of event categories be , the classification result ,in, is the weight matrix, is the bias vector; Finally, in order to deeply understand the semantic information of the event, the attention mechanism is introduced to capture the long-distance dependencies between features. At the same time, the attention mechanism can automatically select the features related to the task and assign them higher weights. This method enables the network to dynamically focus on the most critical information, improving the accuracy of the classification task. The attention weight The calculation of is as follows: ,in , is the context vector, and is a trainable parameter, For the The hidden state of time steps, is the context information. The final semantic representation r for: .

[0076] Compared to existing technologies, the proposed method effectively addresses the problem of understanding the spatiotemporal structure of image and video data in complex scenarios by combining a variant of Rotational Position Embedding (RoPE) with dynamically predicted Laplacian pyramid decomposition kernel weights. Through feature extraction and transformation, pyramid decomposition, and optimization of multi-component loss functions, it improves image and video processing accuracy and enhances event understanding capabilities in multimodal information fusion. By optimizing the training loss function, it further improves the quality and authenticity of generated images. This method is widely applicable to multimodal data analysis in autonomous driving and high-speed, abnormally exposed environments, demonstrating strong real-time, accuracy, and robustness.

[0077] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multimodal large-model context understanding method for high-speed abnormal exposure scenarios, characterized by: The following steps are involved: S1, encodes the complex spatiotemporal structure of video data and extracts temporal features; S2, fusing the temporal features with the text features to form multimodal input data; S3, dynamically predicts the weights of the Laplacian pyramid decomposition kernel based on multimodal input data; S4, generates the corrected image through multi-level feature pyramid decomposition and reconstruction; S5, optimize model parameters by combining multi-component loss function; S6, based on the corrected image, recognizes and classifies scene events to achieve semantic understanding; The step S1 comprises: S101, obtaining a spatiotemporal sequence of video data, including continuous video frames on a time axis and their spatial pixel distribution; S102, constructing a rotation position embedding variant encoder, and performing asymmetric dimension mapping of spatiotemporal positions through a low-frequency time allocation strategy; S103, generating a spatiotemporal position encoding matrix using a diagonal layout, so that the relative position relationship between the visual markers and the rotational position encoding of the text markers maintain symmetry constraints; S104: extracting temporal features to obtain visual feature vectors.

2. The multimodal large-model context understanding method for high-speed abnormal exposure scenarios according to claim 1 is characterized by: The step S102 specifically includes: The time dimension is assigned to the high-order part of the embedding vector, and the number of dimensions is positively correlated with the length of the time series; The spatial dimensions are assigned to the low-order part of the embedding vector, and the horizontal and vertical positions occupy adjacent dimensions in an interleaved pattern.

3. The multimodal large-model context understanding method for high-speed abnormal exposure scenarios according to claim 2 is characterized by: The step S2 comprises: S201, performing video text information processing, encoding the text data using a pre-trained language model, and generating a text feature vector; S202: Jointly modeling the visual feature vector and the text feature vector through a multimodal fusion mechanism to form multimodal input data.

4. The multimodal large-model context understanding method for high-speed abnormal exposure scenarios according to claim 3 is characterized by: The step S3 comprises: S301, receiving multimodal input data and extracting its joint feature representation; S302, constructing a super network, the super network including an asymmetric convolutional layer group with gradually increasing number of channels; S303, performing multi-level downsampling on the multimodal features through the hypernetwork to generate a dynamic weight matrix related to the input features, wherein the weight matrix is ​​used to parameterize the Laplacian pyramid decomposition kernel; S304: Process the input data using a dynamically weighted Laplacian pyramid decomposition kernel and output a multi-scale decomposition result.

5. The multimodal large model context understanding method for high-speed abnormal exposure scenarios according to claim 4 is characterized by: The hypernetwork includes alternating asymmetric convolutional layer groups, each group including at least one horizontal convolution kernel (3×1) and a vertical convolution kernel (1×3).

6. The multimodal large-model context understanding method for high-speed abnormal exposure scenarios according to claim 5, characterized in that: The step S4 comprises: S401, inputting an original image, constructing a feature pyramid comprising multiple resolution levels, wherein the resolution of each level decreases according to a preset ratio; S402, perform iterative processing on each level: Dynamically generate convolution kernels based on the current level features, and perform low-pass filtering on the image to generate a predicted image; Upsample the predicted image to the next higher resolution and calculate the high-frequency residual with the original image; S403 , aggregating high-frequency residuals of all levels and low-frequency predictions of the final level to reconstruct a corrected output image.

7. The multimodal large-model context understanding method for high-speed abnormal exposure scenarios according to claim 6, characterized in that: The step S5 comprises: S501, constructing a generator network, wherein the generator network is used to output a corrected image; The generator network is trained using the following joint loss function: Pixel reconstruction loss function: measures the pixel-level difference between the generated image and the target image; Adversarial loss function: measures the ability of generated images to deceive the discriminator; Kernel constraint loss function: constrains the similarity between the kernel generated by the hypernetwork and the preset target kernel; S502. Back-propagate the gradient according to the joint loss function to synchronously optimize the parameters of the generator and the hypernetwork.

8. The multimodal large model context understanding method for high-speed abnormal exposure scenarios according to claim 7, characterized in that: The step S6 comprises: S601, receiving the corrected multimodal image data, and extracting high-level semantic features layer by layer through a deep neural network; S602, inputting the high-level semantic features into a fully connected layer and a Softmax function for event classification and recognition; S603, dynamically aggregates the contextual dependencies of semantic features based on the attention mechanism, captures the long-range dependencies between features, and generates the final representation vector of the event semantics; S604: Execute event recognition and classification tasks according to the final representation vector.

9. The multimodal large model context understanding method for high-speed abnormal exposure scenarios according to claim 8, characterized in that: The deep neural network includes alternately stacked nonlinear transformation layers and feature compression layers.

Images