Multi-modal large model context understanding method for high-speed abnormal exposure scene

Abstract

The invention relates to the technical field of intelligent traffic, and discloses a multi-modal large model context understanding method for a high-speed abnormal exposure scene, which comprises the following steps of: 1, aiming at a complex space-time structure of video data, encoding by adopting a RoPE (Rotation Position Element) variant to obtain a time sequence characteristic, and 2, combining the time sequence characteristic with a text characteristic to obtain a multi-modal large model context understanding result; the method comprises the following steps of: 1, inputting time sequence + text multi-modal data, 3, dynamically predicting the weight of a Laplacian pyramid decomposition kernel according to the characteristics of the input multi-modal data, 4, extracting and converting the characteristics in each layer of a characteristic pyramid, reconstructing the characteristics to original resolution layer by layer, and generating a corrected image, and 5, in the training process, generating a corrected image. And 6, inputting the corrected image into an event understanding module, identifying and classifying various events in the scene, and realizing semantic understanding.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and particularly to a multimodal large model context understanding method for high-speed abnormal exposure scenarios. Background Art

[0002] With the rapid development of intelligent transportation, how to process image and video data of abnormal exposure scenarios in complex and dynamic environments in real time has become an urgent challenge. Especially in high-speed scenarios, there are significant problems such as drastic light changes, motion blur, and exposure distortion, which pose great challenges to traditional image processing and understanding technologies.

[0003] Currently, image processing methods for abnormal exposure scenarios mainly focus on dehazing, exposure correction, and noise suppression. However, these technologies often face problems of insufficient accuracy and poor real-time performance when dealing with high-speed dynamic scenarios. Although traditional methods based on convolutional neural networks (CNNs) have achieved remarkable results in static image processing, there are still significant processing bottlenecks in dynamic video scenarios, especially in scenarios with high-speed motion or large-scale light changes. In addition, existing multimodal data fusion technologies also have limitations in the modeling of spatio-temporal features and information transmission. Especially in complex context understanding tasks, how to effectively integrate information from different modalities (such as video and text) remains an unsolved problem.

[0004] The multimodal large model context understanding method for high-speed abnormal exposure scenarios proposed by the present invention can more accurately capture spatio-temporal features in video data, especially in scenarios with high speed and large-scale light changes, by innovatively combining a variant of Rotary Position Embedding (RoPE) and a technique for dynamically predicting the kernel weights of Laplacian pyramid decomposition. At the same time, through multi-level feature extraction and fusion in the feature pyramid, the calibration accuracy and stability of images and videos are improved, ensuring high-quality image reconstruction. In addition, an integrated multi-component loss function is used for optimization, effectively overcoming the accuracy and real-time bottlenecks of traditional methods when dealing with complex dynamic scenarios, thereby improving the collaborative understanding ability of multimodal information, and is particularly suitable for complex environments with high speed, dynamics, and abnormal exposure. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art, and to propose a multimodal large model context understanding method for high-speed abnormal exposure scenarios, which can effectively improve the processing accuracy of image and video data and the event understanding ability by generating dynamic kernels to individually process the multimodal features of abnormal exposure scenario images.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions: A multi-modal large model context understanding method for high-speed abnormal exposure scenarios, comprising the following steps: S1, encode the complex spatio-temporal structure of video data and extract temporal features; S2, fuse the temporal features with text features to form multi-modal input data; S3, dynamically predict the weights of the Laplacian pyramid decomposition kernel based on the multi-modal input data; S4, generate a corrected image through multi-level feature pyramid decomposition and reconstruction; S5, optimize the model parameters by combining a multi-component loss function; S6, recognize and classify scene events based on the corrected image to achieve semantic understanding.

[0007] Preferably, the step S1 includes: S101, obtain the spatio-temporal sequence of video data, including consecutive video frames on the time axis and their spatial pixel distributions; S102, construct a rotated position embedding variant encoder, and perform asymmetric dimension mapping on spatio-temporal positions through a low-frequency time allocation strategy; S103, generate a spatio-temporal position encoding matrix using a diagonal layout, so that the relative position relationship between visual tokens maintains a symmetry constraint with the rotated position encoding of text tokens; S104, extract temporal features to obtain visual feature vectors.

[0008] Preferably, the step S102 specifically includes: The time dimension is allocated to the high-order part of the embedding vector, and the number of dimensions is positively correlated with the length of the time series; The space dimension is allocated to the low-order part of the embedding vector, and the horizontal and vertical positions occupy adjacent dimensions in an interleaved pattern.

[0009] Preferably, the step S2 includes: S201, perform video text information processing, use a pre-trained language model to encode text data, and generate text feature vectors; S202, jointly model the visual feature vector and the text feature vector through a multi-modal fusion mechanism to form multi-modal input data.

[0010] Preferably, the step S3 includes: S301, receive multi-modal input data and extract its joint feature representation; S302, construct a hypernetwork, which includes a group of asymmetric convolutional layers with gradually increasing number of channels; S303, perform multi-level downsampling on the multi-modal features through the hypernetwork to generate a dynamic weight matrix related to the input features, where the weight matrix is used to parameterize the Laplacian pyramid decomposition kernel; S304, process the input data using the dynamically weighted Laplacian pyramid decomposition kernel and output the multi-scale decomposition result.

[0011] Preferably, the hypernetwork includes groups of asymmetric convolutional layers arranged alternately, and each group includes at least one horizontal convolutional kernel (3×1) and one vertical convolutional kernel (1×3).

[0012] Preferably, the step S4 includes: S401, input the original image, construct a feature pyramid including multiple resolution levels, where the resolution of each level decreases according to a preset ratio; S402, perform iterative processing on each level: Dynamically generate a convolutional kernel based on the features of the current level, perform low-pass filtering on the image to generate a predicted image; Upsample the predicted image to the resolution of the previous level and calculate the high-frequency residual with the original image; S403, aggregate the high-frequency residuals of all levels and the low-frequency prediction of the final level, and reconstruct to obtain the corrected output image.

[0013] Preferably, the step S5 includes: S501, construct a generator network, where the generator network is used to output the corrected image; The generator network is trained using the following joint loss function: Pixel reconstruction loss function: measure the pixel-level difference between the generated image and the target image; Adversarial loss function: measure the ability of the generated image to deceive the discriminator; Kernel constraint loss function: constrain the similarity between the kernel generated by the hypernetwork and the preset target kernel; S502. Backpropagate the gradient according to the joint loss function and synchronously optimize the parameters of the generator and the hypernetwork.

[0014] Preferably, the step S6 includes: S601, receive the corrected multi-modal image data, and extract high-level semantic features layer by layer through a deep neural network; S602, input the high-level semantic features into a fully connected layer and a Softmax function for event classification and recognition; S603, dynamically aggregate the context dependencies of the semantic features based on the attention mechanism, capture the long-range dependencies between the features, and generate the final representation vector of the event semantics; S604. Perform event recognition and classification tasks based on the final representation vector.

[0015] Preferably, the deep neural network includes non - linear transformation layers and feature compression layers stacked alternately.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: The multi - modal large - model context understanding method for high - speed abnormal exposure scenarios proposed by the present invention can more accurately capture spatio - temporal features in video data by innovatively combining a variant of Rotary Position Embedding (RoPE) and a technique for dynamically predicting the kernel weights of Laplacian pyramid decomposition, especially in scenarios with high speed and large - range illumination changes. At the same time, through multi - level feature extraction and fusion in the feature pyramid, the calibration accuracy and stability of images and videos are improved, ensuring high - quality image reconstruction. In addition, optimization is carried out using a comprehensive multi - component loss function, effectively overcoming the accuracy and real - time bottlenecks of traditional methods when dealing with complex dynamic scenarios, thereby improving the collaborative understanding ability of multi - modal information, and being particularly suitable for complex environments with high speed, dynamics, and abnormal exposure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flowchart of a multi - modal large - model context understanding method for high - speed abnormal exposure scenarios proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0019] For ease of understanding of the embodiments of the present application, the following will be further explained with specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation to the embodiments of the present application.

[0020] As Figure 1 shown, a multi - modal large - model context understanding method for high - speed abnormal exposure scenarios includes the following steps in sequence: (1) For the complex spatio - temporal structure of video data, a variant of Rotary Position Embedding (RoPE) is used for encoding to obtain temporal features. RoPE is a method that improves the model's ability to capture sequential time - dependent relationships, which can enhance the temporal understanding of video sequences and is crucial for high - speed anomaly detection; (2)Combine the temporal features with the text features to form a multi-modal data input of temporal + text. This integration enables the model to not only understand the visual context in the video but also any text information, thus providing a more comprehensive understanding of the scene, which is crucial for complex anomaly detection in high-speed environments; (3)Dynamically predict the weights of the Laplacian pyramid decomposition kernel based on the features of the input multi-modal data, enabling the model to adaptively adjust its feature extraction process to better handle different levels of resolution in image and video data, thereby optimizing the processing of multi-resolution inputs; (4)Refine and transform the features at each layer of the feature pyramid, reconstructing layer by layer to the original resolution to generate a corrected image. This multi-scale approach enables the model to retain detailed spatial information while focusing on higher-level abstractions, thereby improving the accuracy of anomaly detection at different scales; (5)During training, combine the pixel reconstruction loss, adversarial loss, and kernel loss to optimize the model parameters. This combined loss function (pixel reconstruction, adversarial, and kernel loss) during training is a powerful method to improve the model's ability in feature extraction and anomaly detection. The adversarial loss ensures that the generated features are real, while the pixel reconstruction and kernel losses fine-tune the model to retain key details, especially for high-speed and high-risk scenarios; (6)Input the corrected image into the event understanding module to identify and classify various events in the scene, achieving semantic understanding. This step bridges the gap between the original visual data and actionable insights, enabling the model to understand and interpret complex events, which is a key aspect of real-time monitoring and anomaly detection in applications.

[0021] Specifically, step (1) means: First, Low-Frequency Time Allocation (LTA) reduces the oscillation phenomenon in time modeling by allocating higher dimensions to the time axis, thus better capturing long-range dependencies. The frame indices in the video and the token indices in the text are essentially different. This method scales the time indices, using higher dimensions for temporal features in longer contexts and lower dimensions for spatial features, which are resolution-limited and have a fixed range. At the same time, the dimensions of the spatial features are interleaved to avoid gaps between horizontal and vertical positions. Specifically, the horizontal position and the vertical position interleave and occupy lower dimensions, while time occupies higher dimensions. For fair comparison, their values are 48, 48, and 32 respectively. Then, adopt a diagonal layout method to ensure spatial symmetry and maintain the relative position relationship between visual and text tokens. Specifically, arrange the entire input along the diagonal, and the 3D position of the central patch of each video frame is , other patches are offset in all directions; finally, there is an Adjustable Time Span (ATS), introducing a scaling factor , to align the time information between visual and text tokens and adapt to contexts of different lengths. Specifically, let denote the token index, denote the total number of tokens in the text, denote the total number of video frames, denote the total number of tokens in the ending text. For text tokens , the time, horizontal, and vertical indices are directly set to the original token index . For video input , denotes the index of the current frame relative to the start of the video and controls the space in the time dimension. For the ending text , the time, horizontal, and vertical indices are kept consistent, thus creating a linear process. For a multimodal input consisting of text with tokens, subsequent video with patches per frame, and ending text with tokens, for text tokens or visual tokens of the VideoRoPE position index or visual tokens is defined as:

[0022] where w and h denote the horizontal and vertical indices of the visual patches within the frame respectively.

[0023] The specific step (2) refers to: First, perform video-text information processing, encoding the text data using a pre-trained language model, especially descriptions, instructions, or event tags related to the video content, to obtain text information ; then, jointly model the visual features and text features obtained in the previous steps through a multimodal fusion mechanism (the design of the multimodal fusion function allows video and text information to be processed in the same context rather than separately. This helps the model establish associations between different modalities, making the processing results more consistent and accurate. Especially in video understanding, visual and text data are often complementary, and joint modeling can fully utilize the complementary information between them to improve the performance of the model.) to obtain a joint representation , where denotes the multimodal fusion function, denotes the temporal feature representation obtained in the previous steps.

[0024] Step (3) specifically refers to: Using HyperNetwork to process the input multi-modal features to generate personalized kernels customized according to the features. Compared with fixed convolutional kernels, the dynamically generated kernels can better capture the complex relationships in the input data, especially in the case of multi-modal data fusion (video and text features). This architecture consists of a hypernetwork composed of six convolutional layers, and the number of channels in each convolutional layer gradually increases. By gradually increasing the number of channels in the convolutional layers, the network can gradually extract simple features from the lower layers to complex feature representations in the higher layers, which helps to improve the model's learning ability for multi-scale and multi-dimensional data. These convolutional layers use irregular convolutional kernels (such as 3×1 and 1×3) and pooling operations, which can better capture details and changes in different directions, enhancing the network's perception ability of diverse spatial information to achieve significant downsampling. The architecture details are as follows: The initial convolutional layer uses a convolutional kernel of size 3×1, followed by ReLU activation and max pooling operations; the second convolutional layer uses a 1×3 convolutional kernel, followed by ReLU activation and max pooling to continue the downsampling process; the third and fourth convolutional layers use 3×1 and 1×3 convolutional kernels, and each convolutional kernel is followed by a ReLU activation layer and a pooling layer; the fifth convolutional layer uses a 3×3 convolutional kernel, followed by max pooling and a final 1×1 convolution to precisely reduce the dimension to 5×5. Compared with large convolutional kernels, the 1×1 convolutional kernel only adjusts the number of channels, and the precise control of the feature dimension helps to avoid excessive computational burden.

[0025] Step (4) specifically refers to: The input image , where represents the image height, represents the image width. First, calculate the low-pass predicted image by applying the dynamically generated kernel for convolution operation. The formula is as follows: , where represents the neighborhood around the pixel , and are the kernel weights. Then, to ensure reversible reconstruction, calculate the high-frequency residual as follows: , where The function uses bicubic interpolation to Upsample to the original size; then, this process is iteratively applied to subsequent-level images, further decomposing the image into multiple low-frequency and high-frequency components. Through processing at different scales, the model can analyze the image content from both global and local levels, avoiding information loss that may occur when processing the image only at a single scale. This method is particularly suitable for processing complex image data, such as scenes with high-speed motion and large lighting changes. The decomposition of the image is carried out through multiple pyramid levels, including three levels in this architecture, corresponding to resolutions , , and , and the specific mathematical expressions are as follows:

[0026]

[0027]

[0028]

[0029] where is the downsampling function, and the final pyramid consists of multiple levels. The features of these pyramid levels help to express the low-frequency and high-frequency features of the image at different scales.

[0030] Step (5) specifically refers to: A multi-component loss function that combines a pixel reconstruction loss, an adversarial loss, and a kernel loss is used to effectively train the generator. The pixel reconstruction loss measures the difference between the generated image and the target highlight image, ensuring the consistency of the generated image with the target image at the pixel level and ensuring detail restoration. It is usually calculated using the mean squared error (MSE): , represents the generated image, represents the target image; the adversarial loss measures the ability of the generated image to deceive the discriminator, and the specific form depends on the type used; the kernel loss is used to constrain the kernel generated by the hypernetwork to be close to the target kernel (such as a Gaussian kernel), and is also calculated using the mean squared error: , where the target kernel refers to the Gaussian kernel used in the LPTN framework. The loss is used to update the model so that the kernel predicted by the network is close to the Gaussian kernel, but is also dynamic for the input. Finally, the total loss function of the generator is: , where and respectively control the contributions of the pixel reconstruction loss and the kernel loss to the total loss. By minimizing the above loss function, the parameters of the generator and the hypernetwork (if applicable) are optimized to improve the quality and authenticity of the generated image.

[0031] Step (6) specifically refers to: First, feature extraction is performed. The corrected multi-modal feature representation is input into a deep neural network, and high-level semantic features are extracted layer by layer. After layers of neural networks, the feature representation is , where represents the non-linear transformation function of the th layer. Through the multi-layer neural network, the model can gradually abstract from low-level features to high-level semantic features, making the final feature representation able to accurately capture the complex patterns in the input data, which is particularly important for processing multi-modal data (such as combinations of videos and texts) because the relationships of these data are usually complex and dynamic. Then, using the extracted high-level features , event classification and recognition are performed through a fully connected layer and a Softmax function. Let the number of event categories be , then the classification result is , where is the weight matrix, is the bias vector. Finally, in order to deeply understand the semantic information of the event, an attention mechanism is introduced to capture the long-range dependence relationships between features. At the same time, the attention mechanism can automatically select the features relevant to the task and assign higher weights to them. This method enables the network to dynamically focus on the most critical information and improve the accuracy of the classification task. The calculation of the attention weight is as follows: , where , is the context vector, and are trainable parameters, is the hidden state at the th time step, is the context information. The final semantic representation r is: .

[0032] Compared with the prior art, the method proposed by the present invention effectively solves the problem of spatio-temporal structure understanding of image and video data in complex scenarios by combining a variant of Rotary Position Embedding (RoPE) and the technology of dynamically predicting the kernel weights of Laplacian pyramid decomposition. Through feature refinement and transformation, pyramid decomposition, and optimization of the multi-component loss function, it can improve the processing accuracy of image and video data and enhance the event understanding ability in multi-modal information fusion. By optimizing the training loss function, the quality and authenticity of the generated images are further improved. This method can be widely applied to multi-modal data analysis in autonomous driving and high-speed abnormal exposure environments, and has strong real-time performance, accuracy, and robustness.

[0033] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, should be covered by the protection scope of the present invention.

Claims

1. A method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios, characterized in that: It includes the following steps: S1. Encode the complex spatio-temporal structure of video data and extract temporal features; S2. Fuse the temporal features with text features to form multi-modal input data; S3. Dynamically predict the weights of the Laplacian pyramid decomposition kernel based on the multi-modal input data; S4. Generate a corrected image through multi-level feature pyramid decomposition and reconstruction; S5. Optimize the model parameters by combining a multi-component loss function; S6. Identify and classify scene events based on the corrected image to achieve semantic understanding.

2. The method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios according to claim 1, characterized in that: The step S1 includes: S101. Obtain the spatio-temporal sequence of video data, including consecutive video frames on the time axis and their spatial pixel distributions; S102. Construct a rotated position embedding variant encoder to perform asymmetric dimensional mapping on spatio-temporal positions through a low-frequency time allocation strategy; S103. Generate a spatio-temporal position encoding matrix using a diagonal layout to make the relative position relationship between visual tokens maintain symmetry constraints with the rotated position encoding of text tokens; S104. Extract temporal features to obtain visual feature vectors.

3. The method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios according to claim 2, characterized in that: The step S102 specifically includes: The time dimension is allocated to the high-order part of the embedding vector, and the number of dimensions is positively correlated with the length of the time series; The spatial dimension is allocated to the low-order part of the embedding vector, and the horizontal and vertical positions occupy adjacent dimensions in an interleaved pattern.

4. The method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios according to claim 3, characterized in that: The step S2 includes: S201. Perform video text information processing, use a pre-trained language model to encode text data, and generate text feature vectors; S202. Jointly model the visual feature vector and the text feature vector through a multi-modal fusion mechanism to form multi-modal input data.

5. The method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios according to claim 4, characterized in that: The step S3 includes: S301. Receive multi-modal input data and extract its joint feature representation; S302. Construct a hypernetwork, which includes a group of asymmetric convolutional layers with gradually increasing number of channels; S303. Perform multi-level downsampling on the multi-modal features through the hypernetwork to generate a dynamic weight matrix related to the input features, and the weight matrix is used to parameterize the Laplacian pyramid decomposition kernel; S304. Process the input data using a dynamically weighted Laplacian pyramid decomposition kernel and output a multi-scale decomposition result.

6. The method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios according to claim 5, characterized in that: The hypernetwork includes a group of asymmetric convolutional layers arranged alternately, and each group includes at least one horizontal convolutional kernel (3×1) and a vertical convolutional kernel (1×3).

7. The method for context understanding of a multi-modal large model for high-speed abnormal exposure scenarios according to claim 6, characterized in that: The step S4 includes: S401. Input the original image and construct a feature pyramid including multiple resolution levels, where the resolution of each level decreases according to a preset ratio; S402. Perform iterative processing on each level: Dynamically generate a convolutional kernel based on the current level features, perform low-pass filtering on the image to generate a predicted image; Upsample the predicted image to the resolution of the previous level and calculate the high-frequency residual with the original image; S403. Aggregate the high-frequency residuals of all levels and the low-frequency prediction of the final level, and reconstruct to obtain the corrected output image.

8. A multimodal large model context understanding method for high-speed abnormal exposure scenarios according to claim 7, characterized in that: The step S5 includes: S501. Construct a generator network, and the generator network is used to output the corrected image; The generator network is trained using the following joint loss function: Pixel reconstruction loss function: Measures the pixel-level difference between the generated image and the target image; Adversarial loss function: Measures the ability of the generated image to deceive the discriminator; Kernel constraint loss function: Constrains the similarity between the kernel generated by the hypernetwork and the preset target kernel; S502. Backpropagate the gradient according to the joint loss function to synchronously optimize the parameters of the generator and the hypernetwork.

9. A multimodal large model context understanding method for high-speed abnormal exposure scenarios according to claim 8, characterized in that: The step S6 includes: S601. Receive the corrected multi-modal image data and extract high-level semantic features layer by layer through a deep neural network; S602. Input the high-level semantic features into the event classification and recognition through a fully connected layer and a Softmax function; S603. Dynamically aggregate the context dependencies of semantic features based on the attention mechanism, capture the long-range dependencies between features, and generate the final representation vector of event semantics; S604. Perform event recognition and classification tasks according to the final representation vector.

10. A multimodal large model context understanding method for high-speed abnormal exposure scenarios according to claim 9, characterized in that: The deep neural network includes non-linear transformation layers and feature compression layers stacked alternately.

Images