Image-enhanced multi-modal driving video analysis method

Abstract

The invention belongs to the technical field of multi-modal driving video analysis methods, and particularly relates to an image-enhanced multi-modal driving video analysis method, which comprises the following steps of: pre-training an image, converting a clear image and a degraded image into potential space representation, adding noise to the clear image by using a diffusion probability model, and guiding a recovery task through task prompt. Meanwhile, the model is finely adjusted by using Contro Net, alignment of potential space representation and task prompt is ensured in cooperation with a self-attention and text cross attention mechanism, the difference between an image and a target image is recovered through L2 loss minimization, training of an image recovery model is completed, key frame extraction, target detection and three-dimensional positioning are performed on an enhanced video, and the image recovery efficiency is improved. The method comprises the following steps: constructing semantic text description, carrying out multi-modal coding through a visual language model, generating structured perception representation, carrying out structured spatial reasoning by utilizing a small-scale language model, and training the model by adopting a strategy optimization algorithm, so that the model generates a reasoning process and an accurate answer under the condition of inputting scene description and questions.

Description

Technical Field

[0001] This invention belongs to the technical field of multimodal driving video analysis methods, and particularly relates to an image-enhanced multimodal driving video analysis method. Background Technology

[0002] With the rapid development of autonomous driving technology, the requirements for environmental perception in autonomous driving systems are constantly increasing. Especially in complex and dynamic driving scenarios, efficiently and accurately understanding video data has become a key challenge. Existing autonomous driving perception technologies typically rely on the fusion of multiple sensors, such as cameras and LiDAR, to acquire information about the surrounding environment. However, traditional image processing methods have limited recovery capabilities when faced with degraded images, and their ability to handle inter-frame temporal consistency and dynamic changes of multiple targets is insufficient in dynamic scenes.

[0003] Existing image enhancement methods mainly focus on denoising and restoration of static images. However, in real-time video restoration, handling temporal consistency and spatial detail restoration within video sequences remains a challenge. Video restoration not only requires restoring the image quality of each frame but also ensuring smooth transitions between frames to avoid inconsistencies caused by noise or motion blur. Furthermore, with the increasing complexity of autonomous driving scenarios, the ability to process multimodal data has become crucial for improving perception accuracy.

[0004] Traditional multimodal learning methods often rely on relatively simple feature fusion techniques, lacking in-depth modeling of temporal information and spatial details. Although frameworks based on convolutional neural networks and recurrent neural networks can process image and video data, they still have bottlenecks in accuracy and efficiency when dealing with complex dynamic scenes. Especially when processing large-scale, multi-view dynamic videos, existing methods have high computational costs and struggle to meet real-time requirements.

[0005] To address this issue, an image-enhanced multimodal driving video analysis method is designed. Summary of the Invention

[0006] The purpose of this invention is to address the aforementioned technical problems by providing an image-enhanced multimodal driving video analysis method.

[0007] In view of this, the present invention provides an image-enhanced multimodal driving video analysis method, comprising the following steps:

[0008] Step S1: Image pre-training. The clear image and the degraded image are converted into latent spatial representations. Noise is added to the clear image using a diffusion probability model and the restoration task is guided by task cues. At the same time, ControlNet is used to fine-tune the model. Self-attention and text cross-attention mechanisms are used to ensure that the latent spatial representation is aligned with the task cues. The difference between the restored image and the target image is minimized by L2 loss to complete the training of the image restoration model.

[0009] Step S2: Extract keyframes, detect objects, and locate 3D objects in the enhanced video; construct a semantic text description; and perform multimodal encoding through a visual language model to generate a structured perceptual representation.

[0010] Step S3: Use a small-scale language model for structured spatial reasoning, and train the model using a policy optimization algorithm to generate reasoning processes and accurate answers under the input scenario description and question.

[0011] Preferably, in step S1, the process of adding noise to the clear image using the diffusion probability model includes: forward diffusion of the clear image x0, adding Gaussian noise according to a noise scheduling table at each step t∈[1,T] to obtain the degraded image x0. t ,satisfy

[0012] in,

[0013] (α t () represents the predefined noise figure;

[0014] The training objective is to minimize the error between the predicted noise and the actual noise, satisfying:

[0015] Where, (∈ θ ) represents the model's prediction noise, and c represents the guidance and prompt information.

[0016] Preferably, in step S1, the fine-tuning of ControlNet includes: ControlNet receiving control input c and extracting multi-layer control features through branch networks, fusing them with the main network features in the form of residuals at each layer of the diffusion model, wherein the fusion method satisfies:

[0017] in, This is the feature map of the l-th layer of the main U-Net network. These are features extracted by the ControlNet branch from the control input c; during training, the parameters of the main diffusion model are frozen, and only the ControlNet branch network is updated. The training objective satisfies:

[0018] Preferably, in step S1, the self-attention mechanism is used to capture the long-distance dependencies of each part within the image, and the text cross-attention mechanism is used to model the relationship between the input image and the task prompt. The two work together to align the latent spatial representation with the task prompt.

[0019] The L2 loss function is used to keep the recovered image consistent with the target image at the pixel level.

[0020] Preferably, the key frame extraction in step S2 includes: calculating the inter-frame overlap rate based on the visual overlap between consecutive frames through feature point matching; when the overlap rate is lower than a set threshold, the current frame is identified as a key frame and retained, forming a sparse, non-redundant frame sequence.

[0021] Preferably, the target detection and instance segmentation in step S2 includes: using a YOLO detector to extract the target's category, bounding box, and confidence score, and using a SAM model to provide pixel-level mask information for each target;

[0022] By assigning a unique ID to each target using the intersection-union ratio and trajectory matching algorithm, consistent tracking across frames is achieved.

[0023] Preferably, the three-dimensional localization in step S2 includes: generating a pixel-level depth map using a MiDaS monocular depth estimation model, extracting the center point and the average depth value of the mask within the target mask area, and obtaining the three-dimensional position (x, y, d) of the target in the image coordinate system, where x and y are image coordinates and d is the relative depth.

[0024] Preferably, the semantic text description construction in step S2 includes: generating a description statement for each target according to the template of category + relative position + movement state + depth value, and combining the description statements of all targets in spatial order to form a scene-level language paragraph to express the spatial distribution and movement state of key targets in the frame.

[0025] Preferably, the multimodal encoding in step S2 includes: pairing scene-level language paragraphs with the original image and inputting them into a visual language model; converting the image into a visual token via a visual encoder and the text into a text token via a language encoder; and then cross-fusing the two in a Transformer structure through a multi-head attention mechanism to output a semantic representation vector containing significant object features and language attributes and relationships.

[0026] Preferably, step S3 includes:

[0027] Input prompt construction: Collect semantic description text for each frame, arrange it chronologically, and add task questions to form a prompt. <scene>Description of the scene of the tag package and <question>Input prompts for task-related questions tagged with tags;

[0028] Model training: Load a small-scale language model without fine-tuning, freeze the basic structure and only open the parameters of the high-level policy layer, train it using a swarm intelligence policy algorithm, evaluate candidate outputs through format reward, accuracy reward and logical consistency reward and sum them up by weight;

[0029] Inference Output: The model output includes the inference process identified by labels and <richmediareference>The structured text of the final answer, identified by tags.

[0030] The beneficial effects of this invention are:

[0031] Improving image restoration quality and temporal consistency: By combining the diffusion probability model with ControlNet, we can not only effectively process degraded images such as noise and haze to achieve high-quality single-frame image enhancement, but also ensure smooth transitions between frames by modeling the temporal relationship of video sequences, avoiding inconsistencies between frames in dynamic scenes caused by enhancement processing, and providing more reliable visual input for subsequent analysis.

[0032] Enhanced multimodal perception and semantic understanding capabilities: Leveraging technologies such as keyframe extraction, target detection, and 3D localization, the system can accurately capture various targets and their motion states in driving scenarios. Visual information is then transformed into understandable linguistic information through structured semantic text descriptions. Further multimodal encoding via a visual language model achieves deep fusion of image and text information, enabling the system to gain a more comprehensive and accurate semantic understanding of driving scenarios, encompassing not only the visual features of targets but also their attributes and spatial relationships.

[0033] Enhancing Structured Spatial Reasoning and Decision Support Capabilities: By utilizing small-scale language models for structured spatial reasoning and combining them with policy optimization algorithms for training, the model can generate reasonable reasoning processes and accurate answers based on scene descriptions and task problems. This capability enables the system to achieve spatial understanding of complex driving scenarios even under low-resource conditions, providing effective logical support for autonomous driving decisions and improving the rationality and reliability of those decisions.

[0034] Improving the environmental adaptability and real-time performance of autonomous driving systems: Technical solutions designed for complex and dynamic driving scenarios effectively address image degradation issues under different environments, enhancing the system's adaptability to complex environments. Simultaneously, by reducing redundant data processing through keyframe extraction and employing small-scale language models to lower computational overhead, processing efficiency is improved while maintaining analytical accuracy, better meeting the real-time requirements of autonomous driving and providing strong support for the safe and stable operation of autonomous driving systems. Attached Figure Description

[0035] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0037] It should be noted that all directional and positional terms used in this invention, such as "up," "down," "left," "right," "front," "back," "vertical," "horizontal," "inner," "outer," "top," "lower," "lateral," "longitudinal," and "center," are only used to explain the relative positional relationships and connections between components in a specific state (as shown in the accompanying drawings). They are merely for the convenience of describing the invention and do not require the invention to be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on the invention. Furthermore, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0038] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0039] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0040] The method includes step S1: Image pre-training. First, clear images and degraded images (such as noise, haze, etc.) are converted into latent spatial representations. Then, noise is added to the clear image using a diffusion probability model, and the restoration task is guided by task cues. At this point, ControlNet is used to fine-tune the model, combined with self-attention and text cross-attention mechanisms, to ensure that the latent spatial representation is aligned with the task cues. The difference between the restored image and the target image is minimized through L2 loss, thereby completing the training of the image restoration model.

[0041] Step S2: Extract keyframes, detect objects, and locate 3D objects in the enhanced video. Construct a semantic text description and perform multimodal encoding through a visual language model to generate a structured perception representation.

[0042] Step S3: Use a small-scale language model for structured spatial reasoning, and train the model using a policy optimization algorithm to generate reasoning processes and accurate answers under the input scenario description and question.

[0043] During the pre-training process for image restoration, both clear and degraded images (e.g., noise, haze, etc.) are first converted into latent space representations. This step is the first step in the image restoration model, and its purpose is to transform the image from the original pixel space into a simpler, more abstract latent space, enabling the model to learn and optimize on a more efficient representation.

[0044] In this step, we use a diffusion probability model (DDPM) to progressively add noise to the sharp image and guide the model's restoration process with task prompts. The core idea of ​​the diffusion model is to gradually transform the image into noise and then gradually remove the noise through a reverse process to restore the image. During training, the model learns how to remove noise from the degraded image with added noise, thereby generating a restored image that is as close as possible to the original sharp image. We use a diffusion probability model (DDPM) to progressively add noise to the sharp image and guide the model's restoration process with task prompts. In the operation, we first perform a forward diffusion process on the sharp image (x0): in each step (t∈[1,T]), Gaussian noise is added according to a given noise schedule, resulting in a series of degraded images (x0, x0, ...). t This process can be represented as:

[0045] in (α t ) is a predefined noise figure.

[0046] During the training phase, the model receives the degraded image (x) at any time step (t). t The goal is to predict the noise (∈) added to the original image and its corresponding time step index (t). t This means training is performed by minimizing the error between the predicted noise and the actual noise (such as MSE loss). Specifically, the training objective is:

[0047] where (∈ θ ) represents the model's prediction noise, and c represents guidance and prompting information, such as image classification labels, semantic masks, etc.

[0048] During the backsampling process, from standard Gaussian noise Initially, the model receives the current image (x) at each time step t. t Using time step (t) and cue conditions (c), predict the noise component (∈) of the current image. θ (x t ,t,c)), and generate (x) based on the estimated value after denoising. t-1 This process is repeated until (t=0), eventually yielding the restored image. Furthermore, the influence of conditional information in the reverse process can be controlled by adjusting the cue guidance weight (such as classifier-free guidance), thereby improving the relevance and quality of the generated image.

[0049] To ensure high-quality alignment between the restored image and the target image, we introduced ControlNet during model training. ControlNet is a model that fine-tunes based on control signals. During training, we incorporated ControlNet, a method that achieves fine-grained control through an additional network without modifying the structure of the original diffusion model (such as U-Net). The core operation of ControlNet is: receiving a control input (such as an edge map, depth map, semantic map, etc.) as a guiding condition, extracting multi-layer control features through a branch network parallel to the backbone network, and then injecting these features into the features of each layer of the main network as residuals. Specifically, in each layer of the diffusion model, the fusion method of the intermediate features of the first layer is as follows:

[0050] in, This is the feature map of the main U-Net network at layer 1. The features extracted by the ControlNet control branch from the control input c (such as an edge map) are added together to obtain the fused features. The data is then passed to the next layer for processing.

[0051] During training, we typically freeze the parameters of the main diffusion model and only update the ControlNet branch network to maintain the stability of the generated quality. The overall training objective remains consistent: minimizing the difference between the model's predicted noise and the actual added noise. The loss function is as follows:

[0052] Among them, (x t () is the image after noise has been added, t is the time step, c is the control input, (∈ θ () is the model's prediction of noise under the condition of introducing control information.

[0053] The entire process is as follows: First, noise is added to the clear image (x0) to obtain the degraded image (x). t Simultaneously prepare the control input c corresponding to the image, such as its edge map; and set (x t The model takes t and c as inputs, the backbone network performs standard diffusion model operations, the control branch extracts control information, and fuses it with backbone features at each layer through residual connections; finally, the model outputs the predicted noise, calculates the loss, and only updates the control branch weights. In this way, ControlNet can flexibly adjust the generation direction of the diffusion model, making the generated image more consistent with the control conditions, while maintaining the generalization ability of the original model.

[0054] The image generation process can be controlled by additional conditional inputs (such as depth maps, key points, etc.). In image restoration tasks, ControlNet uses this conditional information to adjust the details of the restoration process, enabling the model to better meet the task requirements when generating restored images.

[0055] Furthermore, the model combines self-attention and cross-attention mechanisms to enhance the image restoration process. During training, self-attention captures long-distance dependencies between different parts of the image, while cross-attention helps the model model the relationship between the input image and task cues in multimodal learning. These mechanisms ensure that the latent spatial representation aligns with the task cues, thereby improving the quality of the restored image.

[0056] By using the L2 loss function, we minimize the difference between the generated restored image and the target image. The L2 loss function helps the model optimize during training, ensuring that the generated image maintains pixel-level consistency with the target image. Ultimately, this pre-training process enables the model to handle different types of image degradation and generate sharp restored images through efficient learning.

[0057] Step S2 includes the following steps: After enhancing the image input, the system enters the perceptual modeling stage and begins the multimodal semantic extraction and encoding process. First, keyframes are extracted from the enhanced video data. The extraction process is based on the visual overlap between consecutive frames, and the inter-frame overlap rate is calculated through feature point matching. When the overlap rate is lower than a set threshold, the current frame is considered a key information frame and is retained, forming a sparse, non-redundant frame sequence.

[0058] Next, object detection and instance segmentation are performed on each frame of the image, using the YOLO detector and the SAM model respectively. The detector extracts the category, bounding box, and confidence score of all identifiable objects in the image. The instance segmentation module provides pixel-level mask information for each object, which is used for subsequent fine-grained semantic description. To ensure consistent tracking of the identified objects across multiple frames, each object is assigned a unique ID using an intersection-union algorithm and trajectory matching algorithm, achieving consistency across frames.

[0059] Each detected target is then input into the depth estimation module. The system uses the MiDaS monocular depth estimation model to generate a pixel-level depth map for the entire image. Within the target mask region, the center point and the average depth value of the mask are extracted to obtain the target's three-dimensional position estimate in the image coordinate system, represented as (x, y, d), where x and y are the image coordinates and d is the relative depth.

[0060] After acquiring the target's category, location, movement trend, and depth information, the system constructs a structured language description. Each target is described as: "Category + Relative Position + Movement Status + Depth Value," for example, "A white sedan moving forward appears slightly to the left ahead, approximately 15 meters away." This template uniformly generates semantic descriptions for all targets, forming frame-by-frame target description text.

[0061] Subsequently, the system constructs a complete scene description text. For each keyframe, the descriptions of all targets are combined in spatial order to form scene-level language paragraphs. For example: "In the current field of view, there is a white truck moving slowly in front of the vehicle, and a pedestrian moving towards the middle of the road in the distance to the right." These paragraphs accurately express the spatial distribution and motion state of all key targets in the frame.

[0062] These language segments are then paired one-to-one with the original images and fed into a large-scale visual-language model for joint encoding. The system processes the input using a frozen large-scale visual-language model (such as a VLM backbone), where images are converted into a set of visual tokens by a visual encoder, and text is converted into a set of text tokens by a language encoder.

[0063] Internally, these tokens are cross-fused within the Transformer architecture using a multi-head attention mechanism. Spatial attention for visual tokens initially focuses on local details of the object, such as its outline, color, and structure. Subsequently, it incorporates contextual cues from linguistic tokens, guiding attention further towards areas that highly match the description. For example, a text token like "red traffic light" will guide the model to focus on a small red object in a highlighted area in visual space.

[0064] The model encoder outputs a fused semantic representation vector, which not only contains salient object features from the image but also explicitly preserves the attributes and relationships expressed in the language. This representation then enters the decoder module, supporting further extraction of the position, actions, and relationships of various entities in the scene.

[0065] During the training phase, the system uses accompanying tasks (such as target location prediction and relationship classification) as supervision signals to guide the model in optimizing the image-text joint attention mechanism and contextual understanding capabilities. The training samples consist of multiple scene frame pairs, each pair including an enhanced image, target description text, and location annotation. The loss function covers metrics such as target recognition error, location offset, and semantic matching.

[0066] The output of the entire perception module is a multimodal semantic representation for each frame, including unified encoding of the target entity, 3D position estimation, semantic labels, and motion state information. These representations are encapsulated into structured perception output and fed into the inference module for further processing.

[0067] 4. The method according to claim 1, wherein step S3 comprises:

[0068] After image enhancement and semantic perception using a large-scale visual-language model, the system enters the spatial reasoning stage. This stage uses a small-scale language model as the core component, responsible for parsing the temporal semantic information in the video sequence and outputting the logical reasoning process and answer to the specific question. The reasoning task is based on the graph-text structured semantic representation generated in the previous stage, and is decoded and generated in conjunction with the task question, ultimately achieving spatial understanding capability under low-resource conditions.

[0069] The inference process begins with input construction. The system first collects the semantic description text output by the perception module for each frame. This text, arranged chronologically, records information such as the target's category, relative position, motion trend, and depth value. Then, the question text for the current task is added to the end of the input to construct a complete input prompt. This prompt is organized with a unified structure and includes several paragraphs. <scene>The tag contains a scene description text, and a <question>The task of tagging. For example: <scene> A white sedan is approaching from slightly to the left ahead, about 10 meters away.< / scene> ... <question> From which direction did the car enter the frame?< / question> ".

[0070] After input preparation is complete, a small-scale, untuned language model is loaded, and its basic structure is frozen, with only high-level policy layer parameters exposed for training. During inference, the model outputs structured text, containing... <think>The reasoning process paragraphs identified by the labels, and <answer>The final answer sentence identified by the label. For example: <think> The vehicle slowly enters the frame from the right, and its position continuously shifts to the left across three consecutive frames...< / think> <answer> right side< / answer> The model generates different versions of reasoning paths and answers from multiple candidate outputs.

[0071] During training, a policy optimization-based training method is used to adjust the model. Specifically, a swarm intelligence policy algorithm is used to iteratively optimize the output policy distribution. For each input sample, the model generates a set of candidate outputs, which are evaluated using three reward items. The first is a format reward: checking whether the output correctly contains... <think>and <answer>Labels should be provided, and the format should be standardized. Secondly, there's an accuracy bonus: [the bonus will be applied to]... <answer>The paragraph is compared to the marked answer at the string level; a complete match is rewarded. Finally, there is a reward for logical consistency: <question>and <think>Given a frozen reference model as input, predict the derivation answer of the reasoning process if it matches... <answer>If they match, then the logic is considered consistent.

[0072] After each candidate output receives three reward scores, they are weighted and summed according to the weights corresponding to the current training stage to form the final reward value. The entire training process is divided into three stages: the first two rounds focus on format rewards to help the model establish output specifications; the middle two rounds focus on accuracy to improve answer quality; and the subsequent stages also consider logical consistency to ensure the rationality of the reasoning path. The weights of the three reward items are set to 7:3:0, 3:7:0, and 1:7:2, respectively.

[0073] The model policy optimization objective is based on the formula of the swarm intelligence policy algorithm. The objective function incorporates the pruned probability ratio, the advantage function, and the KL divergence term. In each iteration, the gradient is updated based on the ratio of the old and new policies to the same set of output probabilities, combined with the actual reward, thereby achieving efficient and stable policy optimization. The KL divergence penalty term is used to prevent the policy from changing too quickly and to maintain the quality of generation.

[0074] After training, the model is permanently used in the inference phase. Whenever the system receives a set of video semantic descriptions and question text, it can invoke the inference module to generate structured response text. In the final output, the system automatically extracts... <answer>The text within the label serves as the answer field, available for subsequent action decision-making or analysis modules to access. Meanwhile, the reserved... <think>Paragraphs can be used for process visualization, result interpretation, or anomaly analysis.

[0075] This small model inference module can run stably on edge devices or embedded platforms with limited computing resources and has the same logical reasoning capabilities as the large model output. It is trained using reinforcement learning rather than fine-tuning, enabling it to adapt to new task inputs while exhibiting strong generalization and structural stability.

[0076] This small model inference module can run stably on edge devices or embedded platforms with limited computing resources and has the same logical reasoning capabilities as the large model output. It is trained using reinforcement learning rather than fine-tuning, enabling it to adapt to new task inputs while exhibiting strong generalization and structural stability.

[0077] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.< / think> < / answer> < / answer> < / think> < / question> < / answer> < / answer> < / think> < / answer> < / think> < / question> < / scene> < / richmediareference> < / question> < / scene>

Claims

1. A multimodal driving video analysis method with image enhancement, characterized in that: Includes the following steps: Step S1: Image pre-training. The clear image and the degraded image are converted into latent spatial representations. Noise is added to the clear image using a diffusion probability model and the restoration task is guided by task cues. At the same time, ControlNet is used to fine-tune the model. Self-attention and text cross-attention mechanisms are used to ensure that the latent spatial representation is aligned with the task cues. The difference between the restored image and the target image is minimized by L2 loss to complete the training of the image restoration model. Step S2: Extract keyframes, detect objects, and locate 3D objects in the enhanced video; construct a semantic text description; and perform multimodal encoding through a visual language model to generate a structured perceptual representation. Step S3: Use a small-scale language model for structured spatial reasoning, and train the model using a policy optimization algorithm to generate reasoning processes and accurate answers under the input scenario description and question.

2. The image enhancement multimodal driving video analysis method according to claim 1, characterized in that: In step S1, the process of adding noise to the clear image using the diffusion probability model includes: forward diffusion of the clear image x0, adding Gaussian noise according to the noise scheduling table at each step t∈[1,T] to obtain the degraded image x0. t ,satisfy in, (α t () represents the predefined noise figure; The training objective is to minimize the error between the predicted noise and the actual noise, satisfying: Where, (∈ θ ) represents the model's prediction noise, and c represents the guidance and prompt information.

3. The image enhancement multimodal driving video analysis method according to claim 2, characterized in that: In step S1, the fine-tuning of ControlNet includes: ControlNet receives control input c and extracts multi-layer control features through branch networks, fusing them with the main network features in the form of residuals at each layer of the diffusion model, wherein the fusion method satisfies: in, This is the feature map of the l-th layer of the main U-Net network. These are features extracted by the ControlNet branch from the control input c; during training, the parameters of the main diffusion model are frozen, and only the ControlNet branch network is updated. The training objective satisfies:

4. The image enhancement multimodal driving video analysis method according to claim 3, characterized in that: In step S1, the self-attention mechanism is used to capture the long-distance dependencies of each part inside the image, and the text cross-attention mechanism is used to model the relationship between the input image and the task prompt. The two work together to align the latent spatial representation with the task prompt. The L2 loss function is used to keep the recovered image consistent with the target image at the pixel level.

5. The image enhancement multimodal driving video analysis method according to claim 1, characterized in that: The keyframe extraction in step S2 includes: calculating the inter-frame overlap rate based on the visual overlap between consecutive frames by matching feature points; when the overlap rate is lower than a set threshold, the current frame is identified as a keyframe and retained to form a sparse, non-redundant frame sequence.

6. The image enhancement multimodal driving video analysis method according to claim 5, characterized in that: The target detection and instance segmentation in step S2 includes: using a YOLO detector to extract the target's category, bounding box, and confidence score, and using a SAM model to provide pixel-level mask information for each target; By assigning a unique ID to each target using the intersection-union ratio and trajectory matching algorithm, consistent tracking across frames is achieved.

7. The image enhancement multimodal driving video analysis method according to claim 6, characterized in that: The three-dimensional localization in step S2 includes: generating a pixel-level depth map using the MiDaS monocular depth estimation model, extracting the center point and the average depth value of the mask within the target mask area, and obtaining the three-dimensional position (x, y, d) of the target in the image coordinate system, where x and y are image coordinates and d is the relative depth.

8. The image enhancement multimodal driving video analysis method according to claim 7, characterized in that: The semantic text description construction in step S2 includes: generating a description statement for each target according to the template of category + relative position + movement state + depth value, and combining the description statements of all targets in spatial order to form a scene-level language paragraph to express the spatial distribution and movement state of key targets in the frame.

9. The image enhancement multimodal driving video analysis method according to claim 8, characterized in that: The multimodal encoding in step S2 includes: pairing scene-level language paragraphs with the original image and inputting them into a visual language model; converting the image into a visual token via a visual encoder and the text into a text token via a language encoder; and then cross-fusing the two in a Transformer structure through a multi-head attention mechanism to output a semantic representation vector containing significant object features and language attributes and relationships.

10. The image enhancement multimodal driving video analysis method according to claim 1, characterized in that: Step S3 includes: Input prompt construction: Collect semantic description text for each frame, arrange it chronologically, and add task questions to form a prompt. <scene>Description of the scene of the tag package and <question> Input prompts for task-related questions tagged with tags;< / question> < / scene> Model training: Load a small-scale language model without fine-tuning, freeze the basic structure and only open the parameters of the high-level policy layer, train it using a swarm intelligence policy algorithm, evaluate candidate outputs through format reward, accuracy reward and logical consistency reward and sum them up by weight; Inference Output: The model output includes the inference process identified by labels and <richmediareference> The structured text of the final answer, identified by tags.< / richmediareference>

Images