A method for constructing a video understanding model, a video understanding method, and a system.

Abstract

The application relates to the technical field of automatic driving scene understanding, and specifically discloses a video understanding model construction method, a video understanding method and a system, which comprises the following steps: automatically processing original images and 3D boundary box labeling information in an original data set to obtain 2D surrounding box coordinates and relative depth maps of key traffic participants, and generating a VQA data set; obtaining open world driving scene picture text pairs and generating thought chain text prompts; based on the thought chain text prompts, the open world driving scene picture text pairs and the VQA data set, performing multi-round fine-tuning training on a visual language model, so as to obtain a multi-modal large language model; and fusing the multi-modal large language model with a target detection model to obtain a video understanding model for automatic driving. The method enables the model to have meter-level distance estimation and speed quantitative reasoning capabilities, solves the problem of key target positioning in an open world scene, and enhances the output interpretability of the model.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving scene understanding technology, specifically to a video understanding model construction method, video understanding method, and system. Background Technology

[0002] With the breakthroughs achieved by traditional Vision Language Models (VLMs) in general vision tasks, researchers have begun to explore their application in autonomous driving scene understanding. However, existing VLMs rely heavily on pre-training with general Internet data and lack specialized training for quantitative geometric reasoning in three-dimensional space, resulting in extremely poor performance in key physical quantity estimation tasks: the model can qualitatively describe "there is a vehicle ahead", but cannot quantitatively output "the distance to the vehicle is 23.5 meters and the relative speed is 12.3 km / h". This lack of quantitative reasoning at the perception layer has become a core bottleneck restricting the reliability of end-to-end autonomous driving systems. Especially in video understanding tasks, existing models struggle to capture the continuous motion trajectories and speed change patterns of objects in dynamic traffic scenes, failing to provide reliable input for the decision-making layer. To improve the spatial perception capabilities of VLMs, recent research has attempted to fine-tune them by constructing spatial reasoning datasets. For example, self-QA methods are used to generate question-answer pairs from LLMs, or geometric relationship descriptions are "generated" based on 3D boxes. However, these methods suffer from several drawbacks: VLMs skip quantitative reasoning at the perception layer and directly focus on decision planning, resulting in insufficient precision in the underlying perception layer; the spatial reasoning data is generated only from 3D boxes, lacking a strict coordinate projection chain from 2D to 3D; and the reliance on trial and error in the OnlineRL environment poses safety risks and makes it difficult to effectively supervise quantitative tasks.

[0003] In summary, traditional visual language models (VLMs) lack quantitative spatial geometric reasoning capabilities in autonomous driving video scene understanding tasks, making it difficult to accurately estimate physical quantities such as distance and speed of key objects in traffic scenes, and their generalization ability is insufficient in open-world scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, the first objective of this invention is to provide a method for constructing a video understanding model for autonomous driving. This method generates a physically verifiable Visual Question Answering (VQA) dataset, enabling the model to perform meter-level distance estimation and quantitative speed reasoning. It avoids capability coupling interference caused by one-time training through a four-wheel progressive LoRA fine-tuning paradigm. By integrating the GroundingDINO expert detection model with task-customized thought chain prompts, it solves the problem of key target localization in open-world scenarios, enhancing the interpretability of model output, thereby improving the video understanding model's quantitative spatial geometric reasoning and environmental perception capabilities in autonomous driving scenarios.

[0005] The second objective of this invention is to provide a video understanding model construction system for autonomous driving.

[0006] A third objective of this invention is to provide a video understanding method.

[0007] The first technical solution adopted in this invention is: a video understanding model construction method, comprising the following steps: S100: Obtain the original dataset, perform automated data processing on the original images and their 3D bounding box annotation information in the original dataset to obtain the 2D bounding box coordinates and relative depth maps of key traffic participants; and generate the VQA dataset based on the 2D bounding box coordinates, the original images and the relative depth maps. S200: Obtain open-world driving scene image-text pairs and generate thought chain text prompts; and perform multi-round fine-tuning training on the visual language model based on the thought chain text prompts, open-world driving scene image-text pairs and VQA dataset to obtain a multimodal large language model; S300: The multimodal large language model and the object detection model are fused to obtain a video understanding model for autonomous driving.

[0008] Preferably, step S100 includes the following sub-steps: S110: Extract the original 3D annotation information and camera parameters of traffic participants from the original dataset; S120: Filter the original 3D annotation information of the traffic participants based on visibility and centrality to obtain the 3D annotation information of key traffic participants; S130: Based on the camera parameters, perform computer graphics coordinate transformation on the 3D annotation information of the key traffic participants to obtain the 2D bounding box coordinates and 3D annotation information of the key traffic participants in the vehicle coordinate system. S140: Input the original image into the depth conversion expert model to obtain a relative depth map; S150: Generate ground truth annotations for key traffic participants based on their 3D annotation information in the vehicle coordinate system; and generate a VQA dataset based on the ground truth annotations of the key traffic participants, the original image, the 2D bounding box coordinates, and the relative depth map.

[0009] Preferably, step S120 includes: The original 3D bounding boxes of key traffic participants are projected onto the camera canvas to obtain 2D bounding boxes, and visibility and centrality are calculated. Based on the visibility and centrality, the comprehensive value weight of each traffic participant is calculated. If the comprehensive value weight of any traffic participant is greater than or equal to a preset threshold, it is determined to be a high-value traffic object, and the original 3D annotation information corresponding to the traffic participant is retained to obtain the 3D annotation information of key traffic participants. If the overall value weight of any traffic participant is less than a preset threshold, it is determined to be a low-value object and is removed.

[0010] Preferably, the deep transformation expert model mentioned in step S140 is DepthanythingV2.

[0011] Preferably, step S150 includes: S151: Generate ground truth labels for key traffic participants based on their 3D annotation information in the vehicle coordinate system. The ground truth labels include: the radar coordinates of the key traffic participants in the current frame in the vehicle coordinate system, the Euclidean distance between the key traffic participants and the vehicle in the current frame, and the average speed of the key traffic participants in the driving scenario. S152: Construct text prompts based on the 2D bounding box coordinates of key traffic participants; S153: Generate a continuous driving scene video based on the original image, and generate a corresponding relative depth video based on the relative depth map; S154: Generate a first VQA dataset based on the text prompt, Euclidean distance, original image, and relative depth map; generate a second VQA dataset based on the text prompt, average speed, driving scene video, and relative depth video; and generate a third VQA dataset based on the text prompt, radar coordinates, driving scene video, and relative depth video.

[0012] Preferably, step S200 includes: The visual language model is trained based on the first VQA dataset to obtain the first round of fine-tuned VLMs model; The VLMs model after the first round of fine-tuning was trained based on the second VQA dataset to obtain the VLMs model after the second round of fine-tuning. The VLMs model after the second round of fine-tuning was trained based on the aforementioned thought chain text prompts and the third VQA dataset to obtain the VLMs model after the third round of fine-tuning. Furthermore, the VLMs model after the third round of fine-tuning is trained based on the open-world driving scene image and text pairs to obtain a multimodal large language model.

[0013] Preferably, step S300 includes: An object detection expert model is embedded into a multimodal large language model. The object detection expert model is used to detect the 2D bounding box coordinates of key traffic participants in a video in an open world scene. The 2D bounding box coordinates of the key traffic participants are then embedded into the text prompts of the multimodal large language model according to rules.

[0014] The second technical solution adopted in this invention is: a video understanding model construction system, including a data processing module, a fine-tuning training module, and a model fusion module; The data processing module is used to acquire the original dataset, automatically process the original images and their 3D bounding box annotation information in the original dataset to obtain the 2D bounding box coordinates and relative depth maps of key traffic participants; and generate a VQA dataset based on the 2D bounding box coordinates, the original images and the relative depth maps. The fine-tuning training module is used to acquire open-world driving scene image-text pairs and generate thought chain text prompts; and to perform multiple rounds of fine-tuning training on the visual language model based on the thought chain text prompts, open-world driving scene image-text pairs and VQA dataset, thereby obtaining a multimodal large language model; The model fusion module is used to fuse the multimodal large language model with the object detection model to obtain a video understanding model for autonomous driving.

[0015] The third technical solution adopted in this invention is: a video understanding method, comprising: real-time acquisition of a video to be detected, inputting the video to be detected, key traffic participant category information to be detected, and thought chain text prompts into the video understanding model described in the first technical solution, to obtain overall environmental perception and analysis of key traffic participants.

[0016] Preferably, it includes: The video to be detected is sampled into several video frames. The key traffic participant category information to be detected and several video frames are input into the target detection expert model in the video understanding model. The target detection expert model performs detection and outputs the 2D bounding box coordinates of the key traffic participants in each frame. The 2D bounding box coordinates of the key traffic participant, several video frames, and thought chain text prompts are input into the multimodal large language model. The multimodal large language model embeds the 2D bounding box coordinates of the key traffic participant into the text prompts of the multimodal large language model through text embedding, which helps the multimodal large language model to locate the key traffic participant's position, and then performs subsequent reasoning based on the thought chain text prompts.

[0017] The beneficial effects of the above technical solution are as follows: (1) The present invention provides a video understanding model construction method for autonomous driving, which focuses on improving the quantitative geometric reasoning ability of the perception layer. By constructing a data pipeline with strict geometric mapping, introducing depth information as an independent training modality, and integrating expert models and thought chain prompts, the visual language model is equipped with meter-level distance estimation and quantitative speed reasoning ability.

[0018] (2) This invention constructs an automated data processing pipeline with strict geometric mapping to form an end-to-end mapping of 3D world coordinates → 2D bounding box → radar coordinates → distance / velocity truth value, and further generates a visual command question answering (VQA) dataset with physical verifiability, enabling the model to have meter-level distance estimation and velocity quantitative reasoning capabilities.

[0019] (3) This invention uses a four-round progressive LoRA fine-tuning paradigm to enable the model to gradually establish a cognitive chain of "vision → geometry → physics" and avoid the ability coupling interference caused by one-time training.

[0020] (4) This invention integrates target detection expert models (such as GroundingDINO) with task-customized thinking chain prompts to form a closed loop of "detection → localization → reasoning", which solves the problem of key target localization in open world scenarios and enhances the interpretability of model output.

[0021] (5) Experiments show that the video understanding model for autonomous driving constructed by the present invention, which is fine-tuned with special data, is significantly better than the 235B parameter large model (e.g., Qwen3-235B-A22B) in the distance estimation task (6.80% error), proving that "high-quality data + targeted training" is better than "blindly increasing the number of parameters", and provides accurate and reliable environmental representation for the perception layer of autonomous driving. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a method for constructing a video understanding model for autonomous driving, as provided in one embodiment of the present invention. Figure 2 A schematic diagram of the workflow of an automated data processing pipeline provided in one embodiment of the present invention; Figure 3 A comparison diagram of standard prompts and thought chain prompts provided for one embodiment of the present invention. Detailed Implementation

[0023] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0024] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0025] Example 1 like Figure 1 As shown, one embodiment of the present invention provides a method for constructing a video understanding model for autonomous driving, including the following steps: S100: Obtain the original dataset, perform automated data processing on the original images and their 3D bounding box annotation information in the original dataset to obtain the 2D bounding box coordinates and relative depth maps of key traffic participants; and generate the VQA dataset based on the 2D bounding box coordinates, the original images and the relative depth maps. Obtaining the original dataset includes: obtaining the original dataset from publicly available autonomous driving datasets, such as the nuScenes public dataset; the original dataset includes, but is not limited to, the original images, traffic participants in the original images and their 3D bounding box annotations, and the corresponding camera intrinsic and extrinsic parameters.

[0026] like Figure 2 As shown, this invention constructs an automated data processing pipeline, and based on this automated data processing pipeline, achieves automated data processing. The automated data processing pipeline is used to perform the following operations to obtain visual question-answering pairs: S110: Extract the original 3D annotation information and camera parameters of traffic participants from the original dataset; The system uses automated scripts to traverse the original dataset, extracting 3D annotation information of traffic participants in each frame, as well as the corresponding camera intrinsic and extrinsic parameters. It also extracts the image path of the forward-looking camera in each frame.

[0027] The nuScenes dataset organizes the raw data in a hierarchical structure, with the smallest granularity being a "sample," which is each frame. Each frame of data is associated with the sensor data (such as CAM_FRONT images) at that moment and the raw 3D annotations of traffic participants.

[0028] Since the input in this invention is a monocular video from a forward-looking camera, and the original dataset contains 360-degree panoramic annotations, viewpoint matching and filtering are required. Specifically, the extraction of the original 3D annotation information includes: Panoramic reading: The original dataset is traversed by an automated script to read the 3D bounding box annotation information of all traffic participants in each frame. Traffic participant categories include vehicles, pedestrians, motorcycles, bicycles, obstacles, and cones, etc. Field of view filtering: Using the camera extrinsic matrix, the corner coordinates of the 3D bounding box are projected onto the 2D image plane of the front-view camera; if the coordinates projected onto the 2D image plane are all outside the image boundary, it is determined that the annotation is not within the front-view field of view and is excluded. Category filtering: Based on task requirements, pre-defined categories of no interest (such as distant small traffic participants or specific non-target traffic participants) are excluded, and only valid target annotations are retained to obtain the original 3D annotation information of traffic participants.

[0029] Camera parameter capture includes: In the original dataset, each frame is labeled with its associated camera category (e.g., CAM_FRONT). The script directly reads the intrinsic and extrinsic matrices of the corresponding camera by matching the frame ID with the sensor calibration table to obtain the camera parameters for subsequent coordinate transformation.

[0030] S120: Filter the original 3D annotation information of the traffic participants based on visibility and centrality to obtain the 3D annotation information of key traffic participants; To ensure the model prioritizes learning high-value traffic objects, the concept of a "center of vision region" is introduced for refined screening, ensuring the model focuses on learning high-value traffic objects (i.e., key traffic participants). High-value traffic objects are defined as key traffic participants with clear features, located within the driver's critical field of vision, and significantly contributing to model training convergence; specifically: 1) High data quality: It occupies an appropriate proportion in the image without obvious occlusion or blurring; 2) Key location: It is located in the central area of ​​the image that the driver or autonomous driving system is most concerned about; 3) Important category: It belongs to the detection category of vehicles, pedestrians, bicycles and motorcycles.

[0031] The field of view center (FOV-Center) is defined as a 2D rectangular area of ​​a specific size in the center of the front-view camera's canvas. The size of this area can be preset according to the scope of attention in the actual driving scenario (e.g., occupying 30%-50% of the total canvas area).

[0032] Based on visibility and centrality, this invention filters the original 3D annotation information extracted above to ensure that key traffic participants (vehicles, pedestrians, motorcycles, and bicycles) that need to be continuously monitored in the video are clearly visible in every frame, thereby obtaining key 3D annotation information, namely key traffic participants.

[0033] The original 3D annotation information of the traffic participants extracted above is filtered based on visibility and centrality, including: For each traffic participant's original 3D annotation information, its original 3D annotation box is projected onto the camera canvas to obtain a 2D box, and visibility and centrality are calculated; and based on the visibility and centrality, the comprehensive value weight of each traffic participant is calculated; if the comprehensive value weight of any traffic participant is greater than or equal to a preset threshold, it is determined to be a high-value traffic object, and the original 3D annotation information corresponding to that traffic participant is retained to obtain the 3D annotation information of key traffic participants; if the comprehensive value weight of any traffic participant is less than the preset threshold, it is determined to be a low-value object and is removed.

[0034] Visibility represents the visible size of traffic participants within the image frame, and it is calculated using the following formula: In the formula, Visibility; For the 2D frame area of ​​traffic participants; The total canvas area; The smaller the value, the smaller the area occupied by traffic participants in the image, and the less visible they are.

[0035] Centrality characterizes the degree to which traffic participants are close to the center of the image. Centrality is calculated using the following formula: In the formula, Centrality; The overlapping area between the 2D frame of the traffic participant and the "center of vision area"; For the 2D frame area of ​​traffic participants; The smaller the value, the closer the traffic participant is to the edge of the image, and the lower the centrality.

[0036] The overall value weight for each traffic participant is calculated using the following formula: In the formula, Weighted by comprehensive value; and Weighting coefficients (e.g.) ); Visibility; Centrality.

[0037] Filtering decision: Set a preset threshold ; like If a traffic object is identified as a high-value traffic object, its original 3D annotation information is retained to obtain the 3D annotation information of key traffic participants. like Objects deemed low-value (such as distant, blurry traffic participants or edge distractions) are removed.

[0038] This invention effectively filters out the labeling information of traffic participants that do not meet the requirements by filtering based on visibility and centrality, thereby improving the overall quality of the training dataset.

[0039] S130: Based on the camera parameters, perform computer graphics coordinate transformation on the 3D annotation information of the key traffic participants to obtain the 2D bounding box coordinates and 3D annotation information of the key traffic participants in the vehicle coordinate system. The filtered 3D annotation information of key traffic participants is subjected to computer graphics coordinate transformation, including: The rotation matrix is ​​obtained based on the rotation quaternion. The 3D annotation information (i.e., 3D bounding box coordinates) in the original world coordinate system is transformed to the vehicle coordinate system based on the rotation matrix to obtain the 3D annotation information in the vehicle coordinate system. Based on the camera parameters, the 3D annotation information in the vehicle coordinate system is transformed to the camera coordinate system to obtain the 3D bounding box projection. The 3D bounding box in the camera coordinate system is projected into 2D bounding box coordinates in the view of the forward-looking camera to obtain the 2D bounding box coordinates of the key traffic participants.

[0040] For example, rotation matrices can be obtained using rotation quaternions. Transform 3D annotation information from the world coordinate system to the vehicle coordinate system. Based on the extrinsic parameter matrix between the camera and the vehicle, the 3D coordinates in the vehicle coordinate system are transformed to the camera coordinate system. Combined with the camera's intrinsic parameter matrix, the 3D points in the camera coordinate system are projected into 2D image plane coordinates. That is, by combining the camera's intrinsic and extrinsic parameter matrix K, the 3D camera coordinates in the camera coordinate system are projected into 2D image plane coordinates (u, v), thereby determining the 2D bounding box coordinates. .

[0041] S140: Input the original image into the depth conversion expert model to obtain a relative depth map; The original images in the original dataset are input into the depth transformation expert model, which converts the RGB information in the original images into relative depth information to obtain the corresponding relative depth map; the depth transformation expert model is, for example, DepthanythingV2.

[0042] S150: Generate ground truth annotations for key traffic participants based on 3D annotation information in the vehicle coordinate system of key traffic participants; and generate a visual command question answering (VQA) dataset containing ground truth distance / speed / radar coordinates based on the ground truth annotations of the key traffic participants, the original image, the 2D bounding box coordinates and the relative depth map.

[0043] Generating a Visual Command Question Answering (VQA) dataset containing ground truth distance / velocity / radar coordinates includes the following sub-steps: S151: Generate ground truth annotations for key traffic participants, including: obtaining the radar coordinates ([x,y],m) of the key traffic participant in the current frame's vehicle coordinate system based on the 3D annotation information of the key traffic participant in the vehicle coordinate system; calculating the Euclidean distance (m) between the key traffic participant and the vehicle in the current frame based on the radar coordinates of the key traffic participant in the vehicle coordinate system; and obtaining the average speed (displacement / time interval) of the key traffic participant relative to the vehicle in the driving scenario based on the radar coordinates of the key traffic participant in each video frame according to the temporal difference strategy, i.e., obtaining the average speed (m / s) of the key traffic participant in the driving scenario.

[0044] The truth labels include: the radar coordinates ([x,y],m) of the key traffic participant in the current frame's vehicle coordinate system, the Euclidean distance (m) between the key traffic participant and the vehicle in the current frame, and the average speed (m / s) of the key traffic participant in the driving scenario.

[0045] S152: Construct rule-based text prompts based on the 2D bounding box coordinates of key traffic participants in the image to assist the visual language model in visually locating key traffic participants in the image. S153: Generate a continuous driving scene video based on the original image, and generate a corresponding relative depth video based on the relative depth map; The original images are segmented and aggregated according to specific scenes and durations to form a series of continuous driving scene videos. Since each original image has a corresponding relative depth map, several relative depth videos of driving scenes are generated based on the corresponding relative depth maps.

[0046] S154: Generate a first VQA dataset based on the text prompts, the Euclidean distance between key traffic participants and the vehicle in the current frame from the ground truth annotations, the original image, and the relative depth map; generate a second VQA dataset based on the average speed of key traffic participants in the driving scene from the text prompts, the ground truth annotations, the driving scene video, and the relative depth video; and generate a third VQA dataset based on the radar coordinates of key traffic participants in the vehicle coordinate system in the current frame from the text prompts, the ground truth annotations, the driving scene video, and the relative depth video. Based on the text prompts, the Euclidean distance between key traffic participants and the vehicle in the current frame in the ground truth annotation, the original image and the relative depth map, the present invention generates the first VQA dataset through a preset instruction template, that is, generates object detection and distance quantitative estimation question-answer pairs in static images; Based on the text prompts, the average speed of key traffic participants in the driving scenario in the truth value annotation, the driving scenario video and the relative depth video, a second VQA dataset is generated through a preset instruction template, that is, a quantitative reasoning question-and-answer pair of key traffic participants in the driving scenario video is generated. Based on the text prompts, the radar coordinates of key traffic participants in the current frame's vehicle coordinate system in the truth value annotation, the driving scene video, and the relative depth video, a third VQA dataset is generated through a preset instruction template, that is, a quantitative reasoning question-and-answer pair of the coordinates of key traffic participants in the vehicle coordinate system is generated. The first VQA dataset, the second VQA dataset, and the third VQA dataset constitute a visual command question answering (VQA) dataset containing ground truth distance / velocity / radar coordinates.

[0047] The first VQA dataset is, for example: messages = [ { "role": "user", "content": [ {"type": "image", "image": image_path (image path of the front-view camera corresponding to the current frame)}, {"type": "text", "text": "What category does the object within the 2D bounding box [[x1, y1], [x2, y2]] belong to, and what is its Euclidean distance from the ego vehicle in meters? "(在2D包围框[[x1, y1], [x2, y2]]中的物体属于什么类别?它与自车的欧氏距离为多少米?)}, , }, { "role": "assistant", "content": {"type": "text", "text": "Car. 15.60m."}, , } . The second VQA dataset is, for example: messages = { "role": "user", "content": {"type": "video", "video": video_path (the front view camera video path corresponding to the current scene, aggregated from images)}, {"type": "text", "text": "What category does the object enclosed by the red bounding box belong to, and what is its average speed relative to the ego vehicle in meters per second? "(被红色框包围的物体属于什么类别?它相对与自车的平均速度是多少米每秒?)}, , }, { "role": "assistant", "content": {"type": "text", "text": "Car. 6.50m / s."}, ], } ]. The third VQA dataset is, for example: messages = [ { "role": "user", "content": [ {"type": "image", "image": image_path (image path of the front-view camera corresponding to the current frame)}, {"type": "text", "text": "For the object within the 2D bounding box[[x1, y1], [x2, y2]], what are its radar coordinates [x, y] expressed in the ego-vehicle coordinate frame? Unit: meters." (What are the radar coordinates [x, y] of the object within the 2D bounding box [[x1, y1], [x2, y2]] in the ego-vehicle coordinate system, in meters?)} ], }, { "role": "assistant", "content": [ {"type": "text", "text": "[10.00, 5.05]."}, ], } ]. This invention constructs an automated data processing pipeline that incorporates computer graphics transformations. It precisely transforms the 3D world coordinates in the original dataset using computer graphics (rotation matrix → vehicle coordinates → camera coordinates → 2D projection) to form an end-to-end mapping chain of "2D visual input → 3D physical quantity truth value". Furthermore, it generates a VQA dataset containing truth distance / velocity / radar coordinates, enabling the model to have meter-level distance estimation and quantitative velocity inference capabilities.

[0048] S200: Obtain open-world driving scene image-text pairs and generate thought chain text prompts; perform multiple rounds of fine-tuning training on the visual language model based on the thought chain text prompts, open-world driving scene image-text pairs, and VQA dataset to obtain a multimodal large language model; (1) Obtaining open-world driving scene image-text pairs includes: Image-text pairs are introduced from external open-world autonomous driving datasets (such as CODA-LM) to obtain open-world driving scene image-text pairs, thereby enhancing the model's generalization ability to edge scenes and long-tailed distributions; the open-world driving scene image-text pairs include long-tailed scenes and rich semantic description information; The VQA dataset and the open-world driving scene image-text pairs are aligned in a unified format and packaged into a unified file to obtain a supervised fine-tuning dataset.

[0049] (2) Generate thought chain text prompts; like Figure 3 As shown, differentiated thought chain (CoT) text prompts are generated for different tasks. Task-customized thought chain (CoT) text prompts generate differentiated inference chains for different tasks, including speed inference and overall traffic scene perception, so that the VLMs model inference process conforms to human cognitive logic and enhances the interpretability of the output. For example, the inference chain is "2D frame → radar coordinates → distance" in the distance estimation task; in the speed inference task, the VLMs model is guided to first infer the radar coordinates of each video frame, then calculate the displacement, and finally deduce the speed, that is, to use the "2D frame → radar coordinates → displacement → speed" inference chain, which simulates the logical thinking process of human drivers. Compared with standard prompts, this invention enhances the interpretability of VLMs model output by introducing thought chain prompts.

[0050] (3) Perform multiple rounds of fine-tuning on the visual language model, including: The first VQA dataset is used to train the Vision Language Models (VLMs) to obtain the first-round fine-tuned VLMs model; the second VQA dataset is used to train the first-round fine-tuned VLMs model to obtain the second-round fine-tuned VLMs model; the thought chain text prompts and the third VQA dataset are used to train the second-round fine-tuned VLMs model to obtain the third-round fine-tuned VLMs model; and the open-world driving scene image text pairs are used to train the third-round fine-tuned VLMs model to obtain a multimodal large language model.

[0051] The visual language model mentioned above is, for example, Qwen2VL. In the above four rounds of progressive model fine-tuning training, supervised fine-tuning strategy (SFT) and low-rank adaptive technique (LoRA) are used to efficiently fine-tune the parameters, reducing the consumption of computing resources while achieving task adaptation. The training process adopts a cosine annealing learning rate adjustment strategy to dynamically adjust the learning pace in order to obtain better convergence accuracy, ensure that the VLMs model parameters converge and avoid overfitting.

[0052] 1) First round of fine-tuning training; The first round of fine-tuning training is based on the first VQA dataset to train the visual language model's ability to identify key traffic participants and estimate distances, thereby enhancing the VLMs model's ability to identify key traffic participant categories and quantitatively estimate distances in a scene, achieving static perception and quantitative distance estimation. The specific training process is as follows: The original images and relative depth maps in the first VQA dataset are aligned and fused through frame aggregation to construct a multi-channel input tensor; for example, the paths of the original image and the corresponding relative depth map are input into the prompt, and the preprocessor built into VLMs is converted into a multi-channel tensor. The multi-channel input tensor, text prompts from the first VQA dataset, and Euclidean distances between key traffic participants and the vehicle in the current frame from ground truth annotations are input into the visual language model to calculate the loss value between the visual language model's predicted text and the ground truth annotation text. The visual language model parameters are updated by backpropagation based on the loss value. After training is completed, the visual language model weights are saved to obtain the VLMs model after the first round of fine-tuning, which is used for training initialization in subsequent stages.

[0053] The VQA dataset contains depth-related prompts. When a relative depth map is input in the VQA dataset, prompts such as "This is the relative depth map of the current traffic scene" or "This is the relative depth video of the current traffic scene" will appear to guide the VLMs model in learning.

[0054] This invention introduces a relative depth map as an additional modality, enabling VLMs models to understand the geometry of an image and thus accurately answer questions such as "how far away is the object?"

[0055] (2) Second round of fine-tuning training; The second round of fine-tuning training is based on the second VQA dataset to train the VLMs model after the first round of fine-tuning to improve its quantitative speed inference ability. This enhances the VLMs model's ability to quantitatively perceive and estimate the average speed of key traffic participants in videos, enabling the VLMs model to directly learn the mapping from spatiotemporal depth changes to speed values. The specific training process is as follows: Temporal multimodal input construction: Driving scene videos and relative depth videos in the second VQA dataset are temporally aligned and feature fused using the preprocessor built into VLMs to construct a video input tensor containing spatiotemporal information; Temporal instruction fine-tuning training: The video input tensor and the average speed of key traffic participants in the driving scene in the text prompts and ground truth annotations in the second VQA dataset are input into the VLMs model after the first round of fine-tuning. The inter-frame motion relationship is modeled through the temporal attention mechanism, and the loss value related to the speed estimation task is calculated. Parameter update and model saving: The VLMs model parameters after the first round of fine-tuning are updated by backpropagation based on the loss value. After training is completed, the VLMs model weights after the first round of fine-tuning are saved to obtain the VLMs model after the second round of fine-tuning, which is used for training initialization in the subsequent chain inference stage.

[0056] This invention significantly improves velocity estimation accuracy by introducing relative depth video.

[0057] (3) Third round of fine-tuning training; The third round of fine-tuning training, based on the aforementioned thought chain text prompts and the third VQA dataset, trains the radar coordinate estimation capability of the VLMs model after the second round of fine-tuning. This guides the VLMs model to first infer the coordinates of key traffic participants in the video frame within the vehicle's radar coordinate system, then calculate the displacement from the coordinates, and finally infer the average velocity of the key traffic participants from the displacement. This guides the VLMs model to establish more rigorous physical reasoning logic, enhancing the interpretability and reasoning accuracy of the VLMs model. The specific training process is as follows: Chain-based reasoning prompt construction: Generate thought chain text prompts containing multiple physical reasoning steps, wherein the reasoning steps include at least spatial coordinate reasoning, displacement calculation and velocity derivation; Intermediate step supervised data generation: Based on the thought chain text prompts, annotated reasoning text containing the complete reasoning process is generated for driving scene videos and relative depth videos in the third VQA dataset; the annotated reasoning text presents intermediate reasoning results and the final answer step by step according to the reasoning steps; The generation of labeled inference text includes: based on the thought chain text prompts, inferring the displacement of key traffic participants in consecutive frames of driving scene video and relative depth video from the radar coordinates of key traffic participants in the vehicle coordinate system (or radar coordinate system); inferring the average speed of key traffic participants from the displacement and time interval of key traffic participants, and obtaining labeled inference text; among them, the radar coordinates use two-dimensional coordinates [x,y], because the z coordinate has a negligible impact on speed, so the BEV perspective approach is adopted.

[0058] Chained instruction fine-tuning training: The thought chain text prompts, driving scene videos and relative depth videos in the third VQA dataset are input into the VLMs model after the second round of fine-tuning, and the loss value between the inference sequence generated by the VLMs model after the second round of fine-tuning and the labeled inference text is calculated. Parameter update and model saving: The VLMs model parameters after the second round of fine-tuning are updated by backpropagation based on the loss value. After training is completed, the VLMs model weights after the second round of fine-tuning are saved to obtain the VLMs model after the third round of fine-tuning, which is used for the training initialization of the subsequent general scene perception stage.

[0059] (4) Fourth round of fine-tuning training; The fourth round of fine-tuning training is based on open-world driving scene images and text pairs to train the overall scene perception capability of the VLMs model after the third round of fine-tuning, thereby enhancing the VLMs model's ability to accurately identify and describe the attributes of various traffic elements in traffic scenes; the specific training process is as follows: Semantic Alignment Input Construction: The open-world driving scene image-text pairs are input into the VLMs model after the third round of fine-tuning to construct the semantic understanding task input; Generalization fine-tuning training: Calculate the loss value between the text description generated by the VLMs model after the third round of fine-tuning and the real text annotation in the open world driving scene image text pair. Update the parameters of the VLMs model after the third round of fine-tuning through backpropagation to enhance the ability of the VLMs model after the third round of fine-tuning to identify and describe the attributes of traffic elements. Model Output: After training, the VLMs model weights after the third round of fine-tuning are saved, resulting in a multimodal large language model with geometric reasoning capabilities and general scene awareness. This stage mainly utilizes general image and text data to prevent the VLMs model from overfitting to specific datasets and improve generalization ability.

[0060] This invention improves the ability of VLMs models to perform perception and understanding tasks through four rounds of progressive model fine-tuning.

[0061] S300: The multimodal large language model and the object detection model are fused to obtain a video understanding model for autonomous driving; The aforementioned multimodal large language model only has the ability to identify the categories of key traffic participants and does not have the ability to detect 2D bounding boxes. In order to further enhance the applicability of the multimodal large language model in the open world, an external object detection expert model is introduced for assistance. The object detection expert model provides the coordinates of the 2D bounding boxes of key traffic participants in the image, and then integrates them into the text prompt.

[0062] The object detection expert model and the multimodal large language model (i.e., VLMs after four rounds of fine-tuning training) are fused at the interface level (i.e., interface matching) to obtain the video understanding model (QwenDrivePro) for autonomous driving.

[0063] Object detection expert models (such as GroundingDINO) are used to detect the 2D bounding box coordinates of key traffic participants in each sampled frame of a video in an open-world scene; that is, the object detection expert model is used to output the normalized 2D bounding box coordinates of key traffic participants in the canvas in each frame; the 2D bounding box coordinates of key traffic participants output by the object detection expert model are directly embedded into the text prompts of the multimodal large language model according to the rules to assist the multimodal large language model in visual localization; The normalized 2D bounding box coordinates refer to the top-left to bottom-right xy coordinates that are proportionally divided in width and height, in the format [[0.05,0.1], [0.12,0.2]]. Normalization is to facilitate the direct input of the 2D bounding box coordinates into the text prompts of the multimodal large language model.

[0064] This invention seamlessly embeds the open set detection capability of an object detection expert model (e.g., GroundingDINO) into a multimodal large language model through a coordinate format conversion interface (normalization → pixel coordinates), forming a closed loop of "detection → localization → inference"; it solves the problem of key object localization in open-world scenarios and enhances the interpretability of model output.

[0065] Furthermore, step S300 also includes: performing performance verification on the video understanding model (QwenDrivePro) for autonomous driving.

[0066] The performance of QwenDrivePro was verified through ablation experiments and open-world scene testing, and the effects of relative depth information introduction and thought chain prompts on improving the quantitative reasoning ability of video understanding models were evaluated.

[0067] In the open-world scenario test, the video understanding model constructed by this invention can provide a structured description of open internet scenarios, accurately identify multiple types of traffic objects (buses, SUVs, motorcycles, traffic cones, etc.), and has strong overall perception performance. In the distance quantitative estimation task, on a test set of 1000 samples, the average relative error of the distance estimation of the video understanding model QwenDrivePro constructed in this invention is only 6.80%, which is significantly better than existing open source large models such as InternVL. In the speed quantitative reasoning task, the video understanding model that integrates thought chain cues and GroundingDINO auxiliary detection box inputs achieved a relative error of 16.42% on the 10km / h test set, while the untuned model had almost no such capability. The ablation experiments show that after introducing relative depth information, the percentage error of the video understanding model constructed in this invention on the quantitative velocity estimation task decreased from 51.8% to 46.4%, and the accuracy of the quantitative velocity estimation performance was improved. This proves that depth information can effectively supplement spatial features and provide valuable supplementary features for the model.

[0068] Example 2 One embodiment of the present invention provides a video understanding model construction system, including a data processing module, a fine-tuning training module, and a model fusion module; The data processing module is used to acquire the original dataset, automatically process the original images and their 3D bounding box annotation information in the original dataset to obtain the 2D bounding box coordinates and relative depth maps of key traffic participants; and generate a VQA dataset based on the 2D bounding box coordinates, the original images and the relative depth maps. The fine-tuning training module is used to acquire open-world driving scene image-text pairs and generate thought chain text prompts; based on the thought chain text prompts, open-world driving scene image-text pairs and VQA dataset, the visual language model is fine-tuned and trained in multiple rounds to obtain a multimodal large language model; The model fusion module is used to fuse the multimodal large language model with the object detection model to obtain a video understanding model for autonomous driving.

[0069] Example 3 An embodiment of the present invention provides a video understanding method, comprising: real-time acquisition of a video to be detected, inputting the video to be detected, key traffic participant category information to be detected, and thought chain text prompts into a video understanding model to obtain overall environmental perception and analysis of key traffic participants (i.e. key objects).

[0070] The video to be detected is sampled into several video frames. The key traffic participant category information to be detected and several video frames are input into the target detection expert model in the video understanding model. The target detection expert model performs detection and outputs the 2D bounding box coordinates of the key traffic participants in each frame. The 2D bounding box coordinates of the key traffic participant, several video frames, and thought chain text prompts are input into the multimodal large language model. The multimodal large language model embeds the 2D bounding box coordinates of the key traffic participant into the text prompts of the multimodal large language model through text embedding, which helps the multimodal large language model to locate the key traffic participant in the visual input. Then, the thought chain text prompts and previously learned abilities are used for subsequent reasoning. For example, if you want to know the Euclidean distance, ask for the Euclidean distance; if you want to know the average speed of the key traffic participant, ask for its speed.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0072] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0075] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing a video understanding model, characterized in that, Includes the following steps: S100: Obtain the original dataset, perform automated data processing on the original images and their 3D bounding box annotation information in the original dataset to obtain the 2D bounding box coordinates and relative depth maps of key traffic participants; and generate the VQA dataset based on the 2D bounding box coordinates, the original images and the relative depth maps. S200: Obtain open-world driving scene image-text pairs and generate thought chain text prompts; and perform multi-round fine-tuning training on the visual language model based on the thought chain text prompts, open-world driving scene image-text pairs and VQA dataset to obtain a multimodal large language model; S300: The multimodal large language model and the object detection model are fused to obtain a video understanding model for autonomous driving.

2. The video understanding model construction method according to claim 1, characterized in that, Step S100 includes the following sub-steps: S110: Extract the original 3D annotation information and camera parameters of traffic participants from the original dataset; S120: Filter the original 3D annotation information of the traffic participants based on visibility and centrality to obtain the 3D annotation information of key traffic participants; S130: Based on the camera parameters, perform computer graphics coordinate transformation on the 3D annotation information of the key traffic participants to obtain the 2D bounding box coordinates and 3D annotation information of the key traffic participants in the vehicle coordinate system. S140: Input the original image into the depth conversion expert model to obtain a relative depth map; S150: Generate ground truth annotations for key traffic participants based on their 3D annotation information in the vehicle coordinate system; and generate a VQA dataset based on the ground truth annotations of the key traffic participants, the original image, the 2D bounding box coordinates, and the relative depth map.

3. The video understanding model construction method according to claim 2, characterized in that, Step S120 includes: The original 3D bounding boxes of key traffic participants are projected onto the camera canvas to obtain 2D bounding boxes, and visibility and centrality are calculated. Based on the visibility and centrality, the comprehensive value weight of each traffic participant is calculated. If the comprehensive value weight of any traffic participant is greater than or equal to a preset threshold, it is determined to be a high-value traffic object, and the original 3D annotation information corresponding to the traffic participant is retained to obtain the 3D annotation information of key traffic participants. If the overall value weight of any traffic participant is less than a preset threshold, it is determined to be a low-value object and is removed.

4. The video understanding model construction method according to claim 2, characterized in that, The deep transformation expert model mentioned in step S140 is DepthanythingV2.

5. The video understanding model construction method according to claim 2, characterized in that, Step S150 includes: S151: Generate ground truth labels for key traffic participants based on their 3D annotation information in the vehicle coordinate system. The ground truth labels include: the radar coordinates of the key traffic participants in the current frame in the vehicle coordinate system, the Euclidean distance between the key traffic participants and the vehicle in the current frame, and the average speed of the key traffic participants in the driving scenario. S152: Construct text prompts based on the 2D bounding box coordinates of key traffic participants; S153: Generate a continuous driving scene video based on the original image, and generate a corresponding relative depth video based on the relative depth map; S154: Generate a first VQA dataset based on the text prompt, Euclidean distance, original image, and relative depth map; generate a second VQA dataset based on the text prompt, average speed, driving scene video, and relative depth video; and generate a third VQA dataset based on the text prompt, radar coordinates, driving scene video, and relative depth video.

6. The video understanding model construction method according to claim 5, characterized in that, Step S200 includes: The visual language model is trained based on the first VQA dataset to obtain the first round of fine-tuned VLMs model; The VLMs model after the first round of fine-tuning was trained based on the second VQA dataset to obtain the VLMs model after the second round of fine-tuning. The VLMs model after the second round of fine-tuning was trained based on the aforementioned thought chain text prompts and the third VQA dataset to obtain the VLMs model after the third round of fine-tuning. Furthermore, the VLMs model after the third round of fine-tuning is trained based on the open-world driving scene image and text pairs to obtain a multimodal large language model.

7. The video understanding model construction method according to claim 1, characterized in that, Step S300 includes: An object detection expert model is embedded into a multimodal large language model. The object detection expert model is used to detect the 2D bounding box coordinates of key traffic participants in a video in an open world scene. The 2D bounding box coordinates of the key traffic participants are then embedded into the text prompts of the multimodal large language model according to rules.

8. A video understanding model construction system, characterized in that, It includes a data processing module, a fine-tuning training module, and a model fusion module; The data processing module is used to acquire the original dataset, automatically process the original images and their 3D bounding box annotation information in the original dataset to obtain the 2D bounding box coordinates and relative depth maps of key traffic participants; and generate a VQA dataset based on the 2D bounding box coordinates, the original images and the relative depth maps. The fine-tuning training module is used to acquire open-world driving scene image-text pairs and generate thought chain text prompts; and to perform multiple rounds of fine-tuning training on the visual language model based on the thought chain text prompts, open-world driving scene image-text pairs and VQA dataset, thereby obtaining a multimodal large language model; The model fusion module is used to fuse the multimodal large language model with the object detection model to obtain a video understanding model for autonomous driving.

9. A video understanding method, characterized in that, include: The video to be detected is collected in real time, and the video to be detected, the key traffic participant category information to be detected, and the thought chain text prompts are input into the video understanding model according to any one of claims 1-7 to obtain overall environmental perception and analysis of key traffic participants.

10. The video understanding method according to claim 9, characterized in that, include: The video to be detected is sampled into several video frames. The key traffic participant category information to be detected and several video frames are input into the target detection expert model in the video understanding model. The target detection expert model performs detection and outputs the 2D bounding box coordinates of the key traffic participants in each frame. The 2D bounding box coordinates of the key traffic participant, several video frames, and thought chain text prompts are input into the multimodal large language model. The multimodal large language model embeds the 2D bounding box coordinates of the key traffic participant into the text prompts of the multimodal large language model through text embedding, which helps the multimodal large language model to locate the key traffic participant's position, and then performs subsequent reasoning based on the thought chain text prompts.

Images