Cross-view spatial visual language model training method for autonomous driving scenarios

Abstract

This invention discloses a cross-view spatial visual language model training method for autonomous driving scenarios. The method first acquires bird's-eye view images of the autonomous driving scenario, generates multi-camera images with target object markers through 3D bounding box projection, and uses an input-level bird's-eye view injection method to unify the bird's-eye view images and multi-camera images into a visual question-and-answer format to adapt to various spatial perception tasks. A parameter-efficient fine-tuning method is used to jointly optimize and train the pre-trained visual language model QwenVL 2.5 while keeping most pre-trained parameters frozen. This invention significantly improves the spatial perception capability of the visual language model, enhancing multi-view reasoning and global cognition through various spatial estimation tasks and 3D geometric constraints. Simultaneously, the parameter-efficient fine-tuning method trains only a small number of parameters, reducing resource consumption and maintaining inference speed, making it suitable for real-time deployment at the edge of autonomous driving.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, multimodal learning and autonomous driving technology, and in particular relates to a cross-view spatial visual language model training method for autonomous driving scenarios. Background Technology

[0002] Visual-language models have gradually become an important research direction in the field of multimodal artificial intelligence in recent years. By jointly modeling visual information and natural language information, these models have the ability to perform image understanding, visual question answering, multimodal reasoning, and semantic analysis of complex scenes. They have shown strong generalization ability in tasks such as general image understanding, visual question answering, and embodied intelligence, and are gradually being applied to fields such as autonomous driving, robotics, and intelligent transportation.

[0003] However, existing vision-language model-based autonomous driving methods still face four major challenges in spatial perception and spatial reasoning: First, existing training data often suffers from a significant long-tail distribution problem, meaning that while there are abundant semantic question-answering samples for common general scenarios, spatial perception question-answering samples for a large number of driving scenarios are scarce. This results in low accuracy in answering questions such as target distance estimation, target height estimation, deep reasoning, true distance estimation between targets from multiple perspectives, and spatial orientation relationship judgment. Second, research shows that models still rely primarily on language pattern matching and data priors in many tasks, rather than truly establishing stable spatial cognitive abilities. How to effectively improve the model's understanding of real 3D spatial structures has become a key challenge. Furthermore, edge devices have limited computing resources, making it difficult to directly deploy large-parameter vision-language models, additional 3D encoding modules, and feature fusion structures. Finally, fine-tuning and training the model may significantly damage its inherent semantic knowledge and language capabilities.

[0004] Therefore, there is an urgent need for a training method for autonomous driving visual language models that can effectively enhance the spatial perception and spatial reasoning capabilities of visual language models while taking into account model efficiency, deployment cost, and semantic capabilities. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of the above-mentioned technical solutions by proposing a cross-view spatial visual language model training method for autonomous driving scenarios, so as to improve the visual language model's ability to understand real three-dimensional spatial relationships, especially in core tasks such as target distance estimation, target height estimation, and multi-view target real distance estimation.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a cross-view spatial visual language model training method for autonomous driving scenarios, comprising the following steps:

[0008] S1. Organize multi-camera images with target object markers in the driving scene, bird's-eye view images representing the entire driving scene, and natural language questions into a unified visual question-and-answer format to construct training data and evaluation data for the spatial perception task.

[0009] S2. Input the training data into the pre-trained visual language model and perform efficient fine-tuning of the parameters of the visual language model in a joint optimization manner to complete the spatial perception task in the evaluation data.

[0010] Based on the above scheme, each step can be implemented in the following preferred manner.

[0011] As a preferred embodiment of the first aspect above, in S1, the spatial perception task includes: a target distance estimation task, a target height estimation task, and a multi-view real distance estimation task.

[0012] As a preferred embodiment of the first aspect mentioned above, in the target distance estimation task, multi-camera images, bird's-eye view images, and natural language questions from the training data are input into the visual language model, which is then required to output a predicted target distance value.

[0013] As a preferred embodiment of the first aspect mentioned above, in the target height estimation task, multi-camera images, bird's-eye view images, and questions in natural language form from the training data are input into the visual language model, which is then required to output a predicted value of the target object's height.

[0014] As a preferred embodiment of the first aspect, in the multi-view true distance estimation task, two multi-camera images, a bird's-eye view image, and a question in natural language form from the training data are input into the visual language model. The two target objects marked in the two multi-camera images are located at different viewpoints, and the visual language model is required to output the distance prediction value between the two target objects.

[0015] As a preferred option in the first aspect mentioned above, the Adam optimizer is used to minimize the total loss during the fine-tuning of the visual language model. :

[0016]

[0017] in, This represents the mean absolute error loss in the target distance estimation task; This represents the mean absolute error loss in the target height estimation task; This represents the mean absolute error loss for the multi-view true distance estimation task. Cross-entropy loss for generating text for visual language models; , , , The weighting coefficients for the losses of each task.

[0018] As a preferred option for the first aspect mentioned above, the visual language model adopts QwenVL 2.5. The parameters that need to be fine-tuned include the visual-language merger parameters, the Qwen2.5 language decoder parameters, and the low-rank adaptation parameters, while keeping the parameters in the word embedding layer and the visual encoder frozen.

[0019] In a second aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the cross-view spatial visual language model training method for autonomous driving scenarios as described in any of the solutions in the first aspect above.

[0020] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the cross-view spatial visual language model training method for autonomous driving scenarios as described in any of the solutions of the first aspect above.

[0021] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0022] The memory is used to store computer programs;

[0023] The processor is configured to, when executing the computer program, implement the cross-view spatial visual language model training method for autonomous driving scenarios as described in any of the solutions of the first aspect above.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] First, regarding spatial perception capabilities, this invention significantly enhances the visual language model's ability to understand real 3D spatial relationships. By constructing spatial perception tasks such as target distance estimation, target height estimation, and multi-view real distance estimation, and organizing them uniformly into a visual question-and-answer format, the model can learn distance, depth, and spatial topological relationships in the real physical world under natural language supervision. This invention explicitly introduces 3D geometric constraints from autonomous driving scenarios and utilizes multi-camera images from these scenarios, enabling the model to learn geometric correspondences and spatial consistency relationships between different cameras. This results in more stable spatial understanding and cross-view reasoning capabilities, enhancing the model's global spatial cognition and multi-view collaborative reasoning capabilities in complex autonomous driving scenarios.

[0026] Secondly, regarding training efficiency and engineering deployment, this invention employs a parameter-efficient fine-tuning strategy, training only the parameters of the vision-language merger, the Qwen2.5 language decoder, and the low-rank adaptation parameters, while keeping most pre-trained parameters frozen. This method significantly reduces the scale of training parameters and the consumption of training resources, while preserving the general visual semantic knowledge and cross-modal alignment capabilities of large-scale pre-trained models, thus improving model convergence speed and fine-tuning efficiency. Furthermore, since this invention does not require additional complex network structures or high-dimensional 3D feature branches, it maintains the inference speed and deployment compatibility of the original visual-language model, making it suitable for autonomous driving edge devices and real-time inference scenarios. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention;

[0028] Figure 2 This is a schematic diagram of the spatial perception task components in the method of the present invention;

[0029] Figure 3 This is a schematic diagram of the total loss composition in the method of the present invention;

[0030] Figure 4 This is a schematic diagram of a computer electronic device provided by the present invention. Detailed Implementation

[0031] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0032] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0033] like Figure 1 As shown, in a preferred embodiment of the present invention, the cross-view spatial visual language model training method for autonomous driving scenarios includes the following S1-S2 steps. The specific implementation process of each step will be described in detail below.

[0034] S1. The multi-camera images with target object markers in the driving scene, the bird's-eye view image representing the entire driving scene, and the questions in natural language form are uniformly organized into a visual question-and-answer format to construct training data and evaluation data for the spatial perception task.

[0035] It should be noted that in S1 of this invention, the multi-camera images are derived from the open-source dataset nuScenes. In this embodiment, the 3D bounding box, depth information, vehicle status information, map and lane data, and camera parameters of the target object are acquired simultaneously, and the multi-camera images with target object markers are constructed using this information.

[0036] Specifically, in this embodiment, for a target object in a driving scenario, this embodiment first determines the target center coordinates... Target size information If a 3D bounding box is generated, the eight corner points of the target object in the local coordinate system are represented as follows:

[0037]

[0038] in, These represent the x-axis component, y-axis component, and z-axis component of the center coordinates of the target object, respectively. Indicates the length of the target object; This represents the true value of the target object's width. Represents the true height of the target object; This represents the coordinate matrix of the eight corner points of the target object in the vehicle coordinate system.

[0039] Then, in this embodiment, the target object is transformed from the vehicle coordinate system to the camera coordinate system. The coordinate transformation is performed using the camera extrinsic matrix, and the specific calculation formula is as follows:

[0040]

[0041] in, This represents the coordinate matrix of the eight corner points of the target object in the camera coordinate system. For camera rotation matrix; Let be the camera translation vector.

[0042] For each corner point, the specific calculation formula is as follows:

[0043]

[0044] in, Indicates the target object's first The coordinates of each corner point in the camera coordinate system; These represent the x-axis component, y-axis component, and z-axis component of the coordinate, respectively.

[0045] Finally, this embodiment utilizes the camera intrinsic parameter matrix. The normalized points are projected onto a two-dimensional image plane and plotted in the corresponding image. For pixel coordinates in the two-dimensional image plane... The specific calculation formula is as follows:

[0046]

[0047]

[0048] in, They represent the first The column and row coordinates of each projection point in the image plane; , These represent the equivalent focal lengths of the camera along the u-axis and v-axis of the image coordinate system, respectively. , These represent the horizontal and vertical coordinates of the camera principal point in the image coordinate system, that is, the coordinates of the intersection of the optical axis and the image plane in the pixel coordinate system.

[0049] The above process establishes an explicit mapping relationship between the visual language model and the real physical space through realistic 3D geometric relationships. This embodiment utilizes a multi-camera synchronous sampling mechanism and high-precision 3D annotation information in autonomous driving scenarios, enabling the visual language model to simultaneously learn: image texture features, 3D spatial structure, cross-view geometric consistency, and realistic physical scale relationships during the training phase. This effectively alleviates the problem of traditional visual language models relying solely on semantic texture features and lacking real-world spatial understanding. Simultaneously, by accurately projecting 3D bounding boxes onto 2D images, a correspondence between real-world spatial coordinates and 2D visual observations is established. This allows the visual language model to learn distance, scale, depth, and spatial orientation relationships in real-world question-answering tasks, thereby enhancing the model's ability to model spatial relationships in autonomous driving scenarios.

[0050] It should be noted that in S1 of this invention, the bird's-eye view image itself includes lane line structure, road boundary information, target location information, vehicle status information, and map topology. The generated bird's-eye view information is then directly input into the visual language model in image form, and together with multi-camera images, constitutes the model input. Its core lies in explicitly providing the visual language model with the global spatial topology of the scene through input-level geometric information injection. Compared to existing methods that modify the main structure of the visual language model, insert a bird's-eye view encoder, or fuse 3D features, this invention does not require modification of the internal structure of the visual language model to provide the model with additional spatial geometric constraints. Therefore, without increasing the number of model parameters or the amount of inference computation, it can effectively improve the spatial reasoning ability and autonomous driving scene understanding ability of the visual language model.

[0051] It should be noted that in S1 of this invention, as Figure 2 As shown, the spatial perception task includes: target distance estimation task, target height estimation task, and multi-view true distance estimation task.

[0052] In the target distance estimation task, multi-camera images, bird's-eye view images, and natural language questions from the training data are input into the visual language model, which is required to output the target distance prediction value.

[0053] In this invention, the target distance refers to the distance from the vehicle coordinate system to the center coordinate of the target object.

[0054] In the target height estimation task, multi-camera images, bird's-eye view images, and natural language questions from the training data are input into the visual language model, which is required to output the predicted height of the target object.

[0055] In the multi-view true distance estimation task, two multi-camera images, a bird's-eye view image, and a question in natural language form from the training data are input into the visual language model. The two target objects marked in the two multi-camera images are located at different viewpoints. The visual language model is required to output the distance prediction value between the two target objects.

[0056] In the three tasks described above, the predicted distance to the target, the predicted height of the target object, and the predicted distance between two target objects are all numerical values ​​represented in natural language. The natural language questions include role and background guidance, existing information prompts, explicit task requirements, format requirements for the answers, and question-and-answer examples. This invention unifies these three tasks into a natural language visual question-and-answer format. Its core lies in utilizing a unified natural language interface to enable the visual language model to jointly model spatial reasoning and linguistic reasoning capabilities within a unified task framework. Compared to the single detection or trajectory prediction supervision methods in traditional autonomous driving tasks, this invention, by constructing explicit spatial question-and-answer tasks, allows the model to directly learn geometric, distance, and depth relationships in the real physical world. Simultaneously, through cross-viewpoint task construction, the model can learn the geometric consistency relationships between different cameras, thereby enhancing the model's spatial generalization ability in complex autonomous driving scenarios.

[0057] S2. Input the training data into the pre-trained visual language model and perform efficient parameter fine-tuning of the visual language model in a joint optimization manner (Low-Rank Adaptation) to complete the spatial perception task in the evaluation data.

[0058] It should be noted that in the fine-tuning of the visual language model, such as Figure 3As shown, the Adam optimizer is used to minimize the total loss. :

[0059]

[0060] in, This represents the mean absolute error loss in the target distance estimation task; This represents the mean absolute error loss in the target height estimation task; This represents the mean absolute error loss for the multi-view true distance estimation task. Cross-entropy loss for generating text for visual language models; , , , The weighting coefficients for the losses of each task.

[0061] In this embodiment, the visual language model uses QwenVL 2.5. The parameters requiring fine-tuning include the Vision-Language Merger parameters, the Qwen2.5 Language Decoder (LM Decoder) parameters, and the low-rank adaptation parameters, while keeping the parameters in the word embedding layer and the vision encoder frozen. Specifically, during model fine-tuning, the input question text is processed through the word embedding layer to obtain text features. The input multi-camera images and bird's-eye view images are processed by a visual encoder to obtain image features. Visual-language merger and Input, output fused enhanced features Decoding is performed using the Qwen2.5 language decoder. By freezing the word embedding layer and the parameters in the visual encoder, the model can fully preserve the general language semantics and basic structural priors obtained in large-scale data pre-training, effectively preventing knowledge forgetting on small-scale downstream tasks. At the same time, the training module includes a visual-language merging unit, a Qwen2.5 language decoder, and low-rank fitting parameters. This "selective training" strategy not only accelerates training by significantly reducing the number of parameters to be trained, but also effectively improves model performance by confining the optimization space and focusing gradient updates on the modality fusion layer that yields the highest benefit for specific tasks.

[0062] To achieve the aforementioned efficient fine-tuning, this invention, based on the original cross-entropy loss of QwenVL 2.5, introduces an additional mean absolute error loss for each of the three spatial perception tasks. Specifically: QwenVL 2.5 essentially performs a question-answering task, and its output is in text format; therefore, the cross-entropy loss can be calculated. For the target distance estimation task, The predicted target distance and the true target distance The mean absolute error loss is calculated; for the target height estimation task, The distance is calculated from the mean absolute error loss between the predicted height of the target object and the true height of the target object; for multi-view true distance estimation tasks, The distance between two target objects is the predicted distance and the true distance between the two target objects. The mean absolute error loss is calculated, where the true distance value is calculated as follows:

[0063]

[0064] in, and These represent the center coordinates of the two target objects in the vehicle coordinate system; These represent the x-component, y-component, and z-component of the center coordinates of the first target object, respectively. These represent the x-axis component, y-axis component, and z-axis component of the center coordinates of the second target object, respectively.

[0065] Additionally, it should be noted that after fine-tuning, the visual language model can simultaneously perform three spatial perception tasks: First, inputting multi-camera images, bird's-eye view images, and natural language questions from the evaluation data into the visual language model, requiring the model to output a predicted target distance. Second, inputting multi-camera images, bird's-eye view images, and natural language questions from the evaluation data into the visual language model, requiring the model to output a predicted target object height. Third, inputting two multi-camera images, bird's-eye view images, and natural language questions from the evaluation data into the visual language model, where the two target objects marked in the two multi-camera images are located at different viewpoints, requiring the model to output a predicted distance between these two target objects.

[0066] The present invention will now demonstrate the application effect of the cross-view spatial visual language model training method for autonomous driving scenarios described in S1~S2 of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0067] Example

[0068] The specific implementation process of the cross-view spatial visual language model training method for autonomous driving scenarios used in this embodiment is as described above and will not be repeated here.

[0069] For example, in order to verify the effectiveness of the method provided in this embodiment, the method of this embodiment is implemented on QwenVL 2.5, and a comparison is made before and after using this method.

[0070] MAE (Mean Absolute Error) is the most commonly used core indicator for evaluating regression models. It measures the difference between predicted and actual values. A lower MAE value generally indicates better predictive performance. The comparison results are shown in Table 1, demonstrating that the method proposed in this embodiment improves the model's capabilities in spatial perception tasks.

[0071] Table 1. Experimental data before and after using this method

[0072]

[0073] It is understood that the cross-view spatial visual language model training method for autonomous driving scenarios described in S1~S2 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the cross-view spatial visual language model training method for autonomous driving scenarios provided in the above embodiments. This product includes a computer program / instruction, which, when executed by a processor, can implement the cross-view spatial visual language model training method for autonomous driving scenarios as described in the above embodiments.

[0074] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the cross-view spatial visual language model training method for autonomous driving scenarios provided in the above embodiments, such as... Figure 4 As shown, it includes a memory and a processor;

[0075] The memory is used to store computer programs;

[0076] The processor is configured to implement the cross-view spatial visual language model training method for autonomous driving scenarios in the above embodiments when executing the computer program.

[0077] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0078] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the cross-view spatial visual language model training method for autonomous driving scenarios provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the cross-view spatial visual language model training method for autonomous driving scenarios in the above embodiments.

[0079] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S2.

[0080] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0081] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0082] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A cross-view spatial visual language model training method for autonomous driving scenarios, characterized in that, Includes the following steps: S1. Organize multi-camera images with target object markers in the driving scene, bird's-eye view images representing the entire driving scene, and natural language questions into a unified visual question-and-answer format to construct training data and evaluation data for the spatial perception task. S2. Input the training data into the pre-trained visual language model and perform efficient fine-tuning of the parameters of the visual language model in a joint optimization manner to complete the spatial perception task in the evaluation data.

2. The cross-view spatial visual language model training method for autonomous driving scenarios as described in claim 1, characterized in that, In S1, the spatial perception task includes: target distance estimation task, target height estimation task, and multi-view real distance estimation task.

3. The cross-view spatial visual language model training method for autonomous driving scenarios as described in claim 2, characterized in that, In the target distance estimation task, multi-camera images, bird's-eye view images, and natural language questions from the training data are input into the visual language model, which is required to output the target distance prediction value.

4. The cross-view spatial visual language model training method for autonomous driving scenarios as described in claim 2, characterized in that, In the target height estimation task, multi-camera images, bird's-eye view images, and natural language questions from the training data are input into the visual language model, which is required to output the predicted height of the target object.

5. The cross-view spatial visual language model training method for autonomous driving scenarios as described in claim 2, characterized in that, In the multi-view true distance estimation task, two multi-camera images, a bird's-eye view image, and a question in natural language form from the training data are input into the visual language model. The two target objects marked in the two multi-camera images are located at different viewpoints. The visual language model is required to output the distance prediction value between the two target objects.

6. The cross-view spatial visual language model training method for autonomous driving scenarios as described in claim 2, characterized in that, In fine-tuning the visual language model, the Adam optimizer is used to minimize the total loss. : ； in, This represents the mean absolute error loss in the target distance estimation task; This represents the mean absolute error loss in the target height estimation task; This represents the mean absolute error loss for the multi-view true distance estimation task. Cross-entropy loss for generating text for visual language models; , , , The weighting coefficients for the losses of each task.

7. The cross-view spatial visual language model training method for autonomous driving scenarios as described in claim 1, characterized in that, The visual language model uses QwenVL 2.

5. The parameters that need to be fine-tuned include the visual-language merger parameters, the Qwen2.5 language decoder parameters, and the low-rank adaptation parameters, while keeping the parameters in the word embedding layer and the visual encoder frozen.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the cross-view spatial visual language model training method for autonomous driving scenarios as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the cross-view spatial visual language model training method for autonomous driving scenarios as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement, when executing the computer program, the cross-view spatial visual language model training method for autonomous driving scenarios as described in any one of claims 1 to 7.

Images