Automatic driving detection method and system based on efficient fine-tuning visual pre-training model
By using the Multi-Rank LoRA fine-tuning module SaRA in autonomous driving detection, only fine-tuning of the parameters of the initial visual pre-training model is solved, and the problem of poor detection performance under severe weather conditions is achieved, efficient and robust multi-scene detection is achieved, reducing the Computing resources and data requirements.
Patent Information
- Application Number
- CN202411153109.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-22
AI Technical Summary
The existing autonomous driving target detectors perform poorly in severe weather conditions, and the existing fine-tuning technology lacks improvement and optimization in many scenarios, relying on high-cost diversified training data and high computing resources.
Using the Multi-Rank LoRA-based fine-tuning module SaRA, insert it into the initial visual pre-training model, only fine-tuning the parameters of the fine-tuning module, and through singular value decomposition and linear transformation, an efficient detection network is built to adapt to a variety of data. distributed.
It significantly reduces computing resources and time costs, improves detection performance and robustness in multiple scenarios, enhances the generalization of the model, and can perform better than specific optimization methods under limited scenario training data.
Smart Images

Figure CN119007154A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving detection method and system based on an efficient fine-tuning visual pre-training model. Background Art
[0002] With the rapid development of autonomous driving technology, its performance in various complex environments has attracted more and more attention. However, in autonomous driving scenarios, severe weather conditions, such as heavy rain, heavy snow, and dense fog, can significantly reduce the effectiveness of detectors, making accurate target detection more challenging. Existing target detectors often experience significant performance degradation in these severe weather conditions, which not only affects the driving safety and stability of autonomous vehicles, but also poses a potential threat to the safety of passengers and pedestrians. Therefore, improving the target detection capabilities of autonomous driving systems under various complex weather conditions and developing more robust and reliable detection algorithms have become important research topics.
[0003] In recent years, most robust detection works have been optimized for specific scenarios, such as considering different weather conditions, lighting changes, or specific target types, but these works have weak generalization in multiple scenarios. However, with the emergence of large visual models (such as the Segment Anything Model (SAM) proposed by Kirillov et al. (Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., ... & Girshick, R.: Segment anything. In: CVPR (2023)), these models have been widely used in various downstream tasks, but in the target detection task in autonomous driving scenarios, there is still a large gap compared to some traditional specific scene optimization algorithms.
[0004] Due to the large number of parameters of large models, it is too expensive to fully fine-tune these models to adapt to specific downstream tasks. At the same time, fine-tuning the entire model will also significantly lose the generalization of the model obtained in the pre-training stage. Therefore, researchers have developed a large number of fine-tuning techniques, such as the Low-Rank Adaptation (LoRA) proposed by Hu et al. (Hu, EJ, Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W.: Lora: Low-rank adaptation of large language models. In: arXiv preprint (2021)), which can greatly reduce training costs and improve performance by training fine-tuning modules to apply large models to specific downstream tasks. However, existing fine-tuning techniques still lack improvement and optimization in multiple scenarios.
[0005] Although the above-mentioned technologies based on multi-scenario optimization and large visual model application have made significant progress in improving the performance of autonomous driving target detection, these methods still face some challenges. First, these methods often rely on a large amount of high-quality training data, especially data in severe weather and complex lighting conditions. However, it is costly and time-consuming to obtain and annotate enough diverse training data. Secondly, although the pre-trained large model has strong generalization capabilities, fine-tuning in specific scenarios still requires a lot of computing resources and time. Even with fine-tuning techniques such as LoRA, fine-tuning for multiple scenarios is still a complex process. Summary of the invention
[0006] In view of the defects in the prior art, the object of the present invention is to provide an autonomous driving detection method and system based on an efficient fine-tuning visual pre-training model.
[0007] According to one aspect of the present invention, there is provided an autonomous driving detection method based on an efficient fine-tuning visual pre-training model, comprising:
[0008] Get the initial visual pre-training large model;
[0009] Inserting the fine-tuning module into the initial visual pre-trained large model to form a detection network;
[0010] Training the detection network, and only fine-tuning the parameters of the fine-tuning module during the training process;
[0011] Use the trained detection network for autonomous driving detection.
[0012] Preferably, the initial visual pre-trained large model is a SAM model, which includes an image encoder, a prompt encoder and a lightweight mask decoder, and can generate high-quality segmentation masks under various input prompts.
[0013] Preferably, the fine-tuning module is a fine-tuning module SaRA based on Multi-Rank LoRA, wherein Multi-RankLoRA is a fine-tuning layer improved based on LoRA, which is composed of multiple LoRA sub-modules with different Rank values and has the characteristic of being able to adapt to a variety of data distributions.
[0014] In the fine-tuning module SaRA, the data flow is specifically as follows:
[0015] The input image is feature extracted and processed into two branches;
[0016] After a branch undergoes random feature loss, softmax, and Top2 operations, it obtains multiple feature maps, which are concatenated as masks.
[0017] Multiply the mask by the original feature map of another branch to obtain a processed feature map;
[0018] The processed feature map is input into Multi-Rank LoRA for feature extraction.
[0019] Preferably, the inserting of the fine-tuning module into the initial visual pre-training model to form a detection network includes:
[0020] Insert the Multi-Rank LoRA-based fine-tuning module SaRA into the image encoder of the initial SAM model;
[0021] Perform singular value decomposition on the Neck parameters of the initial SAM model to obtain the detection network, that is, the fine-tuned SAM model;
[0022] Where Multi-Rank LoRA includes parameter W up , W down , respectively represent the parameters of the low-rank matrix in multiple LoRA submodules in Multi-Rank LoRA, W up ∈R r×C , W down ∈R C×r ;
[0023] The Neck part of the initial SAM model includes parameters U, V, and S, which represent two basis matrices and a coefficient vector after singular value decomposition, respectively. m×m , V∈R m×n , S∈R m×1 .
[0024] Preferably, the Multi-Rank LoRA-based fine-tuning module SaRA is inserted into the image encoder of the initial SAM model, comprising:
[0025] Get the hidden layer features H in the SAM image encoder, H∈r H×W×C , H, W are height and width, C is the number of channels;
[0026] Using the parameter matrix W down Perform linear transformation on H to reduce the dimension and obtain H down ∈R H×W×r ;
[0027] Using the parameter matrix W up For H down Perform linear transformation to increase the dimension and obtain △H∈R H×W×C ;
[0028] Update the hidden layer feature H according to the formula H←H+s△H, where s is the hyperparameter coefficient.
[0029] Preferably, the initial visual pre-training large model is a Segment Anything Model, i.e., a SAM model, and the singular value decomposition is performed on the Neck parameters of the initial SAM model to obtain a detection network, i.e., a fine-tuning SAM model, including:
[0030] First, the initial SAM model contains a neck layer, which includes two convolutional layers, and the convolutional layer is represented as an m*n parameter matrix;
[0031] Secondly, based on the SVD singular value decomposition principle, the m*n parameter matrix is decomposed into three parameter matrices with shapes of m*m, m*n and n*n respectively;
[0032] Finally, during the training process, the parameter matrices with shapes of m*m and n*n are frozen, and the parameter matrix with shape of m*n is retained for training.
[0033] Preferably, the training of the detection network, and only fine-tuning the parameters of the fine-tuning module during the training process, includes:
[0034] Obtain task training data and divide it into training set and validation set;
[0035] The detection network is trained using the training data. During the training, the detection network is fixed except for S and W. up , W down All parameters except for S and W are calculated on the training set. up , W downIterative learning is performed, and training is stopped when the detection network has learned a preset number of rounds for the training set.
[0036] According to a second aspect of the present invention, there is provided an autonomous driving detection system based on an efficient fine-tuning visual pre-training model, comprising:
[0037] Pre-training module: obtain the initial visual pre-training model;
[0038] Building module: inserting the fine-tuning module into the initial visual pre-training model to form a detection network;
[0039] Fine-tuning module: training the detection network, and only fine-tuning the parameters of the fine-tuning module during the training process;
[0040] Detection module: Use the trained detection network to perform autonomous driving detection.
[0041] According to a third aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can be used to execute the autonomous driving detection method based on an efficient fine-tuning visual pre-training model, or to run the autonomous driving detection system based on an efficient fine-tuning visual pre-training model.
[0042] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, it can be used to execute the autonomous driving detection method based on an efficient fine-tuning visual pre-training model, or to run the autonomous driving detection system based on an efficient fine-tuning visual pre-training model.
[0043] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:
[0044] The autonomous driving detection method and system based on the efficient fine-tuning visual pre-training model proposed in the embodiments of the present invention innovatively integrate the fine-tuning module into the large model and only adjust the parameters of the fine-tuning module, which significantly reduces the scale of parameter learning and effectively saves computing resources and time costs. It shows excellent efficiency and practicality, and improves the detection performance in multiple scenarios compared with existing methods.
[0045] The embodiments of the present invention propose an autonomous driving detection method and system based on an efficient fine-tuning visual pre-training model. By performing SVD decomposition on the Neck part of the SAM image encoder and training only a small number of parameters, the detection performance can be improved with minimal additional training cost.
[0046] The embodiments of the present invention propose an autonomous driving detection method and system based on an efficient fine-tuning visual pre-training model, wherein the model can extract more general features and has stronger robustness; the training of the model requires less data for extreme scenarios such as bad weather and complex lighting than the existing technical methods.
[0047] The autonomous driving detection method and system based on the efficient fine-tuning visual pre-training model proposed in the embodiments of the present invention have been experimentally verified on two benchmark datasets, SHIFT and BDD100k. The results show that the method in the embodiments of the present invention enhances the generalization of the model in multiple scenarios. In actual scenarios where only a limited number of scene training datasets are provided, the method in the embodiments of the present invention is significantly superior to existing robust detection methods that are specifically optimized for various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0049] Figure 1 is a flow chart of an automatic driving detection method based on an efficient fine-tuning visual pre-training model in one embodiment of the present invention;
[0050] Figure 2 It is a schematic diagram of the overall structure of the detection network based on the Multi-Rank LoRA fine-tuning module (SaRA) in a preferred embodiment of the present invention;
[0051] Figure 3 It is a data flow diagram of a fine-tuning module (SaRA) based on Multi-Rank LoRA in a preferred embodiment of the present invention;
[0052] Figure 4 It is a schematic diagram of the SAM Neck fine-tuning principle in a preferred embodiment of the present invention;
[0053] Figure 5 This is a data effect diagram of applying the autonomous driving detection method based on the efficient fine-tuning visual pre-training model to the SHIFT and BDD100k datasets in a specific embodiment of the present invention. DETAILED DESCRIPTION
[0054] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0055] In one embodiment of the present invention, an automatic driving detection method based on an efficient fine-tuning visual pre-training model is provided. Figure 1 As shown, including:
[0056] S1, obtain the initial visual pre-training large model;
[0057] S2, inserts the fine-tuning module into the initial visual pre-trained large model obtained in S1 to form a detection network;
[0058] S3, trains the detection network constructed in S2, and only fine-tunes the parameters of the fine-tuning module during the training process;
[0059] S4, uses the detection network trained in S3 to perform autonomous driving detection.
[0060] The above embodiment innovatively integrates the fine-tuning module into the large model and only adjusts the parameters of the fine-tuning module, which significantly reduces the scale of parameter learning, effectively saves computing resources and time costs, demonstrates excellent efficiency and practicality, and improves detection performance in multiple scenarios compared to existing methods.
[0061] In a preferred embodiment of the present invention, the initial visual pre-trained large model is a Segment Anything Model, i.e., a SAM model, which includes an image encoder, a prompt encoder, and a lightweight mask decoder, which can generate high-quality segmentation masks under various input prompts.
[0062] In a preferred embodiment of the present invention, the fine-tuning module is a fine-tuning module SaRA based on Multi-Rank LoRA. In the fine-tuning module SaRA, Multi-Rank LoRA is a fine-tuning layer improved based on LoRA, which is composed of multiple LoRA sub-modules with different Rank values, and has the characteristics of being able to adapt to a variety of data distributions.
[0063] To facilitate the description of the embodiments, the following embodiments are described in detail by taking the SAM model and the fine-tuning module SaRA based on Multi-Rank LoRA as examples. However, it should be understood that the SAM model and the fine-tuning module SaRA based on Multi-Rank LoRA are only preferred embodiments of the present invention. In other embodiments, other initial visual pre-trained large models and fine-tuning modules may also be used. The scope of the present invention is not limited to this.
[0064] Specifically, Figure 3 As shown, in the fine-tuning module SaRA, the specific data processing process is as follows:
[0065] The input image is feature extracted and then divided into two branches. One branch concatenates multiple feature maps as a mask after random feature loss, softmax, and Top2 operations. The mask part is multiplied by the original feature map of the other branch to obtain the processed feature map, which is input into Multi-Rank LoRA for processing, and features are selected from the extracted features, for example, the features that best represent the weather and environment are selected.
[0066] The above-mentioned feature extraction of the input image may further include feature processing, and the processing here includes but is not limited to feature screening, such as screening out noise features to improve robustness.
[0067] In a preferred embodiment of the present invention, S2 is implemented to insert the fine-tuning module into the initial visual pre-training model to form a detection network, and the specific steps are as follows:
[0068] S2.1: Insert the Multi-Rank LoRA-based fine-tuning module SaRA into the image encoder of the initial SAM model;
[0069] S2.2: Perform singular value decomposition on some parameters of the initial SAM Neck to obtain a fine-tuned SAM model;
[0070] Where Multi-Rank LoRA includes parameter W up , W down Represents the parameters of the low-rank matrix in multiple LoRA submodules in Multi-Rank LoRA, W up ∈R r×C , W down ∈R C×r , where SAM Neck includes parameters U, V, S, which represent two basis matrices and a coefficient vector after SVD decomposition, U∈R m×m , V∈R m×n , S∈R m×1 .
[0071] In some preferred embodiments, a preferred solution is further provided for the insertion process of S2.1, specifically:
[0072] Get the hidden layer features H in the SAM image encoder, H∈r H×W×C , H, W are height and width, C is the number of channels;
[0073] Using the parameter matrix W down Perform linear transformation on H to reduce the dimension and obtain H down ∈R H×W×r ;
[0074] Using the parameter matrix W up For Hdown Perform linear transformation to increase the dimension and obtain △H∈R H×W×C ;
[0075] Update the hidden layer feature H according to the formula H←H+s△H, where s is the hyperparameter coefficient.
[0076] In some preferred embodiments, a preferred method is further provided for the singular value decomposition process of S2.2, such as Figure 3 As shown, specifically:
[0077] First, SAM contains a neck layer, which is mainly composed of two convolutional layers, and the convolutional layer can be expressed as an m*n parameter matrix;
[0078] Secondly, based on the SVD singular value decomposition principle, the m*n matrix can be decomposed into three parameter matrices with shapes of m*m, m*n and n*n respectively;
[0079] Finally, during the training process, the parameter matrices with shapes of m*m and n*n are frozen, and the parameter matrix with shape of m*n is retained for training.
[0080] In the above embodiment, by performing SVD decomposition on the Neck part of the SAM image encoder and training only a small number of parameters, the detection performance can be improved with very little additional training cost.
[0081] In a preferred embodiment, a process of the entire detection network is provided, such as Figure 2 As shown in the figure, the input image on the left represents the classification of the dataset mainly from two perspectives, namely the acquisition time and the weather environment when the image was collected. The middle represents a part of the encoder of the SAM model, namely a single Transformer Block. Most of the parameters pretrainedweights in the SAM model are frozen and inserted into the fine-tuning module (MoE layer). In this fine-tuning module, LoRA with different rank values is first used for processing, mainly to reduce the number of training parameters through low-rank matrix approximation.
[0082] In a preferred embodiment of the present invention, S3 is implemented to train the detection network, and only the parameters of the fine-tuning module are fine-tuned during the training process. The specific process is as follows:
[0083] S3.1 obtain task training data and divide it into a training set and a validation set;
[0084] S3.2 uses the training data to train the detection network. During the training, the detection network is fixed except for S and W. up , W down All parameters except for S and W are calculated on the training set. up , Wdown Iterative learning is performed, and the training is stopped when the detection network has learned the training set for a preset number of rounds. Generally, the preset number of rounds is set to 12.
[0085] The above embodiment only adjusts the parameters of the fine-tuning module, which significantly reduces the scale of parameter learning, effectively saves computing resources and time costs, demonstrates excellent efficiency and practicality, and improves detection performance in multiple scenarios compared to existing methods.
[0086] In some specific embodiments, the pre-trained large model is a SAMDet model, the encoder part of which is the same as the encoder of the SAM model, and the decoder part is designed with reference to the decoder of ViDet. Similarly, when fine-tuning the parameters of the SAMDet model in the autonomous driving 2D detection scenario, only the parameters of the fine-tuning module (SaRA) need to be fine-tuned, which greatly reduces the scale of parameter learning.
[0087] In addition, it should be noted that the detection network of the present invention can extract more general features and has strong robustness. Therefore, the training of the model requires less data for extreme scenes such as bad weather and complex lighting than in the prior art.
[0088] Based on the same inventive concept, in another embodiment of the present invention, an autonomous driving detection system based on an efficient fine-tuning visual pre-training model is provided, comprising:
[0089] Pre-training module: obtain the initial visual pre-training model;
[0090] Building module: inserting the fine-tuning module into the initial visual pre-training model to form a detection network;
[0091] Fine-tuning module: training the detection network, and only fine-tuning the parameters of the fine-tuning module during the training process;
[0092] Detection module: Use the trained detection network to perform autonomous driving detection.
[0093] The specific implementation techniques of each module / unit in the above example of the present invention may refer to the corresponding steps of the autonomous driving detection method based on the efficient fine-tuning visual pre-training model in the above embodiment, which will not be repeated here.
[0094] In order to verify the feasibility and effect of the autonomous driving detection method and system based on the efficient fine-tuning visual pre-training model in the above embodiments, in some specific embodiments of the present invention, an experimental verification is carried out in which a single scene data training is performed on the autonomous driving target detection benchmark data sets such as SHIFT and BDD100k, and then tested in all scenes to verify the generalization of the method and system in the embodiments in multiple scenes. For simplicity, SAMDet+LoRA is represented as SAMDet+L, Multi-LoRA is represented as ML, Multi-Rank LoRA is represented as MRL, and SVD-Neck is represented as SN.
[0095] 1. Comparison with existing detection methods on the simulation dataset SHIFT:
[0096] Table 1 shows the experimental results of the SHIFT dataset in all scenarios after training in sunny daytime scenes.
[0097]
[0098] According to the experimental results in Table 1 (see Figure 5 ), on the simulation data set, the method of the embodiment of the present invention can maintain strong robustness in all scenarios by training only in the standard scenario (sunny daytime), and can be well generalized from the standard scenario (sunny daytime) to various scenario settings, reducing the dependence on high-quality data sets for multiple scenarios.
[0099] 2. Comparison with existing detection methods on the real data set BDD100k:
[0100] Table 2 shows the experimental results of the BDD100k dataset in the sunny daytime scene after training and testing in all scenes (see Figure 5 )
[0101]
[0102] According to the experimental results in Table 2, the method of the embodiment of the present invention can overcome the noise in the real data set and obtain stable performance improvement under different weather and time conditions, which proves that the method of the embodiment of the present invention has strong generalization ability.
[0103] In another specific embodiment of the present invention, the method of the embodiment of the present invention is experimentally verified by training full-scene data on autonomous driving target detection benchmark datasets such as SHIFT and BDD100k and then testing in all scenarios to verify the characterization capability of the present invention in multiple scenarios.
[0104] 1. Comparison with existing detection methods on the simulation dataset SHIFT:
[0105] Table 3 shows the experimental results of training in all scenarios in the SHIFT dataset and testing in all scenarios.
[0106]
[0107] According to the experimental results in Table 3, the method of the embodiment of the present invention can learn better representations for different scenarios, and can improve performance compared to the baseline model and fine-tuning using only the initial LoRA.
[0108] 2. Comparison with existing detection methods on the real data set BDD100k:
[0109] Table 4 shows the experimental results of testing in all scenarios after training in all scenarios in the BDD100k dataset.
[0110]
[0111] According to the experimental results in Table 4, the method in the embodiment of the present invention can overcome the noise in the real data set and obtain stable performance improvement under different weather and time conditions, which proves that the method in the embodiment of the present invention has strong characterization ability.
[0112] In another specific embodiment, the SAM Neck for SVD decomposition in the method of the embodiment of the present invention selects different trainable parameter configurations and then performs experimental verification on the SHIFT data set to verify the parameter efficiency of the embodiment of the method. For simplicity, Coefficient is represented as C and Bases is represented as B.
[0113] Table 5 shows the experimental results obtained by configuring the method of the embodiment of the present invention according to different parameter amounts.
[0114]
[0115] Experimental results show that choosing to train only the Coefficient method can significantly improve model performance while introducing the least amount of parameters, has the highest parameter efficiency, and reduces training costs.
[0116] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-mentioned autonomous driving detection method based on an efficient fine-tuning visual pre-training model, or, to run the above-mentioned autonomous driving detection system based on an efficient fine-tuning visual pre-training model.
[0117] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.
[0118] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.
[0119] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.
[0120] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above-mentioned autonomous driving detection method based on the efficient fine-tuning visual pre-training model, or to run the above-mentioned autonomous driving detection system based on the efficient fine-tuning visual pre-training model.
[0121] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.
[0122] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0123] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0126] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An autonomous driving detection method based on an efficient fine-tuning visual pre-training model, characterized in that: include: Get the initial visual pre-training large model; Inserting the fine-tuning module into the initial visual pre-trained large model to form a detection network; Training the detection network, and only fine-tuning the parameters of the fine-tuning module during the training process; Use the trained detection network for autonomous driving detection.
2. The automatic driving detection method based on the efficient fine-tuning visual pre-training model according to claim 1 is characterized in that: The initial visual pre-training large model is a SAM model, which includes an image encoder, a prompt encoder and a lightweight mask decoder, and can generate high-quality segmentation masks under various input prompts.
3. The automatic driving detection method based on the efficient fine-tuning visual pre-training model according to claim 2 is characterized in that: The fine-tuning module is a fine-tuning module SaRA based on Multi-Rank LoRA, wherein Multi-Rank LoRA is a fine-tuning layer improved based on LoRA, which is composed of a plurality of LoRA submodules with different Rank values, and has the characteristic of being able to adapt to a variety of data distributions; In the fine-tuning module SaRA, data processing includes: The input image is subjected to feature extraction and divided into two branches; After a branch undergoes random feature loss, softmax, and Top2 operations, it obtains multiple feature maps, which are concatenated as masks. Multiply the mask by the original feature map of another branch to obtain a processed feature map; The processed feature map is input into Multi-Rank LoRA for feature extraction.
4. The automatic driving detection method based on efficient fine-tuning visual pre-training model according to claim 3 is characterized in that: The step of inserting the fine-tuning module into the initial visual pre-training model to form a detection network includes: Insert the Multi-Rank LoRA-based fine-tuning module SaRA into the image encoder of the initial SAM model; Perform singular value decomposition on the Neck parameters of the initial SAM model to obtain the detection network, that is, the fine-tuned SAM model; Where Multi-Rank LoRA includes parameter W up , W down , respectively represent the parameters of the low-rank matrix in multiple LoRA submodules in Multi-Rank LoRA, W up ∈R r×C , W down ∈R C×r ; The Neck part of the initial SAM model includes parameters U, V, and S, which represent two basis matrices and a coefficient vector after singular value decomposition, respectively. m×m , V∈R m×n , S∈R m×1 .
5. The automatic driving detection method based on efficient fine-tuning visual pre-training model according to claim 4 is characterized in that: The Multi-Rank LoRA-based fine-tuning module SaRA is inserted into the image encoder of the initial SAM model, including: Get the hidden layer features H in the SAM image encoder, H∈R H×W×C , H, W are height and width, C is the number of channels; Using the parameter matrix W down Perform linear transformation on H to reduce the dimension and obtain H down ∈R H×W×r ; Using the parameter matrix W up For H down Perform linear transformation to increase the dimension and obtain △H∈R H×W×C ; Update the hidden layer feature H according to the formula H←H+s△H, where s is the hyperparameter coefficient.
6. The automatic driving detection method based on efficient fine-tuning visual pre-training model according to claim 4 is characterized in that: The singular value decomposition is performed on the Neck parameters of the initial SAM model to obtain a detection network, that is, a fine-tuned SAM model, including: First, the initial SAM model contains a neck layer, which includes two convolutional layers, and the convolutional layer is represented as an m*n parameter matrix; Secondly, based on the SVD singular value decomposition principle, the m*n parameter matrix is decomposed into three parameter matrices with shapes of m*m, m*n and n*n respectively; Finally, during the training process, the parameter matrices with shapes of m*m and n*n are frozen, and the parameter matrix with shape of m*n is retained for training.
7. The automatic driving detection method based on efficient fine-tuning visual pre-training model according to claim 4 is characterized in that: The step of training the detection network and fine-tuning only the parameters of the fine-tuning module during the training process includes: Obtain task training data and divide it into training set and validation set; The detection network is trained using the training data. During the training, the detection network is fixed except for S and W. up , W down All parameters except for S and W are calculated on the training set. up , W down Iterative learning is performed, and training is stopped when the detection network has learned a preset number of rounds for the training set.
8. An autonomous driving detection system based on an efficient fine-tuned visual pre-training model, characterized in that: include: Pre-training module: obtain the initial visual pre-training model; Building module: inserting the fine-tuning module into the initial visual pre-training model to form a detection network; Fine-tuning module: training the detection network, and only fine-tuning the parameters of the fine-tuning module during the training process; Detection module: Use the trained detection network to perform autonomous driving detection.
9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.
Citation Information
Cited By
Automatic driving track planning and control method and system based on parameter space skill combination and storage medium
CN122047307A
