Visual identification model training method and device for vehicle automatic driving
Through a two-stage knowledge distillation framework, the pure visual Camera and LiDAR models are trained, which improves the depth perception ability of pure visual Camera, solves the problem of insufficient perception of pure visual solutions in harsh environments, and achieves more cost-effective and efficient perception of autonomous driving environments .
Patent Information
- Application Number
- CN202410891449.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-26
AI Technical Summary
The existing pure visual Camera solution lacks the ability to perceive the depth information in the environment, especially in harsh environments, and cannot directly obtain the depth of field information, resulting in limited environmental perception capabilities of the autonomous driving system.
Using a two-stage knowledge distillation framework, using the Camera+LiDAR fusion model as a teacher model, knowledge distillation and mutual learning of pure visual Camera and pure LiDAR models are improved to improve the deep perception ability of pure visual Camera models and realize the fusion from multi-sensors to pure visual. knowledge transfer of the program.
It significantly improves the depth perception capability of the pure visual Camera model, can maintain high accuracy in complex environments, reduces system costs, and provides more reliable environmental perception support.
Smart Images

Figure CN119027898A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of neural network technology, and in particular to a visual recognition model training method for vehicle autonomous driving, a visual recognition model training device for vehicle autonomous driving, an image recognition method for vehicle autonomous driving, and an image recognition device for vehicle autonomous driving. Background Art
[0002] The safe and reliable operation of autonomous vehicles depends greatly on the accurate perception and modeling of the vehicle's surrounding environment. Environmental perception requires not only accurate detection and identification of objects such as obstacles, pedestrians, and other vehicles, but also high-precision estimation of their spatial position, size, and motion state. In order to improve the accuracy and robustness of vehicle perception of the environment, the current mainstream manufacturers' solutions often use a variety of heterogeneous sensors for fusion perception, including lidar, millimeter-wave radar, and visual cameras. Each sensor has its own unique advantages and limitations. By effectively fusing these heterogeneous data, their respective strengths can be fully utilized to improve the overall perception capability of the system. Among the many technical solutions for fusion perception, BEVFusion (Bird's Eye View Fusion) technology has become a hot topic of research due to its efficient environmental modeling method.
[0003] BEVFusion technology aims to build an accurate bird's-eye view (BEV) environment representation by fusing multi-sensor data. The BEV model intuitively displays the planar position and motion information of surrounding objects, providing critical three-dimensional environmental perception support for the autonomous driving decision-making system.
[0004] LiDAR continuously sends light pulses and receives light returned from objects, and detects the round-trip flight time of light pulses to obtain the distance of the target object. It has a long monitoring distance, high accuracy, relatively sensitive response speed, and is not affected by ambient light. However, LiDAR is expensive, and the light beams emitted will be affected in extreme weather such as rain and snow, affecting the construction of three-dimensional maps. It usually needs to rely on the joint assistance of other sensors. The multi-sensor BEVFusion solution that integrates visual data with LiDAR point cloud data can fully utilize the advantages of each sensor to obtain a more accurate three-dimensional environmental model. However, multi-sensor solutions often require higher system costs and algorithm complexity. Compared with the LiDAR+Camera hybrid solution, the pure vision Camera has the advantages of low cost and simple system, and can capture rich semantic and texture information. However, the camera relies on ambient light and is easily affected by harsh environments. It has poor perception capabilities under conditions such as night and bad weather, and cannot directly obtain depth of field information.
[0005] Although the pure vision camera solution lacks direct depth measurement capabilities, it is most similar to the real world perceived by the human eye and is also closest to the form of human driving. However, the biggest problem with the current pure vision camera solution is that it lacks the ability to perceive the depth information of the environment. Summary of the invention
[0006] The purpose of the present invention is to provide a visual recognition model training method for vehicle autonomous driving to solve at least one of the above-mentioned technical problems.
[0007] The present invention provides the following scheme:
[0008] According to one aspect of the present invention, a visual recognition model training method for vehicle autonomous driving is provided, and the visual recognition model training method for vehicle autonomous driving comprises:
[0009] Constructing a teacher model;
[0010] Get pure vision Camera model and pure LiDAR model;
[0011] Performing knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, so as to obtain a distilled pure vision Camera model and a distilled pure LiDAR model;
[0012] The distilled pure vision Camera model and the distilled pure LiDAR model are mutually learned, so as to obtain a mutually learned pure vision Camera model and a mutually learned pure LiDAR model.
[0013] Optionally, the constructing the teacher model includes:
[0014] Obtain an initial fusion model;
[0015] Obtain image training data and point cloud training data;
[0016] Aligning the image training data and the point cloud training data to obtain aligned image training data and aligned point cloud training data;
[0017] The initial fusion model is trained using the aligned image training data and the aligned point cloud training data.
[0018] Optionally, the teacher model f T The pure visual Camera model f S1 And the pure LiDAR model f S2 The process of knowledge distillation includes the following optimization objectives:
[0019]
[0020] in, is the teacher model bev feature, (i=1,2) is the visual Camera model f S1 And the pure LiDAR model f S2 bev feature, H is the height of the feature map, W is the width of the feature map, w k,x,y is the weight corresponding to the (x, y) position of the kth true detection box on the feature map, which is calculated according to the following formula:
[0021]
[0022] (x k ,y k ) is the center coordinate of the kth true detection box, and exp is the natural exponential function.
[0023] Optionally, when performing knowledge distillation on the pure vision Camera model, the features of the penultimate layer of the network of the pure vision Camera model are selected for distillation;
[0024] When performing knowledge distillation on the pure LiDAR model, the features of the penultimate layer of the network of the pure LiDAR model are selected for distillation.
[0025] Optionally, performing knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model to obtain a distilled pure vision Camera model and a distilled pure LiDAR model includes:
[0026] The bevdistill method is used to perform knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively.
[0027] The present application also provides a visual recognition model training device for vehicle autonomous driving, the visual recognition model training device for vehicle autonomous driving comprising:
[0028] A teacher model building module, wherein the teacher model building module is used to build a teacher model;
[0029] A model acquisition module, wherein the model acquisition module is used to acquire a pure vision Camera model and a pure LiDAR model;
[0030] A distillation module, wherein the distillation module is used to perform knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, so as to obtain a distilled pure vision Camera model and a distilled pure LiDAR model;
[0031] A mutual learning module, wherein the mutual learning module is used to mutually learn the distilled pure vision Camera model and the distilled pure LiDAR model, thereby obtaining a pure vision Camera model and a pure LiDAR model that have undergone mutual learning.
[0032] The present application also provides an image recognition method for automatic driving of a vehicle, the image recognition method for automatic driving of a vehicle comprising:
[0033] Obtain a pure vision Camera model trained by the visual recognition model training method for vehicle autonomous driving as described above;
[0034] Obtaining image information to be identified;
[0035] The image information to be recognized is input into the pure vision Camera model to obtain the recognition result.
[0036] The present application also provides an image recognition device for automatic driving of a vehicle, the image recognition device for automatic driving of a vehicle comprising:
[0037] A visual recognition model training device for vehicle autonomous driving, wherein the visual recognition model training device for vehicle autonomous driving is the visual recognition model training device for vehicle autonomous driving as described above;
[0038] An image acquisition module to be identified, wherein the image acquisition module to be identified is used to acquire information of the image to be identified;
[0039] The recognition module is used to input the image information to be recognized into the pure vision Camera model to obtain a recognition result.
[0040] The visual recognition model training method for vehicle autonomous driving of the present application adopts a two-stage knowledge distillation framework, in which the Camera+LiDAR fusion model is used as the teacher model, and the pure vision Camera and pure LiDAR models are used as student models. Knowledge distillation is used to allow the student model to learn the Camera perception ability and LiDAR depth perception ability learned by the teacher model respectively, and can learn the depth perception knowledge in the teacher model to realize the knowledge transfer from multi-sensor fusion to the pure vision solution.
[0041] Considering the modality difference between LiDAR and Camera, this paper proposes a method combining TeacherAssistant with two-stage distillation. In the first stage, the fusion model distills two student models, pure Camera and pure LiDAR. In the second stage, the two student models learn from each other through mutual learning, and the pure Camera model further understands and learns the depth information of LiDAR, thereby significantly improving its depth perception ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a flow chart of a visual recognition model training method for vehicle autonomous driving in one embodiment of the present application.
[0043] Figure 2 It is used to realize Figure 1 A schematic diagram of an electronic device for a visual recognition model training method for vehicle autonomous driving is shown.
[0044] Figure 3 It is a schematic diagram of the principle of teacher model training.
[0045] Figure 4 This is a schematic diagram of the principle of performing knowledge distillation on the pure vision Camera model and the pure LiDAR model through the teacher model.
[0046] Figure 5 This is a schematic diagram of the principle of performing knowledge distillation on the pure vision Camera model and the pure LiDAR model through the teacher model. DETAILED DESCRIPTION
[0048] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] Figure 1 It is a flow chart of a visual recognition model training method for vehicle autonomous driving in one embodiment of the present application.
[0050] like Figure 1 The visual recognition model training method for vehicle autonomous driving shown includes:
[0051] Step 1: Build a teacher model;
[0052] Step 2: Obtain a pure vision Camera model and a pure LiDAR model;
[0053] Step 3: performing knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, so as to obtain a distilled pure vision Camera model and a distilled pure LiDAR model;
[0054] Step 4: Perform mutual learning between the distilled pure vision Camera model and the distilled pure LiDAR model to obtain a mutual-learned pure vision Camera model and a mutual-learned pure LiDAR model.
[0055] The visual recognition model training method for vehicle autonomous driving of the present application adopts a two-stage knowledge distillation framework, in which the Camera+LiDAR fusion model is used as the teacher model, and the pure vision Camera and pure LiDAR models are used as student models. Knowledge distillation is used to allow the student model to learn the Camera perception ability and LiDAR depth perception ability learned by the teacher model respectively, and can learn the depth perception knowledge in the teacher model to realize the knowledge transfer from multi-sensor fusion to the pure vision solution.
[0056] Considering the modality difference between LiDAR and Camera, this paper proposes a method combining TeacherAssistant with two-stage distillation. In the first stage, the fusion model distills two student models, pure Camera and pure LiDAR. In the second stage, the two student models learn from each other through mutual learning, and the pure Camera model further understands and learns the depth information of LiDAR, thereby significantly improving its depth perception ability.
[0057] In this embodiment, constructing a teacher model includes:
[0058] Obtain an initial fusion model;
[0059] Obtain image training data and point cloud training data;
[0060] Aligning the image training data and the point cloud training data to obtain aligned image training data and aligned point cloud training data;
[0061] The initial fusion model is trained using the aligned image training data and the aligned point cloud training data.
[0062] See also Figure 3In this embodiment, the initial fusion model is a Camera+LiDAR fusion model. The BEVFusion method is used to build a Camera+LiDAR fusion model. A large amount of aligned image training data and point cloud training data are used to train the teacher model to achieve high accuracy in depth perception tasks. The image training data and point cloud training data are respectively extracted through their respective Encoder branches to extract corresponding features. and After processing, they are input into BEV Encoder to obtain BEV fusion features. Used for the final tasks, such as segmentation and 3D detection tasks, to obtain the output y of the fusion network T . Figure 3 The above process is demonstrated.
[0063] In this embodiment, the process of performing knowledge distillation on the pure vision Camera model and the pure LiDAR model by the teacher model includes the following optimization objectives:
[0064]
[0065] in, is the teacher model bev feature, (i=1,2) is the visual Camera model f S1 And the pure LiDAR model f S2 bev feature, H is the height of the feature map, W is the width of the feature map, w k,x,y is the weight corresponding to the (x, y) position of the kth true detection box on the feature map, which is calculated according to the following formula:
[0066]
[0067] (x k ,y k ) is the center coordinate of the kth true detection box, and exp is the natural exponential function.
[0068] In this embodiment, when performing knowledge distillation on the pure vision Camera model, the features of the penultimate layer of the network of the pure vision Camera model are selected for distillation;
[0069] When performing knowledge distillation on the pure LiDAR model, the features of the penultimate layer of the network of the pure LiDAR model are selected for distillation.
[0070] In this embodiment, the method of performing knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model to obtain the distilled pure vision Camera model and the distilled pure LiDAR model includes:
[0071] The bevdistill method is used to perform knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively.
[0072] For example, in this embodiment, a pure visual Camera model and a pure LiDAR model are selected as student models f S1 and student model f S2 , using the bevdistill method to S1 and f S2 Distillation is performed by minimizing the student model BEV features Integrate BEV features with teachers The difference between the two and the difference between the final outputs of the two are used to optimize the student model. Specifically, given the BEV fusion feature of the fusion network With the final output y T , train the student model f S1 When adding the following optimization objectives (training f S2 Same as reference):
[0073]
[0074] in, is the teacher model bev feature, (i=1,2) is the visual Camera model, f S1 And the pure LiDAR model f S2 bev features, H is the height of the feature map, and W is the width of the feature map.
[0075] In this embodiment, cross-modal distillation is not suitable for directly acting on all feature maps. For example, the Camera modality can provide relatively complete foreground and background information, while the LiDAR point cloud can only provide the area that the laser can reach, resulting in differences in information between different modalities. Here we use the same processing method as BEVDistill, that is, for each detected true box center point under BEV, a Gaussian mask is generated to assign different weights to the foreground features, and then the BEV features are distilled to ignore the impact of background distillation as much as possible, as follows:
[0076]
[0077] (x k ,y k) is the center coordinate of the kth true detection box, and exp is the natural exponential function.
[0078] In this embodiment, for the outputs of both, considering distilling information directly on logits, due to the small number of detection categories, we choose to distill the features of the penultimate layer of the network (i.e., the features of the layer before logits), as follows:
[0079]
[0080] Among them, KL is KL divergence (Kullback-Leibler Divergence), are the category prediction probabilities of the student model and the teacher model for the i-th prediction box, respectively. are the coordinate prediction positions of the i-th prediction box by the student model and the teacher model, q i The quality score of the fusion teacher model detection result is calculated as follows:
[0081]
[0082] Among them, IoU is the two prediction boxes Intersection over Union,
[0083] In this embodiment, the distilled pure vision Camera model and the distilled pure LiDAR model are mutually learned as follows:
[0084] Student Model S1 and student model f S2 Through mutual learning, the pure Camera model tries to learn the depth information of the pure LiDAR model, and vice versa. Through mutual learning, the two student models continuously optimize their respective depth perception capabilities, especially the pure Camera model's performance in depth perception is significantly improved. At this stage, the optimization goals of the two student models are basically the same as those of the one-way distillation stage: in the training f S1 When the variables related to the teacher network in formulas 2 and 3 are replaced by f S2 That’s it; training f S2 Same reason.
[0085] The visual recognition model training method for vehicle autonomous driving of the present application has the following advantages:
[0086] 1. Improve depth perception capability: Through the technical solution of the present invention, the pure vision Camera model can effectively learn the depth perception capability in the multi-sensor fusion model, and maintain a high depth perception accuracy even in complex environmental conditions, providing more reliable environmental perception support for the autonomous driving system.
[0087] 2. Economical and efficient: There is no need to add expensive LiDAR hardware. The depth perception capability of the pure vision camera can be improved through software-level optimization alone, which significantly reduces the system cost and makes autonomous driving technology more affordable and economical.
[0088] The present application also provides a visual recognition model training device for vehicle autonomous driving, the visual recognition model training device for vehicle autonomous driving comprises a teacher model building module, a model acquisition module, a distillation module and a mutual learning module, wherein:
[0089] The teacher model building module is used to build the teacher model;
[0090] The model acquisition module is used to obtain pure visual Camera model and pure LiDAR model;
[0091] The distillation module is used to perform knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, so as to obtain a distilled pure vision Camera model and a distilled pure LiDAR model;
[0092] The mutual learning module is used to mutually learn the distilled pure vision Camera model and the distilled pure LiDAR model, so as to obtain the mutually learned pure vision Camera model and the mutually learned pure LiDAR model.
[0093] The present application also provides an image recognition method for automatic driving of a vehicle, the image recognition method for automatic driving of a vehicle comprising:
[0094] Obtain a pure vision Camera model trained by the visual recognition model training method for vehicle autonomous driving as described above;
[0095] Obtaining image information to be identified;
[0096] The image information to be recognized is input into the pure vision Camera model to obtain the recognition result.
[0097] The present application also provides an image recognition device for automatic driving of a vehicle, the image recognition device for automatic driving of a vehicle comprising a visual recognition model training device for automatic driving of a vehicle, an image acquisition module to be recognized and a recognition module, wherein:
[0098] The visual recognition model training device for vehicle autonomous driving is the visual recognition model training device for vehicle autonomous driving as described above;
[0099] The to-be-recognized image acquisition module is used to acquire the to-be-recognized image information;
[0100] The recognition module is used to input the image information to be recognized into the pure vision Camera model to obtain a recognition result.
[0101] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.
[0102] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above-mentioned visual recognition model training method for vehicle autonomous driving is implemented.
[0103] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned visual recognition model training method for vehicle autonomous driving.
[0104] Figure 2 It is an exemplary structural diagram of an electronic device that can implement the visual recognition model training method for vehicle autonomous driving provided according to an embodiment of the present application.
[0105] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 501 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.
[0106] That is to say, Figure 2The electronic device shown may also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which may implement the combination of Figure 1 Described is a method for training visual recognition models for autonomous vehicle driving.
[0107] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program code; and one or more processors configured to run the executable program code stored in the memory 504 to execute the visual recognition model training method for vehicle autonomous driving in the above-mentioned embodiment.
[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0109] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0110] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0112] The processor referred to in this embodiment may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0113] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the device / terminal equipment by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0114] In this embodiment, if the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can be executed by the processor. The steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0116] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.
[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A visual recognition model training method for vehicle autonomous driving, characterized in that: The visual recognition model training method for vehicle autonomous driving includes: Constructing a teacher model; Get pure vision Camera model and pure LiDAR model; Performing knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, so as to obtain a distilled pure vision Camera model and a distilled pure LiDAR model; The distilled pure vision Camera model and the distilled pure LiDAR model are mutually learned, so as to obtain a mutually learned pure vision Camera model and a mutually learned pure LiDAR model.
2. The visual recognition model training method for vehicle automatic driving according to claim 1, characterized in that: The construction of the teacher model includes: Obtain an initial fusion model; Obtain image training data and point cloud training data; Aligning the image training data and the point cloud training data to obtain aligned image training data and aligned point cloud training data; The initial fusion model is trained using the aligned image training data and the aligned point cloud training data.
3. The visual recognition model training method for vehicle automatic driving according to claim 2, characterized in that: The teacher model f T The pure visual Camera model f S1 And the pure LiDAR model f S2 The process of knowledge distillation includes the following optimization objectives: in, is the teacher model bev feature, is the visual Camera model f S1 And the pure LiDAR model f S2 bev features, H is the height of the feature map, W is the width of the feature map, w k,x,y is the weight corresponding to the (x, y) position of the kth true detection box on the feature map, which is calculated according to the following formula: (x k ,y k ) is the center coordinate of the kth true detection box, and exp is the natural exponential function.
4. The visual recognition model training method for vehicle automatic driving according to claim 4, characterized in that: When performing knowledge distillation on the pure vision Camera model, the features of the penultimate layer of the network of the pure vision Camera model are selected for distillation; When performing knowledge distillation on the pure LiDAR model, the features of the penultimate layer of the network of the pure LiDAR model are selected for distillation.
5. The visual recognition model training method for vehicle automatic driving according to claim 5, characterized in that: The performing knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, thereby obtaining the distilled pure vision Camera model and the distilled pure LiDAR model, including: The bevdistill method is used to perform knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively.
6. A visual recognition model training device for vehicle autonomous driving, characterized in that: The visual recognition model training device for vehicle automatic driving includes: A teacher model building module, wherein the teacher model building module is used to build a teacher model; A model acquisition module, which is used to acquire a pure vision Camera model and a pure LiDAR model; A distillation module, wherein the distillation module is used to perform knowledge distillation on the pure vision Camera model and the pure LiDAR model respectively through the teacher model, so as to obtain a distilled pure vision Camera model and a distilled pure LiDAR model; A mutual learning module, wherein the mutual learning module is used to mutually learn the distilled pure vision Camera model and the distilled pure LiDAR model, thereby obtaining a pure vision Camera model and a pure LiDAR model that have undergone mutual learning.
7. An image recognition method for automatic vehicle driving, characterized in that: The image recognition method for automatic vehicle driving comprises: Obtaining a pure vision Camera model trained by the visual recognition model training method for vehicle autonomous driving according to any one of claims 1 to 6; Obtaining image information to be identified; The image information to be recognized is input into the pure vision Camera model to obtain the recognition result.
8. An image recognition device for automatic vehicle driving, characterized in that: The image recognition device for automatic vehicle driving comprises: A visual recognition model training device for vehicle autonomous driving, wherein the visual recognition model training device for vehicle autonomous driving is the visual recognition model training device for vehicle autonomous driving according to claim 7; An image acquisition module to be identified, wherein the image acquisition module to be identified is used to acquire information of the image to be identified; The recognition module is used to input the image information to be recognized into the pure vision Camera model to obtain a recognition result.
Citation Information
Cited By
Semantic map construction method adopting multi-expert cross-modal distillation
CN120318643A
Monocular three-dimensional target sensing method and system based on selective knowledge distillation
CN120543827A
