Obstacle detection method and device based on multimodal features

Abstract

The present application discloses an obstacle detection method and device based on multimodal features, which relates to the field of intelligent driving technology. The method includes: determining the space to be tested of the vehicle in the target direction, collecting the current laser point cloud and the current camera image corresponding to the space to be tested; wherein the current laser point cloud is a single-frame laser point cloud collected at the current moment, and the current camera image is a single-frame camera image collected at the current moment; extracting features from the current laser point cloud to obtain a first 3D voxel feature; extracting features from the current camera image to obtain a first 2D image feature; performing multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature; inputting the first fused feature into a pre-trained obstacle detection model to obtain obstacle information in the space to be tested output by the obstacle detection model. The present application performs multimodal feature fusion before obstacle prediction, thereby improving the accuracy of obstacle detection.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to an obstacle detection method and device based on multimodal features. Background Art

[0002] In the field of intelligent driving, detecting the drivable area of ​​a vehicle is a very important part. Specifically, detecting the drivable area can also be understood as detecting whether there are obstacles around the vehicle, and then determining the drivable area of ​​the vehicle based on the obstacle-free area.

[0003] At present, for obstacle detection, the collected 3D (3-dimensional) laser point cloud is usually projected onto the front-view image and then fused with the 2D (2-dimensional) image features, and then obstacle detection is performed using the fused features; or, the 3D laser point cloud is post-fused with the obstacle prediction results obtained through 2D image features to determine whether there is an obstacle.

[0004] However, the above methods all lose the 3D structural information of the laser point cloud to a certain extent, and the obstacle detection accuracy is low, which in turn leads to low safety of autonomous driving vehicles. Summary of the invention

[0005] The main purpose of this application is to provide an obstacle detection method and device based on multimodal features, aiming to solve the technical problems of low obstacle detection accuracy and low safety of autonomous driving vehicles.

[0006] To achieve the above objectives, the present application proposes an obstacle detection method based on multimodal features, the method comprising:

[0007] Determine the space to be measured of the vehicle in the target direction, and collect the current laser point cloud and the current camera image corresponding to the space to be measured; wherein the current laser point cloud is a single-frame laser point cloud collected at the current moment, and the current camera image is a single-frame camera image collected at the current moment;

[0008] Performing feature extraction on the current laser point cloud to obtain a first 3D voxel feature; performing feature extraction on the current camera image to obtain a first 2D image feature;

[0009] Performing multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature;

[0010] The first fusion feature is input into a pre-trained obstacle detection model to obtain obstacle information in the space to be measured output by the obstacle detection model; wherein the obstacle information includes: a 3D occupancy grid and / or semantics corresponding to the 3D occupancy grid, the 3D occupancy grid is used to characterize position information of obstacles in the space to be measured, and the semantics corresponding to the 3D occupancy grid is used to characterize attribute information of obstacles in the space to be measured.

[0011] In one embodiment, the step of performing multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature includes:

[0012] For each target voxel in the first 3D voxel feature, determine n image pixel points projected by the target voxel in the first 2D image feature as n direct projection points; wherein n is an integer greater than 0;

[0013] According to the positions of the n direct projection points on the current camera image, dividing a target area including the positions corresponding to the n direct projection points on the current camera image;

[0014] Determining, according to the target region, a cross-attention region corresponding to the target voxel;

[0015] A cross-attention calculation is performed between the voxel features of all target voxels in the first 3D voxel feature and the image features of the image pixels in the cross-attention area corresponding to each of the target voxels to obtain a first fusion feature.

[0016] In one embodiment, the step of determining the cross-attention region corresponding to the target voxel according to the target region comprises:

[0017] Determine the number of pixel points of the image pixels in the target area;

[0018] Determine the size relationship between the number of pixels and a preset number;

[0019] When it is determined that the number of pixels is less than a preset number, the range of the target area is expanded until the number of pixel points of the image pixels in the expanded target area is equal to the preset number, and the expanded target area is determined to be the cross-attention area;

[0020] In the case where it is determined that the number of pixels is equal to a preset number, determining that the target area is the cross-attention area;

[0021] When it is determined that the number of pixel points is greater than a preset number, the n direct projection points and the m indirect projection points are combined into the cross-attention area; wherein m is the difference between the preset number and n, and the indirect projection points are other image pixel points on the current camera image except the n direct projection points.

[0022] In one embodiment, the voxel features of all target voxels in the first 3D voxel feature are respectively cross-attention calculated with the image features of image pixels in the cross-attention area corresponding to each of the target voxels to obtain the first fusion feature, including:

[0023] The voxel features corresponding to each of the target voxels are taken as a query set, and the image features corresponding to the image pixels in the cross-attention area corresponding to each of the target voxels are taken as a key value set, and cross-attention calculation is performed to obtain a first fusion feature.

[0024] In one embodiment, before inputting the first fusion feature into a pre-trained obstacle detection model to obtain obstacle information in the to-be-detected space output by the obstacle detection model, the method further includes:

[0025] Obtain a pre-trained teacher model; wherein the teacher model adopts an encoder-decoder structure, and the teacher model is trained by pre-collected multi-frame laser point clouds and corresponding single-frame camera images;

[0026] Constructing a student model; wherein the student model adopts an encoder-decoder structure;

[0027] The student model is trained by performing feature distillation between a first intermediate result corresponding to the teacher model and a second intermediate result corresponding to the student model, and the trained student model is used as the obstacle detection model; wherein the first intermediate result is the output result of the encoder of the teacher model, and the second intermediate result is the output result of the encoder of the student model.

[0028] In one embodiment, the step of training the student model by performing feature distillation between the first intermediate result corresponding to the teacher model and the second intermediate result corresponding to the student model, and using the trained student model as the obstacle detection model includes:

[0029] Determine a one-to-one correspondence between N first voxel features in the first intermediate result and N second voxel features in the second intermediate result; wherein N is an integer greater than 0;

[0030] For each first voxel feature in the first intermediate result, the following steps are performed:

[0031] In the case where the first voxel feature is not empty, determining whether a target voxel feature corresponding to the first voxel feature in the second intermediate result is empty;

[0032] When it is determined that the target voxel feature is empty, L2 loss is used as a loss function to calculate a first loss value corresponding to the target voxel feature;

[0033] Determine a feature distillation loss according to first loss values ​​corresponding to the N second voxel features in the second intermediate result;

[0034] The student model is trained using the feature distillation loss, and the trained student model is used as the obstacle detection model.

[0035] In one embodiment, after determining whether the target voxel feature corresponding to the first voxel feature in the second intermediate result is empty, the method further includes:

[0036] When it is determined that the target voxel feature is not empty, KL divergence is used as a loss function to calculate a second loss value corresponding to the target voxel feature;

[0037] The step of determining the feature distillation loss according to the first loss values ​​corresponding to the N second voxel features in the second intermediate result includes:

[0038] Determine the feature distillation loss according to the first loss value or the second loss value corresponding to the N second voxel features in the second intermediate result.

[0039] In one embodiment, the step of determining the feature distillation loss according to the first loss value or the second loss value corresponding to the N second voxel features in the second intermediate result includes:

[0040] Accumulating the first loss values ​​corresponding to the N second voxel features in the second intermediate result to obtain a third loss value, and accumulating the second loss values ​​corresponding to the N second voxel features in the second intermediate result to obtain a fourth loss value;

[0041] The sum of a fifth loss value and the fourth loss value is calculated as the feature distillation loss; wherein the fifth loss value is the product of the third loss value and a preset loss weighting coefficient.

[0042] In one embodiment, the teacher model is trained by the following steps:

[0043] Following the movement of the preset collection vehicle, starting from the first moment, one frame of laser point cloud of the collection vehicle in the target direction is collected at every preset time interval, and a total of x frames of laser point cloud are collected; wherein x is an integer greater than 1;

[0044] At the second moment of collecting the x-th frame of laser point cloud, collecting a single-frame camera image of the collecting vehicle in the target direction;

[0045] According to the collected vehicle motion information, synchronizing the x-frame laser point cloud to the second moment; wherein the motion information is used to infer the correspondence between the x-frame laser point cloud from the first moment to the second moment;

[0046] Performing feature extraction on the x-frame laser point cloud synchronized to the second moment to obtain a second 3D voxel feature; performing feature extraction on the single-frame camera image acquired at the second moment to obtain a second 2D image feature;

[0047] Performing multimodal feature fusion on the second 3D voxel feature and the second 2D image feature to obtain a second fused feature;

[0048] Inputting the second fused feature into a preset candidate model to obtain a first obstacle prediction result output by the candidate model; wherein the candidate model adopts an encoder-decoder structure, and the first obstacle prediction result includes an occupied grid corresponding to the first moment to the second moment and a semantics corresponding to the occupied grid;

[0049] Obtaining a first obstacle prediction true value corresponding to the first moment to the second moment;

[0050] Calculating a training loss function according to the first obstacle prediction true value and the first obstacle prediction result;

[0051] The candidate model is trained according to the training loss function, and the trained candidate model is used as the teacher model.

[0052] In one embodiment, before calculating the training loss function according to the first obstacle prediction true value and the first obstacle prediction result, the method further includes:

[0053] Inputting the second fusion feature into the candidate model to obtain a second obstacle prediction result output by the candidate model; wherein the second obstacle prediction result includes an occupied grid corresponding to a third moment and semantics corresponding to the occupied grid, and the third moment is a moment after the second moment by the preset time;

[0054] Obtaining a second obstacle prediction true value corresponding to the third moment;

[0055] The step of calculating the training loss function according to the first obstacle prediction true value and the first obstacle prediction result includes:

[0056] A training loss function is calculated according to the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result, and the second obstacle prediction result.

[0057] In one embodiment, the first obstacle prediction result includes a first occupied grid result and a first semantic result, and the second obstacle prediction result includes a second occupied grid result;

[0058] The step of calculating a training loss function according to the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result, and the second obstacle prediction result comprises:

[0059] Calculate the Soft-IOU loss between the occupancy grid true value in the first obstacle prediction true value and the first occupancy grid result as a sixth loss value;

[0060] Calculating a cross entropy loss between a semantic truth value in the first obstacle prediction truth value and the first semantic result as a seventh loss value;

[0061] Calculate the Soft-IOU loss between the occupancy grid true value in the second obstacle prediction true value and the second occupancy grid result as an eighth loss value;

[0062] The training loss function is calculated according to the sixth loss value, the seventh loss value and the eighth loss value.

[0063] In addition, to achieve the above-mentioned purpose, the present application also proposes an obstacle detection device based on multimodal features, and the obstacle detection device based on multimodal features includes:

[0064] An acquisition module, used to determine the space to be measured of the vehicle in the target direction, and to acquire the current laser point cloud and the current camera image corresponding to the space to be measured; wherein the current laser point cloud is a single-frame laser point cloud acquired at the current moment, and the current camera image is a single-frame camera image acquired at the current moment;

[0065] A feature extraction module is used to extract features from the current laser point cloud to obtain a first 3D voxel feature; and to extract features from the current camera image to obtain a first 2D image feature;

[0066] a feature fusion module, configured to perform multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature;

[0067] The obstacle detection module is used to input the first fusion feature into a pre-trained obstacle detection model to obtain obstacle information in the space to be detected output by the obstacle detection model.

[0068] In addition, to achieve the above-mentioned purpose, the present application also proposes an obstacle detection device based on multimodal features, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the obstacle detection method based on multimodal features as described above.

[0069] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the obstacle detection method based on multimodal features as described above are implemented.

[0070] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the obstacle detection method based on multimodal features as described above are implemented.

[0071] One or more technical solutions proposed in this application have at least the following technical effects:

[0072] The present application first determines the space to be tested of the vehicle in the target direction, and then collects the current laser point cloud and the current camera image of the space to be tested at the current moment, and then performs feature extraction on the current laser point cloud to obtain a first 3D voxel feature, performs feature extraction on the current camera image to obtain a first 2D image feature, and performs multimodal feature fusion on the first 3D voxel feature and the first 2D image feature. After obtaining the first fused feature, the first fused feature is input into a trained obstacle detection model, so that the model can determine the obstacle information in the space to be tested according to the first fused feature, and then, for example, the drivable area of ​​the vehicle can be further determined based on the obstacle information to assist the vehicle in intelligent driving.

[0073] Compared with the related art, the present application voxelizes the collected laser point cloud and extracts 3D voxel features, thereby retaining the 3D spatial structure features of the point cloud to the maximum extent, and then fuses the 3D voxel features and 2D image features at the feature level, and then uses the fused features to predict obstacles through an obstacle detection model. It supports the interaction of features of different modalities at an earlier stage, that is, multi-modal feature fusion of 3D voxel features and 2D image features is performed before obstacle prediction, which helps to better capture the complex relationship between 3D-2D modalities, improve the accuracy of obstacle detection, and thus improve the safety of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0076] Figure 1 One of the flowcharts provided for the obstacle detection method based on multimodal features of this application;

[0077] Figure 2 The second flowchart diagram of the obstacle detection method based on multimodal features provided in this application;

[0078] Figure 3 This is a flow chart of multimodal feature fusion in the obstacle detection method based on multimodal features of the present application;

[0079] Figure 4 The third flowchart diagram of the obstacle detection method based on multimodal features provided in this application;

[0080] Figure 5 The fourth flowchart diagram of the obstacle detection method based on multimodal features provided in this application;

[0081] Figure 6 This is a schematic diagram of the process of dense-sparse feature distillation in the obstacle detection method based on multimodal features of this application;

[0082] Figure 7 A flowchart of multi-task training of the teacher-student model in the obstacle detection method based on multimodal features of this application;

[0083] Figure 8 A schematic diagram of the process of predicting occupancy grids using a teacher-student model in the obstacle detection method based on multimodal features of this application;

[0084] Fig. 9 A schematic diagram of the structure of the obstacle detection device based on multimodal features provided in this application;

[0085] Fig.10 A schematic diagram of the structure of the obstacle detection device based on multimodal features provided in this application.

[0086] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0087] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0088] Unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by technicians in the technical field to which this application belongs; the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0089] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0090] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0091] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0092] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0093] For obstacle detection, the following methods are usually used in related technologies:

[0094] Related method 1: After the collected 3D laser point cloud is projected onto the front view image, 2D-CNN (Convolutional Neural Networks) is used for feature extraction. The extracted features are fused with the 2D image features, and then the fused features are used for obstacle detection.

[0095] However, this method will lead to the loss of spatial features, especially the loss of 3D structural information of the laser point cloud. Detection only from 2D depth makes the detection process extremely susceptible to occlusion and can only detect objects closest to itself, resulting in low obstacle detection accuracy and low safety of autonomous driving vehicles.

[0096] Related method 2: Obstacle prediction results are first obtained through 2D image features, and then the 3D laser point cloud is post-fused with the obstacle prediction results to characterize the obstacles.

[0097] However, this method uses post-fusion and can only filter the results generated by the two modes respectively, which loses the 3D structural information of the laser point cloud to a certain extent, and the obstacle detection accuracy is low, which in turn leads to low safety of autonomous driving vehicles.

[0098] In view of the above problems, this application provides an obstacle detection method based on multimodal features, which mainly has the following technical points:

[0099] 1) The collected laser point cloud is voxelized and 3D voxel features are extracted, which maximizes the retention of the 3D spatial structure characteristics of the laser point cloud;

[0100] 2) The 3D voxel features and 2D image features are fused at the feature level by cross-attention calculation, which allows different modalities to interact at an earlier stage, helping to better capture the complex relationship between modalities. In addition, the cross-attention method is used to fuse 2D image features and 3D voxel features, which is more conducive to local feature fusion.

[0101] 3) Gridding the entire 3D space and predicting its occupancy, that is, determining the 3D occupancy grid, which can better represent the overall situation of the environment;

[0102] 4) During the training process, the teacher model uses multi-frame input to generate multi-frame features, and uses the teacher model to perform feature distillation on the student model, so that the student model also has the ability to generate dense features, thereby having better scene understanding and achieving spatial completion.

[0103] It should be noted that the execution subject of the embodiments of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions.

[0104] This embodiment and the following embodiments are described in detail below.

[0105] The present application embodiment provides an obstacle detection method based on multimodal features, referring to Figure 1 , Figure 1 One of the flowcharts of the obstacle detection method based on multimodal features provided in the present application, the obstacle detection method based on multimodal features comprises steps S10 to S40:

[0106] Step S10, determining a space to be measured of the vehicle in a target direction, and collecting a current laser point cloud and a current camera image corresponding to the space to be measured;

[0107] The current laser point cloud is a single-frame laser point cloud acquired at the current moment, and the current camera image is a single-frame camera image acquired at the current moment.

[0108] It should be noted that the above-mentioned target direction can be selected as any direction around the vehicle, such as specific directions such as the front, sides, and rear of the vehicle, or a direction at a certain angle can also be selected, and this application does not impose any restrictions on this.

[0109] Specifically, first determine the space to be tested of the vehicle in the target direction, for example, determine the 30m*30m*3m space in front of the vehicle as the space to be tested, and collect the current laser point cloud and current camera image corresponding to the space to be tested, wherein the current laser point cloud is a single-frame laser point cloud collected at the current moment, and the current camera image is a single-frame camera image collected at the current moment. The collected current laser point cloud and current camera image are used to predict obstacles in the space to be tested at the current moment.

[0110] Step S20, performing feature extraction on the current laser point cloud to obtain a first 3D voxel feature; performing feature extraction on the current camera image to obtain a first 2D image feature;

[0111] Specifically, for the feature extraction of the current laser point cloud, the 3D space can be divided into cubic grids. After the laser point cloud is assigned to the corresponding voxels according to the position, 3D-CNN is used for feature extraction, which is recorded as the first 3D voxel feature V, and its feature can be represented as C1*H1*W1*D1, wherein C1 represents the feature dimension, H1 represents the number of cubic grids divided in the height direction of the space to be measured, W1 represents the number of cubic grids divided in the width direction of the space to be measured, and D1 represents the number of cubic grids divided in the depth direction of the space to be measured.

[0112] For feature extraction of the current camera image, 2D-CNN can be used to extract features of the three primary colors (RGB) image, which is recorded as the first 2D image feature I, and its feature can be represented as C2*H2*W2, where C2 represents the feature dimension, H2 represents the number of plane grids divided in the height direction of the image, and W2 represents the number of plane grids divided in the width direction of the image.

[0113] Step S30, performing multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature;

[0114] It should be noted that the multimodal feature fusion mentioned in the embodiment of the present application specifically refers to a method of fusing features of a modality of 3D voxel features with features of a modality of 2D image features.

[0115] Step S40: input the first fusion feature into a pre-trained obstacle detection model to obtain obstacle information in the space to be detected output by the obstacle detection model.

[0116] It should be noted that the above obstacle detection model can be trained in a supervised manner.

[0117] Specifically, the present application first determines the space to be tested of the vehicle in the target direction, collects the current laser point cloud and the current camera image of the space to be tested at the current moment, then performs feature extraction on the current laser point cloud to obtain a first 3D voxel feature, performs feature extraction on the current camera image to obtain a first 2D image feature, and performs multimodal feature fusion on the first 3D voxel feature and the first 2D image feature. After obtaining the first fused feature, the first fused feature is input into a trained obstacle detection model, so that the model can determine the obstacle information in the space to be tested according to the first fused feature. Subsequently, for example, the drivable area of ​​the vehicle can be further determined based on the obstacle information to assist the vehicle in intelligent driving.

[0118] In some embodiments, the obstacle information includes: a 3D occupancy grid and / or semantics corresponding to the 3D occupancy grid;

[0119] The 3D occupancy grid is used to represent the position information of obstacles in the space to be measured, and the semantics corresponding to the 3D occupancy grid is used to represent the attribute information of obstacles in the space to be measured.

[0120] Specifically, the obstacle information output by the obstacle detection model may include a 3D occupancy grid, or may include a 3D occupancy grid and corresponding semantics, or may only include semantics corresponding to the 3D occupancy grid.

[0121] It should be noted that the 3D grid can be understood as many cubic grids / voxels (for example, a length of 0.1m*0.1m*0.1m) divided in a 3D space (for example, a 30m*30m*3m test space). The specific 3D occupancy grid refers to the occupied grid in the 3D space. In the embodiment of the present application, the occupied 3D occupancy grid is used to characterize the distribution of obstacles in a certain spatial environment (for example, the test space), and the relative position of the 3D occupancy grid in the entire 3D space can also characterize the position information of the obstacle in the test space.

[0122] In addition, the semantics corresponding to the 3D occupancy grid can also be used to represent the attribute information of obstacles in the measured space, such as whether the obstacles are trees, walls, other vehicles, etc.

[0123] In some embodiments, the obstacle information in the space to be tested may only include information indicating whether there is an obstacle in the space to be tested, or may further include a 3D occupancy grid corresponding to the obstacle when indicating the existence of an obstacle, so as to characterize the positional relationship of the obstacle in the space to be tested, so as to facilitate the vehicle to determine the drivable area and avoid obstacles during autonomous driving; in addition, when the 3D occupancy grid corresponding to the obstacle is indicated, the semantics corresponding to the 3D occupancy grid may also be included to characterize the properties of the obstacle, such as whether it is a fixed obstacle such as a tree or a wall, or a movable obstacle such as other vehicles or pedestrians.

[0124] In other embodiments, the above-mentioned obstacle detection method can be executed at regular intervals. When the time interval is set to be small, it is believed that the present application can achieve real-time obstacle detection. After detecting the obstacle information in the space to be tested, the above-mentioned obstacle information is displayed in real time through a visual interface set on the vehicle to assist people in the vehicle to perform intelligent driving, thereby improving the user experience.

[0125] In the embodiment of the present application, the expression paradigm for obstacle information is to divide the entire 3D space to be tested into grids, and predict the occupancy status (occupied or unoccupied) of each grid in the space, that is, to represent obstacles through 3D occupancy grids, which can more effectively express the overall situation of the environment.

[0126] This embodiment provides an obstacle detection method based on multimodal features. Compared with the related art, this application voxelizes the collected laser point cloud and extracts 3D voxel features, thereby retaining the 3D spatial structure features of the point cloud to the maximum extent. After feature-level fusion of the 3D voxel features and the 2D image features, the fused features are used to predict obstacles through an obstacle detection model. It supports the interaction of features of different modalities at an earlier stage, that is, the multimodal feature fusion of the 3D voxel features and the 2D image features is performed before obstacle prediction, which helps to better capture the complex relationship between the 3D-2D modalities, improve the accuracy of obstacle detection, and thus improve the safety of autonomous driving vehicles.

[0127] Based on this application Figure 1 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be described in detail later. Figure 2 , Figure 2 The second flowchart diagram of the obstacle detection method based on multimodal features provided by the present application, step S30 specifically includes steps S301 to S304:

[0128] Step S301: for each target voxel in the first 3D voxel feature, determine the n projected by the target voxel in the first 2D image feature. iimage pixels, as n i Direct projection points;

[0129] Among them, n i An integer greater than 0.

[0130] It should be noted that the first 3D voxel feature includes H1*W1*D1 3D voxels (i.e., target voxels). For the i-th voxel (grid) V i ∈V H1*W1*D1 , and N within its range i The original laser radar points are converted into N according to the external parameter matrix between the laser radar and the camera. i The original lidar point is projected to the image pixel v i =(H i， W i ), where H i , W i Respectively represent the coordinates of the image pixel in the height and width directions, and N i The 3D voxels are projected on the first 2D image feature to obtain n i image pixels. Considering that multiple 3D LiDAR points may correspond to the same image pixel after projection, n i ≤N i .

[0131] It should also be noted that due to the uneven distribution of the LiDAR point cloud, different voxels contain different numbers of LiDAR points. Therefore, the corresponding image pixels of each voxel after projection are likely to be different. That is, the number of image pixels corresponding to different 3D voxels is likely to be different. i To represent, i is an integer greater than 1, used to represent its index.

[0132] Step S302: according to the n i The positions of the direct projection points on the current camera image are divided into a region containing the n i The target area corresponding to the position of the direct projection point.

[0133] It should be noted that for the i-th voxel, it can be calculated based on n i The positions of the image pixels define the circumscribed rectangular area Ω i , as the target area, that is, the image feature candidate area for cross-attention with the i-th voxel feature, the target area needs to include these n i image pixels.

[0134] Step S303: determining the cross-attention region corresponding to the target voxel according to the target region.

[0135] Step S304 , performing cross-attention calculations on the voxel features of all target voxels in the first 3D voxel features and the image features of the image pixels in the cross-attention areas corresponding to the target voxels, respectively, to obtain a first fusion feature.

[0136] Specifically, Figure 3 This is a flow chart of multimodal feature fusion in the obstacle detection method based on multimodal features of this application, such as Figure 3 As shown in the figure, the multimodal feature fusion process is implemented through the following steps:

[0137] Step 1: voxel-pixel mapping, for each 3D voxel in the first 3D voxel feature, voxel-pixel mapping is performed on the original lidar points contained in each 3D voxel, and the original lidar points are mapped to the corresponding n direct projection points in the first 2D image feature by projection, and then the cross-attention region corresponding to the 3D voxel is determined according to the n direct projection points;

[0138] Step 2: Cross-modal cross-attention calculation: In the cross-attention area corresponding to each determined 3D voxel, cross-attention calculation is performed on the voxel features of each 3D voxel and the image features of the image pixels in the cross-attention area to obtain the first fusion feature.

[0139] In an embodiment of the present application, the present application provides a specific implementation method of fusing 2D image features and 3D voxel features using a cross-attention method. The cross-attention method is used for multimodal feature fusion, which is more conducive to local feature fusion. Then, by using the obtained fusion features, the obstacle information in the test space can be more accurately detected through the obstacle detection model, thereby improving the safety of the autonomous driving vehicle.

[0140] Based on this application Figure 2 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be described in detail later. Figure 4 , Figure 4 The third flowchart diagram of the obstacle detection method based on multimodal features provided in this application, step S303 specifically includes steps S3031~S3035:

[0141] Step S3031, determining the number of image pixels in the target area.

[0142] Step S3032, determining the size relationship between the number of pixels and a preset number.

[0143] Step S3033, when it is determined that the number of pixels is less than the preset number, the range of the target area is expanded until the number of pixel points of the image pixels in the expanded target area is equal to the preset number, and the expanded target area is determined to be the cross-attention area.

[0144] Step S3034, when it is determined that the number of pixels is equal to a preset number, determining that the target area is the cross-attention area.

[0145] Step S3035, when it is determined that the number of pixel points is greater than a preset number, the n direct projection points and the m indirect projection points are combined into the cross attention area;

[0146] Among them, m is the difference between the preset number and n, and the indirect projection points are other image pixel points on the current camera image except the n direct projection points.

[0147] Specifically, it can be divided into the following three situations:

[0148] Case 1: If the target area Ω i In (that is, by n i If the number of image pixels contained in the bounding rectangle obtained by expanding the directly projected points is less than the preset number τ, the target area is further expanded (for example, the length and width of the bounding rectangle are further expanded) until the number of pixels therein is greater than or equal to the preset number τ, so as to include the relevant pixels in the target area to the maximum extent.

[0149] Case 2: If the number of pixels is equal to the preset number τ, the target area Ω i As a cross-attention area.

[0150] Case 3: If the number of pixels is greater than the preset number τ, while ensuring n i Under the premise of direct projection points, randomly select τ-n i Indirect projection points form a cross-attention area, such as Figure 4 As shown, direct projection points and indirect projection points in the cross-attention area are exemplarily displayed.

[0151] It should also be noted that the preset number can be set according to actual conditions to reasonably set the range of the cross-attention area, and this application does not impose any restrictions on this.

[0152] In an embodiment of the present application, when the extrinsic parameter matrix is ​​not accurate enough when considering voxel feature projection, the present application can include relevant pixel points into the cross-attention area to the maximum extent to extract local features, and to a certain extent compensate for the impact of the inaccurate extrinsic parameter matrix on feature fusion, so that the feature fusion has the tolerance characteristics of the extrinsic parameter matrix; at the same time, considering the computational burden, reasonably setting the range of the cross-attention area can ensure that the calculation range is not too large, while ensuring the multimodal feature fusion effect, reducing the amount of calculation as much as possible.

[0153] In some embodiments, the method of performing cross-attention calculation between the voxel features of all target voxels in the first 3D voxel feature and the image features of the image pixels in the cross-attention area corresponding to each of the target voxels to obtain the first fusion feature may include the following steps:

[0154] The voxel features corresponding to each of the target voxels are taken as a query set, and the image features corresponding to the image pixels in the cross-attention area corresponding to each of the target voxels are taken as a key value set, and cross-attention calculation is performed to obtain a first fusion feature.

[0155] Specifically, in combination with the above voxel-pixel mapping relationship with tolerance characteristics, the first 3D voxel feature V H1*W1*D1 The voxel features of all target voxels in and the corresponding image features I H2*W2 The cross-attention calculation is performed on the external pixel area (i.e., the cross-attention area) inside to obtain the first fused feature after fusion V’ .

[0156] Specifically, the voxel feature corresponding to each 3D voxel in the first 3D voxel feature (feature dimension C1, as the query set Quary ) and the τ image features corresponding to the image pixels in the cross-attention region (feature dimension τ*C2, as a key-value set Value and Key ) performs cross attention calculation to obtain the first fusion feature with dimension (C3*H1*W1*D1) V’ , the specific calculation formula is as follows:

[0157] .

[0158] in, SoftMax () represents the normalized exponential function, sqrt () represents the square root function.

[0159] In an embodiment of the present application, a specific implementation method of how to use cross-attention calculation to perform multimodal feature fusion is provided, and the specific correspondence between the voxel features corresponding to the 3D voxel points, the corresponding image features in the cross-attention area and the cross-attention calculation is determined. The first fusion feature can be calculated using the above-mentioned cross-attention calculation formula.

[0160] Based on this application Figure 1 Corresponding embodiments, in this embodiment, the same or similar contents as the above embodiments can be referred to the above description, and will not be described in detail later. Figure 5 , Figure 5 The fourth flowchart diagram of the obstacle detection method based on multimodal features of the present application is provided. Before step S40, the obstacle detection method based on multimodal features further includes steps S50 to S70:

[0161] Step S50, obtaining a pre-trained teacher model;

[0162] The teacher model adopts an encoder-decoder structure, and the teacher model is trained by pre-collected multi-frame laser point clouds and corresponding single-frame camera images.

[0163] Step S60, constructing a student model;

[0164] Wherein, the student model adopts an encoder-decoder structure.

[0165] Step S70, training the student model by performing feature distillation between the first intermediate result corresponding to the teacher model and the second intermediate result corresponding to the student model, and using the trained student model as an obstacle detection model;

[0166] Among them, the first intermediate result is the output result of the encoder of the teacher model, and the second intermediate result is the output result of the encoder of the student model.

[0167] It should be noted that the present application does not limit the order of step S60 and step S70. The above embodiments are only examples. In practice, you can first obtain a trained teacher model, or you can first build a student model, or you can simultaneously obtain a trained teacher model and build a student model. The present application does not impose any restrictions on this.

[0168] It should also be noted that the present application does not limit the sequence relationship between steps S50~S70 and steps S10~S30. The figure is only an example and the present application does not limit this.

[0169] In some embodiments, the constructed student model can first be supervised trained using a single-frame laser point cloud and a single-frame camera image, and then trained using feature distillation.

[0170] Specifically, the present application adopts a teacher-student model architecture to train an obstacle detection model, wherein both the teacher model and the student model adopt an encoder-decoder structure, the teacher model has been trained by pre-collected multi-frame laser point clouds and corresponding single-frame camera images, and the student model can be trained by a single-frame laser point cloud; since the teacher model is obtained by training with multi-frame laser point clouds, it can extract more dense spatial features compared to the student model trained with a single-frame laser point cloud, so the student model can be trained by using the first intermediate result output by the encoder of the teacher model and the second intermediate result output by the encoder of the student model to perform feature distillation. This dense-to-sparse knowledge distillation loss can transfer the dense semantic knowledge of the teacher model to the student model, so that the student model has the ability to extract multi-frame dense features and better scene understanding capabilities, thereby enhancing the spatial completion effect of the student model and facilitating more accurate detection of obstacle information.

[0171] In an embodiment of the present application, the fused features are further enhanced by an encoder-decoder structure. At the same time, since it is very difficult to perform semantic scene completion directly from the input of a single-frame laser point cloud, in order to overcome the sparsity and incompleteness of a single-frame laser point cloud, the present application adopts a teacher-student model architecture to train an obstacle detection model. Specifically, for the output of the encoder, feature distillation is performed between the teacher model and the student model. Because the input of the teacher model is a dense multi-frame laser point cloud, the teacher model can extract more dense spatial features. This dense-to-sparse knowledge distillation loss can transfer the dense semantic knowledge of the teacher model to the student model, so that the student model has the ability to extract multi-frame dense features and better scene understanding capabilities, thereby enhancing the spatial completion effect of the student model and thereby improving the obstacle detection effect.

[0172] In some embodiments, the specific implementation of training the student model by performing feature distillation between the first intermediate result corresponding to the teacher model and the second intermediate result corresponding to the student model and using the trained student model as the obstacle detection model may include:

[0173] Determine a one-to-one correspondence between N first voxel features in the first intermediate result and N second voxel features in the second intermediate result; wherein N is an integer greater than 0;

[0174] For each first voxel feature in the first intermediate result, the following steps are performed:

[0175] In the case where the first voxel feature is not empty, determining whether a target voxel feature corresponding to the first voxel feature in the second intermediate result is empty;

[0176] When it is determined that the target voxel feature is empty, L2 loss is used as a loss function to calculate a first loss value corresponding to the target voxel feature;

[0177] Determine a feature distillation loss according to first loss values ​​corresponding to the N second voxel features in the second intermediate result;

[0178] The student model is trained using the feature distillation loss, and the trained student model is used as the obstacle detection model.

[0179] In an embodiment of the present application, when the features of the teacher model are non-empty and the corresponding features of the student model are empty, L2 loss can be used as the loss function to smoothly and easily optimize the gradient to complete the redundant features of the student model, so that it is as close as possible to the effect of the teacher model.

[0180] In some embodiments, after determining whether the target voxel feature corresponding to the first voxel feature in the second intermediate result is empty, the following steps may be further performed:

[0181] When it is determined that the target voxel feature is not empty, KL divergence is used as a loss function to calculate a second loss value corresponding to the target voxel feature;

[0182] The specific implementation manner of determining the feature distillation loss according to the first loss values ​​corresponding to the N second voxel features in the second intermediate result may include:

[0183] Determine the feature distillation loss according to the first loss value or the second loss value corresponding to the N second voxel features in the second intermediate result.

[0184] In an embodiment of the present application, when the features of the teacher model are non-empty and the corresponding features of the student model are also non-empty, the KL divergence can be used as the loss function to measure the distribution difference between the two features and encourage the student model to have a distribution on non-empty features close to that of the teacher model.

[0185] For example, Figure 6 Schematic diagram of the process of dense-sparse feature distillation in the obstacle detection method based on multimodal features in this application, such as Figure 6 As shown in FIG. 1 , the one-to-one correspondence between the N first voxel features in the first intermediate result and the N second voxel features in the second intermediate result is first determined, and the decoder intermediate features (i.e., the output results of the encoder) of the teacher and student models are flattened to obtain F T ∈R NT*C and FS ∈R NS*C , where NT and NS are the number of non-empty voxel features, C is the feature dimension, and NS is a subset of NT, that is, the number of non-empty voxel features of the student model is less than or equal to the number of non-empty voxel features of the teacher model.

[0186] like Figure 6 As shown, non-empty features are indicated by horizontal line filling, and empty features are indicated by white or light gray, where light gray is used to represent the following situation 3. The flattened features are compared and can be divided into the following three situations:

[0187] Case 1: The features of the teacher model and the student model are both non-empty.

[0188] Case 2: The features of the teacher model are non-empty and the features of the student model are empty.

[0189] Case 3: The features of the teacher model and the student model are both empty.

[0190] For case 3 where all voxel features are empty, this application does not perform feature distillation on them, but only performs feature distillation on cases 1 and 2 respectively to obtain feature distillation loss.

[0191] 1) For case 1, the features of the teacher model and the student model are both non-empty:

[0192] like Figure 6 For the solid feature pairs in the figure, we need to measure the distribution difference between the two features and encourage the distribution of the student model on non-empty features to be close to the teacher model. For example, KL divergence can be used as the loss function. The specific formula is as follows:

[0193] ;

[0194] in, Characterizes the sum of the second loss values ​​corresponding to the above-mentioned second voxel features.

[0195] 2) For case 2, the features of the teacher model are non-empty and the features of the student model are empty:

[0196] like Figure 6 For the dotted feature pairs in the figure, we can use L2 loss (smooth and easy to optimize gradients) to enhance the student model to complete the redundant features and make it as close to the teacher model as possible. The specific formula is as follows:

[0197] ;

[0198] in, Characterizes the sum of the first loss values ​​corresponding to the above-mentioned second voxel features.

[0199] In some embodiments, the step of determining the feature distillation loss according to the first loss value or the second loss value corresponding to the N second voxel features in the second intermediate result may include:

[0200] Accumulating the first loss values ​​corresponding to the N second voxel features in the second intermediate result to obtain a third loss value, and accumulating the second loss values ​​corresponding to the N second voxel features in the second intermediate result to obtain a fourth loss value;

[0201] The sum of a fifth loss value and the fourth loss value is calculated as the feature distillation loss; wherein the fifth loss value is the product of the third loss value and a preset loss weighting coefficient.

[0202] Specifically, the above is the sum of the second loss values ​​of all second voxel features, that is, the fourth loss value. is the sum of the first loss values ​​of all second voxel features, that is, the third loss value, and the final dense-sparse feature distillation loss L DK is the sum of the fifth loss value and the fourth loss value. The specific formula is as follows: L DK ,in, Characterizes the loss weighting coefficient and can be set according to actual conditions.

[0203] In some embodiments, a loss weighting coefficient may also be set for the fourth loss value for calculation, and this application does not impose any limitation on this.

[0204] In an embodiment of the present application, a specific implementation method of feature distillation between a teacher model and a student model is provided, and different methods are used to calculate the loss in different situations. For example, when the voxel features of the teacher model and the voxel features of the student model are both non-empty, the KL divergence can be used as the loss function, so that the distribution of the student model on the non-empty features is close to that of the teacher model. For another example, when the voxel features of the teacher model are non-empty and the voxel features of the student model are empty, the L2 loss can be used as the loss function, which is smooth and easy to optimize by gradients, and can enhance the student model to complete the redundant features and be as close to the effect of the teacher model as possible. According to the loss values ​​corresponding to the above-mentioned voxel features, the total feature distillation loss is obtained, so as to use it to train the student model and obtain the obstacle detection model.

[0205] In some embodiments, the teacher model can be trained by the following steps:

[0206] Following the movement of the preset collection vehicle, starting from the first moment, one frame of laser point cloud of the collection vehicle in the target direction is collected at every preset time interval, and a total of x frames of laser point cloud are collected; wherein x is an integer greater than 1;

[0207] At the second moment of collecting the x-th frame of laser point cloud, collecting a single-frame camera image of the collecting vehicle in the target direction;

[0208] According to the collected vehicle motion information, synchronizing the x-frame laser point cloud to the second moment; wherein the motion information is used to infer the correspondence between the x-frame laser point cloud from the first moment to the second moment;

[0209] Performing feature extraction on the x-frame laser point cloud synchronized to the second moment to obtain a second 3D voxel feature; performing feature extraction on the single-frame camera image acquired at the second moment to obtain a second 2D image feature;

[0210] Performing multimodal feature fusion on the second 3D voxel feature and the second 2D image feature to obtain a second fused feature;

[0211] Inputting the second fused feature into a preset candidate model to obtain a first obstacle prediction result output by the candidate model; wherein the candidate model adopts an encoder-decoder structure, and the first obstacle prediction result includes an occupied grid corresponding to the first moment to the second moment and a semantics corresponding to the occupied grid;

[0212] Obtaining a first obstacle prediction true value corresponding to the first moment to the second moment;

[0213] Calculating a training loss function according to the first obstacle prediction true value and the first obstacle prediction result;

[0214] The candidate model is trained according to the training loss function, and the trained candidate model is used as the teacher model.

[0215] Specifically, in the process of training the teacher model, supervised training is performed by using data collected by vehicles + manual labeling.

[0216] Specifically, a collection device can be set on the collection vehicle to collect laser point clouds and camera images. As the preset collection vehicle moves, starting from the first moment (for example, moment T1), the collection device collects one frame of laser point cloud of the collection vehicle in the target direction at a preset interval, and collects a total of x frames of laser point cloud, that is, from T1 to T x Multi-frame laser point cloud corresponding to the moment.

[0217] At the second moment of collecting the x-th frame of laser point cloud (i.e., Tx At time ), the collection vehicle collects a single-frame camera image of the above collection vehicle in the target direction.

[0218] Then, based on the collected vehicle motion information, the x-frame laser point cloud is synchronized to the second moment; the motion information here is used to assist in inferring the correspondence between the x-frame laser point clouds from the first moment to the second moment. For example, the motion information is that the distance between the vehicle and a certain pile of laser point clouds at the moment T1 is 10 meters, and the distance from the above pile of laser point clouds becomes 9 meters at the moment T2 after a preset time, until at T x At this moment, the distance from the above-mentioned pile of laser point clouds becomes 0.5 meters. It should be noted that a pile of laser point clouds corresponding to different moments can represent the same obstacle. As the vehicle moves, the relative position between the collected laser point cloud corresponding to a certain obstacle and the vehicle changes, but in fact, it represents the same obstacle. Therefore, when processing, the laser point clouds corresponding to the obstacle at different moments need to be synchronized to T x At all times, these laser point clouds are used to describe the obstacle rather than describing different obstacles.

[0219] It should be noted that for an obstacle, a single-frame laser point cloud can be used to describe it. For example, 10 point clouds can be used to describe an obstacle. If 10 frames of laser point clouds are used to describe an obstacle, after synchronizing 10 frames of laser point clouds, the obstacle can be described by, for example, 10*10=100 point clouds, making the outline description of the obstacle more detailed.

[0220] Compared with the sparse point cloud of a single-frame laser point cloud, the multi-frame laser point cloud is denser and describes obstacles more accurately. The teacher model is trained with multi-frame laser point clouds, which helps to detect obstacles more accurately.

[0221] After synchronizing the x-frame laser point cloud to the second moment, feature extraction is performed on the synchronized x-frame laser point cloud to obtain a second 3D voxel feature, and feature extraction is performed on the single-frame camera image collected at the second moment to obtain a second 2D image feature, and then multimodal feature fusion is performed on the second 3D voxel feature and the second 2D image feature to obtain a second fused feature, and the second fused feature is input into a preset candidate model to obtain a first obstacle prediction result output by the candidate model, where the first obstacle prediction result may include a 3D occupancy grid corresponding to the first moment to the second moment and a semantic corresponding to the 3D occupancy grid.

[0222] Then obtain the first obstacle prediction true value corresponding to the first moment to the second moment, and calculate the training loss function according to the difference between the first obstacle prediction true value and the first obstacle prediction result, so as to train the candidate model using the training loss function, and use the trained candidate model as the above-mentioned teacher model.

[0223] In an embodiment of the present application, a teacher model is obtained by training with multi-frame laser point clouds and single-frame camera images. Specifically, the multi-frame laser point clouds need to be synchronized to the second moment and then the corresponding voxel features need to be extracted. Multimodal feature fusion is performed with the image features extracted from the single-frame camera image collected at the second moment to obtain a first obstacle prediction result, which is compared with, for example, the manually labeled first obstacle prediction true value to obtain a training loss function, and the candidate model is trained with the loss function to obtain the teacher model. Training the teacher model with dense multi-frame laser point clouds is conducive to extracting denser spatial features, and subsequently training the student model through feature distillation.

[0224] In some embodiments, before calculating the training loss function according to the first obstacle prediction true value and the first obstacle prediction result, the following steps may also be performed:

[0225] Inputting the second fusion feature into the candidate model to obtain a second obstacle prediction result output by the candidate model; wherein the second obstacle prediction result includes an occupied grid corresponding to a third moment and semantics corresponding to the occupied grid, and the third moment is a moment after the second moment by the preset time;

[0226] Obtaining a second obstacle prediction true value corresponding to the third moment;

[0227] The implementation method of calculating the training loss function according to the first obstacle prediction true value and the first obstacle prediction result may include:

[0228] A training loss function is calculated according to the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result, and the second obstacle prediction result.

[0229] Specifically, when training the teacher model, a more complete geometric understanding of obstacles can be obtained by predicting future tasks. Specifically, the second fusion feature can be input into the candidate model, and the candidate model predicts the occupancy grid at the future moment to obtain the second obstacle prediction result output by the candidate model. The second obstacle prediction result specifically includes the third moment (that is, T x+1 The occupancy grid corresponding to the time (at that moment) and the semantics corresponding to the occupancy grid.

[0230] Then obtain the second obstacle prediction true value corresponding to the third moment, and finally calculate the final training loss function to train the teacher model based on the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result and the second obstacle prediction result. Specifically, it is necessary to obtain a loss function by comparing the first obstacle prediction true value and the first obstacle prediction result, and then obtain another loss function by comparing the second obstacle prediction true value and the second obstacle prediction result, and obtain the final training loss function based on these two loss functions.

[0231] In some embodiments, the first obstacle prediction result may include a first occupancy grid result and a first semantic result, and the second obstacle prediction result may include a second occupancy grid result;

[0232] The specific implementation method of calculating the training loss function according to the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result and the second obstacle prediction result may specifically include the following steps:

[0233] Calculate the Soft-IOU loss between the occupancy grid true value in the first obstacle prediction true value and the first occupancy grid result as the sixth loss value L geo_1 ;

[0234] Calculate the cross entropy loss between the semantic truth value in the first obstacle prediction truth value and the first semantic result as the seventh loss value L sem_1 ;

[0235] Calculate the Soft-IOU loss between the occupancy grid true value in the second obstacle prediction true value and the second occupancy grid result as the eighth loss value L geo_2 ;

[0236] According to the sixth loss value L geo_1 , the seventh loss value L sem_1 and the eighth loss value L geo_2 , calculate the training loss function.

[0237] Figure 7 This is a flow chart of multi-task training of the teacher-student model in the obstacle detection method based on multimodal features in this application, such as Figure 7 As shown, for the teacher model, the multiple tasks specifically include:

[0238] 1) Dense semantic occupancy prediction of this frame, the true value is composed of T1, T2...T x Frame-synchronized point cloud + manual annotation generation.

[0239] For dense semantic + occupancy prediction, two loss functions, semantic (such as cross entropy loss) + geometric (such as Soft-IOU loss), are used to constrain the output of semantic occupancy prediction, such as L sem_1 + L geo_1 ;

[0240] 2) Sparse occupancy prediction of future frames, the true value is given by T x+1 Frame sparse point clouds are generated by voxelization.

[0241] For sparse future frame occupancy prediction, geometric (such as Soft-IOU) constraints are used to constrain the future frame occupancy prediction output, such as L geo_2 .

[0242] The training loss function of the final teacher model L Teacher It can be expressed as:

[0243] L Teacher = L sem_1 + * L geo_1 + * L geo_2 ;

[0244] in, The loss weight coefficient corresponding to the dense semantics of the characterization teacher model, The loss weight coefficient corresponding to the sparse semantics that characterizes the teacher model can be set according to actual conditions.

[0245] In an embodiment of the present application, this multi-task prediction enables the teacher model to further deepen its understanding of the scene: on the one hand, future prediction models the underlying dynamics of the scene, and this understanding allows multiple observations in time to be aggregated for a more complete geometric understanding; on the other hand, complete scene understanding allows future predictions to focus on object motion without having to worry about sampling noise or occlusion changes faced by traditional point cloud prediction methods. Therefore, semantic scene completion can greatly improve future predictions and provide a comprehensive understanding of the scene. Future predictions and semantic scene completion are synergistic, and if properly modeled, they can bring huge benefits to each other, thereby effectively improving the feature extraction capabilities of the teacher model.

[0246] It should also be noted that if Figure 7 As shown, for the student model, the multi-tasks specifically include:

[0247] 1) Dense semantic occupancy prediction of this frame, the true value is the same as that of the teacher model, composed of T1, T2...T x Frame-synchronized point cloud + manual annotation generation.

[0248] For dense semantic + occupancy prediction, two loss functions, semantic (such as cross entropy loss) + geometric (such as Soft-IOU loss), are used to constrain the output of semantic occupancy prediction, such as L sem_2 + L geo_3 ;

[0249] 2) Encoder output feature distillation, the true value is the encoder output feature of the teacher model. During the training of the student model, the network parameters of the teacher model will be frozen and cannot be changed.

[0250] The dense-sparse distillation loss function here is the feature distillation loss mentioned above L DK , the feature extraction capability of the teacher model can be distilled to the student model.

[0251] Feature distillation loss of the final student model L Student It can be expressed as:

[0252] L Student = L sem_2 + * L geo_3 + L DK ;

[0253] in, The loss weight coefficient corresponding to the dense semantics of the student model, The loss weighting coefficient corresponding to the feature distillation that characterizes the student model can be set according to actual conditions.

[0254] The following example illustrates the obstacle detection method based on multimodal features provided by the present application.

[0255] Figure 8 This is a flow chart of the teacher-student model predicting the occupancy grid in the obstacle detection method based on multimodal features in this application, such as Figure 8 As shown in the figure, taking the teacher model as an example, it mainly includes the following parts from left to right:

[0256] (1) Multi-frame input: Input multiple frames of 3D laser point cloud (corresponding to T1, T2, ..., T x At time T, the multi-frame laser point cloud is combined with the vehicle's motion information to synchronize to Tx At the same time, enter T x A single-frame camera image at a given moment;

[0257] (2) Multimodal feature extraction:

[0258] <1> Point cloud voxel features: The 3D space is divided into a cubic grid, and the laser point cloud is assigned to the corresponding voxels according to the position. Then, 3D-CNN is used to extract features, which are recorded as voxel features V;

[0259] <2> Image features: Use 2D-CNN to extract features from RGB images, denoted as image feature I.

[0260] (3) Multimodal feature fusion with tolerance characteristics;

[0261] <1> Voxel-to-pixel mapping, the specific process can refer to the description in the above embodiment;

[0262] <2> Cross-modal cross-attention calculation is performed to perform multi-modal feature fusion. The specific process can be referred to the description in the above embodiment and will not be repeated here. Finally, the fusion feature V’ .

[0263] (4) Dense-sparse feature distillation;

[0264] Fusion features after fusion V’ An encoder-decoder structure is used to further enhance the features. At the same time, in order to overcome the sparsity and incompleteness of single-frame point clouds in the student model, it is very difficult to complete the semantic scene directly from a single-frame input. For the output of the encoder, feature distillation is performed between the teacher model and the student model.

[0265] Because the input of the teacher model is a multi-frame dense point cloud, the teacher model can extract more dense spatial features. This dense-to-sparse knowledge distillation loss can transfer the dense semantic knowledge of the teacher model to the student model, enabling the student model to extract multi-frame dense features and better scene understanding capabilities, thereby enhancing the spatial completion effect of the student model.

[0266] The specific characteristic distillation process can be described in the above embodiment, which will not be repeated here. Finally, the characteristic distillation loss is obtained: L DK .

[0267] (5) Multi-task prediction;

[0268] like Figure 7 As shown, for the teacher model, the multi-task prediction includes: the dense semantic occupancy prediction of the current frame and the sparse occupancy prediction of the future frame. For details, please refer to the description of the above embodiment;

[0269] For the student model, the multi-task prediction includes: dense semantic occupancy prediction of the current frame and encoder output feature distillation. For details, please refer to the description of the above embodiment.

[0270] In the embodiments of the present application, at least the following beneficial effects are included:

[0271] 1) Multimodal feature-level fusion with sensor extrinsic tolerance characteristics: To avoid the limitations of a single modality and the disturbance of the voxel-pixel mapping relationship caused by the inaccurate sensor extrinsic matrix, this application designs a multimodal fusion method with sensor extrinsic tolerance characteristics, combines the cross-attention mechanism to perform local feature fusion of laser voxel features and visual image features, uses visually rich color and texture features to provide semantic information, and uses the universal detection and precise ranging characteristics of laser point clouds to predict the 3D grid occupancy state. Feature-level fusion allows different modalities to interact at an earlier stage, which helps to better capture the complex relationship between modalities.

[0272] 2) Dense-sparse knowledge distillation: In order to overcome the difficulty of complete and dense semantic scene completion from a single frame input (sparse and incomplete point cloud), this application designs a teacher-student model architecture and performs dense semantic knowledge distillation. Specifically, the teacher model uses multi-frame dense point cloud input, so the teacher model can extract more dense spatial features, and the dense-to-sparse knowledge distillation loss is used to transfer the teacher's dense semantic knowledge to the student model, so that the student model also has the ability to extract multi-frame dense features and better scene understanding, thereby enhancing the student model's spatial completion effect, and finally using the student model for obstacle detection.

[0273] 3) Multi-task prediction: A complete semantic occupancy prediction task + future prediction task is used for the teacher model. On the one hand, future prediction models the underlying dynamics of the scene. This understanding allows multiple observations in time to be aggregated to obtain a more complete geometric understanding. On the other hand, complete scene understanding allows future prediction to focus on object motion without worrying about sampling noise or occlusion changes faced by traditional point cloud prediction methods. Therefore, semantic scene completion can greatly improve future predictions and provide a comprehensive understanding of the scene. The two tasks work together and promote each other, which is conducive to the training of obstacle models.

[0274] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the obstacle detection method based on multimodal features of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0275] This application also provides an obstacle detection device based on multimodal features, please refer to Fig. 9 , Fig. 9A schematic diagram of the structure of an obstacle detection device based on multimodal features provided in the present application, wherein the obstacle detection device based on multimodal features comprises:

[0276] The acquisition module 901 is used to determine the space to be measured of the vehicle in the target direction, and to acquire the current laser point cloud and the current camera image corresponding to the space to be measured; wherein the current laser point cloud is a single-frame laser point cloud acquired at the current moment, and the current camera image is a single-frame camera image acquired at the current moment;

[0277] The feature extraction module 902 is used to extract features from the current laser point cloud to obtain a first 3D voxel feature; and to extract features from the current camera image to obtain a first 2D image feature;

[0278] A feature fusion module 903 is used to perform multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature;

[0279] The obstacle detection module 904 is used to input the first fusion feature into a pre-trained obstacle detection model to obtain obstacle information in the space to be detected output by the obstacle detection model.

[0280] The obstacle detection device based on multimodal features provided by the present application adopts the obstacle detection method based on multimodal features in the above-mentioned embodiments, which can solve the technical problems of low obstacle detection accuracy and low safety of autonomous driving vehicles. Compared with the related art, the beneficial effects of the obstacle detection device based on multimodal features provided by the present application are the same as the beneficial effects of the obstacle detection method based on multimodal features provided by the above-mentioned embodiments, and the other technical features of the obstacle detection device based on multimodal features are the same as the features disclosed in the above-mentioned embodiment methods, which will not be repeated here.

[0281] The present application provides an obstacle detection device based on multimodal features, and the obstacle detection device based on multimodal features includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the obstacle detection method based on multimodal features in the above-mentioned embodiment 1.

[0282] Reference below Fig.10 , Fig.10A structural diagram of the obstacle detection device based on multimodal features of the present application is provided, which shows a structural diagram of the obstacle detection device based on multimodal features suitable for implementing the embodiments of the present application. The obstacle detection device based on multimodal features in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The obstacle detection device based on multimodal features shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0283] like Fig.10 As shown, the obstacle detection device based on multimodal features may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the obstacle detection device based on multimodal features are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the obstacle detection device based on multimodal features to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an obstacle detection device based on multimodal features with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0284] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0285] The obstacle detection device based on multimodal features provided by the present application adopts the obstacle detection method based on multimodal features in the above-mentioned embodiment, which can solve the technical problems of low obstacle detection accuracy and low safety of autonomous driving vehicles. Compared with the related art, the beneficial effects of the obstacle detection device based on multimodal features provided by the present application are the same as the beneficial effects of the obstacle detection method based on multimodal features provided by the above-mentioned embodiment, and the other technical features in the obstacle detection device based on multimodal features are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0286] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0287] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0288] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the obstacle detection method based on multimodal features in the above-mentioned embodiment.

[0289] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0290] The computer-readable storage medium may be included in the obstacle detection device based on multimodal features; or may exist independently without being assembled into the obstacle detection device based on multimodal features.

[0291] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the obstacle detection device based on multimodal features, the obstacle detection device based on multimodal features performs the following steps:

[0292] Determine the space to be measured of the vehicle in the target direction, and collect the current laser point cloud and the current camera image corresponding to the space to be measured; wherein the current laser point cloud is a single-frame laser point cloud collected at the current moment, and the current camera image is a single-frame camera image collected at the current moment;

[0293] Performing feature extraction on the current laser point cloud to obtain a first 3D voxel feature; performing feature extraction on the current camera image to obtain a first 2D image feature;

[0294] Performing multimodal feature fusion on the first 3D voxel feature and the first 2D image feature to obtain a first fused feature;

[0295] The first fusion feature is input into a pre-trained obstacle detection model to obtain obstacle information in the space to be measured output by the obstacle detection model; wherein the obstacle information includes: a 3D occupancy grid and / or semantics corresponding to the 3D occupancy grid, the 3D occupancy grid is used to characterize position information of obstacles in the space to be measured, and the semantics corresponding to the 3D occupancy grid is used to characterize attribute information of obstacles in the space to be measured.

[0296] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0297] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0298] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0299] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned obstacle detection method based on multimodal features, and can solve the technical problems of low obstacle detection accuracy and low safety of autonomous driving vehicles. Compared with the existing related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the obstacle detection method based on multimodal features provided in the above-mentioned embodiments, and will not be elaborated here.

[0300] The present application also provides a computer program product, including a computer program, which implements the steps of the obstacle detection method based on multimodal features as described above when executed by a processor.

[0301] The computer program product provided by the present application can solve the technical problems of low obstacle detection accuracy and low safety of autonomous driving vehicles. Compared with the related art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the obstacle detection method based on multimodal features provided by the above embodiment, and will not be elaborated here.

[0302] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An obstacle detection method based on multimodal features, characterized in that: The method comprises: Determine the space to be measured of the vehicle in the target direction, and collect the current laser point cloud and the current camera image corresponding to the space to be measured; wherein the current laser point cloud is a single-frame laser point cloud collected at the current moment, and the current camera image is a single-frame camera image collected at the current moment; Performing feature extraction on the current laser point cloud to obtain a first 3D voxel feature; performing feature extraction on the current camera image to obtain a first 2D image feature; For each target voxel in the first 3D voxel feature, determine n image pixel points projected by the target voxel in the first 2D image feature as n direct projection points; wherein n is an integer greater than 0; According to the positions of the n direct projection points on the current camera image, dividing a target area including the positions corresponding to the n direct projection points on the current camera image; Determine the number of pixel points of the image pixels in the target area; Determine the size relationship between the number of pixels and a preset number; When it is determined that the number of pixels is less than a preset number, the range of the target area is expanded until the number of pixel points of the image pixels in the expanded target area is equal to the preset number, and the expanded target area is determined to be a cross-attention area; Performing cross-attention calculations on the voxel features of all target voxels in the first 3D voxel feature and the image features of the image pixels in the cross-attention area corresponding to each of the target voxels, respectively, to obtain a first fusion feature; The first fusion feature is input into a pre-trained obstacle detection model to obtain obstacle information in the space to be measured output by the obstacle detection model; wherein the obstacle information includes: a 3D occupancy grid and / or semantics corresponding to the 3D occupancy grid, the 3D occupancy grid is used to characterize position information of obstacles in the space to be measured, and the semantics corresponding to the 3D occupancy grid is used to characterize attribute information of obstacles in the space to be measured.

2. The method according to claim 1, characterized in that The method further comprises: In the case where it is determined that the number of pixels is equal to a preset number, determining that the target area is the cross-attention area; When it is determined that the number of pixel points is greater than a preset number, the n direct projection points and the m indirect projection points are combined into the cross-attention area; wherein m is the difference between the preset number and n, and the indirect projection points are other image pixel points on the current camera image except the n direct projection points.

3. The method according to claim 1, characterized in that The step of performing cross-attention calculations on the voxel features of all target voxels in the first 3D voxel feature and the image features of the image pixels in the cross-attention area corresponding to each of the target voxels to obtain the first fusion feature comprises: The voxel features corresponding to each of the target voxels are taken as a query set, and the image features corresponding to the image pixels in the cross-attention area corresponding to each of the target voxels are taken as a key value set, and cross-attention calculation is performed to obtain a first fusion feature.

4. The method according to any one of claims 1 to 3, characterized in that Before inputting the first fusion feature into a pre-trained obstacle detection model to obtain obstacle information in the to-be-detected space output by the obstacle detection model, the method further includes: Obtain a pre-trained teacher model; wherein the teacher model adopts an encoder-decoder structure, and the teacher model is trained by pre-collected multi-frame laser point clouds and corresponding single-frame camera images; Constructing a student model; wherein the student model adopts an encoder-decoder structure; The student model is trained by performing feature distillation between a first intermediate result corresponding to the teacher model and a second intermediate result corresponding to the student model, and the trained student model is used as the obstacle detection model; wherein the first intermediate result is the output result of the encoder of the teacher model, and the second intermediate result is the output result of the encoder of the student model.

5. The method according to claim 4, characterized in that The step of training the student model by performing feature distillation between the first intermediate result corresponding to the teacher model and the second intermediate result corresponding to the student model, and using the trained student model as the obstacle detection model includes: Determine a one-to-one correspondence between N first voxel features in the first intermediate result and N second voxel features in the second intermediate result; wherein N is an integer greater than 0; For each first voxel feature in the first intermediate result, the following steps are performed: In the case where the first voxel feature is not empty, determining whether a target voxel feature corresponding to the first voxel feature in the second intermediate result is empty; When it is determined that the target voxel feature is empty, L2 loss is used as a loss function to calculate a first loss value corresponding to the target voxel feature; Determine a feature distillation loss according to first loss values ​​corresponding to the N second voxel features in the second intermediate result; The student model is trained using the feature distillation loss, and the trained student model is used as the obstacle detection model.

6. The method according to claim 5, characterized in that After determining whether the target voxel feature corresponding to the first voxel feature in the second intermediate result is empty, the method further includes: When it is determined that the target voxel feature is not empty, KL divergence is used as a loss function to calculate a second loss value corresponding to the target voxel feature; The step of determining the feature distillation loss according to the first loss values ​​corresponding to the N second voxel features in the second intermediate result includes: Determine the feature distillation loss according to the first loss value or the second loss value corresponding to the N second voxel features in the second intermediate result.

7. The method according to claim 6, characterized in that The step of determining the feature distillation loss according to the first loss value or the second loss value corresponding to the N second voxel features in the second intermediate result includes: Accumulating the first loss values ​​corresponding to the N second voxel features in the second intermediate result to obtain a third loss value, and accumulating the second loss values ​​corresponding to the N second voxel features in the second intermediate result to obtain a fourth loss value; The sum of a fifth loss value and the fourth loss value is calculated as the feature distillation loss; wherein the fifth loss value is the product of the third loss value and a preset loss weighting coefficient.

8. The method according to claim 4, characterized in that The teacher model is trained specifically through the following steps: Following the movement of the preset collection vehicle, starting from the first moment, one frame of laser point cloud of the collection vehicle in the target direction is collected at a preset time interval, and a total of x frames of laser point cloud are collected; wherein x is an integer greater than 1; At the second moment of collecting the x-th frame of laser point cloud, collecting a single-frame camera image of the collecting vehicle in the target direction; According to the collected vehicle motion information, synchronizing the x-frame laser point cloud to the second moment; wherein the motion information is used to infer the correspondence between the x-frame laser point cloud from the first moment to the second moment; Performing feature extraction on the x-frame laser point cloud synchronized to the second moment to obtain a second 3D voxel feature; performing feature extraction on the single-frame camera image acquired at the second moment to obtain a second 2D image feature; Performing multimodal feature fusion on the second 3D voxel feature and the second 2D image feature to obtain a second fused feature; Inputting the second fused feature into a preset candidate model to obtain a first obstacle prediction result output by the candidate model; wherein the candidate model adopts an encoder-decoder structure, and the first obstacle prediction result includes an occupied grid corresponding to the first moment to the second moment and a semantics corresponding to the occupied grid; Obtaining a first obstacle prediction true value corresponding to the first moment to the second moment; Calculating a training loss function according to the first obstacle prediction true value and the first obstacle prediction result; The candidate model is trained according to the training loss function, and the trained candidate model is used as the teacher model.

9. The method according to claim 8, characterized in that Before calculating the training loss function according to the first obstacle prediction true value and the first obstacle prediction result, the method further includes: Inputting the second fusion feature into the candidate model to obtain a second obstacle prediction result output by the candidate model; wherein the second obstacle prediction result includes an occupied grid corresponding to a third moment and semantics corresponding to the occupied grid, and the third moment is a moment after the second moment by the preset time; Obtaining a second obstacle prediction true value corresponding to the third moment; The step of calculating a training loss function according to the first obstacle prediction true value and the first obstacle prediction result comprises: A training loss function is calculated according to the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result, and the second obstacle prediction result.

10. The method according to claim 9, characterized in that The first obstacle prediction result includes a first occupied grid result and a first semantic result, and the second obstacle prediction result includes a second occupied grid result; The step of calculating a training loss function according to the first obstacle prediction true value, the second obstacle prediction true value, the first obstacle prediction result, and the second obstacle prediction result comprises: Calculate the Soft-IOU loss between the occupancy grid true value in the first obstacle prediction true value and the first occupancy grid result as a sixth loss value; Calculating a cross entropy loss between a semantic truth value in the first obstacle prediction truth value and the first semantic result as a seventh loss value; Calculate the Soft-IOU loss between the occupancy grid true value in the second obstacle prediction true value and the second occupancy grid result as an eighth loss value; The training loss function is calculated according to the sixth loss value, the seventh loss value and the eighth loss value.

11. An obstacle detection device based on multimodal features, characterized in that: The device comprises: An acquisition module, used to determine the space to be measured of the vehicle in the target direction, and to acquire the current laser point cloud and the current camera image corresponding to the space to be measured; wherein the current laser point cloud is a single-frame laser point cloud acquired at the current moment, and the current camera image is a single-frame camera image acquired at the current moment; A feature extraction module is used to extract features from the current laser point cloud to obtain a first 3D voxel feature; and to extract features from the current camera image to obtain a first 2D image feature; a feature fusion module, for determining, for each target voxel in the first 3D voxel feature, n image pixels projected by the target voxel in the first 2D image feature as n direct projection points; wherein n is an integer greater than 0; dividing a target area including the positions corresponding to the n direct projection points on the current camera image according to the positions of the n direct projection points on the current camera image; determining the number of image pixels in the target area; judging the size relationship between the number of pixels and a preset number; if it is determined that the number of pixels is less than the preset number, expanding the range of the target area until the number of image pixels in the expanded target area is equal to the preset number, and determining that the expanded target area is a cross-attention area; performing cross-attention calculations between the voxel features of all target voxels in the first 3D voxel feature and the image features of the image pixels in the cross-attention area corresponding to each target voxel, to obtain a first fusion feature; The obstacle detection module is used to input the first fusion feature into a pre-trained obstacle detection model to obtain obstacle information in the space to be detected output by the obstacle detection model.

Images