Obstacle detection method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide an obstacle detection method, device, electronic device and storage medium, which are applied to the field of intelligent driving technology. By performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data, three-dimensional image feature information is obtained. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the present application can use fisheye cameras instead of laser radars, thereby reducing the production cost of intelligent assisted driving systems, and further reducing the production cost of vehicles with intelligent assisted driving functions. In addition, the embodiments of the present application can perform distortion mapping and three-dimensional mapping on the two-dimensional fisheye image data, thereby reducing the number of cameras while achieving full-view coverage of the vehicle, reducing the amount of computer calculations, improving the real-time performance of obstacle detection, and effectively improving the obstacle perception capability of the autonomous driving system and improving the safety of the autonomous driving system.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to an obstacle detection method, device, electronic device and storage medium. Background Art

[0002] Assisted driving or autonomous driving is an important function of today's vehicles, mainly including lane keeping, cruise control, automatic parking, brake assist, reversing assist, signal light detection and other auxiliary functions. These functions can use machine systems to replace human drivers as the driving subject of the vehicle to operate the vehicle under the specified driving scenarios, thereby avoiding operational errors caused by human judgment errors, improving the driving safety of the vehicle while also improving the driving comfort of the driver, and therefore have attracted widespread attention.

[0003] In related technologies, the vehicle-mounted machine system needs to detect the driving environment around the vehicle in real time through laser radar. For example, by receiving the laser radar signal reflected by the obstacle, the three-dimensional spatial information of the obstacle, such as distance, direction, height, etc., is obtained, and then the road planning or driving operation is performed according to the three-dimensional spatial information of the obstacle, so as to achieve the purpose of bypassing the obstacle in time to avoid vehicle scratches. However, the high-precision laser emission and receiving modules in the laser radar, as well as the precise optical and mechanical components, make the laser radar production cost high, which leads to the high cost of obstacle detection. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an obstacle detection method, device, electronic device and storage medium to reduce the cost of obstacle detection. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a method for detecting an obstacle is provided, the method comprising:

[0006] Acquire multi-view fisheye image data;

[0007] Inputting the multi-view fisheye image data into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes;

[0008] Inputting each of the two-dimensional image features into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and three-dimensional mapping to obtain three-dimensional image features of multiple sizes;

[0009] Each of the three-dimensional image features is input into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle detection result.

[0010] In a possible implementation, the feature extraction network includes a backbone network and a feature pyramid network;

[0011] The multi-view fisheye image data is input into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes, including:

[0012] Downsampling the multi-view fisheye image data using the backbone network to obtain downsampled image features of multiple sizes;

[0013] The feature pyramid network is used to perform feature fusion on the downsampled image features to obtain two-dimensional image features of multiple sizes.

[0014] In a possible implementation, the multi-view feature aggregation projection network includes a plurality of two-dimensional-three-dimensional mapping networks;

[0015] The step of inputting each of the two-dimensional image features into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and three-dimensional mapping to obtain three-dimensional image features of multiple sizes includes:

[0016] Each of the two-dimensional-three-dimensional mapping networks is used to perform distortion mapping and three-dimensional mapping on the two-dimensional image features of its corresponding size to obtain three-dimensional image features of multiple sizes, wherein each of the two-dimensional-three-dimensional mapping networks corresponds to one size.

[0017] In a possible implementation, the two-dimensional-three-dimensional mapping network includes: an attention network, a first residual connection and normalization network, a feedforward neural network, and a second residual connection and normalization network;

[0018] The method of using each of the two-dimensional-three-dimensional mapping networks to perform distortion mapping and three-dimensional mapping on the two-dimensional image features of the corresponding size to obtain three-dimensional image features of multiple sizes includes:

[0019] Using the attention network to perform distortion mapping and three-dimensional mapping on the input two-dimensional image feature to obtain a first image feature;

[0020] Using the first residual connection and normalization network to perform residual connection and normalization processing on the input first image feature to obtain a second image feature;

[0021] Using the feedforward neural network to perform feature enhancement on the input second image feature to obtain a third image feature;

[0022] The second residual connection and normalization network is used to perform residual connection and normalization processing on the input third image feature to obtain a three-dimensional image feature.

[0023] In a possible implementation, the using the attention network to perform distortion mapping and three-dimensional mapping on the input two-dimensional image feature to obtain the first image feature includes:

[0024] Using the attention network, we transform: , get the first image feature:

[0025] in, represents the first image feature, represents the query point of the attention network, represents the two-dimensional image features, A set of query points representing the attention network; Represents the mapping operation of the deformable convolutional attention module in the attention network; represents a projection function of a three-dimensional image feature to a two-dimensional image feature; the multi-view fisheye image data includes fisheye images of multiple viewpoints, Represents the two-dimensional image features of the fisheye image at the i-th viewing angle.

[0026] In a possible implementation, the mapping relationship between the query point and the two-dimensional point is:

[0027] ;

[0028] in, Represents the two-dimensional pixel coordinates of the j-th query point of the attention network mapped to the two-dimensional image features of the fisheye image of the i-th perspective; , , , Represents the camera internal parameters; represents the x-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system; It represents the y-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system;

[0029] = ; = ;

[0030] in, Represents the radial distance from the pixel of the distorted two-dimensional image feature to the center of the circle, represents the incident angle of the jth query point;

[0031] ;

[0032] in, is the distortion parameter, r represents the polar coordinates of the normalized query point;

[0033] ;

[0034] in, represents the x-axis coordinate of the j-th normalized query point, represents the y-axis coordinate of the j-th normalized query point,

[0035] ;

[0036] in, represents the coordinates of the jth normalized query point, represents the x-axis coordinate of the j-th query point, represents the y-axis coordinate of the j-th query point, Represents the z-axis coordinate of the j-th query point.

[0037] In a possible implementation, the feature map upsampling network includes multiple three-dimensional deconvolution networks and a probability prediction module;

[0038] The step of inputting each of the three-dimensional image features into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle detection result includes:

[0039] Upsampling the first three-dimensional image feature using the first three-dimensional deconvolution network to obtain a first upsampled image feature;

[0040] Fusing the i-1th upsampled image feature with the ith three-dimensional image feature, and inputting the fused feature into the ith three-dimensional deconvolution network for upsampling to obtain the ith upsampled image feature, where 1<i≤N, i is an integer, and N is the total number of three-dimensional deconvolution networks;

[0041] The probability prediction module is used to perform probability prediction on the Nth upsampled image feature to obtain the placeholder category of each placeholder grid of the three-dimensional scene, wherein the three-dimensional obstacle detection result includes the placeholder category of each placeholder grid, and the placeholder category of the placeholder grid is the maximum probability category of the voxel in the placeholder grid.

[0042] In a second aspect of an embodiment of the present application, a method for training an obstacle detection model is provided, the method comprising:

[0043] Acquire sample fisheye image data and a three-dimensional obstacle calibration true value of the three-dimensional scene corresponding to the sample fisheye image data;

[0044] Inputting the sample fisheye image data into a feature extraction network of an obstacle detection model for feature extraction to obtain two-dimensional sample image features of multiple sizes;

[0045] Inputting each of the two-dimensional sample image features into a multi-view feature aggregation projection network of an obstacle detection model for distortion mapping and three-dimensional mapping to obtain three-dimensional sample image features of multiple sizes;

[0046] Inputting each of the three-dimensional sample image features into a feature map upsampling network of an obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle prediction result;

[0047] The loss of the obstacle detection model is calculated according to the three-dimensional obstacle calibration true value and the three-dimensional obstacle prediction result, and the obstacle detection model is adjusted according to the loss.

[0048] In a third aspect of an embodiment of the present application, an obstacle detection device is provided, the device comprising:

[0049] An image acquisition module, used for acquiring multi-view fisheye image data;

[0050] A feature extraction module, used for inputting the multi-view fisheye image data into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes;

[0051] An image mapping module, used for inputting each of the two-dimensional image features into the multi-view feature aggregation projection network of the obstacle detection model to perform distortion mapping and three-dimensional mapping to obtain three-dimensional image features of multiple sizes;

[0052] The probability prediction module is used to input each of the three-dimensional image features into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle detection result.

[0053] In a possible implementation, the feature extraction network includes a backbone network and a feature pyramid network;

[0054] The feature extraction module comprises:

[0055] A downsampling submodule, specifically used to downsample the multi-view fisheye image data using the backbone network to obtain downsampled image features of multiple sizes;

[0056] The down-sampling feature fusion submodule is specifically used to use the feature pyramid network to perform feature fusion on the down-sampled image features to obtain two-dimensional image features of multiple sizes.

[0057] In a possible implementation, the multi-view feature aggregation projection network includes a plurality of two-dimensional-three-dimensional mapping networks;

[0058] The image mapping module comprises:

[0059] The image mapping submodule is specifically used to use each of the two-dimensional-three-dimensional mapping networks to perform distortion mapping and three-dimensional mapping on the two-dimensional image features of the corresponding size to obtain three-dimensional image features of multiple sizes, wherein each of the two-dimensional-three-dimensional mapping networks corresponds to one size.

[0060] In a possible implementation, the two-dimensional-three-dimensional mapping network includes: an attention network, a first residual connection and normalization network, a feedforward neural network, and a second residual connection and normalization network;

[0061] The image mapping submodule comprises:

[0062] An image mapping unit, specifically used to use the attention network to perform distortion mapping and three-dimensional mapping on the input two-dimensional image feature to obtain a first image feature;

[0063] a first normalization unit, specifically configured to perform residual connection and normalization processing on the input first image feature by using the first residual connection and normalization network to obtain a second image feature;

[0064] A feature enhancement unit, specifically used to perform feature enhancement on the input second image feature using the feedforward neural network to obtain a third image feature;

[0065] The second normalization unit is specifically used to use the second residual connection and normalization network to perform residual connection and normalization processing on the input third image feature to obtain a three-dimensional image feature.

[0066] In a possible implementation, the image mapping unit includes:

[0067] The image transformation subunit is specifically used to utilize the attention network to transform: , get the first image feature:

[0068] in, represents the first image feature, represents the query point of the attention network, represents the two-dimensional image features, A set of query points representing the attention network; represents the mapping operation of the deformable convolution attention module of the attention network; P represents the projection function of the three-dimensional image feature to the two-dimensional image feature; the multi-view fisheye image data includes fisheye images of multiple perspectives, Represents the two-dimensional image features of the fisheye image at the i-th viewing angle.

[0069] In a possible implementation, the mapping relationship between the query point and the two-dimensional point is:

[0070] ;

[0071] in, Represents the two-dimensional pixel coordinates of the j-th query point of the attention network mapped to the two-dimensional image features of the fisheye image of the i-th perspective; , , , Represents the camera internal parameters; represents the x-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system; It represents the y-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system;

[0072] = ; = ;

[0073] in, Represents the radial distance from the pixel of the distorted two-dimensional image feature to the center of the circle, represents the incident angle of the jth query point;

[0074] ;

[0075] in, is the distortion parameter, r represents the polar coordinates of the normalized query point;

[0076] ;

[0077] in, represents the x-axis coordinate of the j-th normalized query point, represents the y-axis coordinate of the j-th normalized query point,

[0078] ;

[0079] in, represents the coordinates of the jth normalized query point, represents the x-axis coordinate of the j-th query point, represents the y-axis coordinate of the j-th query point, Represents the z-axis coordinate of the j-th query point.

[0080] In a possible implementation, the feature map upsampling network includes multiple three-dimensional deconvolution networks and a probability prediction module;

[0081] The obstacle detection module comprises:

[0082] An upsampling submodule, specifically used to upsample the first three-dimensional image feature using the first three-dimensional deconvolution network to obtain a first upsampled image feature;

[0083] The upsampling feature fusion submodule is specifically used to fuse the i-1th upsampled image feature with the ith three-dimensional image feature, and input the fused feature into the ith three-dimensional deconvolution network for upsampling to obtain the ith upsampled image feature, where 1<i≤N, i is an integer, and N is the total number of three-dimensional deconvolution networks;

[0084] The probability prediction submodule is specifically used to use the probability prediction module to perform probability prediction on the Nth upsampled image feature to obtain the placeholder category of each placeholder grid of the three-dimensional scene, wherein the three-dimensional obstacle detection result includes the placeholder category of each placeholder grid, and the placeholder category of the placeholder grid is the maximum probability category of the voxel in the placeholder grid.

[0085] In a fourth aspect of an embodiment of the present application, a device for training an obstacle detection model is provided, the device comprising:

[0086] A sample acquisition module, used to acquire sample fisheye image data and a three-dimensional obstacle calibration true value of a three-dimensional scene corresponding to the sample fisheye image data;

[0087] A sample feature extraction module, used for inputting the sample fisheye image data into a feature extraction network of an obstacle detection model for feature extraction, to obtain two-dimensional sample image features of multiple sizes;

[0088] A sample image mapping module, used to input each of the two-dimensional sample image features into a multi-view feature aggregation projection network of an obstacle detection model for distortion mapping and three-dimensional mapping, so as to obtain three-dimensional sample image features of multiple sizes;

[0089] A sample image prediction module, used to input the features of each of the three-dimensional sample images into a feature map upsampling network of an obstacle detection model for upsampling and probability prediction, to obtain a three-dimensional obstacle prediction result;

[0090] The loss calculation module is used to calculate the loss of the obstacle detection model according to the three-dimensional obstacle calibration true value and the three-dimensional obstacle prediction result, and adjust the obstacle detection model according to the loss.

[0091] Another aspect of the embodiments of the present application further provides an electronic device, including:

[0092] Memory, used to store computer programs;

[0093] The processor is used to implement any obstacle detection method or obstacle detection model training method described in any of the embodiments of the present application when executing the program stored in the memory.

[0094] In another aspect of the implementation of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned obstacle detection methods or obstacle detection model training methods is implemented.

[0095] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned obstacle detection methods or obstacle detection model training methods.

[0096] Beneficial effects of the embodiments of the present application:

[0097] The embodiments of the present application provide an obstacle detection method, device, electronic device and storage medium. The obstacle detection method, device, electronic device and storage medium can extract features from the acquired multi-view fisheye image data, and perform distortion mapping and three-dimensional mapping on the acquired image features to convert the image features from two-dimensional to three-dimensional, thereby performing probability prediction based on the three-dimensional image features to achieve three-dimensional obstacle detection.

[0098] By applying the method of the embodiment of the present application, three-dimensional image feature information can be obtained by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use a fisheye camera instead of laser radar, thereby reducing the cost of obstacle detection and further reducing the production cost of vehicles with intelligent assisted driving functions. In addition, in response to the image distortion phenomenon of the fisheye camera, the method of the embodiment of the present application can achieve image correction by performing distortion mapping on the two-dimensional image features to improve the accuracy of obstacle detection.

[0099] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0101] Figure 1A schematic diagram of an application of an autonomous driving system provided in an embodiment of the present application;

[0102] Figure 2 A schematic diagram of a flow chart of an obstacle detection method provided in an embodiment of the present application;

[0103] Figure 3 A schematic diagram of the backbone network and feature pyramid network structure provided in the embodiment of the present application;

[0104] Figure 4 A schematic diagram of the structure of a multi-view feature aggregation projection network provided in an embodiment of the present application;

[0105] Figure 5 A schematic diagram of a process flow for refining step 203 provided in an embodiment of the present application;

[0106] Figure 6 A schematic diagram of a structure of a two-dimensional-three-dimensional mapping network provided in an embodiment of the present application;

[0107] Figure 7 A schematic diagram of a structure of a feature map upsampling network provided in an embodiment of the present application;

[0108] Figure 8 A flowchart of an obstacle detection model training method provided in an embodiment of the present application;

[0109] Fig. 9 A schematic diagram of the structure of an obstacle detection device provided in an embodiment of the present application;

[0110] Fig.10 A schematic diagram of the structure of an obstacle detection model training device provided in an embodiment of the present application;

[0111] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0112] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0113] An automated driving system (ADS) refers to a system that will continuously perform all dynamic driving tasks (DDT) within its own operational domain design (ODD). That is, the machine system is allowed to fully assume the task of automatic vehicle control only under the specified appropriate driving scenario conditions. When the vehicle meets the ODD conditions, the system is activated, and the system replaces the human driver as the driving subject of the vehicle. Among them, the dynamic driving task DDT refers to the continuous lateral (left and right steering) and longitudinal motion control (acceleration, deceleration, and constant speed) of the vehicle, as well as the detection and response to targets and events in the vehicle's driving environment. The design operating range ODD refers to the conditions under which the automated driving system can operate safely. The set conditions may include geographic location, road type, speed range, weather, time, traffic laws and regulations, etc.

[0114] Environmental perception is the foundation and core of the autonomous driving system. It is mainly used to detect obstacles in the vehicle's surrounding environment. The accuracy of this detection is related to the safety of the autonomous driving system. And with the development of autonomous driving technology, the need for vehicles to perceive the surrounding environment in complex environments has become more urgent. In related technologies, traditional autonomous driving systems often rely on high-cost sensors such as LiDAR to achieve environmental perception. In one example, the LiDAR emits a laser signal and receives a laser signal reflected by an obstacle. Based on the received reflected laser signal, three-dimensional point cloud data is generated, and then an accurate digital three-dimensional model is generated based on the three-dimensional point cloud data to achieve high-precision obstacle detection. For example Figure 1 As shown, during the driving process, the vehicle in front shows an abnormality and places a warning sign to alert the vehicle behind. After receiving the laser signal reflected by the warning sign, the vehicle behind determines that there is an obstacle in front of its own vehicle, and thus controls the vehicle to detour to avoid the obstacle.

[0115] However, LiDAR uses high-precision laser transmitting and receiving modules, as well as precision optical and mechanical components, which results in high manufacturing and assembly costs. During the use of LiDAR, precise calibration and positioning are required, which usually requires professional personnel to operate, thus increasing the initial installation and subsequent maintenance costs of LiDAR. In addition, some mechanical parts of LiDAR are prone to wear and tear during use and require regular maintenance and replacement, which increases the operating costs of LiDAR.

[0116] In addition, in severe weather conditions (such as heavy rain, heavy snow, and haze), the environmental perception ability of LiDAR will be affected by the weather and perform poorly, thus affecting the reliability of the autonomous driving system. For example, in rainy and snowy weather, raindrops and snowflakes will strongly reflect and scatter the laser beam, and small particles in haze weather will cause the laser beam to have a scattering effect. LiDAR relies on the reflected light beam to measure the distance and shape. Raindrops and snowflakes will interfere with these measurements, resulting in increased data noise and decreased perception accuracy. In addition, when the sensor is facing the light source, it is necessary to distinguish between the reflected light from the object and the light from other ambient light sources. Therefore, under strong light conditions, the environmental perception ability of LiDAR is weak, and false detection and missed detection may occur.

[0117] In another embodiment, multiple small cameras can be installed on the vehicle to detect obstacles, but the viewing angle of the small camera is relatively narrow, and the limited viewing angle coverage will inevitably lead to perception blind spots. In order to achieve full coverage of the viewing angle around the vehicle, the number of small cameras needs to be increased, but the increase in the number will lead to an increase in the amount of calculation and the increase in the computing cost. In addition, the automatic perception system needs to obtain the surrounding obstacle detection results in real time. The increase in the amount of calculation may cause a delay in the detection results, making it impossible for the automatic perception system to make real-time judgments.

[0118] In order to solve at least one of the above problems, a first aspect of an embodiment of the present application provides an obstacle detection method, the method comprising: Figure 2 Steps shown:

[0119] Step 201: Acquire multi-view fisheye image data.

[0120] Among them, the multi-view fisheye image data refers to the multi-view images taken by the fisheye camera. Fisheye cameras have the characteristics of wide shooting angles, image distortion, high light transmittance, and large depth of field. Therefore, when setting fisheye cameras on vehicles, compared with other cameras, fewer fisheye cameras can be installed while ensuring that the viewing angles around the vehicle are fully covered, saving camera costs. The acquired multi-view fisheye image data can be an RGB (Red, Green, Blue) image or a grayscale image. In practical applications, the multi-view fisheye image can also be pre-processed and then input into the obstacle detection model for feature extraction, wherein the pre-processing methods include but are not limited to grayscale processing, binarization processing, histogram correction, image smoothing processing, denoising processing and other image processing methods.

[0121] In practical applications, the method of the embodiment of the present application can be implemented through a terminal device. In an example, the terminal device can be a server, a vehicle system, a computer or other electronic device.

[0122] Step 202: Input the multi-view fisheye image data into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes.

[0123] The obstacle detection model is a pre-trained deep learning model, which can be a neural network model such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and Generative Adversarial Networks (GAN). In one example, the pre-trained obstacle detection model is VGG Net (Visual Geometry Group Network). In another example, a CNN network is selected as the obstacle detection model, and the sample image is passed through the convolution layer, pooling layer, and fully connected layer in the CNN network to achieve feature extraction and prediction of the sample image. According to the true value and prediction result of the sample image, the loss of the CNN network is calculated, and the CNN network is adjusted according to the loss to obtain a pre-trained obstacle detection model.

[0124] When performing feature extraction on multi-view fisheye image data, the obstacle detection model can perform feature extraction multiple times to obtain multi-layer semantic information of the multi-view fisheye image data. In an example, the multi-view fisheye image data is downsampled multiple times and then the image features of the multi-view fisheye image data after each downsampling are extracted, thereby obtaining multiple two-dimensional image features of different sizes, thereby obtaining both the global feature information of the multi-view fisheye image data and the local feature information of the multi-view fisheye image data.

[0125] Step 203: Input each 2D image feature into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and 3D mapping to obtain 3D image features of multiple sizes.

[0126] The multi-view feature aggregation projection network can first perform distortion mapping on the two-dimensional image features of each size, perform distortion correction on them, and then perform three-dimensional mapping on the corrected two-dimensional image features to obtain the three-dimensional image features after the two-dimensional image feature mapping at that size.

[0127] Due to the image distortion characteristics of fisheye cameras, some areas of the collected multi-view fisheye image data will have serious distortion effects. Therefore, it is necessary to perform distortion mapping to achieve the purpose of image distortion correction. Among them, the multi-view feature aggregation projection network can be a related camera network model. For example, the Kannala-Brandt (fisheye camera model) distortion model can be used to achieve distortion mapping and three-dimensional mapping of two-dimensional image features.

[0128] When performing distortion mapping and three-dimensional mapping, mapping can be performed based on pre-calibrated internal and external reference information. The internal reference information is related to the sensor in the vehicle (i.e., the camera installed in the vehicle), and can include parameters such as the focal length, eccentricity, pixel aspect ratio, and distortion coefficient of the camera. These parameters determine how the sensor processes image information internally and directly affect the quality and accuracy of the image. External reference information refers to the conversion relationship between the camera and vehicle coordinate systems, which is related to the position and direction of the camera in the vehicle. Through distortion mapping and three-dimensional mapping, the two-dimensional image features are transferred from the plane coordinate system to the three-dimensional coordinate system.

[0129] Step 204: Input each 3D image feature into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a 3D obstacle detection result.

[0130] In step 203, a plurality of 3D image features of different sizes are obtained. In order to fuse the feature information of each 3D image, it is necessary to adjust their sizes to the same and then superimpose them. It is necessary to enlarge the 3D image features of smaller sizes to the same size as the 3D image features of larger sizes through upsampling or deconvolution. After superimposing the 3D image features, a final 3D image feature is obtained, and a probability prediction is performed on the 3D image feature to obtain a 3D obstacle detection result.

[0131] Probabilistic prediction refers to classification prediction, and its prediction results can include the obstacle category represented by the image features and the probability of representing the obstacle category. For example, the probability that the obstacle is a roadblock is 90% in the obtained three-dimensional obstacle detection results. In one example, the probability prediction is performed through the argmax function (arguments of the maxima, optimization function) of the obstacle detection model. This function can be used to determine the location of the maximum value of the input value and is often used to determine the best prediction result of the classifier in the deep learning model.

[0132] By applying the method of the embodiment of the present application, three-dimensional image feature information can be obtained by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use a fisheye camera instead of laser radar, thereby reducing the cost of obstacle detection, and further reducing the production cost of vehicles with intelligent assisted driving functions. In addition, due to the image distortion effect of the fisheye camera, the method of the embodiment of the present application can achieve image correction by performing distortion mapping on the two-dimensional image features. Therefore, compared with other cameras, the embodiment of the present application can utilize the wide shooting angle of the fisheye camera to achieve full-view coverage of the vehicle while reducing the number of cameras, thereby reducing the amount of computer calculations, improving the real-time performance of obstacle detection, and effectively improving the obstacle perception capability of the autonomous driving system, and improving the safety of the autonomous driving system.

[0133] In a possible implementation, the feature extraction network includes a backbone network and a feature pyramid network; then step 201 may be implemented by the following steps:

[0134] Step 1: Use the backbone network to downsample the multi-view fisheye image data to obtain downsampled image features of multiple sizes.

[0135] The backbone network includes multiple downsampling modules, which are used to downsample the multi-view fisheye image data to reduce its size. Figure 3 As shown, the backbone network includes 4 downsampling modules for feature extraction of multi-view fisheye image data. The multi-view fisheye image is recorded as the first image. After passing through the first downsampling module, a second downsampling image feature image with reduced resolution and increased number of channels is obtained; after passing through the second downsampling module, a third downsampling image feature image is obtained; after passing through the third downsampling module, a fourth downsampling image feature image is obtained; and finally, after passing through the fourth downsampling module, a fifth downsampling image feature image is obtained.

[0136] During the downsampling process, the resolution of the downsampled image features output by the backbone network gradually decreases, but the number of channels of the downsampled image features increases layer by layer. As the information extraction process proceeds, the information in the downsampled image features is highly abstracted and summarized, containing richer semantic information.

[0137] Step 2: Use the feature pyramid network to fuse the downsampled image features to obtain two-dimensional image features of multiple sizes.

[0138] After the multi-view fisheye image data is downsampled to reduce the image size, the higher resolution image has shallow features with low semantic information, and the lower resolution image has deep features with high semantic information. In order to obtain high-resolution and high-semantic image features, it is necessary to fuse the downsampled image features output by the backbone network at different levels through the feature pyramid.

[0139] When fusion of features is performed, it is necessary to expand the downsampled image features with a smaller resolution to the same resolution as the adjacent downsampled image features before fusion of features is performed. There are many ways to expand image features, for example, image features can be enlarged by operations such as deconvolution, upsampling, and bilinear interpolation.

[0140] like Figure 3 As shown, first, all feature maps need to be passed through a network layer, such as a convolution layer, to unify the image features to the same number of channels. Starting from the downsampled image feature with the smallest resolution (the fifth image), the first network layer in the feature pyramid network is used to perform a convolution operation on the fifth image to obtain the fifth image feature, and the fifth image feature is upsampled to the same resolution as the fourth image, and the upsampled fifth image feature is fused with the fourth image and input into the second network layer to obtain a two-dimensional image feature (the fourth image feature). Correspondingly, the fourth image feature is upsampled to the same resolution as the third image, and the upsampled fourth image feature is fused with the third image and input into the third network layer to obtain a two-dimensional image feature (the third image feature); the third image feature is upsampled to the same resolution as the second image, and the upsampled third image feature is fused with the second image and input into the fourth network layer to obtain a two-dimensional image feature (the second image feature).

[0141] By using the method of the embodiment of the present application, multiple downsampled image features of different sizes can be obtained by downsampling the multi-view fisheye image data through the backbone network, and then through the lateral connection of the feature pyramid network and the top-to-bottom feature fusion, the semantic information of the multi-view fisheye image data at different levels can be obtained, and the two-dimensional image features with high resolution and high semantic information can be obtained. Thus, the size and shape changes of the multi-view fisheye image data are effectively utilized, so that the obstacle detection model can be more adaptable to obstacle detection in complex scenes, and the vehicle's perception of the surrounding environment is improved.

[0142] In one possible implementation, the multi-view feature aggregation projection network includes a plurality of two-dimensional-three-dimensional mapping networks;

[0143] Then step 203 can be implemented by the following steps:

[0144] Each 2D-3D mapping network is used to perform distortion mapping and 3D mapping on the 2D image features of its corresponding size to obtain 3D image features of multiple sizes, wherein each 2D-3D mapping network corresponds to one size.

[0145] Still Figure 3 Taking the example shown as an example, the two-dimensional image features obtained through the feature pyramid are the second image feature, the third image feature, and the fourth image feature, and the resolutions of these three two-dimensional image features are 256x256x64, 128x128x128, and 64x64x256, respectively. For each two-dimensional image feature, there is a two-dimensional-three-dimensional mapping network corresponding to its size, which is used to map the two-dimensional image feature into a three-dimensional image feature of the same size.

[0146] like Figure 4 As shown, in Figure 3 On the basis of , the multi-view feature aggregation projection network includes three two-dimensional-three-dimensional mapping networks, namely the first two-dimensional-three-dimensional mapping network, the second two-dimensional-three-dimensional mapping network and the third two-dimensional-three-dimensional mapping network. Among them, the first two-dimensional-three-dimensional mapping network is used to perform distortion mapping and three-dimensional mapping on the two-dimensional image features with a resolution of 64x64x256 to obtain three-dimensional image features (fourth three-dimensional image features); the second two-dimensional-three-dimensional mapping network is used to perform distortion mapping and three-dimensional mapping on the two-dimensional image features with a resolution of 128x128x128 to obtain three-dimensional image features (third three-dimensional image features); the third two-dimensional-three-dimensional mapping network is used to perform distortion mapping and three-dimensional mapping on the two-dimensional image features with a resolution of 256x256x64 to obtain three-dimensional image features (second three-dimensional image features).

[0147] By applying the method of the embodiment of the present application, the three-dimensional image feature information can be obtained by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use fisheye camera instead of laser radar, thereby reducing the obstacle detection cost, and further reducing the production cost of vehicles with intelligent assisted driving function. And by performing distortion mapping on the two-dimensional image features to achieve image correction, compared with other cameras, the embodiment of the present application can take advantage of the wide shooting angle of the fisheye camera, while achieving full-view coverage of the vehicle, reducing the number of cameras, thereby reducing the amount of computer calculations, improving the real-time performance of obstacle detection, effectively improving the obstacle perception capability of the autonomous driving system, and improving the safety of the autonomous driving system.

[0148] In a possible implementation, the 2D-3D mapping network includes: an attention network, a first residual connection and a normalization network, a feedforward neural network, and a second residual connection and a normalization network; when each 2D-3D mapping network performs distortion mapping and 3D mapping on the 2D image features of its corresponding size, it can be achieved by Figure 5 The steps shown achieve:

[0149] Step 501: Use the attention network to perform distortion mapping and three-dimensional mapping on the input two-dimensional image features to obtain the first image features.

[0150] Step 502: Use the first residual connection and normalization network to perform residual connection and normalization processing on the input first image feature to obtain a second image feature.

[0151] Step 503: Use a feedforward neural network to perform feature enhancement on the input second image feature to obtain a third image feature.

[0152] Step 504: Use the second residual connection and normalization network to perform residual connection and normalization processing on the input third image features to obtain three-dimensional image features.

[0153] In order to realize the mapping from 2D image features to 3D image features, the attention network is a spatial attention network for fisheye camera distortion, which can realize the 3D mapping of 2D images and output the first image features. In an example, the 2D-3D mapping network is as follows Figure 6 The structure shown, wherein the distortion-aware spatial cross-attention module is the attention network in the embodiment of the present application, is used to execute step 501 to map the two-dimensional image features into three-dimensional first image features.

[0154] The first residual superposition module is used to execute step 502, wherein the first residual superposition module can be an Add&Norm (residual connection and layer normalization) layer in the obstacle detection model. The residual superposition module can use residual connections to allow the input image features to directly cross one or more network layers and be added to the output of the subsequent network layer, thereby alleviating the problems of gradient vanishing and gradient explosion in the obstacle detection network. The normalization layer is used to normalize the output of the network layer in the obstacle detection model, thereby improving the stability of the obstacle detection model during the training process, and improving the convergence speed and performance of the obstacle detection model. The second residual superposition module is used to execute step 504, and its structure can be the same as the first residual superposition module, which is an Add&Norm layer.

[0155] The feed-forward neural network module (FFN) is used to execute step 503. The FFN includes a fully connected layer and an activation function, which can perform complex nonlinear changes on the input image features and enhance the expressiveness of the image features. Moreover, based on the distortion mapping and three-dimensional mapping performed by the attention network, the FFN can further increase the independence between positions and reduce the mutual interference between the positions of the image features.

[0156] By applying the method of the embodiment of the present application, the three-dimensional image feature information can be obtained by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use fisheye camera instead of laser radar, thereby reducing the obstacle detection cost, and further reducing the production cost of vehicles with intelligent assisted driving function. And by performing distortion mapping on the two-dimensional image features to achieve image correction, compared with other cameras, the embodiment of the present application can take advantage of the wide shooting angle of the fisheye camera, while achieving full-view coverage of the vehicle, reducing the number of cameras, thereby reducing the amount of computer calculations, improving the real-time performance of obstacle detection, effectively improving the obstacle perception capability of the autonomous driving system, and improving the safety of the autonomous driving system.

[0157] In a possible implementation, the first image feature can be obtained by using the attention network through the following transformation formula:

[0158] ;

[0159] in, represents the first image feature, represents the query point of the attention network, represents the two-dimensional image features, represents the set of query points of the attention network; Representing the mapping operation of the deformable convolutional attention module of the attention network; Represents a projection function of a three-dimensional image feature to a two-dimensional image feature; the multi-view fisheye image data includes fisheye images of multiple viewpoints, Represents the two-dimensional image features of the fisheye image at the i-th viewing angle.

[0160] The query point in the embodiment of the present application is a hit point selected from the three-dimensional points that do not contain the contour of the ego vehicle, that is, a point selected from the three-dimensional points in the vehicle's surrounding environment that may be obstacles, where the ego vehicle refers to a vehicle that includes sensors for sensing the surrounding environment. The query point can be expressed as , where C represents the number of channels of the feature point, H represents the height of the feature point, W represents the width of the feature point, and Z represents the depth of the feature point. It can be understood that the query point is selected by the attention network. The specific selection process can be referred to in the relevant technology, and is not specifically limited in this application.

[0161] The mapping process of the deformable convolutional attention module includes two parts: attention mapping and summed average mapping. In an example, when the deformable convolutional attention module maps the query point, it can be mapped by the following formula:

[0162] ;

[0163] in, represents the total number of attention heads of the deformable convolutional attention module, represents the number of sampling points, represents the normalized attention weight of the query point, and represents the learnable weights, Indicates the position offset of the sampling point relative to the two-dimensional pixel point in the two-dimensional image feature. Indicates the value obtained by interpolation based on the sampling point position.

[0164] By applying the method of the embodiment of the present application, the three-dimensional image feature information can be obtained by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use fisheye camera instead of laser radar, thereby reducing the obstacle detection cost, and further reducing the production cost of vehicles with intelligent assisted driving function. And by performing distortion mapping on the two-dimensional image features to achieve image correction, compared with other cameras, the embodiment of the present application can take advantage of the wide shooting angle of the fisheye camera, while achieving full-view coverage of the vehicle, reducing the number of cameras, thereby reducing the amount of computer calculations, improving the real-time performance of obstacle detection, effectively improving the obstacle perception capability of the autonomous driving system, and improving the safety of the autonomous driving system.

[0165] In a possible implementation, the mapping relationship between the query point and the two-dimensional point is:

[0166] ;

[0167] in, Represents the 2D pixel coordinates of the j-th query point of the attention network mapped to the 2D image features of the fisheye image of the i-th view; , , , Represents the camera internal parameters; , represents the equivalent focal length of the camera, , represents the optical center of the camera; It represents the x-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system; Represents the y-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system.

[0168] in, It refers to the plane normalized coordinates of the query point mapped to the two-dimensional point , It can be calculated by the following formula:

[0169] ; = ; = ;

[0170] in, Represents the radius of the distorted image, that is, the radial distance from the pixel of the distorted two-dimensional image feature to the center of the circle. represents the incident angle of the j-th query point.

[0171] The radial distance from the pixel of the distorted two-dimensional image feature to the center of the circle can be calculated based on the distortion parameter and the following radial distortion formula:

[0172] ;

[0173] in, is the distortion parameter, r represents the polar coordinates of the normalized query point, and the polar coordinates can be calculated by the following formula:

[0174] ;

[0175] in, represents the x-axis coordinate of the j-th normalized query point, represents the y-axis coordinate of the jth normalized query point. The query point can be normalized by dividing the coordinate value by the z-axis coordinate value, that is:

[0176] ;

[0177] in, represents the coordinates of the jth normalized query point, represents the x-axis coordinate of the j-th query point, represents the y-axis coordinate of the j-th query point, Represents the z-axis coordinate of the j-th query point.

[0178] After calculating the pixel coordinates of the i-th 2D point projected on the camera view by the j-th query point, that is, Then, it is substituted into the attention network transformation formula in the above embodiment to obtain the output result of the attention network mapping the two-dimensional point to the three-dimensional point.

[0179] By applying the method of the embodiment of the present application, the three-dimensional image feature information can be obtained by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use fisheye camera instead of laser radar, thereby reducing the obstacle detection cost, and further reducing the production cost of vehicles with intelligent assisted driving function. And by performing distortion mapping on the two-dimensional image features to achieve image correction, compared with other cameras, the embodiment of the present application can take advantage of the wide shooting angle of the fisheye camera, while achieving full-view coverage of the vehicle, reducing the number of cameras, thereby reducing the amount of computer calculations, improving the real-time performance of obstacle detection, effectively improving the obstacle perception capability of the autonomous driving system, and improving the safety of the autonomous driving system.

[0180] In a possible implementation, the feature map upsampling network includes multiple three-dimensional deconvolution networks and a probability prediction module;

[0181] Then step 204 can be implemented by the following steps:

[0182] Step A: Upsample the first three-dimensional image feature using the first three-dimensional deconvolution network to obtain a first upsampled image feature.

[0183] Step B: Fuse the i-1th upsampled image feature with the i-th three-dimensional image feature, and input the fused feature into the i-th three-dimensional deconvolution network for upsampling to obtain the i-th upsampled image feature.

[0184] Wherein, 1<i≤N, i is an integer, and N is the total number of three-dimensional deconvolution networks.

[0185] Step C: Use the probability prediction module to perform probability prediction on the Nth up-sampled image feature to obtain the placeholder category of each placeholder grid of the three-dimensional scene.

[0186] The three-dimensional obstacle detection result includes the occupancy category of each occupancy grid, and the occupancy category of the occupancy grid is the maximum probability category of the voxels in the occupancy grid.

[0187] In practical applications, the first 3D image feature is usually the one with the smallest resolution. It is upsampled and enlarged to the same resolution as the adjacent 3D image features through the first 3D deconvolution network, so that the i-1th upsampled image feature can be fused with the i-th 3D image feature. And so on, until the fusion reaches the same resolution as the last 3D image feature, and then the fused 3D image features are upsampled and probability predicted.

[0188] In an example, Figure 7 As shown, in Figure 4 In the multi-view feature aggregation projection network, three 3D image features are obtained, with resolutions of 64x64x256, 128x128x128, and 256x256x64, respectively. In step A, the 3D image feature with a resolution of 64x64x256 is first upsampled through the first 3D deconvolution network to obtain the first upsampled image feature with a resolution of 128x128x128, and then the first upsampled image feature is fused with the 3D image feature with a resolution of 128x128x128 to obtain a fused image feature with a resolution of 128x128x128.

[0189] Similarly, the fused image features are upsampled through the second 3D deconvolution network to obtain the second upsampled image features with a resolution of 256x256x64, and then the second upsampled image features are fused with the 3D image features with a resolution of 256x256x64 to obtain a fused image feature with a resolution of 256x256x64. Finally, the fused image features are upsampled through the third 3D deconvolution network to obtain the third upsampled image features of 512x512x3, and the upsampled image features are probabilistically predicted to obtain the occupancy grid prediction results.

[0190] Among them, probability prediction can be performed through occupancy grid networks. In an example, the Nth upsampled image feature is divided into multiple occupancy grids. For each occupancy grid, the probability category of each voxel of the Nth upsampled image feature in the occupancy grid is predicted. Among them, voxel refers to the smallest element in the upsampled image feature, and the probability category refers to the classification result and probability of the voxel. For example, the probability category of a certain voxel is a voxel with an 80% probability of being a roadblock. The probability category of each voxel in the occupancy grid is counted, and the category with the largest probability is selected as the occupancy category of the occupancy grid. Among them, the occupancy category represents the occupancy status and category information of the occupancy grid, such as the occupancy grid is not occupied, or the occupancy grid is occupied, and the category is pedestrian. The three-dimensional obstacle detection result includes the occupancy category of each occupancy grid of the Nth upsampled image feature, so that the obstacle detection result around the vehicle can be represented.

[0191] By applying the method of the embodiment of the present application and performing probability prediction by fusing the semantic information of multiple three-dimensional image features, the prediction accuracy of the occupancy grid network can be improved, thereby improving the recognition accuracy of obstacles and improving the vehicle's environmental perception.

[0192] In a second aspect of an embodiment of the present application, a method for training an obstacle detection model is provided, the method comprising: Figure 8 Steps shown:

[0193] Step 801: Obtain sample fisheye image data and a three-dimensional obstacle calibration truth value of a three-dimensional scene corresponding to the sample fisheye image data.

[0194] Step 802: Input the sample fisheye image data into the feature extraction network of the obstacle detection model for feature extraction to obtain two-dimensional sample image features of multiple sizes.

[0195] Step 803: Input each 2D sample image feature into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and 3D mapping to obtain 3D sample image features of multiple sizes.

[0196] Step 804: Input the features of each three-dimensional sample image into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle prediction result.

[0197] Step 805: Calculate the loss of the obstacle detection model according to the three-dimensional obstacle calibration true value and the three-dimensional obstacle prediction result, and adjust the obstacle detection model according to the loss.

[0198] The true value of the three-dimensional obstacle calibration of the sample fisheye image data refers to the location and category of the obstacles in the actual environment corresponding to the sample fisheye image data. When adjusting the obstacle detection model according to the loss, a loss threshold can be set in advance, and the obstacle detection model is adjusted and re-predicted and the loss value is calculated. When the loss value is less than the loss threshold, the obstacle detection model is considered to have completed training.

[0199] By applying the method of the embodiment of the present application, the obstacle detection model can be trained in advance, so that the trained obstacle detection model can be used to perform probability prediction on multi-view fisheye image data, thereby improving the accuracy of obstacle detection.

[0200] In a third aspect of the embodiments of the present application, an obstacle detection device is provided, the device comprising: Fig. 9 The structure shown:

[0201] The image acquisition module 901 is used to acquire multi-view fisheye image data.

[0202] The feature extraction module 902 is used to input the multi-view fisheye image data into the feature extraction network of the pre-trained obstacle detection model to perform feature extraction to obtain two-dimensional image features of multiple sizes.

[0203] The image mapping module 903 is used to input each 2D image feature into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and 3D mapping to obtain 3D image features of multiple sizes.

[0204] The probability prediction module 904 is used to input each 3D image feature into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a 3D obstacle detection result.

[0205] In a possible implementation, the feature extraction network includes a backbone network and a feature pyramid network;

[0206] Feature extraction module, including:

[0207] The downsampling submodule is specifically used to downsample the multi-view fisheye image data using the backbone network to obtain downsampled image features of multiple sizes.

[0208] The down-sampling feature fusion submodule is specifically used to use the feature pyramid network to fuse the down-sampled image features to obtain two-dimensional image features of multiple sizes.

[0209] In one possible implementation, the multi-view feature aggregation projection network includes a plurality of two-dimensional-three-dimensional mapping networks;

[0210] Image map module, including:

[0211] The image mapping submodule is specifically used to use each two-dimensional-three-dimensional mapping network to perform distortion mapping and three-dimensional mapping on the two-dimensional image features of the corresponding size to obtain three-dimensional image features of multiple sizes, wherein each two-dimensional-three-dimensional mapping network corresponds to one size.

[0212] In a possible implementation, the two-dimensional-three-dimensional mapping network includes: an attention network, a first residual connection and normalization network, a feedforward neural network, and a second residual connection and normalization network;

[0213] Image mapping submodule, including:

[0214] The image mapping unit is specifically used to use the attention network to perform distortion mapping and three-dimensional mapping on the input two-dimensional image features to obtain the first image features.

[0215] The first normalization unit is specifically used to perform residual connection and normalization processing on the input first image feature by using the first residual connection and normalization network to obtain the second image feature.

[0216] The feature enhancement unit is specifically used to use a feedforward neural network to perform feature enhancement on the input second image feature to obtain a third image feature.

[0217] The second normalization unit is specifically used to perform residual connection and normalization processing on the input third image features by using the second residual connection and normalization network to obtain three-dimensional image features.

[0218] In a possible implementation, the image mapping unit includes:

[0219] The image transformation subunit is specifically used to use the attention network to transform: , get the first image feature:

[0220] in, represents the first image feature, represents the query point of the attention network, represents the two-dimensional image features, represents the set of query points of the attention network; Represents the mapping operation of the deformable convolutional attention module of the attention network; A projection function representing a three-dimensional image feature to a two-dimensional image feature; the multi-view fisheye image data includes fisheye images of multiple viewpoints; Represents the two-dimensional image features of the fisheye image at the i-th viewing angle.

[0221] In a possible implementation, the mapping relationship between the query point and the two-dimensional point is:

[0222] ;

[0223] in, Represents the 2D pixel coordinates of the j-th query point of the attention network mapped to the 2D image features of the fisheye image of the i-th view; , , , Indicates the camera internal parameters; It represents the x-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system; It represents the y-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system;

[0224] = ; = ;

[0225] in, Represents the radial distance from the pixel of the distorted two-dimensional image feature to the center of the circle, represents the incident angle of the jth query point;

[0226] ;

[0227] in, is the distortion parameter, r represents the polar coordinates of the normalized query point;

[0228] ;

[0229] in, represents the x-axis coordinate of the j-th normalized query point, represents the y-axis coordinate of the j-th normalized query point,

[0230] ;

[0231] in, represents the coordinates of the jth normalized query point, represents the x-axis coordinate of the j-th query point, represents the y-axis coordinate of the j-th query point, Represents the z-axis coordinate of the j-th query point.

[0232] In a possible implementation, the feature map upsampling network includes multiple three-dimensional deconvolution networks and a probability prediction module;

[0233] Obstacle detection module, including:

[0234] An upsampling submodule, specifically used to upsample the first three-dimensional image feature using the first three-dimensional deconvolution network to obtain a first upsampled image feature;

[0235] The upsampling feature fusion submodule is specifically used to fuse the i-1th upsampled image feature with the ith three-dimensional image feature, and input the fused feature into the ith three-dimensional deconvolution network for upsampling to obtain the ith upsampled image feature, where 1<i≤N, i is an integer, and N is the total number of three-dimensional deconvolution networks;

[0236] The probability prediction submodule is specifically used to use the probability prediction module to perform probability prediction on the Nth upsampled image feature to obtain the occupancy category of each occupancy grid of the three-dimensional scene, wherein the three-dimensional obstacle detection result includes the occupancy category of each occupancy grid, and the occupancy category of the placeholder grid is the maximum probability category of the voxel in the placeholder grid.

[0237] The device using the embodiment of the present application can obtain three-dimensional image feature information by performing distortion mapping and three-dimensional mapping on the collected two-dimensional fisheye image data. Compared with the method of directly obtaining three-dimensional image feature information by laser radar, the embodiment of the present application can use fisheye camera instead of laser radar, thereby reducing the obstacle detection cost, and further reducing the production cost of vehicles with intelligent assisted driving function. In addition, due to the image distortion effect of the fisheye camera, the device of the embodiment of the present application can achieve image correction by performing distortion mapping on the two-dimensional image features, so that compared with other cameras, the embodiment of the present application can utilize the wide shooting angle of the fisheye camera to achieve full-view coverage of the vehicle while reducing the number of cameras, thereby reducing the amount of computer calculations, improving the real-time performance of obstacle detection, effectively improving the obstacle perception capability of the autonomous driving system, and improving the safety of the autonomous driving system.

[0238] In a fourth aspect of an embodiment of the present application, an obstacle detection model training device is provided, the device comprising: Fig.10 The structure shown:

[0239] The sample acquisition module 1001 is used to acquire sample fisheye image data and a three-dimensional obstacle calibration true value of a three-dimensional scene corresponding to the sample fisheye image data.

[0240] The sample feature extraction module 1002 is used to input the sample fisheye image data into the feature extraction network of the obstacle detection model for feature extraction to obtain two-dimensional sample image features of multiple sizes.

[0241] The sample image mapping module 1003 is used to input each two-dimensional sample image feature into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and three-dimensional mapping to obtain three-dimensional sample image features of multiple sizes.

[0242] The sample image prediction module 1004 is used to input the features of each 3D sample image into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a 3D obstacle prediction result.

[0243] The loss calculation module 1005 is used to calculate the loss of the obstacle detection model according to the three-dimensional obstacle calibration true value and the three-dimensional obstacle prediction result, and adjust the obstacle detection model according to the loss.

[0244] By using the device of the embodiment of the present application, the obstacle detection model can be trained in advance, so that the trained obstacle detection model can be used to perform probability prediction on multi-view fisheye image data, thereby improving the accuracy of obstacle detection.

[0245] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0246] Memory 1101, used for storing computer programs;

[0247] The processor 1102 is used to implement the following steps when executing the program stored in the memory 1101:

[0248] Acquire multi-view fisheye image data;

[0249] The multi-view fisheye image data is input into the feature extraction network of the pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes;

[0250] Input each 2D image feature into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and 3D mapping to obtain 3D image features of multiple sizes;

[0251] The features of each 3D image are input into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain the 3D obstacle detection result.

[0252] or,

[0253] Obtaining sample fisheye image data and a three-dimensional obstacle calibration truth value of a three-dimensional scene corresponding to the sample fisheye image data;

[0254] The sample fisheye image data is input into the feature extraction network of the obstacle detection model for feature extraction to obtain two-dimensional sample image features of multiple sizes;

[0255] Input each 2D sample image feature into the multi-view feature aggregation projection network of the obstacle detection model for distortion mapping and 3D mapping to obtain 3D sample image features of multiple sizes;

[0256] Input the features of each 3D sample image into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain the 3D obstacle prediction result;

[0257] According to the true value of the 3D obstacle calibration and the 3D obstacle prediction result, the loss of the obstacle detection model is calculated, and the obstacle detection model is adjusted according to the loss.

[0258] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0259] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0260] The communication interface is used for communication between the above electronic device and other devices.

[0261] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0262] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0263] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned obstacle detection methods or obstacle detection model training methods are implemented.

[0264] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any obstacle detection method or obstacle detection model training method in the above embodiments.

[0265] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0266] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0267] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0268] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An obstacle detection method, characterized in that: The method comprises: Acquire multi-view fisheye image data; Inputting the multi-view fisheye image data into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes; Using each 2D-3D mapping network of the obstacle detection model to perform distortion mapping and 3D mapping on the 2D image features of the corresponding size, respectively, to obtain 3D image features of multiple sizes; wherein each 2D-3D mapping network corresponds to one size, and the 2D-3D mapping network includes: an attention network, a first residual connection and normalization network, a feedforward neural network, and a second residual connection and normalization network; Inputting each of the three-dimensional image features into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle detection result; The two-dimensional-three-dimensional mapping networks of the obstacle detection model are used to perform distortion mapping and three-dimensional mapping on the two-dimensional image features of the corresponding sizes to obtain three-dimensional image features of multiple sizes, including: Using the attention network, by transforming: , performing distortion mapping and three-dimensional mapping on the input two-dimensional image feature to obtain a first image feature; in, represents the first image feature, represents a query point of the attention network, wherein the query point is a hit point selected from three-dimensional points that do not contain the contour of the vehicle; represents the two-dimensional image features, represents the set of query points of the attention network; Representing the mapping operation of the deformable convolutional attention module of the attention network; represents a mapping function for mapping the j-th query point of the attention network to the two-dimensional pixel coordinates in the two-dimensional image features of the fisheye image of the i-th perspective; the multi-view fisheye image data includes fisheye images of multiple perspectives, Represents the two-dimensional image features of the fisheye image of the i-th viewing angle; Using the first residual connection and normalization network to perform residual connection and normalization processing on the input first image feature to obtain a second image feature; Using the feedforward neural network to perform feature enhancement on the input second image feature to obtain a third image feature; The second residual connection and normalization network is used to perform residual connection and normalization processing on the input third image feature to obtain a three-dimensional image feature.

2. The method according to claim 1, characterized in that The feature extraction network includes a backbone network and a feature pyramid network; The multi-view fisheye image data is input into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes, including: Down-sampling the multi-view fisheye image data using the backbone network to obtain down-sampled image features of multiple sizes; The feature pyramid network is used to perform feature fusion on the downsampled image features to obtain two-dimensional image features of multiple sizes.

3. The method according to claim 1, characterized in that: The mapping relationship between the query point and the two-dimensional point is: ; in, Represents the two-dimensional pixel coordinates of the j-th query point of the attention network mapped to the two-dimensional image features of the fisheye image of the i-th perspective; , , , Represents the camera internal parameters; represents the x-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system; It represents the y-axis coordinate value of the j-th query point after normalization and transfer to the plane coordinate system; = ； = ; in, Represents the radial distance from the pixel of the distorted two-dimensional image feature to the center of the circle, represents the incident angle of the jth query point; ; in, is the distortion parameter, r represents the polar coordinates of the normalized query point; ; in, represents the x-axis coordinate of the j-th normalized query point, represents the y-axis coordinate of the j-th normalized query point, ; in, represents the coordinates of the jth normalized query point, represents the x-axis coordinate of the j-th query point, represents the y-axis coordinate of the j-th query point, Represents the z-axis coordinate of the j-th query point.

4. The method according to claim 1, characterized in that The feature map upsampling network includes multiple three-dimensional deconvolution networks and a probability prediction module; The step of inputting each of the three-dimensional image features into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle detection result includes: Upsampling the first three-dimensional image feature using the first three-dimensional deconvolution network to obtain a first upsampled image feature; Fusing the i-1th upsampled image feature with the ith three-dimensional image feature, and inputting the fused feature into the ith three-dimensional deconvolution network for upsampling to obtain the ith upsampled image feature, where 1<i≤N, i is an integer, and N is the total number of three-dimensional deconvolution networks; The probability prediction module is used to perform probability prediction on the Nth upsampled image feature to obtain the placeholder category of each placeholder grid of the three-dimensional scene, wherein the three-dimensional obstacle detection result includes the placeholder category of each placeholder grid, and the placeholder category of the placeholder grid is the maximum probability category of the voxel in the placeholder grid.

5. A method for training an obstacle detection model, characterized in that: The method comprises: Acquire sample fisheye image data and a three-dimensional obstacle calibration true value of the three-dimensional scene corresponding to the sample fisheye image data; Inputting the sample fisheye image data into a feature extraction network of an obstacle detection model for feature extraction to obtain two-dimensional sample image features of multiple sizes; Using each 2D-3D mapping network of the obstacle detection model to perform distortion mapping and 3D mapping on the 2D sample image features of the corresponding size, respectively, to obtain 3D sample image features of multiple sizes; wherein each of the 2D-3D mapping networks corresponds to one size, and the 2D-3D mapping network includes: an attention network, a first residual connection and normalization network, a feedforward neural network, and a second residual connection and normalization network; Inputting each of the three-dimensional sample image features into a feature map upsampling network of an obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle prediction result; Calculating a loss of an obstacle detection model according to the three-dimensional obstacle calibration true value and the three-dimensional obstacle prediction result, and adjusting the obstacle detection model according to the loss; The method of using the 2D-3D mapping networks of the obstacle detection model to perform distortion mapping and 3D mapping on the 2D sample image features of the corresponding sizes to obtain 3D sample image features of multiple sizes includes: Using the attention network, by transforming: , performing distortion mapping and three-dimensional mapping on the input two-dimensional sample image feature to obtain a first sample image feature; in, represents the first image feature, represents a query point of the attention network, wherein the query point is a hit point selected from three-dimensional points that do not contain the contour of the vehicle; represents the two-dimensional image features, represents the set of query points of the attention network; Representing the mapping operation of the deformable convolutional attention module of the attention network; represents a mapping function for mapping the j-th query point of the attention network to the two-dimensional pixel coordinates in the two-dimensional image features of the fisheye image of the i-th perspective; the multi-view fisheye image data includes fisheye images of multiple perspectives, Represents the two-dimensional image features of the fisheye image of the i-th viewing angle; Using the first residual connection and normalization network to perform residual connection and normalization processing on the input first image feature to obtain a second image feature; Using the feedforward neural network to perform feature enhancement on the input second image feature to obtain a third image feature; The second residual connection and normalization network is used to perform residual connection and normalization processing on the input third image feature to obtain a three-dimensional image feature.

6. An obstacle detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring multi-view fisheye image data; A feature extraction module, used for inputting the multi-view fisheye image data into a feature extraction network of a pre-trained obstacle detection model for feature extraction to obtain two-dimensional image features of multiple sizes; An image mapping submodule, for performing distortion mapping and three-dimensional mapping on two-dimensional image features of corresponding sizes using each two-dimensional-three-dimensional mapping network of the obstacle detection model to obtain three-dimensional image features of multiple sizes, wherein each two-dimensional-three-dimensional mapping network corresponds to one size; An obstacle detection module, used for inputting each of the three-dimensional image features into the feature map upsampling network of the obstacle detection model for upsampling and probability prediction to obtain a three-dimensional obstacle detection result; The two-dimensional-three-dimensional mapping network includes: an attention network, a first residual connection and normalization network, a feedforward neural network, and a second residual connection and normalization network; The image mapping submodule comprises: The image transformation subunit is specifically used to utilize the attention network to transform: , obtain the first image feature; in, represents the first image feature, represents the query point of the attention network, represents the two-dimensional image features, A set of query points representing the attention network; the query points are hit points selected from three-dimensional points that do not contain the vehicle contour; represents a mapping operation of the deformable convolutional attention module of the attention network; P represents a mapping function for mapping the j-th query point of the attention network to the two-dimensional pixel coordinates in the two-dimensional image features of the fisheye image of the i-th perspective; the multi-view fisheye image data includes fisheye images of multiple perspectives, Represents the two-dimensional image features of the fisheye image of the i-th viewing angle; a first normalization unit, specifically configured to perform residual connection and normalization processing on the input first image feature by using the first residual connection and normalization network to obtain a second image feature; A feature enhancement unit, specifically used to perform feature enhancement on the input second image feature using the feedforward neural network to obtain a third image feature; The second normalization unit is specifically used to use the second residual connection and normalization network to perform residual connection and normalization processing on the input third image feature to obtain a three-dimensional image feature.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-5 when executing a program stored in a memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images