Method and system for obstacle and occupancy grid prediction using fisheye camera

Abstract

The invention relates to a method and a system for obstacle and occupancy grid prediction by using a fisheye camera. According to the technical scheme, firstly, multiple frames of images are received from multiple fisheye cameras, image features and depth probability distribution are extracted from the multiple frames of images through Backbone, and BEV view angle conversion is carried out on the image features to generate BEV features. Then, based on the generated BEV features, an obstacle prediction task and an occupied grid prediction task are completed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision perception, and in particular to a method and system for predicting obstacles and occupancy grids using a fisheye camera. Background Art

[0002] For general obstacles, the industry obtains geometric information (depth or height) to identify general obstacles. This type of method is essentially a three-dimensional reconstruction of the environment to obtain information related to the location of the obstacle, which is more general than the target detection method that relies heavily on semantic information. The 3D target box predicted by the current 3D target detection task is not enough to describe general obstacles, irregular obstacles, suspended obstacles, etc. Therefore, the object is divided into voxels for expression, and the occupancy, category and other information of each voxel are predicted in the 3D BEV (3D bird's-eye view) voxel space. The Lidar point cloud segmentation task is very similar to this, which is dense and computationally intensive, which is not conducive to mass production and deployment. It can also be understood as an extended task of visual semantic segmentation in 3D voxel space. Vision-based solutions are lower in cost and suitable for mass production and deployment of autonomous driving. The Occupancy Network has expanded this field.

[0003] In 2019, the occupancy network was proposed as a new network learning-based 3D reconstruction method, implicitly representing the 3D surface as a continuous decision boundary of a deep neural network classifier. In 2020, a dense front view image to BEV space conversion network was proposed to map image features to the BEV plane to predict an accurate semantic occupancy grid. The core idea of ​​the occupancy network is to learn a 3D space representation called an "occupancy grid" to represent the volume of a 3D object. The network learns to infer the probability of whether each voxel is occupied by an object by inputting 2D projection images from multiple perspectives, thereby reconstructing the 3D shape of the object.

[0004] However, most existing technologies are panoramic camera solutions for high-speed scenes. Fisheye cameras are a lower-cost solution (especially suitable for low-speed scenes) with the advantages of larger FOV and wider viewing angle. However, depth estimation has problems such as local inconsistency, sparse depth at distant vanishing points, difficulty in achieving cross-view consistency, inability to handle occlusion, and inability to identify dynamic and static objects.

[0005] Therefore, there is a need for methods and systems that can improve upon the deficiencies in the prior art. Summary of the invention

[0006] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In view of the above-mentioned problems in the prior art (local inconsistency, sparse depth of distant vanishing points, difficulty in achieving consistency across viewing angles, inability to handle occlusion and identify the dynamic and static state of objects, etc.), the present invention proposes an occupancy (and obstacle) prediction framework under fisheye distortion, which can effectively solve such problems with the help of occupancy grid prediction and occupancy flow.

[0008] In the technical solution of the present invention, multiple frames of images are first received from multiple fisheye cameras, and image features and depth probability distribution are extracted from the multiple frames of images through Backbone, and the image features are converted into BEV (bird's eye view) perspective to generate BEV features. Subsequently, obstacle prediction tasks and occupancy grid prediction tasks are completed based on the generated BEV features.

[0009] Specifically, in one embodiment of the present invention, a method for using a fisheye camera to predict obstacles and occupancy grids is disclosed, the method comprising:

[0010] Receive multiple frames of images from multiple fisheye cameras;

[0011] Extracting image features and depth probability distribution from the multiple frames of images;

[0012] Perform BEV perspective conversion and voxelization on these image features to generate BEV features;

[0013] Predicting a 3D bounding box of an obstacle on the BEV based on the BEV feature and outputting the position, heading angle, size and speed information of the obstacle according to the 3D bounding box;

[0014] generating an occupancy grid prediction result corresponding to the current frame based on the BEV feature; and

[0015] The occupancy grid prediction results corresponding to the past frames before the current frame are superimposed on the current frame to densify the occupancy grid prediction results corresponding to the current frame.

[0016] In one embodiment of the present invention, the image features are image features of different scales extracted by Backbone, and the method further includes performing 2D BBox supervision on the image features of different scales to increase the 2D image feature constraints and performing point cloud true value depth supervision on the depth probability distribution to increase the depth feature constraints.

[0017] In one embodiment of the present invention, performing BEV perspective conversion and voxelization on the image features further comprises:

[0018] The image features are dedistorted by the internal parameters of multiple fisheye cameras and the fisheye distortion coefficients through the Lift-Splat-Shoot method, and the outer product operation is performed with the depth probability distribution to generate the cone features;

[0019] Obtaining a frustum voxel index in the BEV space according to the external parameters of the multi-channel fisheye camera relative to the vehicle coordinate system to perform BEV perspective conversion table lookup preprocessing; and

[0020] The view cone feature and the depth probability distribution are accessed according to the view cone voxel index to complete the BEV table lookup process from 2D features to 3D features.

[0021] In one embodiment of the present invention, generating the BEV signature further comprises:

[0022] Extracting the image features after BEV perspective conversion as BEV features through a BEV encoder network; and

[0023] These BEV features are aggregated to construct a three-dimensional spatial BEV feature with height.

[0024] In one embodiment of the present invention, the method may optionally further comprise:

[0025] constructing a similarity matrix of grid features between multiple frames corresponding to the past frames and the current frame through a temporal encoder; and

[0026] This similarity matrix is ​​learned to complete the end-to-end occupancy grid tracking task.

[0027] In another embodiment of the present invention, a system for obstacle and occupancy grid prediction using a fisheye camera is disclosed, the system comprising:

[0028] An image feature acquisition module, which is configured to receive multiple frames of images from multiple fisheye cameras and extract image features and depth probability distribution from the multiple frames of images;

[0029] A BEV perspective conversion module configured to perform BEV perspective conversion and voxelization on the image features to generate BEV features;

[0030] An obstacle prediction module configured to predict a 3D bounding box of an obstacle on the BEV based on the BEV feature and output the position, heading angle, size and speed information of the obstacle according to the 3D bounding box;

[0031] Occupancy grid prediction module, which is configured to:

[0032] An occupation grid prediction result corresponding to a current frame is generated based on the BEV feature; and an occupation grid prediction result corresponding to a past frame before the current frame is superimposed on the current frame to densify the occupation grid prediction result corresponding to the current frame.

[0033] In one embodiment of the present invention, the image features are image features of different scales extracted by Backbone, and the image feature acquisition module is further configured to perform 2DBBox supervision on the image features of different scales to increase the 2D image feature constraints and to perform point cloud truth depth supervision on the depth probability distribution to increase the depth feature constraints.

[0034] In one embodiment of the present invention, the BEV perspective conversion module is further configured to perform BEV perspective conversion and voxelization on the image features by:

[0035] The image features are dedistorted by the internal parameters of multiple fisheye cameras and the fisheye distortion coefficients through the Lift-Splat-Shoot method, and the outer product operation is performed with the depth probability distribution to generate the cone features;

[0036] Obtaining a frustum voxel index in the BEV space according to the external parameters of the multi-channel fisheye camera relative to the vehicle coordinate system to perform BEV perspective conversion table lookup preprocessing; and

[0037] The view cone feature and the depth probability distribution are accessed according to the view cone voxel index to complete the BEV table lookup process from 2D features to 3D features.

[0038] In one embodiment of the present invention, the BEV perspective conversion module is further configured to generate BEV features by:

[0039] Extracting the image features after BEV perspective conversion as BEV features through a BEV encoder network; and

[0040] These BEV features are aggregated to construct a three-dimensional spatial BEV feature with height.

[0041] In one embodiment of the present invention, the occupancy grid prediction module may optionally be further configured to:

[0042] constructing a similarity matrix of grid features between multiple frames corresponding to the past frames and the current frame through a temporal encoder; and

[0043] This similarity matrix is ​​learned to complete the end-to-end occupancy grid tracking task.

[0044] In yet another embodiment of the present invention, a computer-readable storage medium is disclosed, which stores instructions for using a fisheye camera to perform obstacle and occupancy grid prediction, including:

[0045] Instructions for receiving multiple frames of images from multiple fisheye cameras;

[0046] Instructions for extracting image and depth probability distribution features from the multiple frames of images;

[0047] Instructions for performing BEV perspective conversion and voxelization on these image features to generate BEV features;

[0048] Instructions for predicting a 3D bounding box of an obstacle on the BEV based on the BEV feature and outputting a position, heading angle, length, width, height, and speed information of the obstacle according to the 3D bounding box;

[0049] Instructions for generating an occupancy grid prediction result corresponding to a current frame based on the BEV feature; and

[0050] Instructions for superimposing occupancy grid prediction results corresponding to past frames before the current frame onto the current frame to densify the occupancy grid prediction results corresponding to the current frame.

[0051] After studying the description of the specific exemplary embodiments of the present invention below in conjunction with the accompanying drawings, other aspects, features and embodiments of the present invention will be apparent to those of ordinary skill in the art. Although features of the present invention may be discussed below with respect to certain embodiments and accompanying drawings, all embodiments of the present invention may include one or more of the advantageous features discussed herein. In other words, although one or more embodiments may be discussed to have certain advantageous features, one or more of such features may also be used according to the various embodiments of the present invention discussed herein. In a similar manner, although exemplary embodiments may be discussed below as device, system or method embodiments, it should be appreciated that such exemplary embodiments may be implemented in various devices, systems, and methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to understand in detail the manner in which the above-stated features of the present disclosure are used, a more particular description of the content briefly summarized above may be made with reference to various aspects, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only certain typical aspects of the present disclosure and are not to be considered limiting of its scope, as the description may admit to other equally effective aspects.

[0053] Figure 1 A schematic block diagram of a system for performing obstacle and occupancy grid prediction using a fisheye camera according to an embodiment of the present disclosure is shown.

[0054] Figure 2 A schematic diagram of a surround view multi-task network framework according to an embodiment of the present disclosure and a more detailed schematic block diagram of the BEV view conversion and occupancy grid prediction tasks in the framework are shown.

[0055] Figure 3 A flowchart of a method for using a fisheye camera to perform obstacle and occupancy grid prediction according to an embodiment of the present disclosure is shown.

[0056] Figure 4 A schematic diagram of a computer-readable storage medium configured to store computer-executable instructions according to one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0057] The various embodiments will be described in more detail below with reference to the accompanying drawings that form part of the present invention and illustrate various specific exemplary embodiments. However, the various embodiments can be implemented in many different forms and should not be interpreted as limiting the various embodiments set forth herein; on the contrary, these embodiments are provided to make the present disclosure thorough and complete, and to fully convey the scope of these embodiments to those of ordinary skill in the art. The various embodiments may be implemented in accordance with methods, systems, or devices. Therefore, these embodiments may be implemented in hardware, in full software, or in a combination of software and hardware. Therefore, the following specific embodiments are not limiting.

[0058] The steps in each flow chart may be performed by hardware (e.g., processor, engine, memory, circuit), software (e.g., operating system, application, driver, machine / processor executable instructions), or a combination thereof. As will be appreciated by those skilled in the art, the methods involved in each embodiment may include more or fewer steps than shown.

[0059] In response to the problems in the prior art, the present invention proposes a multi-task framework for 4D BEV target detection of regular obstacles and occupancy prediction of irregular obstacles. The multi-task framework is applicable to a variety of driving scenarios, especially to low-speed driving scenarios. In the multi-task framework, 4D BEV target detection is shared with the occupancy prediction network, the target detection task effectively improves the semantic classification of the occupancy task, and the auxiliary loss of occupancy effectively improves the position accuracy of the target detection task. The present invention also proposes a BEV perspective conversion for the occupancy network of a surround-view fisheye camera, adds distortion coefficients to generate projection relationships offline, and accelerates reasoning and saves computing resources by looking up tables during reasoning. As those skilled in the art will appreciate, the scenarios to which the multi-task framework in the present disclosure is applicable may include, but is not limited to, autonomous driving and assisted driving, and may also be used in any other suitable scenarios.

[0060] The various aspects of the present disclosure will be described in more detail and more comprehensively through block diagrams and method flow charts below.

[0061] Figure 1 FIG. 1 is a schematic block diagram of a system 100 for performing obstacle and occupancy grid prediction using a fisheye camera according to an embodiment of the present disclosure. Figure 2 The system 100 will now be described in detail.

[0062] In one embodiment of the present invention, the system 100 may include an image feature acquisition module 102 , a BEV view conversion module 104 , an obstacle prediction module 106 , and an occupancy grid prediction module 108 .

[0063] In one embodiment of the present invention, the image feature acquisition module 102 may be configured to receive multiple frames of images from multiple fisheye cameras and extract image features and depth probability distribution from the multiple frames of images. In this embodiment, the image features may be image features of different scales extracted by Backbone. As can be appreciated by those skilled in the art, in other embodiments of the present invention, any other suitable method other than Backbone may be used to extract image features (and simultaneously extract depth probability distribution) from multiple frames of images, without being limited to Backbone.

[0064] Figure 2 A schematic diagram of a surround view multi-task network framework according to an embodiment of the present disclosure (top figure) and a more detailed schematic block diagram of the BEV view conversion and occupancy grid prediction tasks in the framework (bottom figure) are shown.

[0065] like Figure 2 As shown in the upper and lower figures in , the image feature acquisition module 102 can extract multiple fisheye images (possibly with distortion) corresponding to multiple frames Frame@t-Frame@tn-1 (or t-1) through Backbone.

[0066] In another embodiment of the present invention, the image feature acquisition module 102 may be further configured to perform 2D BBox (Bounding Box) supervision on these image features of different scales to increase 2D image feature constraints to improve the prediction accuracy of occupancy grids and regular obstacles, and to perform point cloud truth depth supervision on the depth probability distribution to increase depth feature constraints.

[0067] In one embodiment of the present invention, the BEV perspective conversion module 104 may be configured to perform BEV perspective conversion and voxelization on the image features to generate BEV features. Figure 2As shown in the figure above, the BEV perspective conversion module 104 can be implemented as a perspective converter. Figure 2 As shown in the figure below, the BEV perspective conversion module 104 can be implemented as a Lift-Splat-Shoot (LSS) perspective converter. In this embodiment, the BEV perspective conversion can be completed for the extracted image features through the LSS perspective converter.

[0068] exist Figure 2 In the exemplary embodiment shown in the figure below, the BEV perspective conversion module 104 can be further configured to dedistort the image features through the internal parameters of the multi-channel fisheye cameras and the fisheye distortion coefficients through the Lift-Splat-Shoot method and perform an outer product operation with the depth probability distribution to generate a cone feature; obtain the cone voxel index in the BEV space according to the external parameters of the multi-channel fisheye cameras relative to the vehicle coordinate system to perform BEV perspective conversion lookup table preprocessing; and access the cone feature and the depth probability distribution according to the cone voxel index to complete the BEV lookup process from 2D features to 3D features, thereby achieving the purpose of saving memory and computing resources and speeding up the reasoning speed during reasoning.

[0069] In another exemplary embodiment of the present invention, Figure 2 As shown in the figure below, the BEV perspective conversion module 104 can be further configured to generate BEV features through the following operations: extracting the image features after BEV perspective conversion as BEV features through a BEV encoder (BEV-Encoder) network; and aggregating these BEV features into three-dimensional space (XYZ) BEV features with height, which are more suitable for occupancy grid (Occupancy) prediction.

[0070] In one embodiment of the present invention, the obstacle prediction module 106 may be configured to predict a 3D bounding box of an obstacle on the BEV based on the generated BEV features and output the position, heading angle, size (e.g., length, width, and height), and speed information of the obstacle according to the 3D bounding box.

[0071] like Figure 2 As shown in the above figure, the obstacle prediction module 106 can be implemented as a 3D detector, and can predict the position, heading angle, size and speed of the obstacle based on the BEV features generated by the view transformer.

[0072] In one embodiment of the present invention, the obstacle prediction module 106 may be configured to generate obstacle prediction results through CenterpointHead. As those skilled in the art will appreciate, this CenterpointHead is merely exemplary and non-restrictive, and obstacle prediction results may be generated in any other suitable manner to predict the position, heading angle, size, and speed of the obstacle.

[0073] In one embodiment of the present invention, the occupancy grid prediction module 108 may be configured to generate an occupancy grid prediction result corresponding to the current frame based on the BEV feature; and superimpose the occupancy grid prediction result corresponding to the past frame before the current frame (predicted in the same way as the current frame) onto the current frame (e.g. Figure 2 In this embodiment, the occupancy grid prediction task performed by the occupancy grid prediction module 108 can also assist the obstacle prediction task of the obstacle prediction module 106 by providing auxiliary Loss for obstacle prediction (to be described in detail below).

[0074] like Figure 2 As shown in the figure above, the occupancy grid prediction module 108 can be implemented as an occupancy grid predictor, and can predict semantically occupied grids and occupancy flows based on the BEV features generated by the view transformer. In other words, in addition to providing dense occupancy prediction results, the occupancy grid prediction module 108 can also be configured to provide semantic information associated with the occupancy prediction (i.e., predicting semantically occupied grids, such as Figure 2 As shown in the figure above, it predicts the occupancy flow, where the occupancy flow prediction provides the occupancy positions of future frames.

[0075] In this embodiment, the occupancy grid prediction module 108 can be further configured to perform dynamic and static separation of the occupancy grid through multi-frame superposition and based on the predicted voxel velocity, so as to densify the occupancy prediction results and occupancy flow prediction on bev of the current frame, thereby optimizing the position prediction of irregular obstacles such as gates, traffic lights, road signs, road posts, suspended obstacles, etc., and providing the necessary semantic and position information for obstacle avoidance planning other than regular obstacles.

[0076] In one embodiment of the present invention, the occupancy grid prediction module 108 may be configured to generate an occupancy grid prediction result through OccupancyHead. As those skilled in the art will appreciate, this OccupancyHead is merely exemplary and not restrictive, and the occupancy grid prediction result may also be generated in any other suitable manner to predict the occupancy of the BEV space and the corresponding category and speed. At the same time, the dense voxel occupancy prediction result and semantic information provided by OccupancyHead provide auxiliary Loss for obstacle prediction, which may include CrossEntropy Loss for multi-category semantic segmentation and Lovas Softmax Loss for enhanced category distinction (introducing additional orthogonality constraints to make representations of different categories more dispersed in the feature space), and may also include L1Loss of Occupancy Flow.

[0077] In another embodiment of the present invention, Figure 2 As shown in the figure below, the occupancy grid prediction module 108 can be further configured to construct a similarity matrix of multi-frame grid features corresponding to the past frames and the current frame through a temporal encoder; and learn the similarity matrix through an occupancy grid decoder to complete an end-to-end tracking task, which outputs a Tracking ID for each voxel.

[0078] In another embodiment of the present invention, a complete vehicle-side network reasoning and pre- and post-processing process may be designed for the system 100 so that it can be deployed on the vehicle side and meet real-time requirements.

[0079] Figure 3 A flowchart of a method 300 for performing obstacle and occupancy grid prediction using a fisheye camera according to an embodiment of the present disclosure is shown.

[0080] The method 300 starts at step 302 , where a plurality of frames of images are received from a plurality of fisheye cameras.

[0081] Next, the method 300 proceeds to step 304, extracting image features and depth probability distribution from the multiple frames of images. In one embodiment of the present invention, the image features are image features of different scales extracted by Backbone, and the method 300 may also include performing 2DBBox supervision on the image features of different scales to increase 2D image feature constraints and performing point cloud ground truth depth supervision on the depth probability distribution to increase depth feature constraints.

[0082] Subsequently, the method 300 proceeds to step 306, performing BEV perspective conversion and voxelization on the image features to generate BEV features. In one embodiment of the present invention, performing BEV perspective conversion and voxelization on the image features further includes: dedistorting the image features through the internal parameters of the multi-channel fisheye camera and the fisheye distortion coefficient by the Lift-Splat-Shoot method and performing an outer product operation with the depth probability distribution to generate a cone feature; obtaining the cone voxel index in the BEV space according to the external parameters of the multi-channel fisheye camera relative to the vehicle coordinate system to perform BEV perspective conversion lookup table preprocessing; and accessing the cone feature and the depth probability distribution according to the cone voxel index to complete the BEV lookup process from 2D features to 3D features. . In one embodiment of the present invention, generating BEV features further includes: extracting the image features after BEV perspective conversion into BEV features through a BEV encoder network; and aggregating these BEV features to construct a three-dimensional space BEV feature with height.

[0083] Then, the method 300 proceeds to step 308 , predicting a 3D bounding box of the obstacle on the BEV based on the BEV feature and outputting the position, heading angle, length, width, height, and speed information of the obstacle according to the 3D bounding box.

[0084] Next, the method 300 proceeds to step 310 to generate an occupancy grid prediction result corresponding to the current frame based on the BEV feature.

[0085] Finally, the method 300 proceeds to step 312 to superimpose the occupancy grid prediction results corresponding to the past frames before the current frame onto the current frame to densify the occupancy grid prediction results corresponding to the current frame.

[0086] Method 300 ends after step 312. In another embodiment of the present invention, after step 310, method 300 may further optionally include: constructing a similarity matrix of grid features between multiple frames corresponding to the past frame and the current frame through a temporal encoder; and learning the similarity matrix to complete an end-to-end occupied grid tracking task ( Figure 3 not shown).

[0087] Figure 4 A schematic diagram of a computer-readable storage medium 406 configured to store computer-executable instructions 404 is shown according to one embodiment 400 of the present disclosure.

[0088] like Figure 4As shown, in this embodiment 400, an exemplary computer-readable storage medium 406 is provided. Computer-executable instructions 404 (hereinafter also referred to as instructions) are stored on the computer-readable storage medium 406. In this embodiment 400, the instructions 404 may be configured to implement a method (as shown in the dashed box 402) when executed by a processor, such as Figure 3 In another embodiment, the instructions 404 may also be configured to implement a system such as Figure 1 The system 100 shown. As can be appreciated by those skilled in the art, Figure 4 The steps in the dashed box 402 are merely exemplary and non-limiting. A person skilled in the art can design many such computer-readable media that can be configured to perform a desired method or implement a desired system.

[0089] As used herein, the term "computer-readable storage medium" includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-executable instructions or other data. Memory and storage are examples of computer-readable storage media. Computer-readable storage media may be non-transitory and include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computing device.

[0090] Although not required, the various embodiments are described in the general context of "computer-executable instructions" executed by one or more processors. Computer-executable instructions can be distributed via computer-readable storage media. Computer-executable instructions can be implemented as program modules, such as functions, objects, application programming interfaces (APIs), data structures, etc., that perform a particular method or implement a particular system. In general, the functionality of the computer-executable instructions can be combined or distributed in various environments as needed.

[0091] The term "computer-readable storage medium" may also include communication media. Communication media typically embody computer-executable instructions or other data in a "modulated data signal" such as a carrier wave or other transmission mechanism, and includes any information transmission medium. The term "modulated data signal" may include a signal that sets or changes one or more of its characteristics in a manner that encodes the information in the signal. Those skilled in the art will recognize that computer-readable storage media used to store computer-executable instructions can be distributed over a network. For example, a computer-readable storage medium accessible via a network may store computer-executable instructions to implement one or more embodiments provided by the present disclosure. A computing device may access a computer-readable storage medium over a network and download part or all of the computer-executable instructions for execution. Alternatively, some instructions may be executed on one computing device and other instructions on another computing device.

[0092] In summary, the technical solution of the present invention proposes a multi-task framework for regular obstacle 4D BEV target detection and irregular obstacle occupancy prediction to meet the broader and more detailed requirements for irregular obstacle occupancy positions in urban and parking scenarios.

[0093] Embodiments of the present invention have been described above with reference to block diagrams and / or the operating instructions of methods, systems, and computer program products according to embodiments of the present invention. Each function / action noted in the frame may occur in an order different from that shown in any flow chart. For example, depending on the function / action involved, two frames shown in succession may actually be performed substantially simultaneously, or these frames may sometimes be performed in an opposite order.

[0094] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for obstacle and occupancy grid prediction using a fisheye camera, the method comprising: Receive multiple frames of images from multiple fisheye cameras; Extracting image features and depth probability distribution from the multiple frames of images; Performing BEV perspective conversion and voxelization on the image features to generate BEV features; Predicting a 3D bounding box of an obstacle on the BEV based on the BEV features and outputting the position, heading angle, size and speed information of the obstacle according to the 3D bounding box; generating an occupancy grid prediction result corresponding to a current frame based on the BEV feature; as well as The occupancy grid prediction result corresponding to the past frame before the current frame is superimposed on the current frame to densify the occupancy grid prediction result corresponding to the current frame.

2. The method as claimed in claim 1, wherein the image features are image features of different scales extracted by Backbone, and wherein the method further comprises performing 2D BBox supervision on the image features of different scales to increase 2D image feature constraints and performing point cloud truth depth supervision on the depth probability distribution to increase depth feature constraints.

3. The method of claim 1, wherein performing BEV perspective conversion and voxelization on the image features further comprises: Dedistorting the image features by using the internal parameters of the multi-channel fisheye cameras and the fisheye distortion coefficients through a lift-splat-shoot method and performing an outer product operation with the depth probability distribution to generate a cone feature; Obtaining a frustum voxel index in the BEV space according to the external parameters of the multi-channel fisheye camera relative to the vehicle coordinate system to perform BEV perspective conversion table lookup preprocessing; and The view cone feature and the depth probability distribution are accessed according to the view cone voxel index to complete the BEV table lookup process from 2D feature to 3D feature.

4. The method of claim 1 , wherein generating the BEV signature further comprises: Extracting the image features after BEV perspective conversion as the BEV features through a BEV encoder network; and The BEV features are aggregated and constructed into three-dimensional spatial BEV features with height.

5. The method of claim 1, wherein the method further comprises: Constructing a similarity matrix of grid features between multiple frames corresponding to the past frame and the current frame through a temporal encoder; as well as The similarity matrix is ​​learned to complete the end-to-end occupancy grid tracking task.

6. A system for obstacle and occupancy grid prediction using a fisheye camera, the system comprising: An image feature acquisition module is configured to receive multiple frames of images from multiple fisheye cameras and extract image features and depth probability distribution from the multiple frames of images; A BEV perspective conversion module is configured to perform BEV perspective conversion and voxelization on the image features to generate BEV features; an obstacle prediction module configured to predict a 3D bounding box of an obstacle on the BEV based on the BEV features and output the position, heading angle, size and speed information of the obstacle according to the 3D bounding box; Occupancy grid prediction module, which is configured to: generating an occupancy grid prediction result corresponding to a current frame based on the BEV feature; and The occupancy grid prediction result corresponding to the past frame before the current frame is superimposed on the current frame to densify the occupancy grid prediction result corresponding to the current frame.

7. The system of claim 6, wherein the image features are image features of different scales extracted by Backbone, and wherein the image feature acquisition module is further configured to perform 2D BBox supervision on the image features of different scales to increase 2D image feature constraints and to perform point cloud truth depth supervision on the depth probability distribution to increase depth feature constraints.

8. The system of claim 6, wherein the BEV perspective conversion module is further configured to perform BEV perspective conversion and voxelization on the image features by: The image features are dedistorted by the internal parameters of the multi-channel fisheye cameras and the fisheye distortion coefficients through the Lift-Splat-Shoot method, and the outer product operation is performed with the depth probability distribution to generate a cone feature; Obtaining a frustum voxel index in the BEV space according to the external parameters of the multi-channel fisheye camera relative to the vehicle coordinate system to perform BEV perspective conversion table lookup preprocessing; and The view cone feature and the depth probability distribution are accessed according to the view cone voxel index to complete the BEV table lookup process from 2D feature to 3D feature.

9. The system of claim 6, wherein the BEV perspective conversion module is further configured to generate the BEV features by: Extracting the image features after BEV perspective conversion as BEV features through a BEV encoder network; and The BEV features are aggregated and constructed into three-dimensional spatial BEV features with height.

10. The system of claim 6, wherein the occupancy grid prediction module is further configured to: constructing a similarity matrix of multiple inter-frame grid features corresponding to the past frame and the current frame through a temporal encoder; and The similarity matrix is ​​learned to complete the end-to-end occupancy grid tracking task.

11. A computer-readable storage medium having stored thereon instructions for using a fisheye camera to perform obstacle and occupancy grid prediction, wherein the instructions are executed by a processor to implement the method of any one of claims 1 to 5.

Images