Spaceholder grid detection method based on multiple look-around images

Abstract

The invention provides a space-occupying grid detection method based on multiple look-around images. The space-occupying grid detection method is characterized by comprising the following steps: step 1, arranging a plurality of cameras around a vehicle; step 2, the system collects image data of each camera; step 3, the system extracts features of each view angle image by using a weight-sharing convolutional neural network; 4, introducing a visual angle attention mechanism into the system, and establishing cross-visual-angle feature association; according to the omni-directional multi-view-angle occupation detection network provided by the invention, high-precision three-dimensional perception of the surrounding environment of the vehicle is realized through an innovative view angle attention mechanism and a self-adaptive feature fusion technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle environment perception technology, and particularly relates to a placeholder grid detection method based on surround-view multi-images. BACKGROUND

[0002] The traditional vehicle surrounding object detection method mainly relies on fixed three-dimensional rectangular boxes for target detection. This method has limitations in processing irregular-shaped obstacles, such as: unable to accurately detect a tilted bicycle, difficult to accurately identify irregular shapes such as road damage, insufficient detection accuracy for special-shaped obstacles, etc. And previous object detection often relies on radar to obtain point cloud data, which has problems such as high hardware cost and insufficient resolution. The image-based scheme often uses a single perspective, resulting in limited detection system perception range and insufficient obstacle perception ability for the side and rear of the vehicle, which cannot meet the needs of reversing, turning and other scenarios. SUMMARY

[0003] In view of the above technical problems, the present application provides a placeholder grid detection method based on surround-view multi-images to overcome the limitations of traditional detection methods and provide reliable environment perception capability for autonomous driving systems.

[0004] The present application provides a placeholder grid detection method based on surround-view multi-images, characterized by the following steps:

[0005] Step 1: arranging multiple cameras around the vehicle;

[0006] Step 2: the system collects image data from each camera;

[0007] Step 3: the system uses a convolutional neural network with shared weights to extract features from each perspective image;

[0008] Step 4: the system introduces a perspective attention mechanism to establish cross-perspective feature correlation;

[0009] Step 5: the system establishes a stereo matching relationship using adjacent camera pairs, calculates the disparity using a deep learning stereo matching module, and outputs the confidence of each disparity;

[0010] Step 6: for the deep learning module, the system encodes the position and depth features of each point cloud information, dynamically adjusts the weights of different spatial positions according to the feature quality and reliability, and predicts the occupancy probability of each point in the three-dimensional space.

[0011] The number of cameras is 8, located at the front, rear, left, right, front left, front right, rear left and rear right of the vehicle, wherein each adjacent camera is considered as a stereo binocular camera.

[0012] The specific steps of step three are: encoding the position of each camera in step two to obtain the position encoding I i Using the feature extraction network F with shared weights to process, get the corresponding feature map F_i: Wherein, represents a feature extraction network; θ represents network parameters.

[0013] The specific steps of step four are:

[0014] A: Splice the position encoding corresponding to the camera and the features of each camera to obtain the spliced features:

[0015]

[0016] Where PE i is the position encoding of the i-th camera, which contains the position and orientation information of the camera;

[0017] B: Calculate the multi-view attention weight: Wherein: represents the dot product of the i-th view feature and the j-th view feature, which is used to measure the similarity of the two view features; softmax is a normalization function, which ensures that the sum of all attention weights is 1; A ij represents the attention weight from view i to view j, the value range is [0, 1];

[0018] C: The final view enhanced feature is: Wherein: ) represents the weighted sum of all 8 views; A ij is the attention weight from view i to view j; is the position enhanced feature of the j-th view; is the enhanced feature representation after fusing other view information.

[0019] The specific calculation process of step five is:

[0020] A: For each pair of adjacent cameras (C i , C j ), calculate the disparity map D ij through the deep learning stereo matching network: Where StereoMatch represents a stereo matching algorithm;

[0021] B: Calculate the depth map Z ij through the disparity map and camera parameters: Where f represents the camera focal length, and b represents the camera baseline distance;

[0022] C: Use depth information to calculate three-dimensional point cloud coordinates (X ij , Yij , Z ij ):

[0023]

[0024] wherein:

[0025] (u, v) is the pixel coordinate in the image;

[0026] (c x , c y ) is the principal point coordinate of the camera;

[0027] Z ij is the depth value calculated in the foregoing;

[0028] (X ij , Y ij , Z ij ) is the three-dimensional point cloud coordinate in the camera coordinate system;

[0029] D: convert the point cloud from the camera coordinate system to the world coordinate system using the extrinsic matrix of the camera:

[0030]

[0031] wherein R i is the rotation matrix, t i is the translation vector, and represents the pose of the camera C i .

[0032] The specific calculation process of dynamically adjusting the weight of different spatial positions according to the feature quality and reliability in the sixth step is as follows:

[0033] A: project the three-dimensional point into each camera view:

[0034] p i = Proj(p, K i , R i , t i )

[0035] wherein:

[0036] p represents the three-dimensional point in the world coordinate system;

[0037] K i represents the intrinsic matrix of the i-th camera;

[0038] R i represents the rotation matrix of the i-th camera;

[0039] t i represents the translation vector of the i-th camera;

[0040] Proj denotes the projection function from a three-dimensional point to a two-dimensional image plane;

[0041] p i denotes the projected coordinate of point p on the image plane of the i-th camera;

[0042] B: Check if point p is within the field of view of camera C i :

[0043] v i = 1 if point p is within the field of view of camera C i , otherwise 0

[0044] where v i is a visibility indicator variable indicating whether point p is within the field of view of camera C i ;

[0045] C: Compute feature quality score:

[0046]

[0047] where:

[0048] is the enhanced feature map of the i-th camera;

[0049] Quality is a function that evaluates the quality of a feature;

[0050] p i denotes the feature quality score of point p in the i-th camera;

[0051] D: Compute adaptive fusion weight:

[0052]

[0053] where:

[0054] w i is the contribution weight of the i-th camera to point p;

[0055] exp is the exponential function used to emphasize the importance of high-quality features; the denominator term ensures that the sum of all weights is 1;

[0056] E: Final fused feature:

[0057]

[0058] where F p is the fused feature representation of point p;

[0059] F: Based on the fused feature F p , predict the occupancy probability of point p:

[0060] Op = sigma(MLP(F p ))

[0061] wherein:

[0062] sigma is a sigmoid function mapping the output to the interval [0, 1];

[0063] MLP is a multi-layer perceptron used to process the fused features O p e [0, 1] represents the probability that the point p is occupied by an object.

[0064] The beneficial effects of the present application are: the present application is a surround-view multi-image-based occupancy grid detection method, which simultaneously captures the environmental information around the vehicle through multiple cameras, obtains a full range of perspectives, and solves the problem of limited single-perspective perception range; uses multiple sets of binocular images for stereo matching, each adjacent camera can generate disparity information, and higher-resolution depth information is obtained, which makes up for the problem of low resolution of point cloud data and solves the problem of high cost; discards the traditional rectangular frame detection method, and uses an occupancy network to represent obstacles, which can more accurately describe irregular shapes and solve the limitations of three-dimensional frame-based target detection processing of irregular obstacles; the fields of view of adjacent two sets of cameras (one set is two adjacent cameras) will partially overlap, and such a redundant design further improves the reliability of the disparity information, innovatively introduces a perspective attention mechanism to enhance the processing capability of different perspective information, and through the combination application of these innovative technologies, the present application successfully overcomes the limitations of traditional detection methods and provides a more reliable environmental perception capability for an autonomous driving system. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a flow chart of the present application;

[0066] Figure 2 is a schematic diagram of the camera mounting method of the present application. DETAILED DESCRIPTION

[0067] Example 1

[0068] The technical solutions of the present application will be described in detail below in combination with the drawings of the present application.

[0069] 1. Camera arrangement

[0070] The camera mounting method is shown in Figure 2 , eight cameras arranged around the vehicle are denoted as C = {C1, C2,..., C8}, and the images collected by the cameras are denoted as I = {I1, I2,..., I8}, wherein each image H and W represent the height and width of the image, respectively.

[0071] 2, image acquisition module

[0072] The acquisition time of the 8 cameras needs to be as consistent as possible;

[0073] 3, feature extraction module

[0074] The image I of each camera i is processed using a feature extraction network F with shared weights to obtain the corresponding feature map F_i:

[0075]

[0076] where

[0077] represents the feature extraction network;

[0078] θ represents the network parameters;

[0079] where the shared weight design can reduce the number of network parameters, reduce the resource occupation of model deployment, and at the same time ensure that similar feature representations are extracted for each image input.

[0080] 4, view attention module

[0081] In order to establish the feature correlation across views, we introduce position encoding and view attention mechanism. First, the position encoding corresponding to the camera and the features of each camera are spliced to obtain the spliced features:

[0082]

[0083] where PE i is the position encoding of the i-th camera, which contains the position and orientation information of the camera.

[0084] Then calculate the multi-view attention weight:

[0085]

[0086] where:

[0087] represents the dot product of the i-th view feature and the j-th view feature, which is used to measure the similarity of the two view features;

[0088] softmax is a normalization function that ensures the sum of all attention weights is 1;

[0089] A ij represents the attention weight from view i to view j, with a value range of [0, 1];

[0090] The final view-enhanced feature is:

[0091]

[0092] where:

[0093] denotes the weighted sum over all 8 views;

[0094] A ij is the attention weight from view i to view j;

[0095] is the position enhanced feature of the j-th view;

[0096] is the enhanced feature representation after fusing other view information;

[0097] 5. Stereo matching module

[0098] For each pair of adjacent cameras (C i , C j ), the disparity map D ij is computed by a deep learning stereo matching network:

[0099]

[0100] where StereoMatch denotes the stereo matching algorithm, which can use traditional SGBM (Semi-Global Block Matching) or deep learning based stereo matching method;

[0101] By the disparity map and camera parameters, the depth map Z ij is first computed:

[0102]

[0103] where,

[0104] f denotes the camera focal length;

[0105] b denotes the camera baseline distance (distance between adjacent cameras);

[0106] Then, the three-dimensional point cloud coordinates (X ij , Y ij , Z ij ) are calculated using the depth information:

[0107]

[0108] where:

[0109] (u, v) is the pixel coordinate in the image;

[0110] (c x , c y) is the principal point coordinate of the camera;

[0111] Z ij is the depth value calculated in the previous step;

[0112] (X ij , Y ij , Z ij ) is the 3D point cloud coordinate in the camera coordinate system;

[0113] Finally, we can convert the point cloud from the camera coordinate system to the world coordinate system using the extrinsic matrix of the camera:

[0114]

[0115] where R i is the rotation matrix, t i is the translation vector, representing the pose of the camera C i .

[0116] 6. Occupancy prediction module

[0117] Since the point cloud from different views has redundant safety design, for each point p = (x, y, z) in the 3D space, we need to fuse the features from different views to get a more reliable occupancy prediction. To this end, we designed an adaptive fusion module that dynamically adjusts the weights according to the feature quality and reliability.

[0118] First, project the 3D point into each camera view:

[0119] p i = Proj(p, K i , R i , t i )

[0120] where:

[0121] p represents a 3D point in the world coordinate system;

[0122] K i represents the intrinsic matrix of the i-th camera;

[0123] R i represents the rotation matrix of the i-th camera;

[0124] t i represents the translation vector of the i-th camera;

[0125] Proj represents the projection function of a 3D point to a 2D image plane;

[0126] p i represents the projection coordinates of point p on the i-th camera image plane;

[0127] Check if point p is in the field of view of camera C i :

[0128] v i = 1 if point p is in the field of view of camera C i , 0 otherwise

[0129] where v i is a visibility indicator variable indicating whether point p is in the field of view of C i camera.

[0130] Compute feature quality score:

[0131]

[0132] where:

[0133] is the enhanced feature map of the i-th camera;

[0134] Quality is a function that evaluates feature quality, which can be based on feature response intensity, gradient information, or uncertainty estimates;

[0135] p i denotes the feature quality score of point p in the i-th camera;

[0136] Compute adaptive fusion weights:

[0137]

[0138] where:

[0139] w i is the contribution weight of the i-th camera to point p;

[0140] exp is the exponential function, used to emphasize the importance of high-quality features;

[0141] The denominator term ensures that the sum of all weights is 1

[0142] Final fused feature:

[0143]

[0144] where F p is the fused feature representation of point p.

[0145] Based on the fused feature F p , predict the occupancy probability of point p:

[0146] O p = σ(MLP(F p ))

[0147] where:

[0148] σ is a sigmoid function that maps the output to the interval [0, 1].

[0149] MLP is a multi-layer perceptron used to process fused features O p ∈ [0, 1] represents the probability that point p is occupied by an object.

[0150] The basic principles and main features of the present application and the advantages of the present application are shown and described above. The various components mentioned in the present application are common techniques in the prior art, and those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for detecting an occupancy grid based on surround view multi-images, characterized in that The method comprises the following steps: Step 1: arranging a plurality of cameras around the vehicle; Step 2: the system collects image data of each camera; Step 3: the system extracts features of each view image using a convolutional neural network with shared weights; Step 4: the system introduces a view attention mechanism to establish cross-view feature correlation; Step 5: the system establishes a stereo matching relationship using adjacent camera pairs, calculates the disparity through a deep learning stereo matching module, and outputs the confidence of each disparity; Step 6: for the deep learning module, the system encodes the position and depth features of each point cloud information, dynamically adjusts the weights of different spatial positions according to the feature quality and reliability, and predicts the occupancy probability of each point in the three-dimensional space.

2. The surround view multi-image based occupancy grid detection method of claim 1, wherein The number of cameras is 8, which are respectively located at the front, rear, left, right, left front, right front, left rear and right rear positions of the vehicle, wherein each adjacent camera is regarded as a stereo binocular camera.

3. The surround view multi-image based occupancy grid detection method of claim 1, wherein The specific step of step three is: processing the image I of each camera in step two using a feature extraction network F with shared weights to obtain a corresponding feature map F_i: i using a feature extraction network F with shared weights to obtain a corresponding feature map F_i: wherein, denotes a feature extraction network; and θ denotes network parameters.

4. The surround view multi-image based occupancy grid detection method of claim 1, wherein The specific steps of step 4 are: A: splice the position encoding corresponding to the camera and the features of each camera to obtain the spliced features: where PE i is the position encoding of the ith camera, which contains the position and orientation information of the camera; B: Calculate multi-view attention weights: wherein: represents the dot product of the i-th view feature and the j-th view feature, which measures the similarity of the two view features; softmax is a normalization function to ensure that the sum of all attention weights is 1; A ij represents the attention weight from view i to view j, with a value range of [0, 1]; C: the final view-enhanced feature is: where: ) denotes a weighted sum over all 8 views; A ij is the attention weight from view i to view j; is the position-enhanced feature of the j-th view; is the enhanced feature representation that fuses information from other views.

5. The surround view multi-image based occupancy grid detection method of claim 1, wherein The specific calculation process of step 5 is: A: For each pair of adjacent cameras (C i , C j ), compute a disparity map D ij : where StereoMatch denotes a stereo matching algorithm. B: Compute depth map Z from disparity map and camera parameters ij : where f denotes the camera focal length and b denotes the camera baseline distance. C: Calculate three-dimensional point cloud coordinates (X ij , Y ij , Z ij ) using depth information: Wherein: (u, v) is the pixel coordinate in the image; (c x , c y ) are the principal point coordinates of the camera; Z ij is the depth value calculated above; (X ij , Y ij , Z ij ) are three-dimensional point cloud coordinates in the camera coordinate system; D: use the extrinsic matrix of the camera to convert the point cloud from the camera coordinate system to the world coordinate system: where R i is a rotation matrix, t i is a translation vector, and denotes the pose of camera C i .

6. The surround view multi-image based occupancy grid detection method of claim 1, wherein The specific calculation process of step 6 is: A: project the three-dimensional point into each camera view: p i = Proj(p, K i , R i , t i ) Wherein: p represents a three-dimensional point in the world coordinate system; K i denotes the intrinsic matrix of the i-th camera; R i denotes the rotation matrix of the i-th camera; t i represents the translation vector of the i-th camera; Proj represents the projection function of the three-dimensional point to the two-dimensional image plane; p i denotes the projection coordinate of point p on the i-th camera image plane; B: Check if checkpoint p is in the field of view of camera C i : Yes / No v i = 1 if point p is in the field of view of camera C i , otherwise 0 where v i is a visibility indicator variable that indicates whether point p is within the field of view of camera C i ; C: calculate the feature quality score: Wherein: is an enhanced feature map of the i-th camera; Quality is a function for evaluating feature quality; p i represents the feature quality score of point p in the i-th camera; D: calculate the adaptive fusion weight: Wherein: w i is the contribution weight of the ith camera to point p; exp is the exponential function, which is used to emphasize the importance of high-quality features; the denominator ensures that the sum of all weights is 1; E: the final fused feature: where F p is the fused feature representation of point p; F: based on fusion features F p , the occupancy probability of the prediction point p: O p = σ(MLP(F p )) where: σ is the sigmoid function, which maps the output to the [0, 1] interval rate; MLP is a multi-layer perceptron used to process fused features O p ∈ [0, 1] represents the probability that the point p is occupied by an object.

Images