A grid occupancy estimation method, device and medium based on multiple constraints

Abstract

The present invention relates to a grid occupancy estimation method, device and medium based on multiple constraints, the method comprising: receiving multi-view camera images acquired by camera sensors around a vehicle; inputting the multi-view camera images into a grid occupancy estimation model to obtain a grid occupancy prediction result; wherein the grid occupancy estimation model adds two-dimensional reprojection supervision for each view, which can effectively constrain the features of different objects on the opposite side, and also designs a spatial constraint based on object consistency, using the influence of surrounding voxels on semantic annotation of processed current voxels, to solve the problem of outliers in the predicted three-dimensional results. The present invention can effectively improve the algorithm accuracy and operation speed, and improve the estimation effect of the grid occupancy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a grid occupancy estimation method, device and medium based on multiple constraints. Background Art

[0002] The perception technology of autonomous driving is developing in a more refined and comprehensive direction. Traditional perception technology may fail in open world traffic scenarios with unclear shapes or appearances, especially when facing long-tail obstacles with variable shapes, special-shaped obstacles, and obstacles of unknown categories. Therefore, grid occupancy prediction technology, which can accurately perceive complex road conditions and detailed shape and category information, has emerged.

[0003] However, grid occupancy prediction technology is still in its early stages of development, and has not yet achieved a completely ideal effect on the refined perception of the scene. One of the important reasons is the lack of reasonable and precise semantic and geometric training constraints. At present, most algorithms still use the basic loss functions often used in tasks such as target detection to constrain fine-grained perception. These constraints currently have slow convergence, and traditional loss functions may not be able to handle targets with complex geometric shapes well. For example, if a target object has an irregular shape or multiple connected parts, traditional loss functions may not accurately predict the categories of these parts. In addition, traditional loss functions may be too sensitive to outliers or outliers. If there are some outliers in the training data, then these loss functions may be optimized on the category of the outliers, resulting in reduced performance on other categories.

[0004] Huang Yuanhui from Tsinghua University proposed a three-view perspective (TPV) representation method that can effectively describe the fine-grained structure of 3D scenes. It approximates the features of 3D points / 3D grids through features on three orthogonal planes. Although this representation can be used for tasks such as 3D point cloud semantic segmentation and 3D semantic completion, the data in front and behind the vehicle in the side view and front view are different and complex, making it difficult to express them using the same perspective plane. In addition, the loss function in this method only focuses on the classification of pixels, without considering the spatial relationship between pixels. The lack of reasonable constraints on the side view causes the content on both sides to overlap and affect each other, making the accuracy of other perspectives lower except for the top view, resulting in poor gain of other perspectives on the overall accuracy of the algorithm. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a grid occupancy estimation method, device and medium based on multiple constraints, which can effectively improve the algorithm accuracy and operation speed.

[0006] The technical solution adopted by the present invention to solve the technical problem is: to provide a grid occupancy estimation method based on multiple constraints, comprising the following steps:

[0007] Receiving multi-view camera images acquired by camera sensors around the vehicle;

[0008] The multi-view camera image is input into a grid occupancy estimation model to obtain a grid occupancy prediction result; wherein the grid occupancy estimation model includes:

[0009] A feature extraction module, used to extract image features from the multi-view camera image;

[0010] The surround view attention module is used to map the extracted image features to the surround view according to the direction to obtain the surround view features;

[0011] A two-dimensional supervision module, used to predict the category of the two-dimensional feature according to the surround view feature, and use the two-dimensional view true value for supervision;

[0012] A feature fusion module, used for expanding and fusing the surround view features to obtain fine-grained 3D spatial features;

[0013] A consistency loss module is used to perform object consistency supervision based on the fine-grained 3D spatial features to obtain consistency supervision branch features

[0014] The prediction module is used to fuse the consistency supervision branch features with the fine-grained 3D spatial features, and use the prediction head to perform category prediction to obtain the grid occupancy prediction results.

[0015] The two-dimensional supervision module comprises:

[0016] A mapping unit, configured to construct a 3D pillar region orthogonal to the viewing plane according to the scene scale, generate feature sampling points in the 3D pillar region, and map the surround view feature to the surround 2D view using self-attention;

[0017] A category prediction unit, used to perform category prediction on the surround two-dimensional view to obtain a two-dimensional prediction label result of the surround two-dimensional view;

[0018] The 2D computing unit is used to calculate the loss value of the 2D head branch based on the 2D predicted label result and the 2D true value of the surround 2D view.

[0019] The two-dimensional calculation unit is Calculate the loss value of the two-dimensional head branch, where Loss 2d is the loss value of the 2D head branch, is the two-dimensional truth value, Sur v preIt is the two-dimensional prediction label result.

[0020] The consistency loss module includes:

[0021] A voxel prediction unit, used to obtain voxel features around the prediction point of the fine-grained 3D spatial feature, and predict the number of connected voxels of the same category as the current voxel;

[0022] The consistency calculation unit is used to calculate the consistency loss value based on the predicted number of connected voxels and the true value.

[0023] The consistency calculation unit is Calculate the consistency loss value, where Loss sc is the consistency loss value, smooth L1 () indicates smoothing. is the predicted voxel v in the d direction i The number of connected voxels of the same class, is the actual distance in the d direction from the current voxel v i The number of connected voxels of the same class.

[0024] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned grid occupancy estimation method based on multiple constraints when executing the computer program.

[0025] The technical solution adopted by the present invention to solve its technical problem is: providing a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned grid occupancy estimation method based on multiple constraints are implemented.

[0026] Beneficial Effects

[0027] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: Based on the surround projection, the present invention adds a two-dimensional head branch to the framework and introduces a two-dimensional grid constraint to enhance supervision, which can effectively output a grid occupancy prediction result while selectively integrating the learned high-dimensional features to enhance the final semantic grid occupancy estimation accuracy. The present invention also designs a spatial constraint based on object consistency, and uses the influence of surrounding voxels on the semantic annotation of the processed current voxel to solve the problem of outliers in the predicted three-dimensional results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of a grid occupancy estimation method based on multiple constraints according to a first embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0030] The first embodiment of the present invention relates to a grid occupancy estimation method based on multiple constraints. This embodiment proposes a new loss function constraint scheme to train a three-dimensional grid occupancy prediction algorithm to effectively improve the algorithm accuracy and operation speed, and specifically includes the following steps:

[0031] Step 1, receiving multi-view camera images acquired by camera sensors around the vehicle;

[0032] Step 2: input the multi-view camera image into a grid occupancy estimation model to obtain a grid occupancy prediction result.

[0033] like Figure 1 As shown, the grid occupancy estimation model in this embodiment includes:

[0034] The feature extraction module is used to extract image features from the multi-view camera images; the module can use a pre-trained backbone network to realize feature extraction of multi-view camera images.

[0035] The surround view attention module is used to map the extracted image features to the surround view according to the direction, so as to obtain the surround view feature. The surround view feature is composed of five projection view feature planes composed of the five directional view information of the front view, rear view, left view, right view and top view centered on the vehicle coordinate system, and the projection contains visible surface image features.

[0036] A two-dimensional supervision module is used to predict the category of the two-dimensional feature according to the surround view feature and to supervise using the two-dimensional view true value. The two-dimensional supervision module of this embodiment includes:

[0037] A mapping unit, configured to construct a 3D pillar region orthogonal to the viewing plane according to the scene scale, generate feature sampling points in the 3D pillar region, and map the surround view feature to the surround 2D view using self-attention;

[0038] The category prediction unit is used to predict the category of the surround 2D view and obtain the 2D prediction label result of the surround 2D view. In this way, 2D prediction label results corresponding to 5 different views can be obtained. They represent the semantic labels of the 3D surface of the surround view, which are respectively expressed as:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] in, They are the two-dimensional prediction label results of the front view, rear view, left view, right view and bird's-eye view, respectively. bi [] is the binary output prediction result, the dimension is W×H×Z. If the voxel categories in this direction are all empty, Pre bi (x,y,z)=0, if the label is other, then Pre bi (x,y,z)=1.

[0045] The two-dimensional calculation unit is used to calculate the loss value of the two-dimensional head branch according to the two-dimensional prediction label result and the two-dimensional true value of the surround two-dimensional view. Calculate the loss value of the two-dimensional head branch, where Loss 2d is the loss value of the 2D head branch, is the two-dimensional truth value, is the two-dimensional prediction label result. The construction method of the two-dimensional true value is the same as that of the two-dimensional prediction label result.

[0046] This implementation adds two-dimensional reprojection supervision for each perspective to the solution of multi-view representation of three-dimensional space, which can effectively constrain the features of different objects on the opposite side. At the same time, the supervision of perspective improves the utilization of semantics without significantly increasing the amount of calculation.

[0047] A feature fusion module, used for expanding and fusing the surround view features to obtain fine-grained 3D spatial features;

[0048] The consistency loss module is used to perform object consistency supervision based on the fine-grained 3D spatial features to obtain consistency supervision branch features.

[0049] For spatial continuity loss, this embodiment first assumes that two adjacent voxels with the same category label are connected. In order to simplify the calculation, the adjacent elements of the 6 faces of the voxel are used for approximation. Within the object range, the category of the current voxel is consistent with the category of other voxels in the 6 connection directions. Therefore, the consistency loss module of this embodiment includes:

[0050] The voxel prediction unit is used to obtain the voxel features around the prediction point of the fine-grained 3D spatial feature and predict the number of connected voxels of the same category as the current voxel, expressed as: C num (v i ,d)=Count(v i ,d), where d=1,2,3,…,6, represents the specified direction, Count() is the connection prediction function, and in order to ensure efficient execution speed, this implementation uses CUDA code programming to implement the function.

[0051] The consistency calculation unit is used to calculate the consistency loss value according to the predicted number of connected voxels and the true value. Calculate the consistency loss value, where Loss sc is the consistency loss value, smooth L1 () indicates smoothing. is the predicted voxel v in the d direction i The number of connected voxels of the same class, is the actual distance in the d direction from the current voxel v i The number of connected voxels of the same category. Through this constraint, the network further learns the spatial semantic context information of the voxels and solves the problem of outlier voxels in prediction.

[0052] This implementation effectively solves the problem of erroneous outliers caused by discontinuous category distribution, which can make the prediction of object edges in the grid prediction results smoother, and will not cause abnormal points to cause misjudgment in subsequent vehicle decisions, thereby affecting the driving experience.

[0053] The prediction module is used to fuse the consistency supervision branch features with the fine-grained 3D spatial features, and use the prediction head to perform category prediction to obtain the grid occupancy prediction result. While calculating the spatial continuity of each voxel to construct the loss, this implementation selectively integrates the learned high-dimensional features into the 3D label prediction head, filtering out features that are beneficial to category continuity prediction, thereby enhancing the final grid semantic occupancy estimation effect.

[0054] Taking the application implementation on the common standard dataset nuScenes for autonomous driving BEV perception tasks as an example, the specific operation process of this implementation is as follows:

[0055] Step 1: Read and preprocess the data: including the six-view camera image data and lidar data of nuScenes data.

[0056] Dataset usage data introduction: This dataset is based on the nuScenes dataset and contains 28,130 training data and 6,019 verification data. Each sample consists of RGB images from 6 cameras, forming a 360° surround view. It also includes observable sample masks of the camera and lidar. The six-view camera image data includes: front, left front, right front, left, right, and rear, covering 360° images around the vehicle. This data is used as training data, and also includes camera time information, camera internal and external parameters and other information as image annotation information.

[0057] Data preprocessing: The image size is 1600*900 pixels, and the training is carried out in the order of scene and time information. The true value annotation information uses a 200*200*16 three-dimensional vector, which contains 18 categories. Labels from 1 to 16 represent unclassified obstacles, bicycles, buses, cars, construction vehicles, motorcycles, pedestrians, traffic cones, trailers, trucks, drivable areas, other areas, sidewalks, terrain, man-made obstacles, and vegetation. Label 17 represents voxels that are not occupied by anything.

[0058] During the training process, there will be random flipping (RandomFlip), slight scaling (resize) on multiple scales to maintain the original proportion, and random position filling to ensure the diversity of data between each epoch.

[0059] Step 2: Construct the grid occupancy estimation model proposed in this embodiment, set up an optimizer for training the model, and use the current multiple constraints to send the formatted preprocessed data into the network training to obtain a converged grid occupancy estimation model.

[0060] This algorithm uses the R101 commonly used in this task as the backbone network for training. Then FPN is used as the neck to obtain multi-scale features, which are 1 / 16, 1 / 32 and 1 / 64 respectively. The number of channels is 256 dimensions.

[0061] Parameter settings. The perception range is [-40m, -40m, -1m, 40m, 40m, 5.4m]. The space is represented as a cubic voxel with a side length of 0.4m, divided into 18 categories. The sampling is set to [4, 50, 50] according to the voxel grid ratio. The number of sampling points per layer is 4, and the sampling is uniform within the perception range. The maximum number of epochs for training is set to 24. The learning rate is 2×10 -4 .

[0062] Step 3: Feed the nuScenes validation set data into the trained grid occupancy estimation model and output the grid occupancy results.

[0063] It is not difficult to find that the present invention adds a two-dimensional head branch to the framework based on the surround projection, and introduces a two-dimensional grid constraint to enhance supervision, which can effectively output a grid occupancy prediction result while selectively integrating the learned high-dimensional features to enhance the final semantic grid occupancy estimation accuracy. The present invention also designs a spatial constraint based on object consistency, and uses the influence of surrounding voxels on the semantic annotation of the processed current voxel to solve the problem of outliers in the predicted three-dimensional results.

[0064] A second embodiment of the present invention relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the grid occupancy estimation method based on multiple constraints of the first embodiment when executing the computer program.

[0065] The third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of the grid occupancy estimation method based on multiple constraints of the first embodiment.

[0066] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0067] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction method, which is implemented in the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A grid occupancy estimation method based on multiple constraints, characterized in that: The following steps are involved: Receiving multi-view camera images acquired by camera sensors around the vehicle; The multi-view camera image is input into a grid occupancy estimation model to obtain a grid occupancy prediction result; wherein the grid occupancy estimation model includes: A feature extraction module, used to extract image features from the multi-view camera image; The surround view attention module is used to map the extracted image features to the surround view according to the direction to obtain the surround view features; A two-dimensional supervision module is used to predict the category of the two-dimensional feature according to the surround view angle feature and use the two-dimensional view true value for supervision; the two-dimensional supervision module includes: A mapping unit, configured to construct a 3D pillar region orthogonal to the viewing plane according to the scene scale, generate feature sampling points in the 3D pillar region, and map the surround view feature to the surround 2D view using self-attention; A category prediction unit, used to perform category prediction on the surround two-dimensional view to obtain a two-dimensional prediction label result of the surround two-dimensional view; A two-dimensional calculation unit, used to calculate a loss value of a two-dimensional head branch according to a two-dimensional predicted label result and a two-dimensional true value of a surround two-dimensional view; A feature fusion module is used to expand and fuse the surround view features to obtain fine-grained 3D spatial features; a consistency loss module is used to perform object consistency supervision based on the fine-grained 3D spatial features to obtain consistency supervision branch features; The prediction module is used to fuse the consistency supervision branch features with the fine-grained 3D spatial features, and use the prediction head to perform category prediction to obtain the grid occupancy prediction results.

2. The grid occupancy estimation method based on multiple constraints according to claim 1, characterized in that: The two-dimensional calculation unit is Calculate the loss value of the two-dimensional head branch, where Loss 2d is the loss value of the 2D head branch, is the two-dimensional truth value, It is the two-dimensional prediction label result.

3. The grid occupancy estimation method based on multiple constraints according to claim 1, characterized in that: The consistency loss module includes: A voxel prediction unit, used to obtain voxel features around the prediction point of the fine-grained 3D spatial feature, and predict the number of connected voxels of the same category as the current voxel; The consistency calculation unit is used to calculate the consistency loss value based on the predicted number of connected voxels and the true value.

4. The grid occupancy estimation method based on multiple constraints according to claim 3 is characterized in that: The consistency calculation unit is Calculate the consistency loss value, where Loss sc is the consistency loss value, smooth L1 () indicates smoothing. is the predicted voxel v in the d direction i The number of connected voxels of the same class, is the actual distance in the d direction from the current voxel v i The number of connected voxels of the same class.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the grid occupancy estimation method based on multiple constraints as described in any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the grid occupancy estimation method based on multiple constraints as claimed in any one of claims 1 to 4 are implemented.

Images