A method and system for constructing a multi-view camera top-down feature map

Abstract

The present invention relates to an efficient method and system for constructing a multi-view camera bird's-eye view feature map. The method relates to the field of intelligent driving visual perception and is applied to a vehicle equipped with multiple view cameras, each of which covers the surrounding area of ​​the vehicle. The method comprises: constructing a view cone point cloud spatial model comprising an invalid depth layer and multiple valid depth layers; utilizing a deep learning model to acquire and filter invalid pixel data located in the invalid depth layer to obtain target pixel data; and utilizing the deep learning model to process the target pixel data to generate a bird's-eye view feature map. The present invention reduces the computational complexity involved in constructing a multi-view camera bird's-eye view feature map and improves the efficiency of constructing the bird's-eye view feature map.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving visual perception, and in particular to a method and system for constructing a multi-perspective camera bird's-eye view feature map. Background Art

[0002] Currently, weekly testing of intelligent driving vehicles involves installing multiple cameras with a 360-degree coverage of the vehicle's surroundings. Using multiple images captured simultaneously by each camera, a method for constructing a bird's-eye view feature map (BEV) can be generated. This feature map is centered on the vehicle and has three dimensions: X, Y, and C. The X and Y dimensions correspond to the front, back, and left, right, and sides of the vehicle. This grid of feature maps at different locations encodes environmental information about the corresponding locations around the vehicle. The C dimension represents the feature channel. The bird's-eye view feature map can serve as input to a corresponding deep learning task model and output the corresponding task result. For example, an object detection model outputs objects around the vehicle, while a semantic segmentation model outputs pixel-by-pixel semantic classification encodings of lane lines, road edges, or obstacles around the vehicle.

[0003] Currently, a mainstream method for constructing a bird's-eye view feature map for a multi-view camera is to use a deep learning model to estimate the depth of pixels or downsampled feature points, construct a dense view cone point cloud, and then use the camera's intrinsic and extrinsic parameters to organize the point cloud into a bird's-eye view or bird's-eye view feature map. The view cone point cloud involved in the calculation is relatively dense, resulting in large-scale calculations becoming a bottleneck for the deployment and application of this method on the vehicle side. Summary of the Invention

[0004] In order to reduce the amount of calculation when constructing a multi-view camera top-view feature map and improve the efficiency of constructing the top-view feature map, the present invention provides a method and system for constructing a multi-view camera top-view feature map.

[0005] In a first aspect, the present invention provides a method for constructing a multi-view camera top-down feature map, which adopts the following technical solution:

[0006] A method for constructing a multi-view camera top-view feature map, applied to a vehicle, wherein the vehicle is equipped with multiple view cameras, each camera imaging covers the surrounding area of ​​the vehicle, comprising:

[0007] Constructing a frustum point cloud space model including an invalid depth layer and multiple valid depth layers;

[0008] Acquire and filter invalid pixel data located in the invalid depth layer using a deep learning model to obtain target pixel data;

[0009] The target pixel data is processed using the deep learning model to generate a top-view feature map.

[0010] Furthermore, in the above method for constructing a multi-view camera top-view feature map, constructing a view frustum point cloud space model including an invalid depth layer and multiple valid depth layers includes:

[0011] Determining an effective depth receptive field based on a preset depth distribution range, and dividing the effective depth receptive field into a plurality of effective depth layers based on a preset interval distance;

[0012] An invalid depth receptive field is determined outside the range of the valid depth receptive field, and the invalid depth receptive field is the invalid depth layer.

[0013] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, constructing a view cone point cloud space model including an invalid depth layer and multiple valid depth layers further includes:

[0014] The effective depth layer is screened and updated based on a preset height threshold.

[0015] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, the height threshold includes an upper limit value of a fixed height and / or a lower limit value of a fixed height.

[0016] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, the height threshold includes an upper limit value that changes with the depth value of the effective depth layer and / or a lower limit value that changes with the depth value of the effective depth layer.

[0017] Furthermore, in the above method for constructing a multi-view camera top-view feature map, the method of using a deep learning model to obtain and filter invalid pixel data located in the invalid depth layer to obtain target pixel data includes:

[0018] Training a deep prediction model;

[0019] Generating a depth prediction value for each pixel data in the camera imaging based on the depth prediction model;

[0020] The invalid pixel data located at the invalid depth layer is filtered based on the depth prediction value to obtain the target pixel data.

[0021] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, the training of the depth prediction model includes:

[0022] Annotating preset pixel data training samples based on the view cone point cloud spatial model;

[0023] The depth prediction model is trained based on labeled pixel data training samples.

[0024] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, the training depth prediction model further includes:

[0025] Obtaining a depth prediction result of the pixel data training sample based on the trained depth prediction model;

[0026] Calculating the loss value between the depth prediction result and the true value based on a preset loss function;

[0027] The depth prediction model is trained iteratively in reverse based on the loss value.

[0028] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, the training depth prediction model further includes:

[0029] Obtain vehicle laser point cloud information;

[0030] The depth truth information of the pixel data training sample is obtained based on the laser point cloud information.

[0031] Furthermore, in the above method for constructing a multi-view camera top-view feature map, the processing of the target pixel data using the deep learning model to generate the top-view feature map includes:

[0032] Inputting the target pixel data into the deep learning model to obtain a content feature map of the target pixel data and a depth distribution probability corresponding to each feature point in the content feature map;

[0033] Generate a view cone point cloud feature based on the content feature map and the depth distribution probability calculation;

[0034] The view cone point cloud features are mapped to a top view centered on the vehicle to obtain the top view feature map.

[0035] Furthermore, in the above method for constructing a multi-view camera bird's-eye view feature map, the step of processing the target pixel data using the deep learning model to generate the bird's-eye view feature map further includes:

[0036] Sum the features of multiple view frustum point clouds that are mapped to the same location.

[0037] In a second aspect, the present invention provides a system for constructing a multi-view camera top-down feature map, which adopts the following technical solutions:

[0038] A system for constructing a multi-view camera top-down feature map, comprising

[0039] The data acquisition module is connected to the cameras at each viewing angle on the vehicle to obtain the imaging pixel data of each camera;

[0040] Spatial modeling module, used to construct the view frustum point cloud spatial model;

[0041] Network training module, used to train deep learning models;

[0042] A pixel filtering module, which filters invalid pixel data based on the view cone point cloud spatial model using the deep learning model to generate valid pixel data;

[0043] A result generation module is used to generate and output a top-view feature map using the deep learning model.

[0044] Thirdly, the present invention provides a readable storage medium, which adopts the following technical solution:

[0045] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of a method for constructing a multi-view camera top-view feature map as described in any one of the first aspects above.

[0046] In summary, the present invention includes at least one of the following beneficial technical effects:

[0047] 1. The present invention defines a view cone point cloud space model that includes an invalid depth layer and multiple valid depth layers. When modeling the view cone point cloud space, the valid and invalid view cone depth distribution ranges are clearly divided, and an invalid depth layer representing the outside of the valid view cone is specially set on the view cone depth distribution. For pixels or feature points located outside the valid view cone, their true values ​​are classified as belonging to the invalid depth layer. During depth prediction, if the depth distribution prediction result of a pixel or feature point is in the invalid depth layer, all view cone point clouds corresponding to this pixel or feature point are deleted, and they no longer participate in the subsequent calculation steps of constructing the overhead feature map, thereby simplifying the computational complexity of the subsequent calculation steps and improving the construction efficiency;

[0048] 2. Update the effective depth layer in the view frustum point cloud spatial model based on the foreground height information of interest. The height range of the foreground of interest is calculated and used as a basis to set the effective height range of the view frustum point cloud. Points outside the height range are deleted and excluded from the subsequent calculation of the top-down feature map. This further simplifies the computational complexity of subsequent calculation steps and improves construction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flowchart of an embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0050] Figure 2 This is a flowchart of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0051] Figure 3It is a schematic diagram of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0052] Figure 4 It is a schematic diagram of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0053] Figure 5 It is a schematic diagram of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0054] Figure 6 This is a flowchart of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0055] Figure 7 This is a flowchart of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention.

[0056] Figure 8 This is a flowchart of another embodiment of a method for constructing a multi-view camera top-view feature map according to the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] The method steps described in the embodiments of the present invention may be executed in the order described in the specific implementation manner, or the execution order of each step may be adjusted according to actual needs, provided that the technical problem can be solved. The execution order of each step will not be listed here one by one.

[0059] The following is combined with Figure 1-8 The present invention is described in further detail.

[0060] An embodiment of the present invention discloses a method for constructing a multi-view camera bird's-eye view feature map.

[0061] Reference Figure 1 A method for constructing a multi-view camera top-view feature map, applied to a vehicle, wherein the vehicle is equipped with multiple view cameras, each of which covers the surrounding area of ​​the vehicle, comprises:

[0062] S1, constructing a frustum point cloud space model including an invalid depth layer and multiple valid depth layers;

[0063] S2, using a deep learning model to acquire and filter invalid pixel data located in the invalid depth layer to obtain target pixel data;

[0064] S3: Process the target pixel data using the deep learning model to generate a top-view feature map.

[0065] Specifically, in step S1, the view cone point cloud space modeling is performed, and the view cone point cloud space model is divided into multiple valid depth layers and an invalid depth layer. When the depth of the imaging pixel point is in the valid depth layer, it can be determined that the pixel point is the target pixel data. When the imaging pixel point is in the invalid depth layer, the pixel point is invalid pixel data. In step S2, the depth of the imaging pixel is determined using the deep learning model, and based on this depth, it is determined whether the pixel point is in the invalid depth layer or the valid depth layer of the view cone point cloud space model. At the same time, the invalid pixel data in the invalid depth layer is deleted to obtain the target pixel data and is processed again using the deep learning model in step S3 to obtain a top-down feature map of the target pixel data.

[0066] An invalid depth layer is set in the view cone point cloud space model, and invalid pixels in the imaging pixels are filtered based on this invalid depth layer, which greatly reduces the density of the point cloud. In the subsequent construction of the multi-view camera bird's-eye view feature map, the computational complexity of the construction process is greatly reduced, which is beneficial to the deployment and application on the vehicle side.

[0067] Further, as a specific embodiment of the present invention, refer to Figure 2 , Step S1: Constructing a frustum point cloud space model including an invalid depth layer and multiple valid depth layers, including:

[0068] S11, determining an effective depth receptive field based on a preset depth distribution range, and dividing the effective depth receptive field into a plurality of effective depth layers based on a preset interval distance;

[0069] S12: determining an invalid depth receptive field outside the range of the valid depth receptive field, where the invalid depth receptive field is the invalid depth layer.

[0070] Further, refer to Figure 3 As a specific embodiment of the present invention, the preset depth distribution range can be selected as 4m-44m, and the preset interval distance can be selected as 1m. Then, step S11 specifically includes: defining the depth distribution range within the effective depth receptive field cone as 4m-44m, with an interval of 1m, resulting in a total of 41 depth distribution values, such as d_0, d_1, to d_40 in the figure. The specific implementation of step S12 is: additionally setting a 42nd depth distribution value d_41, which represents the depth range greater than d_max = 44m.

[0071] Based on steps S11 and S12, the entire view frustum point cloud space is layered. The first 41 layers (Layer 0 to Layer 40) are used to represent interesting foreground information near the vehicle. The imaging pixels falling on the first 41 layers are the target pixel data. The final Layer 41 is used to represent uninteresting foreground or background information far from the vehicle (more than 44 meters). The imaging pixels falling on Layer 41 are invalid pixel data.

[0072] Further, refer to Figure 2 As a specific embodiment of the present invention, step S1: constructing a view frustum point cloud space model including an invalid depth layer and multiple valid depth layers, further comprising:

[0073] S13: Filter and update the effective depth layer based on a preset height threshold.

[0074] Specifically, according to the pre-stated height range value of the foreground of interest, the effective range of the view cone point cloud is set in the height direction, and the view cone point cloud beyond the height range is deleted, thereby updating the effective depth layer, further reducing the imaging pixel data located on the effective depth layer, and then further reducing the amount of calculation for subsequent construction of the overhead feature map, thereby improving calculation efficiency.

[0075] Further, refer to Figure 4 As a specific embodiment of the present invention, the height threshold includes an upper limit value of a fixed height and / or a lower limit value of a fixed height. Figure 4 As shown, for effective depth layers of different depths, fixed upper and lower limits are set, the upper and lower parts of the view cone point cloud space are deleted, and the middle part is the effective point cloud space.

[0076] Further, refer to Figure 5 As a specific embodiment of the present invention, the height threshold includes an upper limit value that changes with the depth value of the effective depth layer and / or a lower limit value that changes with the depth value of the effective depth layer. Figure 5 As shown in Figure 1, for effective depth layers of different depths, the upper and lower limits of each layer are proportional to the depth value of this layer. Similarly, the middle part is the effective point cloud space.

[0077] Further, refer to Figure 6 As a specific embodiment of the present invention, step S2: using a deep learning model to obtain and filter invalid pixel data located in the invalid depth layer to obtain target pixel data may include:

[0078] S21, training depth prediction model;

[0079] S22, generating a depth prediction value for each pixel data in the camera imaging based on the depth prediction model;

[0080] S23 , filtering invalid pixel data located at the invalid depth layer based on the depth prediction value to obtain the target pixel data.

[0081] When using the invalid depth layer to filter invalid pixel data in the imaging pixel data, the depth prediction model is first trained in step S21. After the training is completed, the depth of each pixel data in the imaging is predicted based on the depth prediction model in step S22 to obtain a depth prediction value. Finally, in step S23, the invalid pixel data located on the invalid depth layer is deleted to obtain the target pixel data.

[0082] Further, refer to Figure 7 As a specific embodiment of the present invention, step S21: training the depth prediction model may include:

[0083] S211, labeling preset pixel data training samples based on the view cone point cloud space model;

[0084] S212: Train the depth prediction model based on the labeled pixel data training samples.

[0085] Specifically, the depth prediction model can be selected as a probability prediction model such as a convolutional neural network (CNN) and a recurrent neural network (RNN). The depth prediction model is not specifically limited in the present invention. Any model that can be used to predict category probability can be used in the specific implementation of the present invention, and can simplify the computational complexity during model training and classification.

[0086] When training the depth prediction model, step S211 performs true value annotation on the pixel data training samples. For example, when the true depth value of a certain pixel or feature point [h, w] is greater than D_max=44m, its depth distribution one-hot true value is set to [0, 0, 0, ..., 0, 1]. The matrix size is 1x42. Items 1-41 are set to 0, and item 42 is set to 1, indicating that the pixel or feature point [h, w] is located on an invalid depth layer (Layer 41). Similarly, when the one-hot true depth distribution value of another pixel or feature point [h, w] is [1, 0, 0, ..., 0, 0], it means that the pixel or feature point is on a valid depth measurement (Layer 0).

[0087] After the pixel data training samples are annotated with true values ​​in step S211, the depth prediction model is trained using the annotated pixel data training samples in step S212. The depth prediction model determined after multiple iterations of training can predict the depth distribution layer of the imaging pixel.

[0088] Furthermore, as a specific embodiment of the present invention, step S22: generating a depth prediction value of each pixel data in the camera imaging based on the depth prediction model, specifically:

[0089] When a pixel with a one-hot true value of [0, 0, 0, …, 0, 1] is input into the trained depth prediction model, its prediction result is [0.021, 0.023, 0.017, …, 0.842], which is also a matrix with the same dimension as the true value.

[0090] Furthermore, as a specific embodiment of the present invention, step S23: filtering invalid pixel data located in the invalid depth layer based on the depth prediction value to obtain the target pixel data, specifically:

[0091] The threshold for determining whether a pixel or feature point's depth distribution prediction probability belongs to a certain depth distribution layer is set to 0.8. If the last item 0.842 in the above prediction results [0.021, 0.023, 0.017, ..., 0.842] is greater than 0.8, the pixel or feature point [h, w] can be determined to be located in Layer 41. This pixel or feature point is considered invalid pixel data, and the corresponding series of view cone points are deleted to obtain a sparse view cone point cloud, which is the target pixel data.

[0092] Furthermore, as a specific embodiment of the present invention, step S21: training the depth prediction model further includes:

[0093] S213, obtaining a depth prediction result of the pixel data training sample based on the trained depth prediction model;

[0094] S214, calculating a loss value between the depth prediction result and the true value based on a preset loss function;

[0095] S215 , reversely iteratively train the depth prediction model based on the loss value.

[0096] Specifically, when training the depth prediction model using pixel data training samples, in step S213, the depth prediction model is used to predict the pixel data training samples to generate a depth prediction result. In step S214, a loss function is used to calculate the loss between the depth prediction result and the true value. After the loss value is obtained, in step S215, the depth prediction model is trained multiple times in reverse iterations using the loss value, thereby further improving the accuracy of the depth prediction model prediction results.

[0097] Furthermore, as a specific embodiment of the present invention, when the depth prediction model is trained with a pixel input whose one-hot true value is [0, 0, 0, …, 0, 1], its prediction result is [0.021, 0.023, 0.017, …, 0.842]. The loss value between the true value and the prediction result is calculated using the Binary cross entropy method. After multiple back-propagation iterations of the depth prediction model, the depth prediction value will become closer and closer to the true value.

[0098] Furthermore, as a specific embodiment of the present invention, step S211, based on the view cone point cloud space model, labels the preset pixel data training samples, including: obtaining the vehicle laser point cloud information; obtaining the depth truth information of the pixel data training samples based on the laser point cloud information. In actual operation, when performing truth labeling on the pixel data training samples, the detected point cloud information can be extracted from the vehicle-mounted laser radar, and the distance data in the point cloud information can be considered as the depth truth information. The pixel data training samples can be labeled with the depth truth by mapping the laser point cloud information to the pixel data training samples. Labeling the depth truth information by the laser radar facilitates the automatic labeling of the training samples, thereby improving the convenience of sample production.

[0099] Further, refer to Figure 8 As a specific embodiment of the present invention, step S3: using the deep learning model to process the target pixel data to generate a top-view feature map includes:

[0100] S31, inputting the target pixel data into the deep learning model to obtain a content feature map of the target pixel data and a depth distribution probability corresponding to each feature point in the content feature map;

[0101] S32, generating a view cone point cloud feature based on the content feature map and the depth distribution probability calculation;

[0102] S33: Map the view cone point cloud features to a top view centered on the vehicle to obtain the top view feature map.

[0103] Specifically, after filtering out the sparse target pixel data in step S2, the target pixel data is input into the deep learning model in step S31, resulting in a content feature map corresponding to the target pixel data and the depth distribution probability corresponding to each feature point in the content feature map. In step S32, the view frustum point cloud features are calculated based on the content feature map obtained in step S31 and its corresponding depth distribution probability. Finally, in step S33, the view frustum point cloud features are mapped to a top-down view centered on the vehicle to obtain the final top-down feature map.

[0104] Furthermore, as a specific implementation of the present invention, step S31 can specifically be: using a deep learning model to process each image, and output two results: one is the content feature map after downsampling, and the other is the depth distribution probability corresponding to each feature point in the feature map.

[0105] Furthermore, as a specific implementation of the present invention, step S32 may specifically be: performing an outer product operation on the content feature map obtained in step S31 and the depth distribution probability value, thereby adding depth information to the content feature, thereby obtaining the view frustum point cloud feature.

[0106] Furthermore, as a specific implementation of the present invention, step S33 can be specifically as follows: through the internal and external parameters of the camera, combined with the position of each view cone point cloud in the vehicle coordinate system, the view cone point cloud features are placed at the corresponding position of the top view centered on the vehicle, thereby obtaining the top view feature map.

[0107] Furthermore, as a specific embodiment of the present invention, step S3: using the deep learning model to process the target pixel data to generate a top-view feature map, further includes:

[0108] Step S34 , summing up the multiple view cone point cloud features mapped to the same position. Specifically, when the view cone point cloud feature is placed at a corresponding position in the top view centered on the vehicle, the multiple view cone point cloud features mapped to the same position are summed up.

[0109] The embodiments of the present invention reduce the computational complexity of constructing a multi-view camera top-view feature map in the following aspects:

[0110] First, the present invention defines a view cone point cloud space model that includes an invalid depth layer and multiple valid depth layers. When modeling the view cone point cloud space, the valid and invalid view cone depth distribution ranges are clearly divided, and an invalid depth layer representing the outside of the valid view cone is specially set on the view cone depth distribution. For pixels or feature points located outside the valid view cone, their true values ​​are classified as belonging to the invalid depth layer. During depth prediction, if the depth distribution prediction result of a pixel or feature point is in the invalid depth layer, all view cone point clouds corresponding to this pixel or feature point are deleted, and they no longer participate in the subsequent calculation steps of constructing the overhead feature map, thereby simplifying the computational complexity of the subsequent calculation steps and improving construction efficiency.

[0111] Second, the effective depth layer in the view frustum point cloud spatial model is updated based on the height information of the foreground of interest. The height range of the foreground of interest is calculated and used as a basis to set the effective range of the view frustum point cloud in the height direction. Points outside the height range are deleted and excluded from the subsequent calculation of the top-down feature map. This further simplifies the computational complexity of subsequent calculation steps and improves construction efficiency.

[0112] An embodiment of the present invention discloses a system for constructing a multi-view camera bird's-eye view feature map.

[0113] A system for constructing a multi-view camera bird's-eye view feature map includes a data acquisition module, a space modeling module, a network training module, a pixel filtering module, and a result generation module.

[0114] The data acquisition module is connected to the cameras at each viewing angle on the vehicle and is used to obtain the imaging pixel data of each camera; the module is connected to the cameras at each viewing angle on the vehicle through the data interface data, reads the imaging pixel data of each camera and sends it to the subsequent module for processing.

[0115] The spatial modeling module is used to build a view cone point cloud spatial model. In the spatial modeling module, a regressible view cone point cloud spatial model is defined to filter and select pixel data.

[0116] The network training module is used to train deep learning models.

[0117] The pixel filtering module uses the deep learning model to filter invalid pixel data based on the cone point cloud space model to generate valid pixel data, which is used to filter out target pixel data located in the effective depth and effective height receptive field based on the cone point cloud space model.

[0118] The result generation module is used to generate and output a bird's-eye view feature map using the deep learning model.

[0119] The embodiment of the present invention also discloses a computer-readable storage medium.

[0120] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for constructing a multi-view camera overhead feature map. The computer-readable storage medium may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. The computer program includes computer program code. The computer program code may be in source code form, object code form, an executable file, or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying a computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for constructing a multi-view camera top-view feature map, applied to a vehicle, wherein the vehicle is equipped with multiple view cameras, each of which covers the surrounding area of ​​the vehicle. include: Constructing a view frustum point cloud space model including an invalid depth layer and multiple valid depth layers, including: determining an effective depth receptive field based on a preset depth distribution range, and dividing the effective depth receptive field into multiple effective depth layers based on a preset interval distance; determining an invalid depth receptive field outside the range of the effective depth receptive field, the invalid depth receptive field being the invalid depth layer; Acquiring and filtering invalid pixel data located in the invalid depth layer using a deep learning model to obtain target pixel data, including: generating a depth prediction value for each pixel data in the camera imaging based on a pre-trained depth prediction model; filtering the invalid pixel data located in the invalid depth layer based on the depth prediction value to obtain the target pixel data; The target pixel data is processed using the deep learning model to generate a top-view feature map.

2. The method for constructing a multi-view camera top-view feature map according to claim 1, wherein: The constructing of the view frustum point cloud space model including an invalid depth layer and a plurality of valid depth layers further includes: The effective depth layer is screened and updated based on a preset height threshold.

3. The method for constructing a multi-view camera top view feature map according to claim 2, characterized in that: The altitude threshold includes an upper limit value of a fixed altitude and / or a lower limit value of a fixed altitude.

4. The method for constructing a multi-view camera top view feature map according to claim 2, wherein: The height threshold includes an upper limit value that changes with the depth value of the effective depth layer and / or a lower limit value that changes with the depth value of the effective depth layer.

5. The method for constructing a multi-view camera top-view feature map according to claim 1, wherein: The training depth prediction model includes: Annotating preset pixel data training samples based on the view cone point cloud spatial model; The depth prediction model is trained based on labeled pixel data training samples.

6. The method for constructing a multi-view camera top view feature map according to claim 5, characterized in that: described Training a depth prediction model also involves: Obtaining a depth prediction result of the pixel data training sample based on the trained depth prediction model; Calculating the loss value between the depth prediction result and the true value based on a preset loss function; The depth prediction model is trained iteratively in reverse based on the loss value.

7. The method for constructing a multi-view camera top-view feature map according to claim 5, characterized in that: The pixel data training samples preset based on the view cone point cloud space model annotation include: Obtain vehicle laser point cloud information; The depth truth information of the pixel data training sample is obtained based on the laser point cloud information.

8. The method for constructing a multi-view camera top-view feature map according to claim 1, wherein: The generating of a top-view feature map by processing the target pixel data using the deep learning model includes: Inputting the target pixel data into the deep learning model to obtain a content feature map of the target pixel data and a depth distribution probability corresponding to each feature point in the content feature map; Generate a view cone point cloud feature based on the content feature map and the depth distribution probability calculation; The view cone point cloud features are mapped to a top view centered on the vehicle to obtain the top view feature map.

9. The method for constructing a multi-view camera top-view feature map according to claim 8, wherein: The generating a top-view feature map by processing the target pixel data using the deep learning model further includes: Sum the features of multiple view frustum point clouds that are mapped to the same location.

10. A system for constructing a multi-view camera top-view feature map, applied to the method according to any one of claims 1 to 9, characterized in that: include: The data acquisition module is connected to the cameras at each viewing angle on the vehicle to obtain the imaging pixel data of each camera; Spatial modeling module, used to construct the view frustum point cloud spatial model; Network training module, used to train deep learning models; A pixel filtering module, which filters invalid pixel data based on the view cone point cloud spatial model using the deep learning model to generate valid pixel data; A result generation module is used to generate and output a top-view feature map using the deep learning model.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for constructing a multi-view camera top-view feature map as described in any one of claims 1 to 9 are implemented.

Images