Target detection method, device and equipment, medium and vehicle

Abstract

The invention relates to a target detection method, device and equipment, a medium and a vehicle. The method comprises the following steps: acquiring point cloud data and image data which are acquired by a radar and a camera at the same moment and respectively acquired in a vehicle surrounding environment; generating a point cloud original feature map under the BEV space based on the point cloud data; performing feature coding on the original feature map to generate a point cloud semantic feature map in a BEV space; encoding the image data into a first image semantic feature map under a camera coordinate system through an image feature encoder; projecting the first image semantic feature map to a BEV space to generate a second image semantic feature map in the BEV space; fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map; and inputting the fused feature map into a target detector to enable the target detector to output target information. According to the invention, the sensing effect and robustness of the target detection scheme can be improved by reducing the loss of the feature information in the fusion sensing process.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a target detection method, device, equipment, medium and vehicle. Background Art

[0002] Object detection technology has a wide range of applications in the field of autonomous driving. At present, many existing technical solutions use radar and visual camera post-fusion to detect objects. The target detection scheme of the post-fusion method in the prior art usually includes the following three steps: 1. The radar link outputs the detection target under the bird's eye view (BEV); 2. The camera link outputs the detection target under the bird's eye view (BEV); 3. Correlate the detection targets output by the radar link and the camera link to obtain the final target detection result.

[0003] Target detection based on post-fusion requires high accuracy of detection results of a single link. If the position of the target detected by the camera link or radar link is inaccurate, the association effect will be greatly affected, resulting in inaccurate final target detection results and unsatisfactory perception effect under BEV. Therefore, how to avoid the impact of post-fusion in the target detection process and provide accuracy and robustness of the target detection solution has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a target detection method, device, equipment, medium and vehicle.

[0005] In a first aspect, the present disclosure provides a target detection method, comprising:

[0006] Obtain point cloud data and image data of the vehicle surrounding environment respectively obtained by the radar and camera at the same time;

[0007] Generate a point cloud original feature map in BEV space based on the point cloud data;

[0008] Performing feature encoding on the original feature map of the point cloud to generate a semantic feature map of the point cloud in the BEV space;

[0009] Encoding the image data into a first image semantic feature map in a camera coordinate system through an image feature encoder;

[0010] Projecting the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space;

[0011] Fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map;

[0012] The fused feature map is input into a target detector so that the target detector outputs target information.

[0013] Optionally, generating a point cloud original feature map in a BEV space based on the point cloud data includes:

[0014] Dividing a range interval of a preset size in the BEV space and gridding the range interval according to a preset number of grids to obtain a grid diagram of a preset size;

[0015] The point cloud data is filled into the grid map to generate the original feature map of the point cloud.

[0016] Optionally, the point cloud data includes position information, speed information and amplitude information of each point;

[0017] Filling the point cloud data into the grid map to obtain the original feature map of the point cloud includes:

[0018] According to the position information of each point in the point cloud data, searching for a grid at a corresponding position in the grid map;

[0019] The speed information and amplitude information of each point in the point cloud data are filled into the grid at the corresponding position to obtain the original feature map of the point cloud.

[0020] Optionally, projecting the first image semantic feature map into a BEV space to generate a second image semantic feature map in the BEV space includes:

[0021] According to the position of each grid in the grid map, searching for a semantic feature corresponding to the position of the grid in the first image semantic feature map;

[0022] The semantic features are filled into the grid to generate the second image semantic feature map.

[0023] Optionally, searching, according to the position of each grid in the grid map, for a semantic feature corresponding to the position of the grid in the first image semantic feature map includes:

[0024] According to the position of each grid in the grid image, searching the image data for a pixel point corresponding to the position of the grid;

[0025] The semantic feature of the pixel point in the first image semantic feature map is determined as the semantic feature corresponding to the position of the grid.

[0026] Optionally, the fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map includes:

[0027] The point cloud semantic feature map and the second image semantic feature map are stacked along the feature dimension to generate the fused feature map.

[0028] In a second aspect, the present disclosure provides a target detection device, comprising:

[0029] An acquisition module is used to obtain point cloud data and image data obtained by the radar and camera respectively from the vehicle surrounding environment at the same time;

[0030] A point cloud processing module, used to generate a point cloud original feature map in the BEV space based on the point cloud data;

[0031] A point cloud encoding module, used for encoding the original feature map of the point cloud to generate a semantic feature map of the point cloud in the BEV space;

[0032] An image encoding module, used for encoding the image data into a first image semantic feature map in a camera coordinate system through an image feature encoder;

[0033] A projection module, used for projecting the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space;

[0034] A fusion module, used for fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map;

[0035] The target detection module is used to input the fused feature map into the target detector so that the target detector outputs target information.

[0036] Optionally, when the point cloud processing module generates a point cloud original feature map in the BEV space based on the point cloud data, it is specifically used to divide the range interval of a preset size in the BEV space and grid the range interval according to a preset number of grids to obtain a grid map of a preset size; fill the point cloud data into the grid map to generate the point cloud original feature map.

[0037] Optionally, the point cloud data includes position information, speed information and amplitude information of each point;

[0038] When the point cloud processing module fills the point cloud data into the grid map to obtain the original feature map of the point cloud, it is specifically used to search for the grid at the corresponding position in the grid map according to the position information of each point in the point cloud data; and fills the speed information and amplitude information of each point in the point cloud data into the grid at the corresponding position to obtain the original feature map of the point cloud.

[0039] Optionally, when the projection module projects the first image semantic feature map into the BEV space to generate the second image semantic feature map in the BEV space, it is specifically used to search for the semantic feature corresponding to the position of the grid in the first image semantic feature map according to the position of each grid in the grid map; and fill the semantic feature into the grid to generate the second image semantic feature map.

[0040] Optionally, when the projection module searches for the semantic feature corresponding to the position of the grid in the first image semantic feature map according to the position of each grid in the grid map, the projection module is specifically used to search for the pixel point corresponding to the position of the grid in the image data according to the position of each grid in the grid map; and determine the semantic feature of the pixel point in the first image semantic feature map as the semantic feature corresponding to the position of the grid.

[0041] Optionally, when the fusion module fuses the point cloud semantic feature map and the second image semantic feature map to generate the fused feature map, the fusion module is specifically used to stack the point cloud semantic feature map and the second image semantic feature map along a feature dimension to generate the fused feature map.

[0042] In a third aspect, the present disclosure provides a computer device, comprising:

[0043] Memory;

[0044] Processor; and

[0045] Computer programs;

[0046] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in any one of the first aspects.

[0047] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first aspects.

[0048] In a fifth aspect, the present disclosure provides a vehicle, comprising the target detection device as described in the second aspect and / or the computer device as described in the third aspect.

[0049] The target detection method, device, equipment, medium and vehicle provided by the present invention obtain point cloud data and image data respectively obtained by radar and camera collecting the vehicle surrounding environment at the same time, use the point cloud data to generate a point cloud original feature map in BEV space, and then perform feature encoding on the original feature map to generate a point cloud semantic feature map in BEV space, and use an image feature encoder to encode the image data into a first image semantic feature map, and then project the first image semantic feature map into BEV space to generate a second image semantic feature map in BEV space, and then fuse the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map, and finally detect the fused feature map through a target detector to output target information. Compared with target detection based on post-fusion, the loss of feature information in the fusion perception process is reduced, so that the features perceived by the radar and camera in the fusion perception process are fully utilized, thereby effectively improving the perception effect and robustness of the target detection scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 A flow chart of a target detection method provided by an embodiment of the present disclosure;

[0053] Figure 2 A schematic diagram of the structure of a target detection device provided in an embodiment of the present disclosure;

[0054] Figure 3 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0057] Target detection based on post-fusion requires high accuracy of detection results of a single link. If the position of the target detected by the camera link or radar link is inaccurate, the association effect will be greatly affected, resulting in inaccurate final target detection results, making the perception effect under BEV unsatisfactory. Moreover, the post-fusion method discards a wealth of intermediate features, such as visual information such as the texture and shape of the target and physical information such as radar reflection signals. These intermediate features are of great significance for improving the accuracy and robustness of target detection. Therefore, discarding these features in the post-fusion process will limit the performance upper limit of target detection. The above defects will increase the risk of false detection and missed detection of target detection, reduce the overall target detection performance, and affect the accuracy and robustness of the target detection scheme of the post-fusion method.

[0058] Figure 1 This is a flow chart of a target detection method provided in an embodiment of the present disclosure. The method can be executed by a target detection device. The target detection device can be implemented in software and / or hardware. The target detection device can be configured in an electronic device, such as a server or a terminal, wherein the terminal includes a vehicle computer, a vehicle-mounted computer, etc. The target detection method includes the following steps:

[0059] S101, obtaining point cloud data and image data respectively obtained by radar and camera collecting vehicle surrounding environment at the same time.

[0060] During the driving process of the vehicle, the radar and camera mounted on the vehicle simultaneously scan and photograph the environment around the vehicle, generate point cloud data and image data of the environment around the vehicle at the same time, and transmit the point cloud data and image data to the vehicle computer for subsequent fusion perception processing. Among them, the relative position relationship between the radar and the camera on the vehicle is known, and in addition, the coordinates of the points in the point cloud collected by the radar and the pixels in the image collected by the camera in the vehicle coordinate system are also known; the camera and radar are commonly used perception sensors that constitute the vehicle perception system. The camera in the embodiment of the present disclosure can be, for example, a wide-angle camera, a surround view camera, etc., and the radar can be, for example, a millimeter-wave radar, a laser radar, etc. The point cloud data generated by the millimeter-wave radar is usually sparse compared to other types of sensors (such as laser radar), but the performance is often more stable than the laser radar under severe weather conditions, and can accurately measure the position and speed of obstacles.

[0061] S102: Generate a point cloud original feature map in the BEV space based on the point cloud data.

[0062] The vehicle computer generates a feature map in the BEV space by performing parameterized operations on the point cloud data. First, the point cloud data of the vehicle's surrounding environment is preprocessed, such as removing ground points, noise filtering, data format conversion, and coordinate system conversion. Among them, the conversion of the coordinate system point cloud data from the three-dimensional coordinate system to the BEV coordinate system can be achieved by projecting the z-axis to the xy plane. The converted point cloud data will have two-dimensional coordinates, and the height information of each point can be retained as its height attribute value. Then, the point cloud data in the BEV coordinate system is gridded, and the point cloud data in each grid is counted and processed. Finally, the eigenvalues ​​of each grid are combined into a multi-channel image, which is the original feature map of the point cloud in the BEV space. Each channel represents a feature, and the feature includes the attributes of the point cloud data, such as the speed of each point, the reflection intensity, etc.

[0063] In one embodiment, before generating the original feature map of the point cloud in the BEV space based on the point cloud data, it also includes: dividing the range interval of a preset size in the BEV space and gridding the range interval according to a preset number of grids to obtain a grid map of a preset size.

[0064] After preprocessing, the point cloud data includes the position information (x, y), speed information (v) and amplitude information (f) of each point, where the amplitude information represents the reflection intensity of the point. The vehicle computer grids a range interval of a preset size in the BEV space to generate a grid map of size H*W, where H and W represent the settings according to specific tasks, and the grid map includes a preset number of grids. H*W is a preset size set according to a specific task, and the preset number of grids is also set according to the specific task. For example, the scanning distance of the radar is 80m, while the distance that the camera can theoretically shoot is close to infinity. You can take half of the radar scanning distance, set the width and height of the preset size to 40, and set the preset number of grids to 400.

[0065] Correspondingly, a point cloud original feature map in the BEV space is generated based on the point cloud data, including: filling the point cloud data into the grid map to generate the point cloud original feature map.

[0066] After the vehicle computer divides the BEV space into a grid map, it fills the information corresponding to each point in the point cloud data into the corresponding grid in the grid map to generate the original feature map of the point cloud.

[0067] In this case, the point cloud data is filled into the grid map to obtain the original feature map of the point cloud, including: searching for the grid at the corresponding position in the grid map according to the position information of each point in the point cloud data; filling the speed information and amplitude information of each point in the point cloud data into the grid at the corresponding position to obtain the original feature map of the point cloud.

[0068] According to the position information (x, y) of each point in the point cloud data, the corresponding position of each point in the grid map and the grid where each point is located are found. For example, the position information (x, y) of each point can be rounded, and the grid containing the position information in the grid map can be found according to the rounded position information to achieve the effect of discretizing the position information of each point. After finding the grid corresponding to the point in the grid map according to the position information of the point, the speed information (v) and amplitude information (f) of the point are filled into the grid at the corresponding position to obtain the original feature map of the point cloud. Since the value of the amplitude information is usually large, in order to facilitate calculation, the amplitude information can be logarithmized when filling the amplitude information to obtain the intensity value of each point in the point cloud data, and then the intensity value and speed information are filled into the grid at the corresponding position. In this process, it may happen that the position information of multiple points corresponds to the position of the same grid in the grid map, so that the speed information and amplitude information of multiple points will be filled in the same grid. At this time, the speed information and amplitude information corresponding to the grid can be obtained by using some aggregation operations on the speed information and amplitude information of multiple points. The aggregation operation includes taking the average of the speed information and amplitude information of multiple points as the speed information and amplitude information corresponding to the grid, or taking the maximum value of the speed information and amplitude information of multiple points as the speed information and amplitude information corresponding to the grid. The original feature map of the point cloud generated in this way contains 4 feature dimensions, including the horizontal axis coordinate X of each grid in the BEV space, the vertical axis coordinate Y of each grid in the BEV space, the speed information V of each grid, and the amplitude information F of each grid, wherein the horizontal axis coordinate X and the vertical axis coordinate Y of each grid in the BEV space can be uniformly represented by the center point of the grid, that is, the coordinates (X, Y) of each grid in the original feature map of the point cloud take the coordinates of the center point of the grid.

[0069] The disclosed embodiment divides the range interval in the BEV space and grids it to obtain a grid map in the BEV space, then searches for the grid at the corresponding position in the grid map according to the position information of each point in the point cloud data, and fills the speed information and amplitude information of the point into the grid at the corresponding position to generate an original feature map of the point cloud in the BEV space, so that the features contained in the point cloud data and the image data can be fused in the BEV space later.

[0070] S103: Perform feature encoding on the original feature map to generate a point cloud semantic feature map in the BEV space.

[0071] Feature encoding of the original feature map is to map the original feature space to a high-dimensional semantic feature space through a certain mapping relationship, so that the encoded features have stronger expressive power and better generalization. Feature encoding can be achieved through an encoder. Common encoders include Convolutional Neural Networks (CNN), which process data with a similar grid structure (such as images). Feature encoding of the original feature map to generate a high-dimensional semantic feature map in the BEV space allows the machine to learn more information from the point cloud data. For example, the original feature map of the point cloud contains 4 feature dimensions, and the encoded semantic feature map of the point cloud can have 128 or more feature dimensions.

[0072] S104: Encode the image data into a first image semantic feature map in a camera coordinate system through an image feature encoder.

[0073] The image data contains rich features, where each pixel has values ​​of three color channels (usually red, green, and blue channels, i.e., RGB), so each pixel has three feature dimensions. Similar to S103, the image data is encoded into an image semantic feature map through feature encoding, where the image feature encoder can also be a convolutional neural network.

[0074] Because the image data to be encoded is image data in the camera coordinate system, the image semantic feature map is the semantic feature map in the camera coordinate system, recorded as the first image semantic feature map. In order to perform feature fusion in the BEV space later, it is necessary to convert the semantic feature map in the camera coordinate system into the semantic feature map in the BEV space.

[0075] S105. Project the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space.

[0076] When the first image semantic feature map is projected into the BEV space, the vehicle computer obtains the semantic features of each pixel in the first image semantic feature map by performing a semantic segmentation task on the image data, and then converts the coordinates of the pixel in the pixel coordinate system to the coordinates in the camera coordinate system, and then converts the coordinates in the camera coordinate system to the coordinates in the vehicle coordinate system to obtain the pixel in the vehicle coordinate system. Then the pixel in the vehicle coordinate system is projected into the BEV space. Since the three-dimensional point is transformed into a two-dimensional point, the height value of the pixel needs to be set during projection, which can usually be achieved by ignoring the height component and setting it to a constant. After the pixel is projected into the BEV space to obtain a two-dimensional point, the semantic features of the pixel are combined with the two-dimensional point to form a two-dimensional group, and then the two-dimensional point falling in the grid map is determined according to the position of the grid in the grid map pre-divided in the above embodiment, and then the semantic features in the two-dimensional group to which the two-dimensional point belongs are filled into the grid according to the grid where the two-dimensional point falls, to obtain the second image semantic feature map in the BEV space. When multiple two-dimensional points fall on the same grid, the aggregation operation is also used to obtain the semantic features of the grid. The aggregation operation includes taking the average or maximum value of the semantic features in the two-dimensional group to which the multiple two-dimensional points belong as the semantic features of the grid.

[0077] In one embodiment, the first image semantic feature map is projected into the BEV space to generate a second image semantic feature map in the BEV space, including: searching for the semantic feature corresponding to the position of each grid in the grid map in the first image semantic feature map; and filling the semantic feature into the grid to generate a second image semantic feature map.

[0078] When generating the original feature map of the point cloud, the position of the grid in the grid map represents the position of the point cloud in the BEV space. The point cloud is generated by the radar scanning the environment around the vehicle, that is, the point cloud can represent the information of the environment around the vehicle. In order to facilitate the subsequent fusion of the semantic feature maps of the point cloud and the image in the BEV space, the semantic feature maps of the point cloud and the image in the BEV space need to be aligned. In other words, the semantic features of the point cloud and the semantic features of the image at a certain position in the BEV space need to be the semantic features of the same position in the real environment around the vehicle.

[0079] For example, point A1 with position information (x1, y1) in the point cloud data is generated by radar scanning a cyclist m at position A in the vehicle environment. In S102, the vehicle computer fills the speed information and amplitude information of point A1 into the grid a at the corresponding position in the grid map based on the position information (x1, y1), and produces the original feature map of the point cloud. Correspondingly, in the point cloud semantic feature map generated in S103, the point cloud semantic features contained in grid a include the semantic features of point A1. It can be understood that the semantic features of point A1 learned by machine can help it identify the cyclist m. At the same moment, the camera shoots the cyclist m at position A and generates pixel B1 in the image data. Therefore, when projecting the first semantic feature map into the BEV space, the semantic features encoded according to pixel B1 of cyclist m in the first semantic feature map need to be projected to the position of grid a. In this way, the position of grid a in the BEV space includes the semantic features of point A1 and pixel B1, and the semantic features of point A1 and pixel B1 are respectively sensed by the radar and the camera for cyclist m at position A. Therefore, the features fused by the semantic features of point A1 and pixel B1 in grid a of machine learning can enable it to more accurately identify cyclist m.

[0080] Based on the above principle, when the vehicle computer projects the first image semantic feature map into the BEV space, it searches for the semantic feature corresponding to the position of each grid in the grid map in the first image semantic feature map, and then fills the corresponding semantic feature into the grid to generate a second image semantic feature map aligned with the point cloud semantic feature map in the BEV space, so that the point cloud semantic features and image semantic features contained in the same grid in the grid map are the semantic features obtained by the radar and camera perceiving the same position in the real vehicle environment.

[0081] In this case, according to the position of each grid in the grid map, the semantic feature corresponding to the position of the grid in the first image semantic feature map is searched, including: according to the position of each grid in the grid map, searching for the pixel point corresponding to the position of the grid in the image data; determining the semantic feature of the pixel point in the first image semantic feature map as the semantic feature corresponding to the position of the grid.

[0082] When the vehicle computer searches for the semantic feature corresponding to the position of the grid in the first image semantic feature map, for each grid, it can search for the pixel point corresponding to the position in the image data according to the position of the grid, and then search for the semantic feature of the pixel point in the first image semantic feature map through the pixel point, and then fill the semantic feature of the pixel point into the grid. After repeating this process to fill the semantic features of the pixel points in each grid, a second semantic feature map is generated.

[0083] When searching for the pixel points corresponding to the position of each grid in the image data, one implementation method is as described above, by converting the coordinate points in the pixel coordinate system into the pixel points in the vehicle coordinate system, and then projecting the pixel points in the vehicle coordinate system into the BEV space, obtaining the coordinates of each pixel point in the image data in the BEV space, and determining which grid of the grid diagram the coordinates of each pixel point in the BEV space fall into, thereby determining the pixel points corresponding to the position of the grid. Another implementation method may be to project the grid diagram range into the vehicle coordinate system. Since the two-dimensional coordinate range is converted into a three-dimensional coordinate range, a preset height range, that is, a z-axis range, may be added. The preset z-axis range may depend on the application scenario, such as twice the vehicle height. Then, the grid diagram range in the vehicle coordinate system is converted into the grid diagram range in the pixel coordinate system, and then the image data pixel points enclosed by the range of each grid in the grid diagram range in the pixel coordinate system are determined, thereby determining the pixel points corresponding to the position of the grid, so that all the pixel points in the image data do not need to be projected into the BEV space to find the pixel points corresponding to the position of the grid.

[0084] The disclosed embodiment searches for the pixel point corresponding to the position of the grid in the image data for each grid in the grid map, then searches for the semantic feature of the pixel point in the first image semantic feature map, and finally fills the semantic feature of the pixel point into the grid, thereby projecting the first image semantic feature map into the BEV space and generating a second image semantic feature map in the BEV space that is aligned with the point cloud semantic feature map.

[0085] S106 , fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map.

[0086] The vehicle computer generates a fused feature map by superimposing the feature dimensions of the point cloud semantic feature map and the second image semantic feature map. For example, if the feature dimension of the point cloud semantic feature map is N1 and the feature dimension of the second image semantic feature map is N2, then the feature dimension of the generated fused feature map is N1+N2. The fused feature map includes features obtained by the radar and camera perceiving the same position in the real vehicle environment, which can improve the perception effect of target detection.

[0087] In one embodiment, fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map includes: stacking the point cloud semantic feature map and the second image semantic feature map along a feature dimension to generate a fused feature map.

[0088] In the disclosed embodiment, simple feature dimension stacking can be achieved by splicing the point cloud semantic feature map and the second image semantic feature map along the feature dimension. The feature dimension of the spliced ​​fusion feature map is also the sum of the feature dimensions of the point cloud semantic feature map and the second image semantic feature map. In addition, the point cloud semantic feature map and the second image semantic feature map can be fused by point-by-point convolution.

[0089] S107, input the fused feature map into the target detector so that the target detector outputs target information.

[0090] The vehicle computer inputs the fused feature map into the target detector, which learns and recognizes the semantic features in the fused feature map and outputs the detected target information.

[0091] Commonly used target detectors include Faster R-CNN, RetinaNet, and Libra R-CNN. The disclosed embodiment uses the method for generating a fused feature map introduced in the previous embodiment to process radar point cloud data and image data as original training data, obtain a fused feature map corresponding to the original training data, and use the target detection true value annotated by the original training data as the target detection true value corresponding to the fused feature map. Then, the fused feature map corresponding to the original training data and its corresponding target detection true value are used as training data for the target detector, and the target detector is retrained using the training data to obtain a target detector in the disclosed embodiment that can learn and recognize semantic features in the fused feature map.

[0092] The disclosed embodiment acquires point cloud data and image data respectively obtained by radar and camera collecting the vehicle surrounding environment at the same time, and uses the point cloud data to generate an original feature map of the point cloud in the BEV space, and then performs feature encoding on the original feature map to generate a point cloud semantic feature map in the BEV space, and uses an image feature encoder to encode the image data into a first image semantic feature map, and then projects the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space, and then fuses the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map, and finally detects the fused feature map through a target detector to output target information. Compared with target detection based on post-fusion, the loss of feature information in the fusion perception process is reduced, so that the features perceived by the radar and camera in the fusion perception process are fully utilized, thereby effectively improving the perception effect and robustness of the target detection scheme.

[0093] Figure 2 Schematic diagram of the structure of the target detection device provided in the embodiment of the present disclosure. The target detection device may be a terminal as in the above embodiment, or the target detection device may be a component or assembly in the terminal. The target detection device provided in the embodiment of the present disclosure may execute the processing flow provided in the target detection method embodiment, such as Figure 2 As shown, the target detection device 200 includes:

[0094] An acquisition module 201 is used to acquire point cloud data and image data respectively obtained by radar and camera collecting vehicle surrounding environment at the same time;

[0095] A point cloud processing module 202 is used to generate a point cloud original feature map in the BEV space based on the point cloud data;

[0096] The point cloud encoding module 203 is used to perform feature encoding on the original feature map to generate a point cloud semantic feature map in the BEV space;

[0097] An image encoding module 204, configured to encode the image data into a first image semantic feature map in a camera coordinate system through an image feature encoder;

[0098] A projection module 205 is used to project the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space;

[0099] A fusion module 206, used for fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map;

[0100] The target detection module 207 is used to input the fused feature map into the target detector so that the target detector outputs target information.

[0101] In some embodiments, when the point cloud processing module 202 generates a point cloud original feature map in the BEV space based on the point cloud data, it is specifically used to divide the range interval of a preset size in the BEV space and grid the range interval according to a preset number of grids to obtain a grid map of a preset size; fill the point cloud data into the grid map to generate a point cloud original feature map.

[0102] In some embodiments, the point cloud data includes position information, speed information and amplitude information of each point; when the point cloud processing module 202 fills the point cloud data into the grid map to obtain the original feature map of the point cloud, it is specifically used to find the grid at the corresponding position in the grid map according to the position information of each point in the point cloud data; fill the speed information and amplitude information of each point in the point cloud data into the grid at the corresponding position to obtain the original feature map of the point cloud.

[0103] In some embodiments, when the projection module 205 projects the first image semantic feature map into the BEV space to generate the second image semantic feature map in the BEV space, it is specifically used to search for the semantic feature corresponding to the position of each grid in the grid map in the first image semantic feature map according to the position of the grid in the grid map; fill the semantic feature into the grid to generate the second image semantic feature map.

[0104] In some embodiments, when the projection module 205 searches for the semantic feature corresponding to the position of the grid in the first image semantic feature map according to the position of each grid in the grid map, it is specifically used to search for the pixel point corresponding to the position of the grid in the image data according to the position of each grid in the grid map; and determine the semantic feature of the pixel point in the first image semantic feature map as the semantic feature corresponding to the position of the grid.

[0105] In some embodiments, when the fusion module 206 fuses the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map, it is specifically used to stack the point cloud semantic feature map and the second image semantic feature map along the feature dimension to generate a fused feature map.

[0106] Figure 2 The target detection device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0107] Figure 3 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 3 , which shows a structural schematic diagram of an electronic device 300 suitable for implementing the embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to implement the target detection method of the embodiment described in the present disclosure. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0109] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0110] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the target detection method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0111] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0113] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0116] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0117] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0118] The embodiments of the present disclosure also provide a vehicle, which includes the target detection device and / or computer equipment as described in the above embodiments.

[0119] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A target detection method, characterized in that: include: Obtain point cloud data and image data of the vehicle surrounding environment respectively obtained by the radar and camera at the same time; Generate a point cloud original feature map in BEV space based on the point cloud data; Performing feature encoding on the original feature map of the point cloud to generate a semantic feature map of the point cloud in the BEV space; Encoding the image data into a first image semantic feature map in a camera coordinate system through an image feature encoder; Projecting the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space; Fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map; The fused feature map is input into a target detector so that the target detector outputs target information.

2. The method according to claim 1, characterized in that Before generating the original feature map of the point cloud in the BEV space based on the point cloud data, the method further includes: Dividing a range interval of a preset size in the BEV space and gridding the range interval according to a preset number of grids to obtain a grid diagram of a preset size; The generating of the original feature map of the point cloud in the BEV space based on the point cloud data comprises: The point cloud data is filled into the grid map to generate the original feature map of the point cloud.

3. The method according to claim 2, characterized in that The point cloud data includes the position information, speed information and amplitude information of each point; Filling the point cloud data into the grid map to generate the point cloud original feature map includes: According to the position information of each point in the point cloud data, searching for a grid at a corresponding position in the grid map; The speed information and amplitude information of each point in the point cloud data are filled into the grid at the corresponding position to obtain the original feature map of the point cloud.

4. The method according to claim 2, characterized in that The projecting the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space includes: According to the position of each grid in the grid map, searching for a semantic feature corresponding to the position of the grid in the first image semantic feature map; The semantic features are filled into the grid to generate the second image semantic feature map.

5. The method according to claim 4, characterized in that The searching, according to the position of each grid in the grid map, for a semantic feature corresponding to the position of the grid in the first image semantic feature map comprises: According to the position of each grid in the grid image, searching the image data for a pixel point corresponding to the position of the grid; The semantic feature of the pixel point in the first image semantic feature map is determined as the semantic feature corresponding to the position of the grid.

6. The method according to claim 1, characterized in that The fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map includes: The point cloud semantic feature map and the second image semantic feature map are stacked along the feature dimension to generate the fused feature map.

7. A target detection device, characterized in that: include: An acquisition module is used to obtain point cloud data and image data obtained by the radar and camera respectively from the vehicle surrounding environment at the same time; A point cloud processing module, used to generate a point cloud original feature map in the BEV space based on the point cloud data; A point cloud encoding module, used for encoding the original feature map of the point cloud to generate a semantic feature map of the point cloud in the BEV space; An image encoding module, used for encoding the image data into a first image semantic feature map in a camera coordinate system through an image feature encoder; A projection module, used for projecting the first image semantic feature map into the BEV space to generate a second image semantic feature map in the BEV space; A fusion module, used for fusing the point cloud semantic feature map and the second image semantic feature map to generate a fused feature map; The target detection module is used to input the fused feature map into the target detector so that the target detector outputs target information.

8. A computer device comprising: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A vehicle, characterized in that: It comprises the target detection device as claimed in claim 7 and / or the computer device as claimed in claim 8.

Images