Object detection method and system based on multiple modes and medium
By adopting a multimodal fusion method in object detection technology and combining the features of laser point clouds and environmental images to predict objects, the problem of poor detection results in the prior art is solved, and higher detection accuracy and effect are achieved.
Patent Information
- Application Number
- CN202411676514.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-03-14
AI Technical Summary
In actual environment, the existing object detection technology has the potential to be blocked because the target is easily blocked, and the laser point cloud data contains noise and the density of different regions varies greatly, resulting in poor detection effect.
The multimodal object detection method is adopted to obtain laser point cloud data and multiple environmental images, and preprocess the pillar data and real environment images. The object prediction is performed by combining the feature fusion model, the object information and its confidence are output, and the detection results are screened based on the confidence.
By fusing the characteristics of laser point clouds and environmental images, the accuracy of object detection is improved, the detection effect is enhanced, and the limitations of single-modal detection are overcome.
Smart Images
Figure CN119625475A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of object detection technology, and in particular to a multimodal object detection method, system and medium. Background Art
[0002] Object detection technology refers to determining the location of an object based on the collected information, and can be applied to scenes related to object detection, such as autonomous driving of cars. In the prior art, data analysis is usually performed based on the data collected by the LiDAR to determine the location of the object. However, in the actual environment, the detected target is easily blocked, and the collected laser point cloud not only contains noise, but also has density changes in different areas. These factors lead to poor object detection results. Therefore, how to improve the detection effect of object detection has become a technical problem that needs to be solved at present. Summary of the invention
[0003] In view of the above problems, the purpose of the present invention is to provide a multimodal object detection method, system and medium to improve the detection effect of object detection. The specific scheme is as follows:
[0004] In a first aspect, an embodiment of the present application provides a multimodal object detection method, the method comprising:
[0005] Acquire laser point cloud data collected within a preset time, and multiple environmental images; the multiple environmental images are environmental images collected at different viewing angles based on the current location;
[0006] Preprocessing the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images; at least part of the pillar data in the multiple pillar data includes the laser point cloud data;
[0007] Input the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features, and perform object prediction based on the fusion features, and output object information of objects contained in the current environment and the confidence of the object information; the fusion features include the features contained in the plurality of pillar data and the plurality of real environment images;
[0008] According to a preset comparison threshold and the confidence level of the object information, the object information of the objects contained in the current environment is screened to obtain an object detection result.
[0009] Optionally, the inputting the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features includes:
[0010] Inputting the multiple real environment images into the preset feature fusion model to obtain image features; the image features are features contained in the multiple real environment images;
[0011] Inputting the plurality of pillar data into the preset feature fusion model to obtain laser point cloud features; the laser point cloud features are features contained in the laser point cloud data;
[0012] Filling the laser point cloud feature based on the size of the image feature to obtain the laser point cloud feature to be merged; the size of the laser point cloud feature to be merged is the same as the size of the image feature;
[0013] The image features and the features of the laser point cloud to be spliced are combined to obtain fusion features.
[0014] Optionally, the inputting the multiple real environment images into the preset feature fusion model to obtain image features includes:
[0015] Inputting the multiple real environment images into the preset feature fusion model to obtain the features contained in the multiple real environment images;
[0016] According to a preset depth estimation network, depth prediction is performed on the multiple real environment images to obtain depth information of the multiple real environment images;
[0017] According to the depth information of the multiple real environment images, the features contained in the multiple real environment images are converted into a three-dimensional space to obtain the image features.
[0018] Optionally, the inputting the plurality of pillar data into the preset feature fusion model to obtain laser point cloud features includes:
[0019] Inputting the plurality of pillar data into the preset feature fusion model to obtain features of the plurality of pillar data;
[0020] splicing and arranging the features of the plurality of pillar data according to the centroids contained in the plurality of pillar data to obtain splicing features;
[0021] Performing a projective transformation on the stitching feature to obtain a pseudo image corresponding to the stitching feature;
[0022] The pseudo image is encoded and decoded to obtain the laser point cloud feature.
[0023] Optionally, the performing object prediction based on the fusion feature and outputting object information of the object contained in the current environment and the confidence of the object information includes:
[0024] Based on the fusion features, heat map prediction, offset prediction, scale prediction, center point prediction, direction prediction and speed prediction are performed respectively to obtain a comprehensive prediction result;
[0025] Generating a confidence level of the object information according to the comprehensive prediction result;
[0026] Using the comprehensive prediction result as object information of objects contained in the current environment;
[0027] Output object information of objects contained in the current environment and the confidence level of the object information.
[0028] Optionally, the preprocessing of the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images includes:
[0029] Divide the area where the laser point cloud data is located into a plurality of pillars, and use the data contained in the plurality of pillars as the plurality of pillar data; each of the plurality of pillars contains the same amount of data;
[0030] The distortion in the multiple environment images is removed to obtain multiple real environment images.
[0031] In a second aspect, an embodiment of the present application provides a multimodal object detection system, the system comprising:
[0032] An acquisition module is used to acquire laser point cloud data collected within a preset time, and multiple environmental images; the multiple environmental images are environmental images collected at different viewing angles based on the current location;
[0033] A processing module, used for preprocessing the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images; at least part of the pillar data in the multiple pillar data includes the laser point cloud data;
[0034] A prediction module, used for inputting the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features, and performing object prediction based on the fusion features to obtain object information of objects contained in the current environment and the confidence of the object information; the fusion features include features contained in the plurality of pillar data and the plurality of real environment images;
[0035] The screening module is used to screen the object information of the objects contained in the current environment according to a preset comparison threshold and the confidence of the object information to obtain an object detection result.
[0036] Optionally, the prediction module is specifically used to:
[0037] Inputting the multiple real environment images into the preset feature fusion model to obtain image features; the image features are features contained in the multiple real environment images;
[0038] Inputting the plurality of pillar data into the preset feature fusion model to obtain laser point cloud features; the laser point cloud features are features contained in the laser point cloud data;
[0039] Filling the laser point cloud feature based on the size of the image feature to obtain the laser point cloud feature to be merged; the size of the laser point cloud feature to be merged is the same as the size of the image feature;
[0040] The image features and the features of the laser point cloud to be spliced are combined to obtain fusion features.
[0041] Optionally, the prediction module is specifically used for:
[0042] Based on the fusion features, heat map prediction, offset prediction, scale prediction, center point prediction, direction prediction and speed prediction are performed respectively to obtain a comprehensive prediction result;
[0043] Generating a confidence level of the object information according to the comprehensive prediction result;
[0044] Using the comprehensive prediction result as object information of objects contained in the current environment;
[0045] Output object information of objects contained in the current environment and the confidence level of the object information.
[0046] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned multimodal-based object detection methods.
[0047] Compared with the prior art, this application has the following beneficial effects:
[0048] By obtaining the laser point cloud data collected within a preset time, and multiple environmental images, the laser point cloud data and the multiple environmental images are preprocessed to obtain multiple pillar data and multiple real environment images. The multiple pillar data and the multiple real environment images are input into the preset feature fusion model to obtain fusion features, and object prediction is performed based on the fusion features, and the object information of the objects contained in the current environment and the confidence of the object information are output. According to the preset comparison threshold and the confidence of the object information, the object information of the objects contained in the current environment is screened to obtain the object detection result. Since the fusion features in this application include the features contained in the multiple pillar data and the multiple real environment images, the advantages of the two-modal feature fusion are complementary, providing more effective data for the prediction of subsequent objects, thereby improving the accuracy of object detection and improving the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application, and a person skilled in the art can also obtain other drawings based on the provided drawings.
[0050] Figure 1 A flowchart of a method for multimodal object detection provided in an embodiment of the present application;
[0051] Figure 2 A schematic diagram of the installation of a laser radar and a standard single-view camera on a vehicle provided in an embodiment of the present application;
[0052] Figure 3 A schematic diagram of a pillar provided for an embodiment of the present application;
[0053] Figure 4 A schematic diagram of a training image provided in an embodiment of the present application;
[0054] Figure 5 A schematic diagram of the splicing and fusion of two-modal features provided in an embodiment of the present application;
[0055] Figure 6 A schematic diagram of the output structure of the object detection result part provided in an embodiment of the present application;
[0056] Figure 7 A schematic diagram of the model training process provided in the embodiment of the present application;
[0057] Figure 8 A schematic diagram of a flow chart of another method for multimodal object detection provided in an embodiment of the present application;
[0058] Fig. 9A schematic diagram of the structure of a system for multimodal object detection provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0060] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.
[0061] like Figure 1 As shown, an embodiment of the present application provides a multimodal object detection method, the method comprising:
[0062] S101: Acquire laser point cloud data collected within a preset time, and a plurality of environmental images; the plurality of environmental images are environmental images collected at different viewing angles based on the current location.
[0063] Laser point cloud data is collected by laser radar, and the laser point cloud data includes three-dimensional coordinates (x, y, z) and reflection intensity i. Environmental images of different perspectives are collected by standard single-view cameras to obtain environmental images of all angles at the current location, which is convenient for subsequent detection of objects of different perspectives. The timestamp corresponding to the collected data is recorded during the collection process, so that the laser point cloud data and multiple environmental images collected within the same preset time can be extracted from the collected data.
[0064] For example, when collecting data, if the laser radar collects laser point cloud data A at 12:01:01:01 milliseconds, the standard single-view camera collects environment images B and C at 12:01:01:01 milliseconds, and collects environment image D at 12:01:01:50 milliseconds. When the preset time is between 12:01:01:01:01 milliseconds and 12:01:01:50 milliseconds, since the timestamps corresponding to laser point cloud data A, environment image B, and environment image C are 12:01:01:01 milliseconds, and the timestamp corresponding to environment image D is 12:01:01:50 milliseconds, the acquired data includes laser point cloud data A, environment image B, environment image C, and environment image D. When the preset time is within 30 milliseconds from 12:01:01 to 12:01:01, since the timestamps corresponding to the laser point cloud data A, environmental image B, and environmental image C are 12:01:01:01, the acquired data include laser point cloud data A, environmental image B, and environmental image C. Thus, by setting the preset time range by yourself, the time difference between the acquired data can be controlled. By setting the preset time to the millisecond level, it can be avoided that the environmental image and laser point cloud data corresponding to a certain object are not in the same position due to the high-speed movement of a certain object. And when the preset time is a certain moment, the time when the laser point cloud data is acquired is exactly the same as the time when multiple environmental images are acquired, so that time synchronization is achieved and the accuracy of subsequent object recognition results is improved.
[0065] It should be noted that multiple environmental images can be collected by multiple standard single-view cameras located at different positions, so that multiple standard single-view cameras can collect environmental images at various angles in the current position. LiDAR and standard single-view cameras collect data at different fields of view (FOV). LiDAR and standard single-view cameras can be located at different heights, thereby avoiding the situation where, when the LiDAR and standard single-view camera are at the same height, the object behind is completely blocked by the object in front at this height and the relevant information of the blocked object cannot be collected, thereby increasing the amount of information collected about the objects in the current environment. Figure 2 As shown, taking a vehicle as an example, a laser radar 202 with a forward FOV of 120° and six standard single-view cameras 203 arranged symmetrically front to back are installed on the vehicle 201. The environmental images captured by the six standard single-view cameras 203 arranged symmetrically front to back are all environmental images around the vehicle.
[0066] S102: Preprocess the laser point cloud data and multiple environment images to obtain multiple pillar data and multiple real environment images; at least part of the multiple pillar data includes laser point cloud data.
[0067] The laser point cloud data is divided into pillars to obtain multiple pillar data, and each pillar data corresponds to the same height of the pillar. Since the multiple environmental images collected may be distorted, it is necessary to repair the multiple environmental images to obtain a real environmental image that is more in line with the actual situation.
[0068] In an optional embodiment, the laser point cloud data and the multiple environment images are preprocessed to obtain multiple pillar data and multiple real environment images, including:
[0069] Divide the area where the laser point cloud data is located into multiple pillars, and use the data contained in the multiple pillars as multiple pillar data; each pillar in the multiple pillars contains the same amount of data;
[0070] The distortion in multiple environment images is removed to obtain multiple real environment images.
[0071] Obtain the distortion coefficient, and use the distortion coefficient to remove the distortion in the environment image collected by the standard single-view camera itself to obtain multiple real environment images. Ignore the height information of the z-axis and divide the area where the laser point cloud data is located into pillars to obtain multiple images such as Figure 3 The pillars shown are of the same height.
[0072] Specifically, the area where the laser point cloud data is located is divided into pillars by ignoring the height information of the z-axis, including: obtaining a top view of the area where the laser point cloud data is located, the top view can represent the x-axis information and y-axis information of each laser point cloud data. The area where the laser point cloud data is located is divided into multiple grids according to a fixed size, and each grid serves as a pillar.
[0073] Since the laser point cloud has the characteristics of discreteness and sparseness, the number of laser point clouds contained in each divided pillar is not the same. To facilitate subsequent calculations, set the hyperparameter N. When the number of laser point clouds in a pillar exceeds N, N are randomly selected from the laser point clouds included in the pillar as the laser point cloud data included in the pillar. When the number of laser point clouds in a pillar is less than N, the number of laser point clouds included in the pillar is padded to N by zero padding as the laser point cloud data included in the pillar. In this way, the amount of data contained in each divided pillar is the same, which is convenient for subsequent calculations.
[0074] Assuming that there are P non-empty pillars in the area where the laser point cloud data is located, since the information contained in each laser point cloud data is a four-tuple (x, y, z, i), the divided area can be represented by a 4×P×N-dimensional tensor. In the embodiment of the present application, N is set to 16, and those skilled in the art can also adjust the value of N according to actual needs.
[0075] S103: Input multiple pillar data and multiple real environment images into a preset feature fusion model to obtain fusion features, and perform object prediction based on the fusion features to obtain object information of objects contained in the current environment and the confidence of the object information; the fusion features include the features contained in the multiple pillar data and multiple real environment images.
[0076] Get the laser point cloud sample data and environment image samples collected at the same time. The laser point cloud sample data is the laser point cloud data used in the process of training the preset feature fusion model, and the environment image samples are the environment images used in the process of training the preset feature fusion model. The laser point cloud sample data and the environment image samples are all pre-processed sample data. By manually checking and comparing the laser point cloud sample data and the environment image samples, the annotation files corresponding to the laser point cloud sample data and the environment image samples are obtained. Figure 4 As shown, the annotation file includes the object border, laser point cloud and annotation content. The annotation content includes the object category, size, relative position and heading. The environmental image sample and the laser point cloud sample data are input into the pre-established model, so that the model generates object prediction results based on the environmental image sample and the laser point cloud sample data. Then the annotation file corresponding to the environmental image sample and the laser point cloud sample data is input into the model, so that the model updates the model parameters according to the error between the object prediction result and the annotation file until the error converges, and the preset feature fusion model is obtained, so that the preset feature fusion model can generate fusion features based on the input multiple pillar data and multiple real environment images, and perform object prediction based on the fusion features, and output the object information of the objects contained in the current environment and the confidence corresponding to the object information.
[0077] It should be noted that the objects contained in the current environment are objects contained in the preset spatial range. The preset spatial range is the spatial range consisting of 80 meters to 80 meters on the left and right, 160 meters in front and 80 meters behind, and 5 meters above and 5 meters below the current location as the center point. For ease of understanding, take the vehicle as an example: establish a vehicle coordinate system with the center point of the vehicle as the origin. Take the left side of the vehicle as the negative half axis of the x-axis, and the right side of the vehicle as the positive half axis of the x-axis. Take the front of the vehicle as the positive half axis of the y-axis, and the rear of the vehicle as the negative half axis of the y-axis. Take the top of the vehicle as the positive half axis of the z-axis, and the bottom of the vehicle as the negative half axis of the z-axis. The objects contained in the preset spatial range are objects within the spatial range consisting of -80m≤x≤80m, -80m≤y≤160m, and -5m≤z≤5m.
[0078] S104: Filter the object information of the objects contained in the current environment according to the preset comparison threshold and the confidence of the object information to obtain the object detection result.
[0079] The confidence of the object information is compared with the preset comparison threshold. If the confidence of the object information is less than the preset comparison threshold, the object information is removed. Since the same object may be recognized multiple times during the detection process, it is necessary to filter out the overlapping frames through NMS to obtain the object frame with the highest score.
[0080] After filtering out object information and overlapping frames with confidence levels lower than a preset comparison threshold, the eight corner points corresponding to the object in three-dimensional space are determined based on the object information (x, y, z, l, w, h, r), and the determined results are used as object detection results.
[0081] By acquiring the laser point cloud data collected within a preset time, and multiple environmental images, the laser point cloud data and the multiple environmental images are preprocessed to obtain multiple pillar data and multiple real environment images. The multiple pillar data and the multiple real environment images are input into the preset feature fusion model to obtain fusion features, and object prediction is performed based on the fusion features, and the object information of the objects contained in the current environment and the confidence of the object information are output. According to the preset comparison threshold and the confidence of the object information, the object information of the objects contained in the current environment is screened to obtain the object detection result. Since the fusion features in the present application include the features contained in the multiple pillar data and the multiple real environment images, the advantages of the two-modal feature fusion are complementary, providing more effective data for the prediction of subsequent objects, thereby improving the detection effect of object detection.
[0082] Since the pillar data and the real environment image are data collected and generated by sensors of different modalities, in order to achieve the fusion of the pillar data and the real environment image, in an optional embodiment, multiple pillar data and multiple real environment images are input into a preset feature fusion model to obtain fusion features, including:
[0083] Inputting multiple real environment images into a preset feature fusion model to obtain image features; the image features are features contained in the multiple real environment images;
[0084] Input multiple pillar data into a preset feature fusion model to obtain laser point cloud features; laser point cloud features are features contained in laser point cloud data;
[0085] Filling the laser point cloud features based on the size of the image features to obtain the laser point cloud features to be merged; the size of the laser point cloud features to be merged is the same as the size of the image features;
[0086] The image features and the features of the laser point cloud to be spliced are merged to obtain fusion features.
[0087] The preset feature fusion model extracts features from the input real environment image and pillar data respectively to obtain image features and laser point cloud features. Since the laser radar and the standard single-view camera are sensors of different modes, and sensors of different modes have different receptive fields, the image features and laser point cloud features have different sizes in the bird's eye view (BEV). Figure 5 As shown in the figure, in order to enable the image features and the laser point cloud features to be spliced, the laser point cloud features are padded with zeros based on the size of the image features, so that the size of the padded laser point cloud features is the same as that of the image features. The padded laser point cloud features are used as the laser point cloud features to be merged. The laser point cloud features to be merged are spliced with the image features in the channel domain, so that the laser point cloud features to be merged and the image features are merged into one feature, and the feature is encoded and decoded by UNet to reduce the semantic interval between the two modalities and obtain the fused feature.
[0088] In an optional embodiment, multiple real environment images are input into a preset feature fusion model to obtain image features, including:
[0089] Inputting multiple real environment images into a preset feature fusion model to obtain features contained in the multiple real environment images;
[0090] According to a preset depth estimation network, depth prediction is performed on multiple real environment images to obtain depth information of the multiple real environment images;
[0091] According to the depth information of the multiple real environment images, the features contained in the multiple real environment images are converted into three-dimensional space to obtain image features.
[0092] Multiple real environment images are input into a preset feature fusion model, the multi-scale features contained in the multiple real environment images are extracted through ResNet, and the multi-scale features in the multiple real environment images are fused based on the FPN structure.
[0093] The fused features are converted from two-dimensional space to three-dimensional space to determine the image features from the BEV perspective, specifically including: predicting and generating a probability vector for each feature point in the real environment image at a series of possible discrete depth values, and performing an outer product between the extracted features and the predicted probability vector, so that each pixel in the real environment image has a corresponding feature at each discrete depth value. Then, each pixel is projected into a three-dimensional coordinate system created based on the current position using the intrinsic and extrinsic parameters of a standard single-view camera, so that the features extracted from the image are converted from a two-dimensional perspective to a three-dimensional space. Finally, multiple grids are divided in the three-dimensional space, and the features corresponding to the pixels contained in each grid are summed up, and dimensionality reduction is performed through convolution to obtain the image features from the BEV perspective.
[0094] In an optional embodiment, multiple pillar data are input into a preset feature fusion model to obtain laser point cloud features, including:
[0095] Inputting multiple pillar data into a preset feature fusion model to obtain features of the multiple pillar data;
[0096] According to the centroids contained in the plurality of pillar data, features of the plurality of pillar data are spliced and arranged to obtain spliced features;
[0097] Project and transform the splicing features to obtain a pseudo image corresponding to the splicing features;
[0098] The pseudo image is encoded and decoded to obtain the laser point cloud features.
[0099] The data of multiple pillars are input into the preset feature fusion model, and the features of the multiple pillar data are extracted through the PillarVFE structure. Based on the centroid of the pillar where the multiple pillar data are located, the features of the multiple pillar data are spliced and arranged to obtain spliced features. For example, after determining the centroid of each pillar, the features corresponding to each laser point cloud are spliced and sorted according to the offset between each laser point cloud and the centroid of the pillar where it is located.
[0100] Through the maximum pooling operation, the pillar dimension is transformed into dimension E. The value of dimension E can be selected according to actual needs. In this way, the features that can represent the content of the pillar are extracted according to actual needs. Then, the spliced features are projected and transformed into a pseudo image through PillarScatter, that is, the P expansion dimension is transformed into H×W according to the spatial position of the pillar, and a pseudo image with a dimension of C×H×W is obtained, which provides spatial dependencies for the network model.
[0101] Unet is used as the feature extraction network for encoding and decoding, and the pseudo image is encoded and decoded to obtain the features under the BEV perspective, and the obtained features are used as the laser point cloud features. Encoding and decoding the pseudo image includes downsampling and upsampling the features based on the pseudo image in sequence, with a jump connection structure between downsampling and upsampling, and feature splicing and fusion are performed through the jump connection structure, so that the feature size is restored to the size when the pillar data is input into the preset feature fusion model.
[0102] In an optional embodiment, object prediction is performed based on the fusion features, and object information of objects contained in the current environment and the confidence of the object information are output, including:
[0103] Based on the fusion features, thermal map prediction, offset prediction, scale prediction, center point prediction, direction prediction and speed prediction are performed respectively to obtain a comprehensive prediction result;
[0104] Generate confidence of object information based on comprehensive prediction results;
[0105] The comprehensive prediction result is used as the object information of the objects contained in the current environment;
[0106] Output the object information of the objects contained in the current environment and the confidence of the object information.
[0107] The first detection head predicts objects in the overlapping area of the data collected by the laser radar and the standard camera, and the second detection head predicts objects in other multi-view areas. Figure 6 As shown in the figure, each detection head consists of 6 parts, namely the heat map part, the center point part, the object length, width and height part, the center point height part, the object orientation part and the object speed part, and each part is implemented by a full convolutional layer.
[0108] The fused features are divided into J×K regions, each of which contains a feature point. When the center of an object falls in a region, the object is predicted based on the feature points contained in the region. Since the first detection head and the second detection head contain the same structure, the object needs to be predicted in 6 parts in any region, namely heat map prediction, offset prediction, scale prediction, center point prediction, orientation prediction and speed prediction. Heat map prediction is used to predict whether there is an object at each heat point, and predict the type of object when there is an object. Offset prediction is used to predict the offset of the center of each object from the heat point. Scale prediction is used to predict the scale of each object, including the length, width and height of the object. Center point prediction is used to predict the height information of the center point of the object, orientation prediction is used to predict the orientation information of the object, and speed prediction is used to predict the speed information of the object. The prediction results of the above 6 parts are used as the object information of the object, so as to obtain the object information of the objects contained in multiple real environment images by predicting based on the fused features. Object information includes the center point of the object (x, y, z), the length, width and height of the object (l, w, h) and the orientation of the object (r).
[0109] For ease of understanding, the following example uses the scenario of vehicle autonomous driving as an example:
[0110] Before using the preset feature fusion model, the preset feature fusion model needs to be trained. The training process is as follows: Figure 7 As shown. The laser point cloud sample data is collected by the laser radar installed on the vehicle, and the environmental image samples of different perspectives are collected by the standard single-view camera, and joint calibration and three-dimensional box annotation are performed according to the collected sample data. The laser point cloud data is divided according to the pillars to obtain multiple pillar data, and the distortion in the environmental image is removed to obtain multiple real environmental images. The multiple pillar data and the multiple real environmental images are input into the network model including the image stream BEV feature extraction, the laser radar stream BEV feature extraction, the modal fusion and the object information prediction functions for training, so that the trained preset feature fusion model can output object information and the confidence corresponding to the object information. Based on the object information and the confidence corresponding to the object information, NMS non-maximum suppression is performed, and the eight corner points corresponding to the object in the three-dimensional space are determined to ensure that the trained preset feature fusion model can realize object detection.
[0111] After the preset feature fusion model is trained, the object detection process of the vehicle during autonomous driving is as follows: Figure 8As shown. After preprocessing the collected environmental images and laser point cloud data respectively, multiple real environmental images collected from different perspectives are input into the preset feature fusion model as image streams, and feature extraction, multi-scale fusion, and 3D / three-dimensional space feature conversion are performed in sequence to obtain image features from the BEV perspective. Multiple pillar data are input into the preset feature fusion model as lidar streams, and feature extraction, feature projection, and BEV feature extraction are performed in sequence to obtain laser point cloud features from the BEV perspective. Based on the image feature size from the BEV perspective, feature filling is performed on the laser point cloud feature size from the BEV perspective, and feature merging is performed based on the filled laser point cloud features from the BEV perspective and the image features from the BEV perspective, and feature encoding is performed on the merged features to obtain fused features. Based on the area where the fused features are located, the corresponding detection head is used to detect objects and output object information. The object information is filtered, and the eight corner points corresponding to the object in the three-dimensional space are determined based on the filtered object information to achieve object detection.
[0112] like Fig. 9 As shown, an embodiment of the present application provides a multi-modal object detection system, the system comprising:
[0113] The acquisition module 901 is used to acquire the laser point cloud data collected within a preset time, and multiple environmental images; the multiple environmental images are environmental images collected at different viewing angles based on the current location;
[0114] The processing module 902 is used to pre-process the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images; at least part of the pillar data in the multiple pillar data includes the laser point cloud data;
[0115] The prediction module 903 is used to input the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features, and perform object prediction based on the fusion features to obtain object information of the objects contained in the current environment and the confidence of the object information; the fusion features include the features contained in the plurality of pillar data and the plurality of real environment images;
[0116] The screening module 904 is used to screen the object information of the objects contained in the current environment according to a preset comparison threshold and the confidence of the object information to obtain the object detection result.
[0117] In an optional embodiment, the prediction module 903 is specifically configured to:
[0118] Inputting multiple real environment images into a preset feature fusion model to obtain image features; the image features are features contained in the multiple real environment images;
[0119] Input multiple pillar data into a preset feature fusion model to obtain laser point cloud features; laser point cloud features are features contained in laser point cloud data;
[0120] Filling the laser point cloud features based on the size of the image features to obtain the laser point cloud features to be merged; the size of the laser point cloud features to be merged is the same as the size of the image features;
[0121] The image features and the features of the laser point cloud to be spliced are merged to obtain fusion features.
[0122] In an optional embodiment, the prediction module 903 is specifically used for:
[0123] Inputting multiple real environment images into a preset feature fusion model to obtain features contained in the multiple real environment images;
[0124] According to a preset depth estimation network, depth prediction is performed on multiple real environment images to obtain depth information of the multiple real environment images;
[0125] According to the depth information of the multiple real environment images, the features contained in the multiple real environment images are converted into three-dimensional space to obtain image features.
[0126] In an optional embodiment, the prediction module 903 is specifically used for:
[0127] Inputting multiple pillar data into a preset feature fusion model to obtain features of the multiple pillar data;
[0128] According to the centroids contained in the plurality of pillar data, features of the plurality of pillar data are spliced and arranged to obtain spliced features;
[0129] Project and transform the splicing features to obtain a pseudo image corresponding to the splicing features;
[0130] The pseudo image is encoded and decoded to obtain the laser point cloud features.
[0131] In an optional embodiment, the prediction module 903 is specifically used for:
[0132] Based on the fusion features, thermal map prediction, offset prediction, scale prediction, center point prediction, direction prediction and speed prediction are performed respectively to obtain a comprehensive prediction result;
[0133] Generate confidence of object information based on comprehensive prediction results;
[0134] The comprehensive prediction result is used as the object information of the objects contained in the current environment;
[0135] Output the object information of the objects contained in the current environment and the confidence of the object information.
[0136] In an optional embodiment, the screening module 904 is specifically used for:
[0137] Divide the area where the laser point cloud data is located into multiple pillars, and use the data contained in the multiple pillars as multiple pillar data; each pillar in the multiple pillars contains the same amount of data;
[0138] The distortion in multiple environment images is removed to obtain multiple real environment images.
[0139] By acquiring the laser point cloud data collected within a preset time, and multiple environmental images, the laser point cloud data and the multiple environmental images are preprocessed to obtain multiple pillar data and multiple real environment images. The multiple pillar data and the multiple real environment images are input into the preset feature fusion model to obtain fusion features, and object prediction is performed based on the fusion features, and the object information of the objects contained in the current environment and the confidence of the object information are output. According to the preset comparison threshold and the confidence of the object information, the object information of the objects contained in the current environment is screened to obtain the object detection result. Since the fusion features in the present application include the features contained in the multiple pillar data and the multiple real environment images, the advantages of the two-modal feature fusion are complementary, providing more effective data for the prediction of subsequent objects, thereby improving the detection effect of object detection.
[0140] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any of the above-mentioned multi-modal-based object detection methods is implemented.
[0141] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0143] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A multimodal object detection method, characterized in that: The method comprises: Acquire laser point cloud data collected within a preset time, and multiple environmental images; the multiple environmental images are environmental images collected at different viewing angles based on the current location; Preprocessing the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images; at least part of the pillar data in the multiple pillar data includes the laser point cloud data; Input the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features, and perform object prediction based on the fusion features, and output object information of objects contained in the current environment and the confidence of the object information; the fusion features include the features contained in the plurality of pillar data and the plurality of real environment images; According to a preset comparison threshold and the confidence level of the object information, the object information of the objects contained in the current environment is screened to obtain an object detection result.
2. The method according to claim 1, characterized in that The step of inputting the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features includes: Inputting the multiple real environment images into the preset feature fusion model to obtain image features; the image features are features contained in the multiple real environment images; Inputting the plurality of pillar data into the preset feature fusion model to obtain laser point cloud features; the laser point cloud features are features contained in the laser point cloud data; Filling the laser point cloud feature based on the size of the image feature to obtain the laser point cloud feature to be merged; the size of the laser point cloud feature to be merged is the same as the size of the image feature; The image features and the features of the laser point cloud to be spliced are combined to obtain fusion features.
3. The method according to claim 2, characterized in that The step of inputting the plurality of real environment images into the preset feature fusion model to obtain image features includes: Inputting the multiple real environment images into the preset feature fusion model to obtain the features contained in the multiple real environment images; According to a preset depth estimation network, depth prediction is performed on the multiple real environment images to obtain depth information of the multiple real environment images; According to the depth information of the multiple real environment images, the features contained in the multiple real environment images are converted into a three-dimensional space to obtain the image features.
4. The method according to claim 2, characterized in that: The step of inputting the plurality of pillar data into the preset feature fusion model to obtain laser point cloud features includes: Inputting the plurality of pillar data into the preset feature fusion model to obtain features of the plurality of pillar data; splicing and arranging the features of the plurality of pillar data according to the centroids contained in the plurality of pillar data to obtain splicing features; Performing a projective transformation on the stitching feature to obtain a pseudo image corresponding to the stitching feature; The pseudo image is encoded and decoded to obtain the laser point cloud feature.
5. The method according to claim 1, characterized in that The object prediction is performed based on the fusion feature, and the object information of the object contained in the current environment and the confidence of the object information are outputted, including: Based on the fusion features, heat map prediction, offset prediction, scale prediction, center point prediction, direction prediction and speed prediction are performed respectively to obtain a comprehensive prediction result; Generating a confidence level of the object information according to the comprehensive prediction result; Using the comprehensive prediction result as object information of objects contained in the current environment; Output object information of objects contained in the current environment and the confidence level of the object information.
6. The method according to claim 1, characterized in that The preprocessing of the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images includes: Divide the area where the laser point cloud data is located into a plurality of pillars, and use the data contained in the plurality of pillars as the plurality of pillar data; each of the plurality of pillars contains the same amount of data; The distortion in the multiple environment images is removed to obtain multiple real environment images.
7. A multimodal object detection system, characterized in that: The system comprises: An acquisition module is used to acquire laser point cloud data collected within a preset time, and multiple environmental images; the multiple environmental images are environmental images collected at different viewing angles based on the current location; A processing module, used for preprocessing the laser point cloud data and the multiple environment images to obtain multiple pillar data and multiple real environment images; at least part of the pillar data in the multiple pillar data includes the laser point cloud data; A prediction module, used for inputting the plurality of pillar data and the plurality of real environment images into a preset feature fusion model to obtain fusion features, and performing object prediction based on the fusion features to obtain object information of objects contained in the current environment and the confidence of the object information; the fusion features include features contained in the plurality of pillar data and the plurality of real environment images; The screening module is used to screen the object information of the objects contained in the current environment according to a preset comparison threshold and the confidence of the object information to obtain an object detection result.
8. The system according to claim 7, characterized in that The prediction module is specifically used for: Inputting the multiple real environment images into the preset feature fusion model to obtain image features; the image features are features contained in the multiple real environment images; Inputting the plurality of pillar data into the preset feature fusion model to obtain laser point cloud features; the laser point cloud features are features contained in the laser point cloud data; Filling the laser point cloud feature based on the size of the image feature to obtain the laser point cloud feature to be merged; the size of the laser point cloud feature to be merged is the same as the size of the image feature; The image features and the features of the laser point cloud to be spliced are combined to obtain fusion features.
9. The system according to claim 7, characterized in that The prediction module is specifically used for: Based on the fusion features, heat map prediction, offset prediction, scale prediction, center point prediction, direction prediction and speed prediction are performed respectively to obtain a comprehensive prediction result; Generating a confidence level of the object information according to the comprehensive prediction result; Using the comprehensive prediction result as object information of objects contained in the current environment; Output object information of objects contained in the current environment and the confidence level of the object information.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the multimodal object detection method according to any one of claims 1 to 6 is implemented.