Multi-sensor data fusion method, target detection method, device and computer equipment

Abstract

The application relates to a multi-sensing data fusion, target detection method, device and computer equipment. The method comprises the following steps: a server acquires camera images and original point cloud data at the same time, performs two-dimensional target detection on the camera images, acquires at least one region of interest in the camera images, performs two-dimensional conversion on the original point cloud data, obtains a two-dimensional image under a pixel coordinate system, acquires at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image, and then extracts target point cloud data from the original point cloud data according to the at least one target region of interest. In the method, the server extracts target point cloud data from the original point cloud data according to the target region of interest in the two-dimensional image through the conversion relationship between the radar original point cloud data and the pixel points, and performs three-dimensional target detection, so that the number of input points of the three-dimensional target detection is reduced, the detection time is shortened, and the three-dimensional detection efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a multi-sensor data fusion, target detection method, device and computer equipment. BACKGROUND

[0002] With the popularization of laser radars, three-dimensional target detection by collecting three-dimensional point cloud data through laser radars has become a common technology in the field. Through three-dimensional target detection, the position, size, category and other information of the target object can be determined, for example, through three-dimensional target detection, the target objects such as vehicles and pedestrians on the road can be detected.

[0003] At present, the algorithm of three-dimensional target detection is mainly based on the point cloud data collected by the radar combined with some data processing algorithms, and the amount of point cloud data collected is large, which consumes a large amount of computing resources when processing data, and accordingly reduces the efficiency of three-dimensional target detection. SUMMARY

[0004] Therefore, it is necessary to provide a multi-sensor data fusion, target detection method, device and computer equipment to solve the above technical problems.

[0005] In a first aspect, a multi-sensor data fusion method is provided, which comprises:

[0006] acquiring a camera image and original point cloud data at the same time;

[0007] performing two-dimensional target detection on the camera image to obtain at least one region of interest in the camera image;

[0008] performing two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in a pixel coordinate system;

[0009] obtaining at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image;

[0010] extracting target point cloud data from the original point cloud data according to the at least one target region of interest.

[0011] In one embodiment, the above extracting target point cloud data from the original point cloud data according to the at least one target region of interest comprises:

[0012] obtaining a target index of the point cloud in the at least one target region of interest according to the point cloud points in the at least one target region of interest and the pixels in the camera image;

[0013] extracting target point cloud data from the original point cloud data according to the target index.

[0014] In one of the embodiments, the obtaining of the target index of the point cloud in the at least one target region of interest according to the point cloud points in the at least one target region of interest and the pixels in the camera image comprises:

[0015] The initial index of the point cloud points in each target region of interest is obtained, and the initial index is determined according to the correspondence between the point cloud points in the target region of interest and the pixels in the camera image.

[0016] The initial index of the point cloud in each target region of interest is subjected to an OR operation to obtain the target index of the point cloud in each target region of interest.

[0017] In one of the embodiments, the obtaining of the target index of the point cloud in the at least one target region of interest according to the point cloud points in the at least one target region of interest and the pixels in the camera image comprises:

[0018] The initial index of the point cloud points in each target region of interest is obtained, and the index of each pixel in the camera image is obtained.

[0019] The initial index of the point cloud in each target region of interest is subjected to an OR operation to obtain the initial target index of the point cloud in each target region of interest.

[0020] The initial target index of the point cloud in each target region of interest is subjected to an AND operation with the initial target index of the point cloud in the two-dimensional image to obtain the target index of the point cloud in each target region of interest.

[0021] In one of the embodiments, the two-dimensional conversion of the original point cloud data to obtain a two-dimensional image in the pixel coordinate system comprises:

[0022] According to the conversion relationship between the world coordinate system and the camera coordinate system, the coordinates of the original point cloud data in the world coordinate system are converted into the coordinates of the original point cloud data in the camera coordinate system.

[0023] According to the conversion relationship between the camera coordinate system and the pixel coordinate system, the coordinates of the original point cloud data in the camera coordinate system are converted into the coordinates of the original point cloud data in the pixel coordinate system.

[0024] According to the coordinates of the original point cloud data in the pixel coordinate system, a two-dimensional image in the pixel coordinate system is obtained.

[0025] In one of the embodiments, the conversion of the coordinates of the original point cloud data in the camera coordinate system into the coordinates of the original point cloud data in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the pixel coordinate system comprises:

[0026] The original point cloud data in the camera coordinate system, the camera intrinsic parameter, the radial distortion coefficient and the lateral distortion coefficient are substituted into the conversion relationship between the camera coordinate system and the pixel coordinate system to calculate the two-dimensional coordinates of the original point cloud data in the pixel coordinate system.

[0027] In one of the embodiments, the above-mentioned substituting the original point cloud data in the camera coordinate system, the camera intrinsic parameter, the radial distortion coefficient and the lateral distortion coefficient into the conversion relationship between the camera coordinate system and the pixel coordinate system to calculate the two-dimensional coordinates of the point cloud points in the original point cloud data in the pixel coordinate system comprises:

[0028] According to the first coordinate of the original point cloud data in the camera coordinate system, the second coordinate and the distance from the first coordinate to the origin of the camera coordinate system are calculated.

[0029] According to the second coordinate, the distance, the radial distortion coefficient and the lateral distortion coefficient, the third coordinate is calculated.

[0030] According to the third coordinate and the camera intrinsic parameter, the two-dimensional coordinates of the original point cloud data in the pixel coordinate system are calculated.

[0031] In one of the embodiments, the method further comprises:

[0032] Based on the rotation matrix and the translation vector in the extrinsic parameter between the camera and the radar, a conversion relationship between the world coordinate system and the camera coordinate system is constructed.

[0033] In one of the embodiments, the above-mentioned two-dimensional target detection on the camera image to obtain at least one region of interest in the camera image comprises:

[0034] The preset neural network model is used for two-dimensional target detection on the camera image to obtain at least one region of interest.

[0035] The second aspect provides a target detection method, which comprises:

[0036] The above-mentioned first aspect provides a multi-sensor data fusion method to obtain target point cloud data.

[0037] The target point cloud is subjected to three-dimensional target detection to obtain a target detection result.

[0038] The third aspect provides a target detection method, which comprises:

[0039] The camera image is obtained and subjected to two-dimensional target detection to obtain at least one region of interest in the camera image.

[0040] The above-mentioned first aspect provides a multi-sensor data fusion method to obtain target point cloud data.

[0041] According to the at least one region of interest in the camera image and the target point cloud, a target detection result is obtained.

[0042] In one embodiment, the target detection result is obtained according to the at least one region of interest in the camera image and the target point cloud data, including:

[0043] According to each region of interest in the camera image, a first initial target detection result is obtained.

[0044] The target point cloud data is subjected to three-dimensional detection to obtain a second initial target detection result.

[0045] The first initial target detection result and the second initial target detection result are subjected to weighted calculation, and a target detection result is obtained according to the weighted calculation result.

[0046] In a fourth aspect, a multi-sensor data fusion device is provided, including:

[0047] A first acquisition module is configured to acquire camera images and original point cloud data at the same time.

[0048] A detection module is configured to perform two-dimensional target detection on the camera images to obtain at least one region of interest in the camera images.

[0049] A conversion module is configured to perform two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in a pixel coordinate system.

[0050] A second acquisition module is configured to acquire at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image.

[0051] An extraction module is configured to extract target point cloud data from the original point cloud data according to the at least one target region of interest.

[0052] In a fifth aspect, a target detection device is provided, including:

[0053] An acquisition module is configured to acquire target point cloud data by using the multi-sensor data fusion method provided in the first aspect.

[0054] A detection module is configured to perform three-dimensional target detection on the target point cloud to obtain a target detection result.

[0055] In a sixth aspect, a target detection device is provided, including:

[0056] A first acquisition module is configured to acquire camera images and perform two-dimensional target detection on the camera images to obtain at least one region of interest in the camera images.

[0057] A second acquisition module is configured to acquire target point cloud data by using the multi-sensor data fusion method provided in the first aspect.

[0058] detecting a target detection result according to the at least one region of interest in the camera image and the target point cloud data.

[0059] In a seventh aspect, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the target detection method of any one of the first aspect, the second aspect, or the third aspect when executing the computer program.

[0060] In an eighth aspect, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the target detection method of any one of the first aspect, the second aspect, or the third aspect when executed by a processor.

[0061] The multi-sensor data fusion method, the target detection method, the device, and the computer device described above, the server obtains camera images and original point cloud data at the same time, performs two-dimensional target detection on the camera images, obtains at least one region of interest in the camera images, performs two-dimensional conversion on the original point cloud data, obtains a two-dimensional image in a pixel coordinate system, obtains at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image, and extracts target point cloud data from the original point cloud data according to the at least one target region of interest. In the method, the server obtains a two-dimensional image corresponding to the original point cloud data through the conversion relationship between the radar coordinate system and the pixel coordinate system, determines the target point cloud data based on the two-dimensional image, filters out point cloud data not located in the target region of interest and invalid point cloud data, and uses the determined target point cloud data for three-dimensional detection, which greatly reduces the number of input points for three-dimensional target detection, shortens the detection time when performing three-dimensional target detection, and improves the efficiency of three-dimensional detection. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 An application environment diagram of the multi-sensor data fusion method and the target detection method in one embodiment;

[0063] Figure 2 A flowchart of the multi-sensor data fusion method in one embodiment;

[0064] Figure 3 A flowchart of the multi-sensor data fusion method in one embodiment;

[0065] Figure 4 A flowchart of the multi-sensor data fusion method in one embodiment;

[0066] Figure 5 A flowchart of the multi-sensor data fusion method in one embodiment;

[0067] Figure 6A flowchart of a multi-sensor data fusion method in one embodiment;

[0068] Figure 7 A flowchart of a multi-sensor data fusion method in one embodiment;

[0069] Figure 8 A flowchart of a target detection method in another embodiment;

[0070] Figure 9 A flowchart of a target detection method in another embodiment;

[0071] Figure 10 A flowchart of a target detection method in one embodiment;

[0072] Figure 11 A flowchart of a multi-sensor data fusion and target detection method in another embodiment;

[0073] Figure 12 A structural block diagram of a multi-sensor data fusion device in one embodiment;

[0074] Figure 13 A structural block diagram of a multi-sensor data fusion device in one embodiment;

[0075] Figure 14 A structural block diagram of a target detection device in one embodiment;

[0076] Figure 15 A structural block diagram of a target detection device in one embodiment;

[0077] Figure 16 An internal structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0078] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0079] The target detection method provided by the present application can be applied in an application environment as shown in the figure. Figure 1 The server 101 communicates with the radar 102 and the two-dimensional image acquisition device 103 through a network. The server 101 can be an independent server or a server cluster composed of multiple servers. The radar 102 is generally a laser radar used for acquiring point cloud data at a certain time. The two-dimensional image acquisition device 103 can be any kind of high-definition camera or other image acquisition device used for acquiring camera images at the same time as the point cloud data.

[0080] The technical solutions of the present application and how the technical solutions solve the above technical problems will be specifically explained below by embodiments and in combination with the drawings. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. It should be noted that the present application Figures 2-7 The multi-sensor data fusion method provided by the embodiments has a server 101 as an execution subject, and can also be a multi-sensor data fusion device. The multi-sensor data fusion device can be part or all of the server 101 in the form of software, hardware, or a combination of software and hardware. In the following method embodiments, the execution subject is taken as an example to be the server 101.

[0081] In one embodiment, as shown in Figure 2 A multi-sensor data fusion method is provided, which involves that a server acquires camera images and original point cloud data at the same time, performs two-dimensional target detection on the camera images, acquires at least one region of interest in the camera images, performs two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in a pixel coordinate system, acquires at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image, and then extracts target point cloud data from the original point cloud data according to the at least one target region of interest. The process includes the following steps:

[0082] S201. Acquire camera images and original point cloud data at the same time.

[0083] The camera images are acquired by a two-dimensional image acquisition device, and the original point cloud data is acquired by a laser radar.

[0084] In this embodiment, to ensure the consistency of the target object, the camera images and the original point cloud data are data of the same scene acquired at the same time. The server can send an instruction to acquire data to the two-dimensional image acquisition device and the radar, so as to acquire the camera images and the original point cloud data at the same time. The server can also receive the data sent by the two-dimensional image acquisition device and the radar. It should be noted that the original point cloud data at a certain time can be one frame of point cloud data, or can be multiple frames of point cloud data, which is not limited in this embodiment.

[0085] S202. Perform two-dimensional target detection on the camera images to obtain at least one region of interest in the camera images.

[0086] The region of interest refers to a region corresponding to a target object obtained by performing two-dimensional target detection on the camera images. Optionally, the two-dimensional bounding box of the target object, the target object category, and the confidence, etc. can be determined according to the region of interest.

[0087] In this embodiment, the server can perform two-dimensional target detection on the camera image by using a preset two-dimensional target detection model, which can be any neural network model. Alternatively, the server can also perform two-dimensional target detection on the camera image by using an image recognition method. It should be noted that there is at least one target object in the camera image, and the category of the target object can include a person, a vehicle, or the like. The server performs two-dimensional target detection on the camera image by using the preset two-dimensional target detection model, and the output result is at least one target object corresponding to a region of interest. This embodiment does not limit the region of interest.

[0088] S203, performing two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in a pixel coordinate system.

[0089] The original point cloud data refers to all point cloud data in a detection region collected by a radar, and the point cloud data is three-dimensional data.

[0090] In this embodiment, the server can perform two-dimensional conversion on the original point cloud data by using a preset conversion relationship, obtain a two-dimensional image corresponding to the original point cloud data according to a two-dimensional coordinate corresponding to the original point cloud data, and the two-dimensional coordinate system can be a pixel coordinate system. Alternatively, the two-dimensional conversion on the original point cloud data can convert three-dimensional coordinates in a world coordinate system in which the original point cloud data is located into two-dimensional coordinates in a pixel coordinate system. Preferably, the three-dimensional coordinates in the world coordinate system in which the original point cloud data is located can be converted into coordinates in a camera coordinate system by using a first conversion, and then the coordinates in the camera coordinate system corresponding to the original point cloud data can be converted into two-dimensional coordinates in the pixel coordinate system by using a second conversion, so as to determine the two-dimensional image according to the two-dimensional coordinates of the original point cloud data in the pixel coordinate system.

[0091] S204, obtaining at least one target region of interest corresponding to at least one region of interest from the two-dimensional image.

[0092] The target region of interest refers to a region of interest corresponding to a target object in a two-dimensional image corresponding to the point cloud data.

[0093] In this embodiment, after obtaining the two-dimensional image corresponding to the original point cloud data, the server determines the target region of interest in the two-dimensional image corresponding to the region of interest detected according to the camera image. Optionally, the server can also establish the correspondence between the camera image and the original point cloud data, so as to determine the target region of interest in the two-dimensional image corresponding to the region of interest according to the correspondence and the region of interest. For example, since the coordinates of each point in the two-dimensional image and the camera image are both two-dimensional coordinates and have a correspondence, the server can determine the coordinate position of the target region of interest in the two-dimensional image according to the boundary box coordinates of the region of interest, which is not limited in this embodiment.

[0094] S205, extracting target point cloud data from the original point cloud data according to at least one target region of interest.

[0095] In this embodiment, the target point cloud data refers to the point cloud data in the original point cloud data corresponding to the target region of interest.

[0096] In this embodiment, the server determines the three-dimensional point cloud data corresponding to the two-dimensional points included in the target region of interest in the two-dimensional image, so as to extract these three-dimensional point cloud data from the original point cloud data as the target point cloud data. Optionally, when determining the three-dimensional point cloud data corresponding to the two-dimensional points in the target region of interest, the server can determine according to the index of each point, and it should be noted that the index is determined according to the mapping relationship between the three-dimensional point cloud data and the two-dimensional points, and the index of the two-dimensional point can be used to determine the three-dimensional point cloud data. Optionally, the server can also directly determine the three-dimensional point cloud data corresponding to the two-dimensional points according to the inverse conversion relationship between the two-dimensional points and the three-dimensional point cloud data, so as to extract these three-dimensional point cloud data from the original point cloud data, which is not limited in this embodiment.

[0097] In the above multi-sensor data fusion method, the server obtains the camera image and the original point cloud data at the same time, performs two-dimensional target detection on the camera image, obtains at least one region of interest in the camera image, performs two-dimensional conversion on the original point cloud data, obtains a two-dimensional image under the pixel coordinate system, obtains at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image, and extracts target point cloud data from the original point cloud data according to the at least one target region of interest. In this method, the server obtains the two-dimensional image corresponding to the original point cloud data through the conversion relationship between the radar coordinate system and the pixel coordinate system, so as to determine the target point cloud data according to the two-dimensional image, to a certain extent, filter out the point cloud data of the non-target region of interest and the invalid point cloud data, use the target point cloud data for three-dimensional detection, greatly reduce the number of input points of three-dimensional target detection, shorten the detection time when performing three-dimensional target detection, and improve the efficiency of three-dimensional detection.

[0098] To improve the accuracy of the target point cloud data, in an embodiment, as shown in Figure 3 The above extracting the target point cloud data from the original point cloud data according to at least one target region of interest comprises:

[0099] S301, obtaining a target index of the point cloud in the at least one target region of interest according to the point cloud points in the at least one target region of interest and the pixels in the camera image.

[0100] The target index of the point cloud refers to the index after the fusion processing, and the corresponding index before the fusion processing is the initial index of the point cloud. The fusion processing refers to the set processing of the point cloud points in multiple target regions of interest. The initial index is determined according to the correspondence between the point cloud points in the target region of interest and the pixels in the camera image. For example, the point cloud points in the original point cloud data that are not in the two-dimensional image, and / or the pixel points in the camera image that do not correspond to the point cloud data, can be set to false as the initial index value. The point cloud points in the original point cloud data that are in the two-dimensional image, and the pixel points in the camera image that correspond to the point cloud data can be set to true as the initial index value.

[0101] In the embodiment, the server determines the corresponding point cloud points of each target region of interest according to the target region of interest, determines the index of the target region of interest to obtain the corresponding point cloud points according to the correspondence between the point cloud points in the target region of interest and the pixels in the camera image, and determines the target index of the point cloud in the at least one target region of interest. Optionally, the server can perform an or operation on the initial indexes of all point cloud points in the target region of interest to obtain the target index of the point cloud in the at least one target region of interest, which is not limited in the embodiment.

[0102] S302, extracting the target point cloud data from the original point cloud data according to the target index.

[0103] In the embodiment, the server determines the corresponding three-dimensional point cloud data of the point cloud of the target region of interest according to the target index of the point cloud of the at least one target region of interest after the determination, and extracts the three-dimensional point cloud data from the original point cloud data as the target point cloud data.

[0104] In the embodiment, the server extracts the target data according to the point cloud points in the two-dimensional image and the pixel points in the camera image, accurately obtains and retains the point cloud data of the target region of interest, reduces the data amount for three-dimensional target detection, and improves the efficiency of three-dimensional target detection.

[0105] The above fusion processing of the initial index of the point cloud in the target region of interest is optional, and in an embodiment, as shown in Figure 4As shown, the target index of the point cloud in the at least one target region of interest is obtained according to the point cloud points in the at least one target region of interest and the pixels in the camera image, and includes:

[0106] S401, obtaining an initial index of the point cloud points in each target region of interest; the initial index is determined according to the correspondence between the point cloud points in the target region of interest and the pixels in the camera image.

[0107] In this embodiment, the server obtains the initial index of the point cloud points in each target region of interest according to the correspondence between the point cloud points in the target region of interest and the pixels in the camera image. For example, the server can determine the target region of interest in the two-dimensional image according to the at least one region of interest, so as to obtain the point cloud points in the target region of interest, and determine the initial index corresponding to the point cloud points in the target region of interest according to the correspondence between the point cloud points and the pixels in the camera image.

[0108] S402, performing or operation on the initial index of the point cloud in each target region of interest to obtain the target index of the point cloud in each target region of interest.

[0109] In this embodiment, for example, the server obtains 8 target regions of interest. Specifically, the server can perform or operation on the initial index of the point cloud in each two target regions of interest in turn to obtain 4 or operation results, and then perform or operation on the 4 or operation results in pairs to obtain the final result of the or operation corresponding to all target regions of interest. The final result is the target index of the point cloud in all target regions of interest. Alternatively, the server can also perform or operation on the initial index of the point cloud in the 8 target regions of interest at the same time to directly obtain the final result of the or operation corresponding to the initial index of the point cloud in all target regions of interest. Further, if the index value of point cloud point A in the first target region of interest is true, and the index value of point cloud point B in the second target region of interest is false, the operation result of the logical or operation is true, that is, point cloud point A and point cloud point B are retained at the same time. This embodiment is not limited in this regard.

[0110] In this embodiment, the server performs or operation on the initial index of the point cloud in the plurality of target regions of interest, that is, performs fusion processing, so as to ensure the preservation rate of the point cloud data corresponding to the target region of interest.

[0111] To further improve the accuracy of the point cloud data corresponding to the target region of interest, in one embodiment, as shown in Figure 5 the target index of the point cloud in the at least one target region of interest is obtained according to the point cloud points in the at least one target region of interest and the pixels in the camera image, and includes:

[0112] S501, obtain initial indexes of point cloud points in each target region of interest, and obtain indexes of each pixel in the camera image.

[0113] In the embodiment, the server obtains the initial indexes of the point cloud points in each target region of interest, similar to step S401.

[0114] S502, perform or operation on the initial indexes of the point cloud in each target region of interest to obtain initial target indexes of the point cloud in each target region of interest.

[0115] In the embodiment, the server performs or operation on the initial indexes of the point cloud in each target region of interest to obtain initial target indexes of the point cloud in all target regions of interest, similar to step S402.

[0116] S503, perform and operation on the initial target indexes of the point cloud in each target region of interest and the initial target indexes of the point cloud in the two-dimensional image to obtain target indexes of the point cloud in each target region of interest.

[0117] In the embodiment, the server performs logical and operation on the indexes of the point cloud in all two-dimensional bounding boxes and the indexes of the point cloud in the image plane point cloud, which aims to ensure that the obtained point cloud data exists in the three-dimensional point cloud data and has a corresponding relationship with the pixel points in the camera image, i.e., exists in the two-dimensional image; if the above two conditions are not met, it is invalid point cloud data; after the and operation, the server can obtain the corrected indexes of the point cloud in all target regions of interest.

[0118] In the embodiment, since the image plane point cloud and the point cloud data have a one-to-one correspondence, when the server obtains the corrected indexes of the point cloud in all target regions of interest, the corresponding point cloud data can be obtained from the original point cloud data according to the correspondence, and the point cloud data is used as the target point cloud data, the target indexes of the point cloud are determined by using the and operation of the indexes, the fusion time is reduced, and the fusion efficiency is improved.

[0119] The three-dimensional coordinates of the original point cloud data in the world coordinate system are converted to two-dimensional coordinates, which can be obtained by multiple conversions. In one embodiment, as shown in FIG. 6, the two-dimensional conversion of the original point cloud data is performed to obtain a two-dimensional image in the pixel coordinate system, including: Figure 6

[0120] S601, according to the conversion relationship between the world coordinate system and the camera coordinate system, convert the coordinates of the original point cloud data in the world coordinate system to the coordinates of the original point cloud data in the camera coordinate system.

[0121] ​In this embodiment, the server can construct the conversion relationship between the world coordinate system and the camera coordinate system according to the preset calibration method. For example, the server can construct the corresponding rotation matrix and translation vector in the conversion relationship between the world coordinate system and the camera coordinate system according to the preset calibration method, so that the server converts the coordinates of the original point cloud data in the world coordinate system into the coordinates of the original point cloud data in the camera coordinate system according to the rotation matrix and the translation vector.

[0122] S602, convert the coordinates of the original point cloud data in the camera coordinate system into the coordinates of the original point cloud data in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the pixel coordinate system.

[0123] In this embodiment, the server determines the corresponding camera intrinsic parameters, distortion coefficients and other parameter information in the current conversion relationship according to the camera, and determines the conversion relationship between the camera coordinate system and the pixel coordinate system according to the parameter information, so as to convert the coordinates of the original point cloud data in the camera coordinate system into the coordinates of the original point cloud data in the pixel coordinate system. Optionally, the parameter information includes camera intrinsic parameters, radial distortion coefficients and transverse distortion coefficients, the conversion relationship between the camera coordinate system and the pixel coordinate system is a relationship formula containing these parameter information, and the server substitutes the coordinates of the original point cloud data in the camera coordinate system into the conversion relationship formula between the camera coordinate system and the pixel coordinate system. Calculate the two-dimensional coordinates of the original point cloud data in the pixel coordinate system.

[0124] S603, obtain a two-dimensional image in the pixel coordinate system according to the coordinates of the original point cloud data in the pixel coordinate system.

[0125] In this embodiment, the server determines a two-dimensional image composed of the coordinates of all original point cloud data in the pixel coordinate system according to the coordinates of the original point cloud data in the pixel coordinate system. Optionally, the server can also establish the corresponding relationship between each point cloud point in the two-dimensional image and each pixel point in the camera image according to the two-dimensional image, so that the server can extract the target point cloud data according to the corresponding relationship.

[0126] In this embodiment, the server converts the three-dimensional coordinates of the original point cloud data into the coordinates in the pixel coordinate system according to the conversion relationship between the world coordinate system and the camera coordinate system and the conversion relationship between the camera coordinate system and the pixel coordinate system, obtains the two-dimensional image corresponding to the original point cloud data, and realizes the conversion and mapping of the three-dimensional coordinates of the original point cloud data and the two-dimensional coordinates.

[0127] When converting the coordinates in the camera coordinate system into the coordinates in the pixel coordinate system, the camera intrinsic parameters can be used for conversion. Optionally, as shown in FIG. 6, the conversion relationship between the camera coordinate system and the pixel coordinate system can be expressed as follows: Figure 7As shown, the above-mentioned coordinates of the original point cloud data in the camera coordinate system, camera intrinsic parameters, radial distortion coefficients and lateral distortion coefficients are substituted into the conversion relationship between the camera coordinate system and the pixel coordinate system to calculate the two-dimensional coordinates of the point cloud points in the original point cloud data in the pixel coordinate system, including:

[0128] S701, the second coordinate and the distance of the second coordinate to the origin of the camera coordinate system are calculated according to the first coordinate of the original point cloud data in the camera coordinate system.

[0129] In this embodiment, the server calculates the second coordinate (x', y') according to the coordinates (x, y, z) of the original point cloud data in the camera coordinate system, and the calculation formula is as follows:

[0130] x' = x / z

[0131] y' = y / z

[0132] r 2 = x' 2 +y' 2

[0133] Where (x', y') is the second coordinate, and r is the distance of the second coordinate to the origin of the camera coordinate system.

[0134] S702, the third coordinate is calculated according to the second coordinate, the distance, the radial distortion coefficient and the lateral distortion coefficient.

[0135] In this embodiment, the server calculates the third coordinate (x", y") according to the second coordinate (x', y') of the original point cloud data, the distance r, the radial distortion coefficient and the lateral distortion coefficient, and the calculation formula is as follows:

[0136]

[0137]

[0138] Where: k1, k2, k3, k4, k5, k6 are radial distortion coefficients; p1, p2 are tangential distortion coefficients.

[0139] S703, the two-dimensional coordinates of the original point cloud data in the pixel coordinate system are calculated according to the third coordinate and the camera intrinsic parameters.

[0140] In this embodiment, the server calculates the two-dimensional coordinates (u, v) of the original point cloud data in the pixel coordinate system according to the third coordinate (x", y") of the original point cloud data and the camera intrinsic parameters, and the calculation formula is as follows:

[0141] u = f x *x" + c x

[0142] v = f y *y" + c y

[0143] wherein f x ,f y ,c x ,c y represents the camera intrinsic parameters.

[0144] In this embodiment, the server can convert the coordinates of the original point cloud data in the camera coordinate system into the coordinates in the pixel coordinate system according to the preset conversion formula, so that the coordinates of the original point cloud data in the pixel coordinate system can be subjected to the target point cloud extraction operation according to the conversion relationship.

[0145] Before converting the original point cloud data between the camera coordinate system and the pixel coordinate system, the server also needs to convert the original point cloud data from the world coordinate system to the camera coordinate system. In one embodiment, the above method further comprises:

[0146] Based on the rotation matrix and the translation vector in the extrinsic parameters between the camera and the radar, a conversion relationship between the world coordinate system and the camera coordinate system is constructed.

[0147] In this embodiment, the server can determine the rotation matrix and the translation vector of the current camera and the radar according to the preset calibration method, so as to convert the coordinates of the original point cloud data into the corresponding coordinates in the camera coordinate system according to the conversion relationship between the camera coordinate system and the radar coordinate system, and the expression is as follows:

[0148]

[0149] wherein (X, Y, Z) represents the coordinates of a point in the radar coordinate system; (x, y, z) represents the coordinates of a point in the camera coordinate system; R represents the rotation matrix; and t represents the translation vector.

[0150] In this embodiment, the server converts the original point cloud data from the world coordinate system to the camera coordinate system according to the conversion relationship between the radar coordinate system and the camera coordinate system, which lays the foundation for the subsequent conversion of the point cloud coordinates in the camera coordinate system to the coordinates in the pixel coordinate system.

[0151] The above two-dimensional target detection on the camera image obtains the region of interest corresponding to the target object, and in one embodiment, the above two-dimensional target detection on the camera image obtains at least one region of interest in the camera image, comprising:

[0152] The preset neural network model is used for two-dimensional target detection on the camera image to obtain at least one region of interest.

[0153] In the embodiment, the server can select any other neural network to detect the camera image according to different detection accuracies, for example, a deep learning neural network, a support vector machine neural network, or the like, and the output result of the model includes the region of interest of at least one target object in the camera image, which is not limited in the embodiment.

[0154] In the embodiment, the server detects the camera image by using the neural network model, and the camera image processing is more time-saving than the point cloud data processing. The target point cloud data is determined by using the obtained two-dimensional region of interest, which greatly saves the time for three-dimensional target detection and improves the efficiency of three-dimensional target detection.

[0155] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with embodiments and in combination with the drawings. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments. It should be noted that the present application Figure 8 The target detection method provided by the embodiment has a server 101 as an execution subject, which can also be a target detection device. The target detection device can be part or all of the server 101 in the form of software, hardware, or a combination of software and hardware. In the following method embodiments, the execution subject is taken as an example of the server 101.

[0156] In one embodiment, as shown in Figure 8 a target detection method is provided, including the following steps:

[0157] S801, the target point cloud data is obtained by using the multi-sensor data fusion method provided in the above Figures 1-7 embodiment.

[0158] In the embodiment, the server can obtain the target point cloud data by using the multi-sensor data fusion method provided in the above Figures 1-7 embodiment. The target point cloud data is the point cloud data extracted from the original point cloud data by the server according to the target region of interest of the two-dimensional image.

[0159] S802, three-dimensional target detection is performed on the target point cloud to obtain a target detection result.

[0160] In this embodiment, the server performs three-dimensional target detection on the target point cloud data determined according to the preset three-dimensional target detection model, where the three-dimensional target detection model can be a deep learning model, or other image processing model. For example, the three-dimensional target detection can be to detect the horizontal edges or vertical edges in the actual space according to the bird's eye view obtained by performing viewpoint transformation on the captured image to be a top view, and to detect three-dimensional objects such as vehicles by using the number of edges, and the output result is the target detection result of each target object. Optionally, the target detection result includes the three-dimensional region of interest, the category, and the confidence of each target object, which is not limited in this embodiment.

[0161] In the above target detection method, the server performs three-dimensional target detection on the target point cloud data determined according to the multi-sensor data fusion method provided in the above embodiment to obtain the target detection result. In this method, since the target point cloud data is the target point cloud data corresponding to the target object determined according to the mapping relationship between the two-dimensional image and the three-dimensional point cloud data, the point cloud points of non-target objects are filtered to a certain extent, and the number of input points of three-dimensional target detection is greatly reduced when performing three-dimensional target detection, thereby shortening the detection time and improving the efficiency of three-dimensional detection.

[0162] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below through embodiments and in combination with the drawings. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. It should be noted that the present application Figures 9-10 The target detection method provided in the embodiment has a server 101 as the execution subject, which can also be a target detection device. The target detection device can be part or all of the server 101 in the form of software, hardware, or a combination of software and hardware. In the following method embodiments, the execution subject is taken as an example to be the server 101.

[0163] In one embodiment, as shown in Figure 9 a target detection method is provided, which includes:

[0164] S901, acquiring a camera image, and performing two-dimensional target detection on the camera image to obtain at least one region of interest in the camera image;

[0165] In this embodiment, the server can select any other neural network to detect the camera image according to the detection accuracy to obtain an output result, which includes the region of interest of at least one target object in the camera image, and the category and confidence corresponding to each region of interest, which is not limited in this embodiment.

[0166] S902, utilize the above Figures 1-7 The multi-sensor data fusion method provided in the embodiment acquires target point cloud data.

[0167] In the embodiment, the server can acquire the target point cloud data according to the above Figures 1-7 The multi-sensor data fusion method provided in the embodiment acquires target point cloud data, which is the point cloud data extracted from the original point cloud data by the server according to the target region of interest of the two-dimensional image.

[0168] S903, obtaining a target detection result according to at least one region of interest in the camera image and the target point cloud data.

[0169] In the embodiment, the server performs three-dimensional target detection according to the target point cloud data to obtain a three-dimensional target detection result. Optionally, the server can also correct the three-dimensional target detection result according to the two-dimensional target detection result of the camera image. For example, the server can perform weighted calculation according to the confidence of the three-dimensional target detection result and the confidence of the two-dimensional detection result to obtain a corrected target detection result. The server can also perform correction calculation on the target detection result according to the category of the three-dimensional target detection result and the category of the two-dimensional detection result, which is not limited in the embodiment.

[0170] In the above target detection method, the server performs three-dimensional target detection according to the target point cloud data determined by the multi-sensor data fusion method provided in the above embodiment and the region of interest obtained by two-dimensional detection to obtain a target detection result. In the method, the target point cloud data is determined according to the mapping relationship between the two-dimensional image and the three-dimensional point cloud data, and the target point cloud data corresponding to the target object is determined to a certain extent. The point cloud points of non-target objects are filtered out. The three-dimensional detection is performed using the target point cloud data and the region of interest, which greatly reduces the number of input points of the three-dimensional target detection, shortens the detection time, improves the efficiency of the three-dimensional detection, and corrects the three-dimensional detection result by the two-dimensional region of interest, thereby improving the accuracy of the three-dimensional detection result.

[0171] Optionally, in one embodiment, as shown in Figure 10 obtaining a target detection result according to at least one region of interest in the camera image and the target point cloud data includes:

[0172] S1001, acquiring a first initial target detection result according to each region of interest in the camera image.

[0173] In the embodiment, the server determines a first initial target detection result corresponding to each region of interest according to each region of interest in the camera image. The first initial target detection result includes the category, confidence, and other information corresponding to each region of interest of the target object in the camera image.

[0174] S1002, three-dimensional detection is performed on the target point cloud data to obtain a second initial target detection result.

[0175] In this embodiment, the server performs three-dimensional target detection on the target point cloud data by using a preset three-dimensional target detection model. The three-dimensional target detection model can be a deep learning model, or can be another image processing model. The second initial target detection result is obtained by performing three-dimensional target detection on the target point cloud data. The second initial target detection result includes the category and confidence of the three-dimensional region of interest corresponding to each target object, and the present embodiment does not limit this.

[0176] S1003, the first initial target detection result and the second initial target detection result are weighted and calculated, and a target detection result is obtained according to the weighted calculation result.

[0177] In this embodiment, the server can perform weighted summation calculation on the weight value corresponding to the first initial target detection result and the weight value corresponding to the second initial target detection result. For example, the calculation formula of the final confidence C of n target objects is as follows:

[0178]

[0179] wherein a is the weight value of the first initial target detection result; C Two-dim_i is the confidence of the i-th first initial target detection result; C Thr-dim_i is the confidence of the i-th second initial target detection result.

[0180] The server can determine the reliability of the current target detection result according to the final confidence obtained by the above calculation. The final confidence is output as the confidence of the target detection result. If the final confidence is low, the server can verify the reliability of the confidence by correcting the target object category.

[0181] In this embodiment, the three-dimensional detection result is corrected by using the two-dimensional region of interest, thereby improving the accuracy of the three-dimensional detection result.

[0182] In order to better illustrate the above method, as Figure 11 shown, the present embodiment provides a target detection method, which specifically comprises:

[0183] S101, obtaining a camera image and original point cloud data at the same time;

[0184] S102, performing two-dimensional target detection on the camera image to obtain at least one region of interest in the camera image;

[0185] S103, converting the coordinates of the original point cloud data in the world coordinate system into coordinates of the original point cloud data in the camera coordinate system according to the conversion relationship between the world coordinate system and the camera coordinate system;

[0186] S104, converting the coordinates of the original point cloud data in the camera coordinate system into coordinates of the original point cloud data in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the pixel coordinate system;

[0187] S105, obtaining a two-dimensional image in the pixel coordinate system according to the coordinates of the original point cloud data in the pixel coordinate system;

[0188] S106, obtaining at least one target region of interest corresponding to at least one region of interest from the two-dimensional image;

[0189] S107, obtaining a target index of the point cloud in the at least one target region of interest according to the point cloud points in the at least one target region of interest and the pixels in the camera image;

[0190] S108, extracting target point cloud data from the original point cloud data according to the target index;

[0191] S109, obtaining a camera image and performing two-dimensional target detection on the camera image to obtain at least one region of interest in the camera image;

[0192] S110, obtaining a target detection result according to the at least one region of interest in the camera image and the target point cloud data.

[0193] In the embodiment, the server extracts target point cloud data from the image plane point cloud corresponding to the original point cloud data according to each two-dimensional bounding box through the conversion relationship between the radar and the camera, greatly reduces the number of input points of three-dimensional target detection, shortens the detection time when performing three-dimensional target detection, and in the process of three-dimensional detection, the input points are point cloud data corresponding to the detection target, which reduces the false detection rate to a certain extent, and improves the efficiency and accuracy of three-dimensional detection.

[0194] The target detection method provided by the above embodiment has similar implementation principles and technical effects to the above method embodiments, which will not be described here.

[0195] It should be understood that, although Figures 2-11 The steps in the flowchart are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, Figures 2-11At least one of the steps in the method can include multiple steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the steps or stages is not necessarily sequential, but can be performed alternately or alternately with at least one of the other steps or steps or stages in the other steps.

[0196] In one embodiment, as shown in Figure 12 A multi-sensor data fusion device is provided, comprising: a first acquisition module 01, a detection module 02, a conversion module 03, a second acquisition module 04 and an extraction module 05, wherein:

[0197] The first acquisition module 01 is configured to acquire camera images and raw point cloud data at the same time;

[0198] The detection module 02 is configured to perform two-dimensional target detection on the camera images to obtain at least one region of interest in the camera images;

[0199] The conversion module 03 is configured to convert the raw point cloud data into a two-dimensional image in a pixel coordinate system;

[0200] The second acquisition module 04 is configured to acquire at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image;

[0201] The extraction module 05 is configured to extract target point cloud data from the raw point cloud data according to the at least one target region of interest.

[0202] In one embodiment, the extraction module 05 is specifically configured to obtain a target index of the point cloud in the at least one target region of interest according to the point cloud points in the at least one target region of interest and the pixels in the camera images; and extract the target point cloud data from the raw point cloud data according to the target index.

[0203] In one embodiment, the extraction module 05 is specifically configured to obtain an initial index of the point cloud points in each target region of interest; the initial index is determined according to the correspondence between the point cloud points in the target region of interest and the pixels in the camera images; and the initial index of the point cloud in each target region of interest is calculated to obtain the target index of the point cloud in each target region of interest.

[0204] In one of the embodiments, the extraction module 05 is specifically configured to obtain initial indexes of the point cloud points in each target region of interest, and obtain indexes of each pixel in the camera image; perform an OR operation on the initial indexes of the point cloud in each target region of interest to obtain initial target indexes of the point cloud in each target region of interest; and perform an AND operation on the initial target indexes of the point cloud in each target region of interest and the initial target indexes of the point cloud in the two-dimensional image to obtain target indexes of the point cloud in each target region of interest.

[0205] In one of the embodiments, the conversion module 03 is specifically configured to convert, according to a conversion relationship between the world coordinate system and the camera coordinate system, coordinates of the original point cloud data in the world coordinate system into coordinates of the original point cloud data in the camera coordinate system; convert, according to a conversion relationship between the camera coordinate system and the pixel coordinate system, the coordinates of the original point cloud data in the camera coordinate system into coordinates of the original point cloud data in the pixel coordinate system; and obtain the two-dimensional image in the pixel coordinate system according to the coordinates of the original point cloud data in the pixel coordinate system.

[0206] In one of the embodiments, the conversion module 03 is specifically configured to substitute the coordinates of the original point cloud data in the camera coordinate system, the camera intrinsic parameter, the radial distortion coefficient and the lateral distortion coefficient into the conversion relationship between the camera coordinate system and the pixel coordinate system, and calculate the two-dimensional coordinates of the original point cloud data in the pixel coordinate system.

[0207] In one of the embodiments, the conversion module 03 is specifically configured to calculate, according to the first coordinates of the original point cloud data in the camera coordinate system, the second coordinates and the distance from the first coordinates to the origin of the camera coordinate system; calculate, according to the second coordinates, the distance, the radial distortion coefficient, the lateral distortion coefficient, the third coordinates; and calculate, according to the third coordinates and the camera intrinsic parameter, the two-dimensional coordinates of the original point cloud data in the pixel coordinate system.

[0208] In one of the embodiments, as shown in Figure 13 the multi-sensor data fusion device further includes a construction module 06 configured to construct, based on a rotation matrix and a translation vector in the extrinsic parameter between the camera and the radar, a conversion relationship between the world coordinate system and the camera coordinate system.

[0209] In one of the embodiments, the detection module 02 is specifically configured to perform two-dimensional target detection on the camera image by using a preset neural network model to obtain at least one region of interest.

[0210] In one of the embodiments, as shown in Figure 14 a target detection device is provided, which includes an acquisition module 11 and a detection module 12, wherein:

[0211] The acquisition module 11 is configured to use the above Figures 1-8The multi-sensor data fusion method provided by the embodiment acquires target point cloud data;

[0212] The detection module 12 is configured to perform three-dimensional target detection on the target point cloud to obtain a target detection result.

[0213] In one embodiment, as shown in Figure 15 A target detection apparatus is provided, which comprises a first acquisition module 21, a second acquisition module 22 and a detection module 23, wherein:

[0214] The first acquisition module 21 is configured to acquire a camera image, perform two-dimensional target detection on the camera image, and obtain at least one region of interest in the camera image.

[0215] The second acquisition module 22 is configured to acquire target point cloud data by using the multi-sensor data fusion method provided in the first aspect.

[0216] The detection module 23 is configured to obtain a target detection result according to the at least one region of interest in the camera image and the target point cloud.

[0217] In one embodiment, the detection module 23 is specifically configured to acquire a first initial target detection result according to each region of interest in the camera image, perform three-dimensional detection on the target point cloud data to acquire a second initial target detection result, perform weighted calculation on the first initial target detection result and the second initial target detection result, and obtain the target detection result according to the weighted calculation result.

[0218] For specific limitations of the multi-sensor data fusion and the target detection apparatus, refer to the limitations of the multi-sensor data fusion and the target detection method in the foregoing, which will not be repeated here. Each module in the multi-sensor data fusion and the target detection apparatus can be realized by software, hardware and a combination thereof in whole or in part. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0219] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 16As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless mode can be achieved through WIFI, operator network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a target detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0220] Those skilled in the art can understand that, Figure 16 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0221] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps:

[0222] Obtain camera images and original point cloud data at the same time;

[0223] Perform two-dimensional target detection on the camera images to obtain at least one region of interest in the camera images;

[0224] Perform two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in the pixel coordinate system;

[0225] Obtain at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image;

[0226] According to the at least one target region of interest, extract target point cloud data from the original point cloud data.

[0227] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps:

[0228] Using the above Figures 1-7The multi-sensor data fusion method provided by the embodiment acquires target point cloud data;

[0229] Three-dimensional target detection is performed on the target point cloud to obtain a target detection result.

[0230] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0231] A camera image is acquired, and two-dimensional target detection is performed on the camera image to obtain at least one region of interest in the camera image;

[0232] The multi-sensor data fusion method provided by the above Figures 1-7 The multi-sensor data fusion method provided by the embodiment acquires target point cloud data;

[0233] According to the at least one region of interest in the camera image and the target point cloud data, a target detection result is obtained.

[0234] The computer device provided by the above embodiment has similar implementation principles and technical effects to the above method embodiments, and thus will not be described here.

[0235] In one embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the following steps:

[0236] Camera images and original point cloud data at the same time are acquired;

[0237] Two-dimensional target detection is performed on the camera images to acquire at least one region of interest in the camera images;

[0238] The original point cloud data is converted into a two-dimensional image in a pixel coordinate system;

[0239] At least one target region of interest corresponding to the at least one region of interest is acquired from the two-dimensional image;

[0240] According to the at least one target region of interest, target point cloud data is extracted from the original point cloud data.

[0241] The computer readable storage medium provided by the above embodiment has similar implementation principles and technical effects to the above method embodiments, and thus will not be described here.

[0242] In one embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the following steps:

[0243] The multi-sensor data fusion method provided by the above Figures 1-7 The multi-sensor data fusion method provided by the embodiment acquires target point cloud data;

[0244] Perform 3D target detection on the target point cloud to obtain the target detection results.

[0245] The computer-readable storage medium provided in the above embodiments has similar implementation principles and technical effects to the above method embodiments, and will not be described again here.

[0246] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0247] Acquire camera images and perform two-dimensional target detection on the camera images to obtain at least one region of interest in the camera images;

[0248] Using the above Figures 1-7 The multi-sensor data fusion method provided in this embodiment acquires target point cloud data;

[0249] The target detection result is obtained based on at least one region of interest in the camera image and the target point cloud data.

[0250] The computer-readable storage medium provided in the above embodiments has similar implementation principles and technical effects to the above method embodiments, and will not be described again here.

[0251] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0252] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0253] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-sensor data fusion method, characterized by, The method comprises: acquiring camera images and original point cloud data at the same time; performing two-dimensional target detection on the camera images to acquire at least one region of interest in the camera images; performing two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in a pixel coordinate system; acquiring at least one target region of interest corresponding to the at least one region of interest from the two-dimensional image; extracting target point cloud data from the original point cloud data according to the at least one target region of interest; the extracting target point cloud data from the original point cloud data according to the at least one target region of interest comprises: acquiring target indexes of point clouds in the at least one target region of interest according to point cloud points in the at least one target region of interest and pixels in the camera images; and extracting target point cloud data from the original point cloud data according to the target indexes.

2. The method of claim 1, wherein, the acquiring target indexes of point clouds in the at least one target region of interest according to point cloud points in the at least one target region of interest and pixels in the camera images comprises: acquiring initial indexes of point cloud points in each target region of interest; the initial indexes are determined according to a corresponding relationship between the point cloud points in the target region of interest and the pixels in the camera images; performing or operation on the initial indexes of point clouds in each target region of interest to obtain target indexes of point clouds in each target region of interest.

3. The method of claim 1, wherein, the acquiring target indexes of point clouds in the at least one target region of interest according to point cloud points in the at least one target region of interest and pixels in the camera images comprises: acquiring initial indexes of point cloud points in each target region of interest, and acquiring indexes of each pixel in the camera images; performing or operation on the initial indexes of point clouds in each target region of interest to obtain initial target indexes of point clouds in each target region of interest; performing and operation on the initial target indexes of point clouds in each target region of interest and initial target indexes of point clouds in the two-dimensional image to obtain target indexes of point clouds in each target region of interest.

4. The method of claim 1, wherein, the performing two-dimensional conversion on the original point cloud data to obtain a two-dimensional image in a pixel coordinate system comprises: converting coordinates of the original point cloud data in a world coordinate system into coordinates of the original point cloud data in a camera coordinate system according to a conversion relationship between the world coordinate system and the camera coordinate system; converting coordinates of the original point cloud data in the camera coordinate system into coordinates of the original point cloud data in a pixel coordinate system according to a conversion relationship between the camera coordinate system and the pixel coordinate system; obtaining the two-dimensional image in the pixel coordinate system according to the coordinates of the original point cloud data in the pixel coordinate system.

5. The method of claim 4, wherein, the converting coordinates of the original point cloud data in the camera coordinate system into coordinates of the original point cloud data in the pixel coordinate system according to a conversion relationship between the camera coordinate system and the pixel coordinate system comprises: The coordinates of the original point cloud data in the camera coordinate system, the camera intrinsic parameters, the radial distortion coefficients and the lateral distortion coefficients are substituted into the conversion relationship between the camera coordinate system and the pixel coordinate system to calculate the two-dimensional coordinates of the original point cloud data in the pixel coordinate system.

6. The method of claim 5, wherein, The method further comprises: Based on the rotation matrix and the translation vector in the extrinsic parameters between the camera and the radar, the conversion relationship between the world coordinate system and the camera coordinate system is constructed. The method comprises: The method comprises:

7. The method of claim 4, wherein, The method comprises: The method comprises:

8. The method according to any one of claims 1 to 7, characterized in that, The method comprises: The method comprises:

9. A target detection method characterized by, The method comprises: The method comprises: The method comprises:

10. A target detection method characterized by, The method comprises: The method comprises: The method comprises: The method comprises:

11. The method of claim 10, wherein, The method comprises: The method comprises: The method comprises: The method comprises:

12. A multi-sensor data fusion apparatus, characterized by comprising: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: An extraction module is configured to extract target point cloud data from the original point cloud data according to the at least one target region of interest. The extraction module is specifically configured to: acquire a target index of the point cloud in the at least one target region of interest according to a point cloud point in the at least one target region of interest and a pixel in the camera image; and extract target point cloud data from the original point cloud data according to the target index.

13. A target detection apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire target point cloud data by using the multi-sensor data fusion method in any one of claims 1-8. A detection module is configured to perform three-dimensional target detection on the target point cloud to obtain a target detection result.

14. A target detection apparatus characterized by comprising: The device comprises: A first acquisition module is configured to acquire a camera image and perform two-dimensional target detection on the camera image to obtain at least one region of interest in the camera image. A second acquisition module is configured to acquire target point cloud data by using the multi-sensor data fusion method in any one of claims 1-8. A detection module is configured to obtain a target detection result according to the at least one region of interest in the camera image and the target point cloud data.

15. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-11.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-11.

Images