Target detection method and system based on image and laser point cloud fusion, and medium

Abstract

The embodiment of the invention discloses a target detection method and system based on image and laser point cloud fusion and a medium. The method comprises the following steps: acquiring a to-be-processed image and laser point cloud data; the image is input into a segmentation model, the segmentation model outputs a plurality of segmentation masks, and each segmentation mask corresponds to one target in the image; determining structural elements of the segmentation mask, and performing erosion operation on the segmentation mask according to the structural elements to obtain a target image; projecting the laser point cloud data to a two-dimensional plane of the target image, and obtaining a pixel coordinate corresponding to each point cloud; segmenting the laser point cloud data into P cylinder units, coding the point cloud in each cylinder unit into a feature vector, and converting the laser point cloud data into a pseudo image according to a pixel coordinate corresponding to each point cloud and an original three-dimensional coordinate of the point cloud; and performing target detection on the pseudo image, and determining vertex coordinates of a target. According to the invention, through combination of image segmentation and laser point cloud data, the precision and stability of target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and laser radar data technology, and in particular to a target detection method, system and medium based on image and laser point cloud fusion. Background Art

[0002] With the rapid development of technologies such as autonomous driving, robotic navigation, and intelligent surveillance, three-dimensional object detection is becoming increasingly important in many application scenarios. Traditional two-dimensional image-based object detection methods primarily rely on image data acquired by cameras. While these methods can extract object contours, they cannot provide depth and spatial position information, resulting in low target recognition accuracy in complex environments. Laser radar (LiDAR), an active sensor, can provide precise three-dimensional point cloud data, exhibits strong anti-interference capabilities, and exhibits stability in harsh lighting conditions, making it an indispensable sensor for three-dimensional object detection.

[0003] However, laser point cloud data is often sparse and noisy. Directly using this point cloud data for target detection often leads to problems such as low target recognition accuracy, difficulty in point cloud clustering, and high computational complexity. To overcome these challenges, researchers typically fuse image data with laser point cloud data, leveraging the texture and semantic information provided by the image data and the depth information provided by the laser point cloud data to improve target detection accuracy.

[0004] Existing fusion methods mostly focus on combining images with laser point cloud data for target recognition. However, significant challenges remain in extracting polygonal regions, accurately extracting target bounding boxes, removing noise points, and improving target accuracy. For multi-target detection in complex scenes, using effective algorithms to process segmentation mask regions and combine the different characteristics of image and laser point cloud data to accurately identify and extract the target's 3D bounding box remains a pressing issue. Summary of the Invention

[0005] Based on this, it is necessary to propose a target detection method, system and medium based on image and laser point cloud fusion to address the above problems.

[0006] A target detection method based on image and laser point cloud fusion, the method comprising:

[0007] Obtain images and laser point cloud data to be processed;

[0008] Inputting the image into a segmentation model, the segmentation model outputs a plurality of segmentation masks, each of the segmentation masks corresponding to an object in the image;

[0009] Determining a structural element of the segmentation mask, and performing an erosion operation on the segmentation mask according to the structural element to obtain a target image;

[0010] Projecting the laser point cloud data onto a two-dimensional plane of the target image to obtain pixel coordinates corresponding to each point cloud;

[0011] The laser point cloud data is divided into P cylindrical units, and the point cloud in each cylindrical unit is encoded into a feature vector, and the laser point cloud data is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud;

[0012] Performing target detection on the pseudo image to determine vertex coordinates of the target.

[0013] The step of determining the structural element of the segmentation mask and performing an erosion operation on the segmentation mask according to the structural element to obtain a target image specifically includes:

[0014] Obtaining a binary segmentation mask from the segmentation mask;

[0015] Extract any pixel in the binary segmentation mask as a local window with a center, and calculate the standard deviation of each pixel in the local window;

[0016] Mapping the standard deviation of each pixel to a structure element value using a linear mapping function;

[0017] According to the structure element value corresponding to each pixel, the structure element of each pixel is obtained, and an erosion operation is performed on the pixel according to the structure element to obtain a target image.

[0018] The method further includes, before projecting the laser point cloud data onto a two-dimensional plane of a target image and obtaining pixel coordinates corresponding to each point cloud:

[0019] Get the neighborhood radius threshold and quantity threshold;

[0020] Calculating the Euclidean distance matrix between each two point clouds in the laser point cloud data, determining that the two point clouds are neighbors if the Euclidean distance matrix between the two point clouds is less than or equal to the neighborhood radius threshold, and obtaining the neighborhood of each point cloud, wherein the neighborhood includes each neighbor of the point cloud;

[0021] For each point cloud, if the number of point clouds in the neighborhood reaches the threshold, the point cloud is taken as the core point, and the point clouds in the neighborhood of the core point are marked as the same cluster. The point clouds that are not the core points and do not belong to the neighborhood of the core point are marked as noise points, and the noise points are filtered.

[0022] The step of projecting the laser point cloud data onto a two-dimensional plane of a target image to obtain pixel coordinates corresponding to each point cloud specifically includes:

[0023] Converting the three-dimensional coordinates of the filtered laser point cloud data from the laser radar coordinate system to the camera coordinate system through an external parameter matrix;

[0024] The converted three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional plane of the target image through the intrinsic parameter matrix to obtain the pixel coordinates corresponding to each point cloud.

[0025] The step of dividing the laser point cloud data into P cylindrical units, encoding the point cloud in each cylindrical unit into a feature vector, and converting the laser point cloud data into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud, specifically includes:

[0026] Divide the filtered laser point cloud data into P cylindrical units, and unify the number of point clouds in each cylindrical unit;

[0027] Encoding the point cloud within each cylindrical unit into a feature vector of the first dimension;

[0028] After performing linear layer, batch normalization and ReLU activation function processing on the feature vector of the first dimension in each of the cylindrical units, a feature vector of the second dimension is obtained;

[0029] Perform a maximum pooling operation on the feature vector of the second dimension to obtain a tensor;

[0030] The tensor is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud.

[0031] The step of dividing the filtered laser point cloud data into P cylindrical units and unifying the number of point clouds in each cylindrical unit specifically includes:

[0032] Segmenting the filtered laser point cloud data into P cylindrical units, each cylindrical unit containing N point clouds;

[0033] The cylindrical units whose number of point clouds exceeds N are selected by a random sampling method, and the cylindrical units whose number of point clouds does not exceed N are filled by a zero filling method.

[0034] The performing target detection on the pseudo image to obtain vertex coordinates of the target specifically includes:

[0035] Applying a channel attention mechanism to the pseudo image to obtain a one-dimensional channel attention map;

[0036] After element-wise multiplication of the one-dimensional channel attention map and the pseudo image feature, a refined feature with channel attention is generated;

[0037] Applying a spatial attention mechanism to the refined features with channel attention to obtain a three-dimensional spatial attention map;

[0038] Multiply the three-dimensional spatial attention map by the refined features with channel attention element by element to obtain the final feature map;

[0039] Performing bounding box center position detection and size detection on the final feature map to obtain a center position prediction value and a size dimension prediction value of the target;

[0040] Convert the center position prediction value and the size dimension prediction value into the vertex coordinates of the bounding box where the object is located.

[0041] The step of performing bounding box center position detection and size detection on the final feature map to obtain a target center position prediction value and a target size dimension prediction value specifically includes:

[0042] The center position of the bounding box is detected on the final feature map to obtain the center position prediction value of the target. The center position prediction value of the target is:

[0043]

[0044] in, is the predicted value of the center position, is the predicted value in the horizontal direction of the center position, is the predicted value in the vertical direction of the center position, is the predicted value of the depth direction of the center position, F is the final feature map, is the weight vector of the dimension corresponding to the predicted value in the horizontal direction of the center position, is the weight vector of the dimension corresponding to the predicted value in the vertical direction of the center position, is the weight vector of the dimension corresponding to the predicted value in the depth direction of the center position, is the bias of the dimension corresponding to the predicted value in the horizontal direction of the center position, The predicted value in the vertical direction of the center position corresponds to the bias of the corresponding dimension, The predicted value in the depth direction of the center position corresponds to the bias of the corresponding dimension;

[0045] The bounding box size detection is performed on the final feature map to obtain the target size dimension prediction value, and the target size dimension prediction value is:

[0046]

[0047] in, is the predicted value of the size dimension, is the length prediction value, is the predicted value of width, is a highly predicted value, is the weight vector of the dimension corresponding to the length prediction value, is the weight vector of the dimension corresponding to the width prediction value, is the weight vector of the dimension corresponding to the height prediction value, is the bias of the length prediction value corresponding to the dimension, is the bias of the width prediction value corresponding to the dimension, is the bias of the dimension corresponding to the height prediction value.

[0048] An object detection system based on image and laser point cloud fusion, the system comprising:

[0049] An image and laser point cloud data acquisition module, used to acquire the image and laser point cloud data to be processed;

[0050] An image segmentation module, configured to input the image into a segmentation model, wherein the segmentation model outputs a plurality of segmentation masks, each of which corresponds to an object in the image;

[0051] a target image acquisition module, configured to determine a structural element of the segmentation mask and perform an erosion operation on the segmentation mask according to the structural element to acquire a target image;

[0052] An image and laser point cloud data alignment module is used to project the laser point cloud data onto a two-dimensional plane of a target image and obtain pixel coordinates corresponding to each point cloud;

[0053] a pseudo image acquisition module, configured to segment the laser point cloud data into P cylindrical units, encode the point cloud within each cylindrical unit into a feature vector, and convert the laser point cloud data into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud;

[0054] The target detection module is used to perform target detection on the pseudo image and determine the vertex coordinates of the target.

[0055] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0056] The embodiments of the present invention have the following beneficial effects:

[0057] Laser point cloud can provide high-precision spatial depth information and accurately locate the position of the target in three-dimensional space; image segmentation focuses on mining the texture and semantic features of the target, giving the target a rich feature description. The present invention achieves complementary advantages by combining image segmentation and laser point cloud data, greatly enhancing the detection capabilities of the present invention in complex environments, overcoming the limitations of a single technology, and significantly improving the accuracy and stability of target detection. Even in low-light conditions that cause image information to be blurred, the target is partially obscured, making feature extraction difficult, or in the event of adverse weather such as rain, snow, and fog that affects the quality of sensor data, it can still rely on the complementary advantages of the two to maintain a high detection accuracy rate and ensure accurate identification and positioning of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] in:

[0060] Figure 1 A flowchart of an embodiment of a target detection method based on image and laser point cloud fusion provided by the present invention;

[0061] Figure 2 A flowchart of another embodiment of a target detection method based on image and laser point cloud fusion provided by the present invention;

[0062] Figure 3 A schematic structural diagram of an embodiment of a target detection system based on image and laser point cloud fusion provided by the present invention;

[0063] Figure 4 This is a schematic structural diagram of an embodiment of the medium provided by the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] like Figure 1 As shown, Figure 1The present invention provides a flow chart of an embodiment of a method for target detection based on image and laser point cloud fusion. The method comprises:

[0066] S101: Acquire the image and laser point cloud data to be processed.

[0067] For example, an image to be processed is acquired from an image acquisition device (such as a camera, a scanner, or an image database), and precise three-dimensional point cloud data is provided by a laser radar.

[0068] S102: Input the image into a segmentation model, and the segmentation model outputs a number of segmentation masks, each of which corresponds to an object in the image.

[0069] For example, an image is input into a segmentation model. The model leverages its powerful feature extraction and segmentation capabilities to segment the image and output several segmentation masks. Each mask corresponds to an object in the image, accurately outlining the object's boundaries. The model also assigns a clear label to each object, identifying its category. Furthermore, the model provides a confidence score for each detected object, reflecting the reliability of the model's recognition of that object.

[0070] S103: Determine a structural element of the segmentation mask, and perform an erosion operation on the segmentation mask according to the structural element to obtain a target image.

[0071] Exemplarily, a binary segmentation mask is obtained from the segmentation mask; any pixel in the binary segmentation mask is extracted as a local window at the center, and the standard deviation of each pixel in the local window is calculated; the standard deviation of each pixel is mapped to a structure element value using a linear mapping function; according to the structure element value corresponding to each pixel, the structure element of each pixel is obtained, and an erosion operation is performed on the pixel according to the structure element to obtain a target image.

[0072] Erosion can be performed on the segmentation mask in an image to reduce or refine the white areas (i.e., foreground objects) in the mask, thereby removing small objects, disconnecting objects, or smoothing object boundaries. Erosion effectively expands the boundaries of the segmentation mask to more closely match the actual object shape, while reducing noise and unnecessary details, providing a clearer and more accurate foundation for subsequent image analysis or processing steps.

[0073] The erosion operation plays a crucial role in data processing. On the one hand, it can effectively remove noise points and points irrelevant to target detection from the laser point cloud, purifying the raw data and improving its quality and usability. On the other hand, the erosion operation can accurately correct the segmentation mask area, reducing false detections caused by segmentation errors. Through the application of this technology, the generation of bounding boxes is more accurate, significantly improving the accuracy and stability of target recognition. Whether facing objects with regular shapes or complex objects of varying shapes and sizes, the present invention can accurately extract their three-dimensional bounding boxes, providing a reliable data foundation for subsequent analysis and decision-making.

[0074] S104: Project the three-dimensional coordinates of the laser point cloud data onto the two-dimensional plane of the target image to obtain the pixel coordinates corresponding to each point cloud.

[0075] Exemplarily, first obtain a neighborhood radius threshold and a number threshold; calculate the Euclidean distance matrix between each two point clouds in the laser point cloud data; if the Euclidean distance matrix of the current two point clouds is less than or equal to the neighborhood radius threshold, determine that the current two point clouds are neighbors of each other, and obtain the neighborhood of each point cloud, including each neighbor of the point cloud; for each point cloud, if the number of point clouds in the neighborhood reaches the number threshold, then the point cloud is taken as the core point, and the point clouds in the neighborhood of the core point are marked as the same cluster, and the point clouds that are not core points and do not belong to the neighborhood of the core point are marked as noise points, and the noise points are filtered.

[0076] Furthermore, a calibration process is used to obtain the extrinsic and intrinsic parameter matrices between the LiDAR and camera. The extrinsic parameter matrix describes the rotational and translational relationship between the LiDAR coordinate system and the camera coordinate system, while the intrinsic parameter matrix contains information such as the camera's focal length and principal point position. For each 3D coordinate in the laser point cloud, the extrinsic parameter matrix is used to transform the filtered 3D coordinates of the laser point cloud data from the LiDAR coordinate system to the camera coordinate system. The intrinsic parameter matrix is then used to project the transformed 3D coordinates in the camera coordinate system onto the 2D plane of the target image to obtain the pixel coordinates corresponding to each point cloud. The laser point cloud is then mapped to the camera image through this projection process, achieving spatial alignment between the laser point cloud and the image.

[0077] S105: Divide the laser point cloud data into P cylindrical units, encode the point cloud in each cylindrical unit into a feature vector, and convert the laser point cloud data into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud.

[0078] For example, the filtered laser point cloud data is segmented into P cylindrical units, each containing N point clouds. Cylindrical units with more than N point clouds are randomly sampled, while those with less than N point clouds are filled with zeros. The point cloud within each cylindrical unit is encoded as a first-dimensional feature vector containing coordinates, reflection intensity, and related offsets, achieving integration and structuring of the point cloud information.

[0079] Furthermore, the first-dimensional feature vector in each cylindrical unit is processed with a linear layer, batch normalization, and ReLU activation function to obtain the second-dimensional feature vector; the second-dimensional feature vector is subjected to a maximum pooling operation to obtain a tensor; and the scatter operator is used to convert the tensor into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud. The pseudo image serves as an important input for subsequent related field tasks and facilitates data processing.

[0080] S106: Perform target detection on the pseudo image to obtain vertex coordinates of the target.

[0081] For example, for a pseudo-image feature in a pseudo-image, a one-dimensional channel attention map is first derived through the spatial attention mechanism and element-wise multiplied with the pseudo-image feature to obtain a refined feature with channel attention. The refined feature with channel attention is then processed through the spatial attention mechanism to generate a three-dimensional spatial attention map. Finally, the final feature map is obtained by element-wise multiplication of the three-dimensional spatial attention map with the refined feature with channel attention.

[0082] Furthermore, the center position of the final feature map is detected to obtain the target center position prediction value, which is:

[0083]

[0084] in, is the predicted value of the center position, is the predicted value in the horizontal direction of the center position, is the predicted value in the vertical direction of the center position, is the predicted value of the depth direction of the center position, F is the final feature map, is the weight vector of the dimension corresponding to the predicted value in the horizontal direction of the center position, is the weight vector of the dimension corresponding to the predicted value in the vertical direction of the center position, is the weight vector of the dimension corresponding to the predicted value in the depth direction of the center position, is the bias of the dimension corresponding to the predicted value in the horizontal direction of the center position, The predicted value in the vertical direction of the center position corresponds to the bias of the corresponding dimension, The predicted value in the depth direction of the center position corresponds to the bias of the corresponding dimension;

[0085] Perform size detection on the final feature map to obtain the target size dimension prediction value, which is:

[0086]

[0087] in, is the predicted value of the size dimension, is the length prediction value, is the predicted value of width, is a highly predicted value, is the weight vector of the dimension corresponding to the length prediction value, is the weight vector of the dimension corresponding to the width prediction value, is the weight vector of the dimension corresponding to the height prediction value, is the bias of the length prediction value corresponding to the dimension, is the bias of the width prediction value corresponding to the dimension, is the bias of the dimension corresponding to the height prediction value.

[0088] Furthermore, the center position prediction value and the size dimension prediction value are converted into the vertex coordinates of the bounding box where the target is located. The vertex coordinates of the bounding box where the target is located are:

[0089]

[0090] Among them, x v1 ~x v8 is the horizontal coordinate value of the eight vertices of the bounding box, y v1 ~y v8 is the vertical coordinate value of the eight vertices of the bounding box, z v1 ~z v8 The depth coordinate values of the eight vertices of the bounding box.

[0091] From the above description, it can be seen that laser point cloud can provide high-precision spatial depth information and accurately locate the position of the target in three-dimensional space; image segmentation focuses on mining the texture and semantic features of the target, giving the target a rich feature description. The present invention achieves complementary advantages by combining image segmentation and laser point cloud data, greatly enhancing the detection ability of the present invention in complex environments, overcoming the limitations of a single technology, and significantly improving the accuracy and stability of target detection. Even in low-light conditions that cause image information to be blurred, the target is partially obscured, making feature extraction difficult, or in the event of adverse weather such as rain, snow, and fog that affects the quality of sensor data, it can still rely on the complementary advantages of the two to maintain a high detection accuracy rate and ensure accurate identification and positioning of the target.

[0092] like Figure 2 As shown, Figure 2 A flow chart of another embodiment of a target detection method based on image and laser point cloud fusion provided by the present invention. A target detection method based on image and laser point cloud fusion, the method comprising:

[0093] S201: Acquire the image and laser point cloud data to be processed.

[0094] S202: Input the image into a segmentation model, and the segmentation model outputs a number of segmentation masks, each of which corresponds to an object in the image.

[0095] It should be noted that steps S201-S202 are Figure 1 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0096] S203: Obtain a binary segmentation mask from the segmentation mask.

[0097] Exemplarily, a binary segmentation mask M(x, y) to be processed is obtained from the segmentation mask, where (x, y) is the coordinate position of the pixel.

[0098] S204: extract any pixel in the binary segmentation mask as a local window with a center, and calculate the standard deviation of each pixel in the local window.

[0099] For example, for each pixel (x, y) in M(x, y), a local window W centered on it is selected. x,y , the window size is n×n, where n is a positive odd number.

[0100] Furthermore, the mean μ(x, y) and standard deviation σ(x, y) of the pixels in the local window are calculated using the following formula:

[0101]

[0102] Where n is the window size, W x,y is the local window, M is the segmentation mask, i and j are the horizontal and vertical coordinate indices used to traverse the elements in the local window.

[0103] S205: Map the standard deviation of each pixel to a structure element value using a linear mapping function.

[0104] For example, according to preset constants α and β, a linear mapping function is used to map the standard deviation to the structural element value s(x, y), and the calculation formula is:

[0105] s(x,y)=α×σ(x,y)+β

[0106] Among them, α and β are constants preset according to experiments and actual needs.

[0107] S206: Obtain the structure element of each pixel according to the structure element value corresponding to each pixel, and perform an erosion operation on the pixel according to the structure element to obtain a target image.

[0108] For example, for each pixel (x, y), the corresponding structuring element B is generated according to its corresponding structuring element value s(x, y). s(x,y) , the calculation formula of the structural element is as follows:

[0109]

[0110] Furthermore, the pixel (x, y) is eroded, and the calculation formula of the erosion result E(x, y) is as follows:

[0111]

[0112] Among them, ∩ represents the logical AND operation, that is, when the structural element B s(x,y) When all covered pixels are foreground pixels (eg, pixel value is 1) in the segmentation mask M, the pixel in E(x, y) is 1, otherwise it is 0.

[0113] S207: Obtain a neighborhood radius threshold and a quantity threshold.

[0114] For example, we define a neighborhood radius threshold of ε as the neighborhood of a point and select an appropriate neighborhood radius. This value determines whether two points belong to the same neighborhood. Generally speaking, this parameter can be selected based on the density of the point cloud.

[0115] Define a MinPts (number threshold): the minimum number of points within the radius. If a point's neighborhood contains at least MinPts points, the point is considered a core point.

[0116] S208: Calculate the Euclidean distance matrix between each two point clouds in the laser point cloud data. If the Euclidean distance matrix between the two current point clouds is less than or equal to the neighborhood radius threshold, determine that the two current point clouds are neighbors of each other, and obtain the neighborhood of each point cloud. The neighborhood includes each neighbor of the point cloud.

[0117] For example, it is assumed that the laser point cloud data is P = {P1, P2, ..., P n}, each point cloud P i =(x i ,y i ,z i ) contains 3D coordinate information. The Euclidean distance matrix between each pair of point clouds is calculated according to the following formula:

[0118]

[0119] Among them, d(P i ,P j ) is the point cloud P i With point cloud P j The Euclidean distance matrix between i Point cloud P i The horizontal coordinate, y i Point cloud P i The vertical coordinate, z i Point cloud P i The depth coordinate, x j Point cloud P j The horizontal coordinate, y j Point cloud P j The vertical coordinate, z j Point cloud P j The depth coordinate of .

[0120] If d(P i ,P j )≤∈ then Pi can be considered as a neighbor of Pj

[0121] Furthermore, the neighborhood of each point cloud is obtained, and the neighborhood includes each neighbor of the point cloud.

[0122] S209: For each point cloud, if the number of point clouds in the neighborhood reaches a threshold, the point cloud is taken as a core point, and the point clouds in the neighborhood of the core point are marked as the same cluster. Point clouds that are not core points and do not belong to the neighborhood of the core point are marked as noise points, and the noise points are filtered.

[0123] For example, for each point cloud, if there are at least MinPts point clouds in the neighborhood of point cloud Pi, it is regarded as a core point and the points in its neighborhood are marked as the same cluster. All density-connected points are grouped into one cluster until there are no more points to be merged.

[0124] Point clouds that are not core points and do not belong to the neighborhood of core points are marked as noise points and filtered.

[0125] S210: Convert the three-dimensional coordinates of the filtered laser point cloud data from the lidar coordinate system to the camera coordinate system through the external parameter matrix; project the three-dimensional coordinates in the converted camera coordinate system onto the two-dimensional plane of the target image through the internal parameter matrix to obtain the pixel coordinates corresponding to each point cloud.

[0126] For example, assume that the three-dimensional coordinates of the laser point cloud are:

[0127] P Lidar =(x lidar ,ylidar ,z lidar )

[0128] Among them, P Lidar is the three-dimensional coordinate of the laser point cloud, x lidar is the horizontal coordinate of the laser point cloud, y lidar is the vertical coordinate of the laser point cloud, z lidar is the depth coordinate of the laser point cloud.

[0129] Furthermore, the transformation relationship between the laser radar coordinate system and the camera coordinate system is represented by the extrinsic matrix [R|t]. The extrinsic matrix is used to transform the three-dimensional coordinates of the filtered laser point cloud data from the laser radar coordinate system to the camera coordinate system. The three-dimensional coordinates of the transformed point cloud are:

[0130] P Cam =RP Lidar +t

[0131] Among them, Pcam is the three-dimensional coordinate of the converted point cloud, R is the rotation matrix, and t is the offset.

[0132] According to the camera intrinsic parameter matrix K, the three-dimensional coordinates in the transformed camera coordinate system are projected onto the two-dimensional plane of the target image. The camera intrinsic parameter matrix K is usually expressed as:

[0133]

[0134] Among them, fx, fy are focal lengths, cx, cy are the coordinates of the principal point.

[0135] The camera coordinates are converted to the pixel coordinates (u, v) corresponding to each point cloud through the projection formula:

[0136]

[0137] Among them, (u, v) is the pixel coordinate corresponding to each point cloud, indicating the position of the point cloud in the image.

[0138] S211: Divide the filtered laser point cloud data into P cylindrical units, and unify the number of point clouds in each cylindrical unit.

[0139] Exemplarily, the filtered laser point cloud data is divided into P cylindrical units, each of which contains N point clouds; for cylindrical units with more than N point clouds, a random sampling method is used to select them, and for cylindrical units with no more than N point clouds, a zero-filling method is used to fill them.

[0140] S212: Encode the point cloud within each cylindrical unit into a feature vector of the first dimension.

[0141] For example, the input point cloud is divided into multiple Pillar units, and each point in the Pillar unit is encoded into a D-dimensional vector v(x, y, z, r, x c ,y c ,z c ,x p ,y p ). Where x, y, z, and r represent the three coordinates and reflection intensity of the point cloud in three-dimensional space respectively; xc, yc, and zc represent the offset to the arithmetic mean point of all point clouds in the Pillar; and xp and yp represent the offset value of the point cloud to the center of the Pillar.

[0142] S213: After performing linear layer, batch normalization and ReLU activation function processing on the feature vector of the first dimension in each cylindrical unit, a feature vector of the second dimension is obtained.

[0143] Exemplarily, each point containing D-dimensional features is processed by a linear layer, batch normalization, and ReLU activation function to obtain a C-dimensional feature vector.

[0144] Specifically, for the feature vector v=(v1,v2,…,v D ), the result after the linear layer is v′=Wv+b, where the weight matrix of the linear layer is W and the bias is b.

[0145] Furthermore, batch normalization is performed to normalize the batch data. For the feature vector v′=(v′1,v′2,…,v′ D ), the batch normalization operation is:

[0146]

[0147] Among them, v′ i is the mean, μ i is the standard deviation, ∈ is a small positive number to prevent division by zero.

[0148] Furthermore, the ReLU activation function is used for processing, as shown in the following formula:

[0149]

[0150] S214: Perform a maximum pooling operation on the feature vector of the second dimension to obtain a tensor.

[0151] S215: Convert the tensor into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud.

[0152] For example, a maximum pooling operation is performed on each Pillar unit, and the maximum value is taken in each dimension for the C-dimensional feature vector of the N point clouds in each Pillar unit. j The c-th dimension feature in , the maximum pooling result is:

[0153] M pj,c =max(v 1c ,v 2c ,…,v Nc )

[0154] Among them, v kc is the c-th dimension feature of the k-th point cloud. In this way, we can get a tensor of dimension (C, P), where P is the number of cylindrical units and C is the dimension.

[0155] Furthermore, the tensor is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud.

[0156] S216: Apply a channel attention mechanism to the pseudo image to obtain a one-dimensional channel attention map.

[0157] S217: After element-wise multiplication of the one-dimensional channel attention map and the pseudo image feature, a refined feature with channel attention is generated.

[0158] S218: Apply a spatial attention mechanism to the refined features with channel attention to obtain a three-dimensional spatial attention map.

[0159] S219: Multiply the three-dimensional spatial attention map and the refined features with channel attention element by element to obtain the final feature map.

[0160] S220: Perform bounding box center position detection and size detection on the final feature map to obtain the target center position prediction value and size dimension prediction value.

[0161] S221: Convert the center position prediction value and the size dimension prediction value into the vertex coordinates of the bounding box where the target is located.

[0162] It should be noted that steps S216-S221 are Figure 1 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0163] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an embodiment of a target detection system based on image and laser point cloud fusion provided by the present invention. A target detection system based on image and laser point cloud fusion, the system includes:

[0164] The image and laser point cloud data acquisition module 11 is used to acquire the image and laser point cloud data to be processed.

[0165] The image segmentation module 12 is used to input the image into a segmentation model, and the segmentation model outputs a plurality of segmentation masks, each of which corresponds to an object in the image.

[0166] A target image acquisition module 13 is used to determine the structural elements of the segmentation mask and perform an erosion operation on the segmentation mask according to the structural elements to acquire a target image;

[0167] The image and laser point cloud data alignment module 14 projects the three-dimensional coordinates of the laser point cloud data onto the two-dimensional plane of the target image to obtain the pixel coordinates corresponding to each point cloud.

[0168] The pseudo image acquisition module 15 is used to divide the laser point cloud data into P cylindrical units, encode the point cloud in each cylindrical unit into a feature vector, and convert the feature vector in each cylindrical unit into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud.

[0169] The target detection module 16 is used to perform target detection on the pseudo image and obtain the vertex coordinates of the target.

[0170] Exemplarily, in the image and laser point cloud data acquisition module 11, the image and laser point cloud data to be processed are acquired. In the image segmentation module 12, the image is input into the segmentation model, and the segmentation model outputs several segmentation masks, each of which corresponds to a target in the image. In the target image acquisition module 13, a binary segmentation mask is obtained from the segmentation mask; any pixel in the binary segmentation mask is extracted as a local window at the center, and the standard deviation of each pixel in the local window is calculated; the standard deviation of each pixel is mapped to a structural element value using a linear mapping function; based on the structural element value corresponding to each pixel, the structural element of each pixel is obtained, and the pixel is eroded according to the structural element to obtain the target image. In the image and laser point cloud data alignment module 14, the three-dimensional coordinates of the filtered laser point cloud data are converted from the lidar coordinate system to the camera coordinate system through the external parameter matrix; the three-dimensional coordinates in the converted camera coordinate system are projected onto the two-dimensional plane of the target image through the internal parameter matrix to obtain the pixel coordinates corresponding to each point cloud. In the pseudo image acquisition module 15, the filtered laser point cloud data is divided into P cylindrical units, and the number of point clouds in each cylindrical unit is unified; the point cloud in each cylindrical unit is encoded as a feature vector of the first dimension; the feature vector of the first dimension in each cylindrical unit is processed by a linear layer, batch normalization and ReLU activation function to obtain a feature vector of the second dimension; the feature vector of the second dimension is subjected to a maximum pooling operation to obtain a tensor; and the tensor is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud. In the target detection module 16, a channel attention mechanism is applied to the pseudo image to obtain a one-dimensional channel attention map; the one-dimensional channel attention map is element-wise multiplied with the pseudo image feature to generate a refined feature with channel attention; a spatial attention mechanism is applied to the refined feature with channel attention to obtain a three-dimensional spatial attention map; the three-dimensional spatial attention map is element-wise multiplied with the refined feature with channel attention to obtain a final feature map; the final feature map is subjected to bounding box center position detection and size detection to obtain the target center position prediction value and size dimension prediction value; the center position prediction value and size dimension prediction value are converted into the vertex coordinates of the bounding box where the target is located.

[0171] like Figure 4 As shown, Figure 4 The structure diagram of an embodiment of the medium provided by the present invention. The medium 30 stores at least one computer program 31, which is executed by the processor to implement the following Figure 1 and Figure 2 In one embodiment, the medium 30 may be a memory chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server.

[0172] Additionally, the processes depicted in the accompanying figures do not necessarily have to be performed in the particular order shown, or sequential order, to achieve desired results. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.

[0173] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer-readable storage medium embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, refer to the description of the method embodiments.

[0174] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other. Therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0175] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0176] For the convenience of description, when describing the above device, various units are divided into functions and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in the same one or more software and / or hardware. It should be understood by those skilled in the art that this specification embodiment can be provided as a method, system, or computer program product. Therefore, this specification embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0181] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0182] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0183] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0184] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0185] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For relevant parts, refer to the description of the method embodiments.

[0186] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A target detection method based on image and laser point cloud fusion, characterized in that: The method comprises: Obtain images and laser point cloud data to be processed; Inputting the image into a segmentation model, the segmentation model outputs a plurality of segmentation masks, each of the segmentation masks corresponding to an object in the image; Determining a structural element of the segmentation mask, and performing an erosion operation on the segmentation mask according to the structural element to obtain a target image; Projecting the laser point cloud data onto a two-dimensional plane of the target image to obtain pixel coordinates corresponding to each point cloud; The laser point cloud data is divided into P cylindrical units, and the point cloud in each cylindrical unit is encoded into a feature vector, and the laser point cloud data is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud; Performing target detection on the pseudo image to determine vertex coordinates of the target.

2. The target detection method based on image and laser point cloud fusion according to claim 1, characterized in that: The determining of the structural element of the segmentation mask and performing an erosion operation on the segmentation mask according to the structural element to obtain a target image specifically includes: Obtaining a binary segmentation mask from the segmentation mask; Extract any pixel in the binary segmentation mask as a local window with a center, and calculate the standard deviation of each pixel in the local window; Mapping the standard deviation of each pixel to a structure element value using a linear mapping function; According to the structure element value corresponding to each pixel, the structure element of each pixel is obtained, and an erosion operation is performed on the pixel according to the structure element to obtain a target image.

3. The target detection method based on image and laser point cloud fusion according to claim 2, characterized in that: Before projecting the laser point cloud data onto the two-dimensional plane of the target image and obtaining the pixel coordinates corresponding to each point cloud, the method further includes: Get the neighborhood radius threshold and quantity threshold; Calculating the Euclidean distance matrix between each two point clouds in the laser point cloud data, determining that the two point clouds are neighbors if the Euclidean distance matrix between the two point clouds is less than or equal to the neighborhood radius threshold, and obtaining the neighborhood of each point cloud, wherein the neighborhood includes each neighbor of the point cloud; For each point cloud, if the number of point clouds in the neighborhood reaches the threshold, the point cloud is taken as the core point, and the point clouds in the neighborhood of the core point are marked as the same cluster. The point clouds that are not the core points and do not belong to the neighborhood of the core point are marked as noise points, and the noise points are filtered.

4. The target detection method based on image and laser point cloud fusion according to claim 3, characterized in that: Projecting the laser point cloud data onto a two-dimensional plane of a target image to obtain pixel coordinates corresponding to each point cloud specifically includes: Converting the three-dimensional coordinates of the filtered laser point cloud data from the laser radar coordinate system to the camera coordinate system through an external parameter matrix; The converted three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional plane of the target image through the intrinsic parameter matrix to obtain the pixel coordinates corresponding to each point cloud.

5. The target detection method based on image and laser point cloud fusion according to claim 4, characterized in that: The laser point cloud data is divided into P cylindrical units, and the point cloud in each cylindrical unit is encoded into a feature vector, and the laser point cloud data is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud, specifically including: Divide the filtered laser point cloud data into P cylindrical units, and unify the number of point clouds in each cylindrical unit; Encoding the point cloud within each cylindrical unit into a feature vector of the first dimension; After performing linear layer, batch normalization and ReLU activation function processing on the feature vector of the first dimension in each of the cylindrical units, a feature vector of the second dimension is obtained; Perform a maximum pooling operation on the feature vector of the second dimension to obtain a tensor; The tensor is converted into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud.

6. The target detection method based on image and laser point cloud fusion according to claim 5, characterized in that: The step of dividing the filtered laser point cloud data into P cylindrical units and unifying the number of point clouds in each cylindrical unit specifically includes: Segmenting the filtered laser point cloud data into P cylindrical units, each cylindrical unit containing N point clouds; The cylindrical units whose number of point clouds exceeds N are selected by a random sampling method, and the cylindrical units whose number of point clouds does not exceed N are filled by a zero filling method.

7. The target detection method based on image and laser point cloud fusion according to claim 6, characterized in that: The performing target detection on the pseudo image to obtain vertex coordinates of the target specifically includes: Applying a channel attention mechanism to the pseudo image to obtain a one-dimensional channel attention map; After element-wise multiplication of the one-dimensional channel attention map and the pseudo image feature, a refined feature with channel attention is generated; Applying a spatial attention mechanism to the refined features with channel attention to obtain a three-dimensional spatial attention map; Multiply the three-dimensional spatial attention map by the refined features with channel attention element by element to obtain the final feature map; Performing bounding box center position detection and size detection on the final feature map to obtain a center position prediction value and a size dimension prediction value of the target; Convert the center position prediction value and the size dimension prediction value into the vertex coordinates of the bounding box where the object is located.

8. The target detection method based on image and laser point cloud fusion according to claim 7, characterized in that: The performing bounding box center position detection and size detection on the final feature map to obtain a target center position prediction value and a target size dimension prediction value specifically includes: The center position of the bounding box is detected on the final feature map to obtain the center position prediction value of the target. The center position prediction value of the target is: in, is the predicted value of the center position, is the predicted value in the horizontal direction of the center position, is the predicted value in the vertical direction of the center position, is the predicted value of the depth direction of the center position, F is the final feature map, is the weight vector of the dimension corresponding to the predicted value in the horizontal direction of the center position, is the weight vector of the dimension corresponding to the predicted value in the vertical direction of the center position, is the weight vector of the dimension corresponding to the predicted value in the depth direction of the center position, is the bias of the dimension corresponding to the predicted value in the horizontal direction of the center position, The predicted value in the vertical direction of the center position corresponds to the bias of the corresponding dimension, The predicted value in the depth direction of the center position corresponds to the bias of the corresponding dimension; The bounding box size detection is performed on the final feature map to obtain the target size dimension prediction value, and the target size dimension prediction value is: in, is the predicted value of the size dimension, is the length prediction value, is the predicted value of width, is a highly predicted value, is the weight vector of the dimension corresponding to the length prediction value, is the weight vector of the dimension corresponding to the width prediction value, is the weight vector of the dimension corresponding to the height prediction value, is the bias of the length prediction value corresponding to the dimension, is the bias of the width prediction value corresponding to the dimension, is the bias of the dimension corresponding to the height prediction value.

9. A target detection system based on image and laser point cloud fusion, characterized in that: The system comprises: An image and laser point cloud data acquisition module, used to acquire the image and laser point cloud data to be processed; An image segmentation module, configured to input the image into a segmentation model, wherein the segmentation model outputs a plurality of segmentation masks, each of which corresponds to an object in the image; a target image acquisition module, configured to determine a structural element of the segmentation mask and perform an erosion operation on the segmentation mask according to the structural element to acquire a target image; An image and laser point cloud data alignment module is used to project the laser point cloud data onto a two-dimensional plane of a target image and obtain pixel coordinates corresponding to each point cloud; a pseudo image acquisition module, configured to segment the laser point cloud data into P cylindrical units, encode the point cloud within each cylindrical unit into a feature vector, and convert the laser point cloud data into a pseudo image according to the pixel coordinates corresponding to each point cloud and the original three-dimensional coordinates of the point cloud; The target detection module is used to perform target detection on the pseudo image and determine the vertex coordinates of the target.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

Images