Water area target detection method based on multi-modal feature fusion and dynamic candidate box optimization

Abstract

The invention provides a water area target detection method based on multi-modal feature fusion and dynamic candidate box optimization, and relates to the technical field of image processing, and the method comprises the steps: obtaining point cloud data and image data of a to-be-detected water area; fusing the image data and the point cloud data according to the coordinates to obtain fusion features; processing the fusion features by using an improved point cloud network to obtain feature vectors; performing identification processing based on the feature vector to obtain a first detection result; screening the candidate box according to the confidence coefficient and a preset proportion parameter to obtain a second detection result; and extracting local features based on the second detection result, and performing recognition processing according to the local features and the feature vectors to obtain a target detection result of the water area to be detected. Through the method, multi-dimensional extraction of the water area information is realized, the identification capability of the model on the low-texture region is effectively enhanced, the influence of noise interference on the detection precision is reduced, the data is more accurate, and the reasoning speed and accuracy of water area target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a water area target detection method combining multimodal feature fusion and dynamic candidate frame optimization. Background Art

[0002] With the continuous advancement and development of radar technology, its application in water scenes such as ocean, water conservancy, and unmanned ships has gradually emerged, and more and more studies have begun to try to use lidar for surface target detection.

[0003] However, most existing research relies on directly transferring land-based target detection models to aquatic environments, failing to fully consider typical water characteristics such as surface reflections, ripple disturbances, and multi-source interference. These methods are susceptible to various complex factors in complex water environments, resulting in insufficient data accuracy and target recognition precision. Summary of the Invention

[0004] To solve the above problems and improve data accuracy and target detection accuracy, an embodiment of the present invention proposes a water area target detection method using multimodal feature fusion and dynamic candidate box optimization. The method includes:

[0005] Obtain point cloud data and image data of the water area to be inspected;

[0006] fusing the image data and the point cloud data according to the coordinates to obtain fusion features corresponding to each coordinate point in the water area to be detected; the fusion features include the point cloud data and image features extracted from the image data;

[0007] Processing the fused features using an improved point cloud network to obtain a feature vector corresponding to the fused features;

[0008] Performing recognition processing based on the feature vector to obtain a first detection result, where the first detection result includes a candidate box and a confidence level corresponding to the candidate box;

[0009] Filtering the candidate boxes according to the confidence level and a preset ratio parameter to obtain a second detection result;

[0010] Local features are extracted from the point cloud data corresponding to the second detection result, and recognition processing is performed based on the local features and the feature vector to obtain a target detection result of the water area to be detected.

[0011] Optionally, the preset ratio parameter includes a preset retention ratio parameter;

[0012] The screening of the candidate boxes according to the confidence level and the preset ratio parameter to obtain a second detection result includes:

[0013] According to the following formula, the confidence threshold is calculated based on the confidence corresponding to each candidate box:

[0014] Θ = S sorted [ k top ] ,

[0015] in, represents the calculated confidence threshold, Indicates the preset retention ratio parameter, S sorted [ ∙ ] A confidence threshold for calculating compliance with the retention ratio parameter;

[0016] A candidate box that meets the confidence threshold is screened out from the candidate boxes to obtain a second detection result.

[0017] Optionally, the preset scale parameter includes a preset confidence interval adjustment parameter;

[0018] The screening of the candidate boxes according to the confidence level and the preset ratio parameter to obtain a second detection result includes:

[0019] According to the confidence corresponding to each candidate box, calculate the confidence mean and confidence standard deviation;

[0020] Determining a selection range based on the confidence level according to the confidence level mean, the confidence level standard deviation, and the confidence level interval adjustment parameter;

[0021] Candidate frames whose confidence levels are within the selection range are screened out from the candidate frames to obtain a second detection result.

[0022] Optionally, extracting local features from the point cloud data corresponding to the second detection result, and performing recognition processing based on the local features and the feature vector to obtain the target detection result of the water area to be detected includes:

[0023] performing pooling processing on the point cloud data corresponding to the second detection result to extract local features;

[0024] According to the local features and the feature vector, a multi-layer perceptron is used to perform recognition processing on the second detection result to obtain a target detection result of the water area to be detected.

[0025] Optionally, the fusing the image data and the point cloud data according to the coordinates to obtain the fusion features corresponding to each coordinate point in the water area to be detected includes:

[0026] Extracting image features corresponding to each pixel coordinate point from the image data;

[0027] Projecting the point cloud of the water area to be detected onto a 2D image plane to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data;

[0028] The image features and the point cloud data are spliced ​​according to the corresponding relationship to obtain fusion features corresponding to each coordinate point in the water area to be detected.

[0029] Optionally, projecting the point cloud of the water area to be detected onto a 2D image plane to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data includes:

[0030] According to the following formula, the point cloud of the water area to be detected is projected onto the 2D image plane to obtain the correspondence between each point in the point cloud and each pixel coordinate point in the image data:

[0031] u v 1 = K cam ⋅ [R|t ] lidar → cam ⋅ x y z 1 lidar ,

[0032] Among them, u represents the horizontal pixel coordinate in the image, v represents the vertical pixel coordinate in the image, cam represents the camera image, lidar represents the radar point cloud data, is the camera intrinsic parameter matrix, [R|t ] lidar → cam is the rotation transformation matrix from radar to camera, and (x, y, z) represents the three-dimensional coordinates of the coordinate point in the point cloud data.

[0033] Optionally, the step of splicing the image features and the point cloud data according to the corresponding relationship to obtain fusion features corresponding to each coordinate point in the water area to be detected includes:

[0034] The image features and the point cloud data are spliced ​​according to the corresponding relationship, and the weights of the point cloud data and the image features are allocated using a channel attention mechanism to obtain fusion features corresponding to each coordinate point in the water area to be detected.

[0035] Optionally, performing recognition processing based on the feature vector to obtain a first detection result includes:

[0036] Separating foreground points from the point cloud data using a classifier according to the feature vector;

[0037] A rasterized regression process is performed based on the point cloud data corresponding to the foreground point and the feature vector to obtain a first detection result.

[0038] Optionally, the point cloud data includes initial point cloud data and rotated point cloud data;

[0039] The step of obtaining point cloud data and image data of the water area to be detected includes:

[0040] Acquiring initial point cloud data and image data of the water area to be detected;

[0041] The initial point cloud data is rotated to obtain rotated point cloud data.

[0042] Optionally, the image feature includes at least one of color information, brightness gradient, texture and semantic label; and the point cloud data includes at least one of three-dimensional coordinate information, rotation angle and reflection intensity.

[0043] The water area target detection method using multimodal feature fusion and dynamic candidate box optimization provided by the embodiment of the present invention has the following advantages:

[0044] In an embodiment of the present invention, the acquired image data and point cloud data are first fused according to the coordinates to obtain fused features corresponding to each coordinate point of the water area to be detected. The fused features are then processed using an improved point cloud network to obtain feature vectors corresponding to the fused features. Recognition processing is performed based on the feature vectors to obtain a first detection result. After that, candidate boxes are screened according to the confidence level and a preset scale parameter to obtain a second detection result. Finally, local features are extracted from the point cloud data corresponding to the second detection result, and recognition processing is performed based on the local features and the feature vectors to obtain a target detection result for the water area to be detected. In the above method, the fused features obtained by fusing the point cloud data and the image data realize multi-dimensional extraction of water area information, effectively enhance the model's ability to recognize low-texture areas, reduce the impact of noise on data in complex water environments, and improve data accuracy. The candidate box screening mechanism guided by confidence can realize dynamic adjustment of the threshold range, thereby achieving precise control of the candidate boxes, reducing the amount of data, reducing the impact of noise interference on detection accuracy, and improving the inference speed. Finally, the local features are combined with the global features, i.e., the feature vectors, for recognition, and the accuracy of water area target detection is further improved on the basis of the above processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments with reference to the following drawings, in which:

[0046] Figure 1 This is a flowchart of an embodiment of a method for detecting water objects using multimodal feature fusion and dynamic candidate box optimization according to the present invention;

[0047] Figure 2 This is a flowchart of another embodiment of a method for detecting water objects using multimodal feature fusion and dynamic candidate box optimization according to the present invention;

[0048] Figure 3 This is a graph showing changes in the loss function during a training process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] Reference Figure 1 , a flowchart of an embodiment of a water area target detection method using multimodal feature fusion and dynamic candidate box optimization of the present invention is provided, which may specifically include the following steps:

[0051] Step S1: Acquire point cloud data and image data of the water area to be detected.

[0052] Point cloud data refers to a collection of data points representing the shape of an object or environmental surface in three-dimensional space. This data can be acquired using a LiDAR sensor and may include the three-dimensional distance (x, y, z) of each point from the origin, as well as the reflection intensity. In embodiments of the present invention, the sensor position and timestamp corresponding to each point cloud frame can also be recorded to improve data accuracy. Furthermore, a three-dimensional bounding box can be used to annotate the point cloud image corresponding to the point cloud data to improve the information annotation of the water area to be inspected. This annotated information includes, but is not limited to, center coordinates, dimensions, and rotation angle information.

[0053] Image data refers to the RGB image of the water area to be inspected, which is captured by a camera. Timestamps are also collected to ensure spatial and temporal consistency between the point cloud data and the image data.

[0054] In addition, in order to reduce the interference of environmental interference factors on target detection and enhance the ability to recognize targets, the point cloud data can also be preprocessed. The preprocessing methods include but are not limited to point cloud filtering, downsampling, adding random noise, and adding rotation data.

[0055] Step S2: fusing the image data and the point cloud data according to the coordinates to obtain fusion features corresponding to each coordinate point in the water area to be detected; the fusion features include the point cloud data and image features extracted from the image data.

[0056] Image features are features extracted from image data, and can include low-dimensional features such as RGB color, brightness gradient, texture, and semantic labels. Image features can be extracted using a pre-trained network, such as YOLO.

[0057] The fusion processing of image data and point cloud data means that after extracting the image features corresponding to the pixels from the image data, the image features corresponding to each coordinate point and the point cloud data are spliced ​​according to the correspondence between the pixel coordinates and the three-dimensional coordinate points in the point cloud data to obtain the fused features. Specifically, the 3D point cloud corresponding to the point cloud data can be projected onto the 2D image plane, and vice versa, that is, one three-dimensional coordinate corresponds to one pixel coordinate, and then the image features and the corresponding point cloud data are spliced ​​together according to this correspondence to obtain the fused features, for example, the following formula:

[0058] F fused =[x,y,z, intensity ,R,G,B, semantic label ] , (1)

[0059] in, Indicates the fusion feature corresponding to the coordinate point, (x, y, z) represents the three-dimensional coordinates of the coordinate point in the point cloud data, intensity represents the reflection intensity, R, G, B represent the brightness values ​​of the red, green, and blue color channels respectively, and semantic label represents the semantic label.

[0060] Referring to the fusion feature example shown in the above formula (1), if the reflection intensity of the point cloud is too different from the image brightness, for example, there are abnormal points of mirror reflection on the water surface, they can be marked as noise and removed, thereby reducing interference factors and improving recognition accuracy.

[0061] Therefore, aligning the RGB image with the laser point cloud data to obtain fusion features realizes the multi-dimensional extraction of water area information, effectively enhances the model's recognition ability for low-texture areas, improves the accuracy of the data, and thus helps to improve the accuracy of water area target detection.

[0062] Step S3: Process the fused features using an improved point cloud network to obtain a feature vector corresponding to the fused features.

[0063] The improved point cloud network (Ponitnet++) mimics the hierarchical structure of a 2D convolutional neural network (CNN) to enable multi-scale local feature extraction. In this embodiment of the present invention, the improved point cloud network is used to process fused features, enabling in-depth exploration of global feature information within the water area to be inspected, obtaining feature vectors at different scales and in different scenarios, and reducing the false detection rate for water object recognition.

[0064] Step S4: performing recognition processing based on the feature vector to obtain a first detection result, where the first detection result includes a candidate box and a confidence level corresponding to the candidate box.

[0065] In an embodiment of the present invention, an anchor-free architecture can be used to process the feature vector to obtain a preliminary recognition result, i.e., a first detection result. The first detection result includes a candidate frame, i.e., a 3D recognition frame, the main information of the 3D frame, and the confidence level corresponding to the candidate frame. The main information of the 3D frame can be expressed as , represents the center position of the 3D box, and Indicates the height, width and length of the 3D box. Indicates the pitch angle.

[0066] In complex aquatic environments, anchor-free methods can directly predict the target's center, boundary points, key points, or size. This representation method can theoretically cover targets of any scale and aspect ratio, making it more adaptable to scenarios with high target diversity, such as water. In other words, using this method for preliminary recognition not only meets the requirements of removing interference information and excluding non-target points, but also reduces the false deletion rate of target points and improves computational efficiency, thereby enhancing the detection capability of targets in water.

[0067] Step S5: Screen the candidate boxes according to the confidence level and a preset ratio parameter to obtain a second detection result.

[0068] Based on the confidence level guidance, the threshold range is determined by the preset ratio parameters and the screening mechanism, and then the first detection result is screened based on the determined threshold range to obtain the second detection result. Among them, the second detection result also corresponds to the filtered candidate box, the candidate box parameters and the confidence level corresponding to the candidate box. The screening mechanism includes but is not limited to a fixed ratio mechanism and a statistical distribution pattern, etc. Different screening mechanisms can be selected according to actual conditions, and this application does not limit this. The ratio parameters can be obtained through pre-training or directly preset.

[0069] It is understandable that based on the confidence level guidance, screening by using the preset scale parameter method can achieve dynamic adjustment of the threshold range, thereby achieving precise control of the candidate box. Therefore, screening the candidate box according to the confidence level and the preset scale parameter helps to select the most useful information for target recognition for subsequent processing, reducing the amount of data to be processed, while reducing the impact of noise interference on detection accuracy and the impact of the special sea environment on the recognition process, while improving the reasoning speed. In addition, the selection of candidate boxes can also reduce the processing pressure in the subsequent stage and improve the real-time performance of the model.

[0070] Step S6: extract local features from the point cloud data corresponding to the second detection result, perform recognition processing based on the local features and the feature vector, and obtain the target detection result of the water area to be detected.

[0071] Among them, the target detection results may include the 3D prediction box, the confidence score and category corresponding to the 3D prediction box, etc.

[0072] Specifically, local features can be extracted by performing region-of-interest (ROI) pooling on the point cloud data corresponding to the second detection result. Local features supplement deeper local information. Using local features and feature vectors for recognition processing corrects and adjusts the second detection result, helping to further improve target detection accuracy.

[0073] In summary, an embodiment of the present invention provides a water area target detection method with multimodal feature fusion and dynamic candidate frame optimization. First, the acquired image data and point cloud data are fused according to the coordinates to obtain the fusion features corresponding to each coordinate point of the water area to be detected, and then the fusion features are processed using the improved point cloud network to obtain the feature vector corresponding to the fusion features, and recognition processing is performed based on the feature vector to obtain a first detection result, and then the candidate frame is screened according to the confidence level and the preset scale parameter to obtain a second detection result, and finally local features are extracted from the point cloud data corresponding to the second detection result, and recognition processing is performed based on the local features and the feature vector to obtain the target detection result of the water area to be detected. In the above method, the fusion features obtained by fusing point cloud data and image data realize the multi-dimensional extraction of water area information, effectively enhance the model's recognition ability for low-texture areas, reduce the impact of noise on data in complex water environments, and improve data accuracy; the candidate box screening mechanism guided by confidence can realize dynamic adjustment of the threshold range, thereby achieving precise control of the candidate box, reducing the amount of data, and improving the inference speed while reducing the impact of noise interference on detection accuracy; finally, local features are combined with global features, namely feature vectors, for identification, and the accuracy of water target detection is further improved on the basis of the above processing process.

[0074] Optionally, the preset ratio parameter includes a preset retention ratio parameter;

[0075] Step S5, screening the candidate boxes according to the confidence level and the preset ratio parameter to obtain a second detection result, includes:

[0076] S51. Calculate the confidence threshold according to the confidence corresponding to each candidate box according to the following formula:

[0077] Θ = S sorted [ k top ] (2)

[0078] in, represents the calculated confidence threshold, Indicates the preset retention ratio parameter, S sorted [ ∙ ] A confidence threshold for calculating compliance with the retention ratio parameter;

[0079] S52: Filter out candidate boxes that meet the confidence threshold from the candidate boxes to obtain a second detection result.

[0080] S sorted [ ∙ ] The function is a sorting function. Specifically, it can sort the confidence levels corresponding to the candidate boxes, and then output a confidence threshold that meets the requirements of the retention ratio parameter. The candidate boxes are then filtered out according to the confidence threshold to obtain the second prediction result. Among them, the confidence threshold serves as the boundary value when filtering the candidate boxes, and the retention ratio parameter can be a ratio or a specific number, which is not limited in this application. In addition, sorting the candidate boxes according to the confidence level can ensure that the candidate boxes with high confidence are processed first to improve the recall rate.

[0081] As an example, if the retention ratio parameter is 50%, the number of candidate boxes in the first detection result is 90, and the number of candidate boxes screened out is 45. The confidence threshold is the lowest value among the confidences corresponding to the 45 candidate boxes, and the candidate boxes with confidences greater than or equal to the confidence threshold are retained. Furthermore, the maximum number of retained candidate boxes N_max and the minimum number of retained candidate boxes N_min can also be preset. For example, N_max is preset to 50, and N_min is preset to 10. If the number of candidate boxes retained after step S52 is greater than 50, only the first 50 are retained; if the number of candidate boxes retained after step S52 is less than 10, the candidate boxes with low confidence are supplemented to 10 to avoid missed detection.

[0082] Therefore, when performing target detection in different water scenes, candidate frames can be screened based on confidence guidance and the retention ratio parameter can be used to achieve adaptive adjustment of the number of candidate frames, thereby achieving precise control of the candidate frames and reducing the impact of noise interference on detection accuracy.

[0083] Optionally, the preset scale parameter includes a preset confidence interval adjustment parameter;

[0084] Step S5, screening the candidate boxes according to the confidence level and the preset ratio parameter to obtain a second detection result, includes:

[0085] According to the confidence corresponding to each candidate box, calculate the confidence mean and confidence standard deviation;

[0086] Determining a selection range based on the confidence level according to the confidence level mean, the confidence level standard deviation, and the confidence level interval adjustment parameter;

[0087] Candidate frames whose confidence levels are within the selection range are screened out from the candidate frames to obtain a second detection result.

[0088] Specifically, the selection range of candidate boxes can be adjusted by the sum of the confidence mean and the parameter standard deviation. The parameter standard deviation is expressed as the product of the confidence standard deviation and the confidence interval adjustment parameter. This method can effectively cope with water environments with large data fluctuations and reduce the false detection rate.

[0089] Optionally, step S6 extracts local features from the point cloud data corresponding to the second detection result, performs recognition processing based on the local features and the feature vector, and obtains the target detection result of the water area to be detected, including:

[0090] S61, performing pooling processing on the point cloud data corresponding to the second detection result to extract local features;

[0091] S62: Perform recognition processing on the second detection result using a multi-layer perceptron according to the local features and the feature vector to obtain a target detection result of the water area to be detected.

[0092] The Multi-Layer Perceptron (MLP) is a feedforward artificial neural network composed of at least three layers of neurons. It transmits information through a fully connected architecture and uses nonlinear activation functions such as ReLU and Sigmoid to achieve complex mapping. This embodiment of the present invention leverages the classification and regression capabilities of the MLP to perform recognition and correction on the second detection result based on the feature vector and newly added local features, further improving target detection accuracy.

[0093] Optionally, the step S2 of fusing the image data and the point cloud data according to the coordinates to obtain the fusion features corresponding to each coordinate point in the water area to be detected includes:

[0094] Step S21: extracting image features corresponding to each pixel coordinate point from the image data;

[0095] Step S22: Projecting the point cloud of the water area to be detected onto a 2D image plane to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data;

[0096] Step S23: splicing the image features and the point cloud data according to the corresponding relationship to obtain fusion features corresponding to each coordinate point in the water area to be detected.

[0097] Specifically, a pre-trained network can be used to extract low-dimensional features such as RGB color, brightness gradient, texture (such as LBP features) and image features such as semantic labels from image data. The 3D point cloud is then projected onto the 2D image plane to obtain the pixel coordinates corresponding to each point and extract the image features at that position. The image features are then used as additional channels to splice with point cloud data such as point cloud coordinates and reflection intensity to obtain fused features.

[0098] Optionally, step S22 projects the point cloud of the water area to be detected onto a 2D image plane to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data, including:

[0099] Step S221: Project the point cloud of the water area to be detected onto a 2D image plane according to the following formula to obtain the correspondence between each point in the point cloud and each pixel coordinate point in the image data:

[0100] u v 1 = K cam ⋅ [R|t ] lidar → cam ⋅ x y z 1 lidar , (3)

[0101] Among them, u represents the horizontal pixel coordinate in the image, v represents the vertical pixel coordinate in the image, cam represents the camera image, lidar represents the radar point cloud data, is the camera intrinsic parameter matrix, [R|t ] lidar → cam is the rotation transformation matrix from radar to camera, and (x, y, z) represents the three-dimensional distance between each point in the point cloud and the origin.

[0102] Specifically, the calibration board is first used to calculate the joint extrinsic matrix of the camera and LiDAR. The rotation transformation matrix from the LiDAR to the camera is then obtained from the joint extrinsic matrix. The corresponding relationship is then calculated according to formula (3). Furthermore, parameter calibration is required before the LiDAR is used to obtain the rotation angle of the LiDAR.

[0103] Optionally, step S23 includes splicing the image features and the point cloud data according to the corresponding relationship to obtain fusion features corresponding to each coordinate point in the water area to be detected, including:

[0104] Step S231: splicing the image features and the point cloud data according to the corresponding relationship, and using the channel attention mechanism to assign weights to the point cloud data and the image features to obtain fusion features corresponding to each coordinate point in the water area to be detected.

[0105] Using the channel attention mechanism to assign weights to image features and point cloud data helps suppress redundant information and reduce noise interference.

[0106] Optionally, the step S4 of performing recognition processing based on the feature vector to obtain a first detection result includes:

[0107] S41, separating foreground points from the point cloud data using a classifier according to the feature vector;

[0108] S42: Perform rasterized regression processing based on the point cloud data corresponding to the foreground point and the feature vector to obtain a first detection result.

[0109] Specifically, the classifier in step S41 can be a lightweight classifier. Furthermore, considering that in general point cloud data, foreground points are usually less than background points, focal loss is selected for separation during classifier training.

[0110] Regarding step S42, as an example, the area near each foreground point is first divided into discrete grids along the X-axis and Z-axis directions, and the target center point problem is converted into a "classification + regression" problem. Specifically, the range corresponding to the separated foreground points is evenly divided with the preset device center point as the origin, and then it is determined whether the center point of the object is in the grid to complete the classification. When the center point is in the grid, the specific position of the center point is solved by regression. Finally, the same method is used to calculate various parameters, such as 3D box parameters, confidence level of the target existence, etc., to obtain a series of predicted 3D boxes.

[0111] Optionally, the point cloud data includes initial point cloud data and rotated point cloud data;

[0112] The step S1 of obtaining the point cloud data and image data of the water area to be detected includes:

[0113] Acquiring initial point cloud data and image data of the water area to be detected;

[0114] The initial point cloud data is rotated to obtain rotated point cloud data.

[0115] Considering the rotation invariance of point cloud data, the point cloud data can be rotated, for example, rotating it along the x-axis and reversing its y-axis coordinate, which can enhance the recognition ability of objects in different directions and orientations.

[0116] Optionally, the image feature includes at least one of color information, brightness gradient, texture and semantic label; and the point cloud data includes at least one of three-dimensional coordinate information, rotation angle and reflection intensity.

[0117] Reference Figure 2 A flowchart of another embodiment of a water area target detection method using multimodal feature fusion and dynamic candidate box optimization provided by the present invention is provided, which may specifically include:

[0118] Step A1, obtaining image data and point cloud data according to the aforementioned step S1;

[0119] Step A2: extracting image features according to the aforementioned step S21, and performing projection, i.e., coordinate transformation, according to the aforementioned step S22, to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data, and then splicing the image features and the point cloud data according to the correspondence to obtain a fused feature;

[0120] Step A3: Process the fused features using the Pointnet++ network as described in step S3 to obtain a feature vector corresponding to the fused features;

[0121] Step A4: Separating foreground points from the point cloud data using a classifier according to the aforementioned step S41 to obtain a foreground point mask; wherein the foreground point mask is used to identify whether a point in the point cloud data is a foreground point or a background point;

[0122] Step A5: performing rasterized regression processing based on the point cloud data corresponding to the foreground point and the feature vector according to the content of the aforementioned step S42 to obtain a first detection result including an initial 3D candidate box;

[0123] Step A6: According to the content described in the aforementioned step S5, the candidate boxes obtained in step A5 are screened based on the confidence level and the preset ratio parameter to obtain a second detection result;

[0124] Step A7: converting the point cloud data corresponding to the second detection result into coordinates in a local canonical coordinate system, and then performing pooling processing on the point cloud data after the canonical coordinate system conversion to extract local features;

[0125] Step A8: According to the content described in the aforementioned step S62, a multi-layer perceptron is used to perform recognition processing on the second detection result based on the local features and the feature vector to obtain a target detection result of the water area to be detected.

[0126] It is understood that the processing described in the embodiments of the present invention can be implemented through model building. Therefore, image data and point cloud data can be collected according to the method described in step S1, and a dataset can be constructed based on this data for model training. The processed and annotated dataset can be divided into a training set and a validation set. The ratio of the training set to the validation set can be 8:1. Of course, the ratio of the training set to the validation set can also be adjusted according to actual conditions, and the present invention is not limited to this.

[0127] During training, you can improve the training effect by adjusting parameters such as the learning rate and batch size. You can also evaluate the model's performance on a validation set, and adjust model parameters based on the results to further optimize the model. Once the model is trained, the point cloud data and other feature data can be input into the trained model. The model processes the input data, detects and identifies objects within it, and outputs bounding boxes (i.e., object detection results). Furthermore, to improve the model's detection accuracy, the bounding boxes output in step A8 can be evaluated to improve the model.

[0128] There are multiple loss functions involved in the training process. The overall loss of the model built based on steps A1-A8 is shown in the following formula:

[0129] , (4)

[0130] in, is the overall loss, which is the weighted sum of each sub-loss and is used for the optimization problem during training; is the foreground point separation loss weight, which controls the foreground point separation loss. is the candidate box regression loss weight, which controls the candidate box regression loss when executing step A5. It is the regression loss weight when executing steps A7-A8, which controls the regression loss when executing steps A7-A8. It is the foreground point separation loss, which measures the accuracy of separating the foreground and background. This is the candidate box regression loss when executing step A5. This loss measures the accuracy of candidate box generation in the first detection result. This is the regression loss when executing steps A7-A8. This loss measures the accuracy of the candidate box in the target detection result.

[0131] Figure 3 A graph showing the change in the loss function during the training process of an embodiment of the present invention is given, where the ordinate represents the training loss, the abscissa represents the training batch, the light-colored curve represents the original value of the training loss calculated for each training batch, and the dark-colored curve is the smoothed curve: it represents the smooth trend of the training loss and can indicate the overall decline. Table 1 shows the experimental comparison results of the method of the embodiment of the present invention and the existing technology (point area convolutional neural network) based on a public dataset. It represents the average precision (AP) of target recognition when the intersection over union (IoU) threshold is set to 0.7.

[0132] Table 1: Experimental comparison results of different detection models

[0133]

[0134] Combine Figure 3 As can be seen from Table 1, the method described in the embodiment of the present invention helps the model converge quickly, significantly improving the accuracy of water target detection and the reasoning speed of the model.

[0135] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0136] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A water area target detection method based on multimodal feature fusion and dynamic candidate box optimization, characterized in that: The method comprises: Obtain point cloud data and image data of the water area to be inspected; fusing the image data and the point cloud data according to the coordinates to obtain fusion features corresponding to each coordinate point in the water area to be detected; the fusion features include the point cloud data and image features extracted from the image data; Processing the fused features using an improved point cloud network to obtain a feature vector corresponding to the fused features; Performing recognition processing based on the feature vector to obtain a first detection result, where the first detection result includes a candidate box and a confidence level corresponding to the candidate box; Filtering the candidate boxes according to the confidence level and a preset ratio parameter to obtain a second detection result; Local features are extracted from the point cloud data corresponding to the second detection result, and recognition processing is performed based on the local features and the feature vector to obtain a target detection result of the water area to be detected.

2. The water area target detection method according to claim 1, characterized in that: The preset ratio parameters include preset retention ratio parameters; The screening of the candidate boxes according to the confidence level and the preset ratio parameter to obtain a second detection result includes: According to the following formula, the confidence threshold is calculated based on the confidence corresponding to each candidate box: ， in, represents the calculated confidence threshold, Indicates the preset retention ratio parameter, A confidence threshold for calculating compliance with the retention ratio parameter; A candidate box that meets the confidence threshold is screened out from the candidate boxes to obtain a second detection result.

3. The water area target detection method according to claim 1, characterized in that: The preset scale parameter includes a preset confidence interval adjustment parameter; The screening of the candidate boxes according to the confidence level and the preset ratio parameter to obtain a second detection result includes: According to the confidence corresponding to each candidate box, calculate the confidence mean and confidence standard deviation; Determining a selection range based on the confidence level according to the confidence level mean, the confidence level standard deviation, and the confidence level interval adjustment parameter; Candidate frames whose confidence levels are within the selection range are screened out from the candidate frames to obtain a second detection result.

4. The water area target detection method according to claim 1, characterized in that: The extracting of local features from the point cloud data corresponding to the second detection result, performing recognition processing based on the local features and the feature vector to obtain the target detection result of the water area to be detected includes: performing pooling processing on the point cloud data corresponding to the second detection result to extract local features; According to the local features and the feature vector, a multi-layer perceptron is used to perform recognition processing on the second detection result to obtain a target detection result of the water area to be detected.

5. The water area target detection method according to claim 1, characterized in that: The fusing of the image data and the point cloud data according to the coordinates to obtain the fusion features corresponding to each coordinate point in the water area to be detected includes: Extracting image features corresponding to each pixel coordinate point from the image data; Projecting the point cloud of the water area to be detected onto a 2D image plane to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data; The image features and the point cloud data are spliced ​​according to the corresponding relationship to obtain fusion features corresponding to each coordinate point in the water area to be detected.

6. The water area target detection method according to claim 5, characterized in that: The step of projecting the point cloud of the water area to be detected onto a 2D image plane to obtain a correspondence between each point in the point cloud and each pixel coordinate point in the image data includes: According to the following formula, the point cloud of the water area to be detected is projected onto the 2D image plane to obtain the correspondence between each point in the point cloud and each pixel coordinate point in the image data: ， Among them, u represents the horizontal pixel coordinate in the image, v represents the vertical pixel coordinate in the image, cam represents the camera image, lidar represents the radar point cloud data, is the camera intrinsic parameter matrix, is the rotation transformation matrix from radar to camera, and (x, y, z) represents the three-dimensional coordinates of the coordinate point in the point cloud data.

7. The water area target detection method according to claim 5, characterized in that: The step of splicing the image features and the point cloud data according to the corresponding relationship to obtain fusion features corresponding to each coordinate point in the water area to be detected includes: The image features and the point cloud data are spliced ​​according to the corresponding relationship, and the weights of the point cloud data and the image features are allocated using a channel attention mechanism to obtain fusion features corresponding to each coordinate point in the water area to be detected.

8. The water area target detection method according to claim 1, characterized in that: The performing identification processing based on the feature vector to obtain a first detection result includes: Separating foreground points from the point cloud data using a classifier according to the feature vector; A rasterized regression process is performed based on the point cloud data corresponding to the foreground point and the feature vector to obtain a first detection result.

9. The water area target detection method according to claim 1, characterized in that: The point cloud data includes initial point cloud data and rotated point cloud data; The step of obtaining point cloud data and image data of the water area to be detected includes: Acquiring initial point cloud data and image data of the water area to be detected; The initial point cloud data is rotated to obtain rotated point cloud data.

10. The water area target detection method according to claim 1, characterized in that: The image features include at least one of color information, brightness gradient, texture and semantic label; the point cloud data includes at least one of three-dimensional coordinate information, rotation angle and reflection intensity.

Images