An adaptive multi-scale target detection method and system for unmanned boat environment

Abstract

The present invention discloses an unmanned boat environment adaptive multi-scale target detection method and system, which relates to the field of unmanned boat and multi-sensor fusion technology. The method comprises: pre-processing the acquired radar data to obtain a radar estimated target set; the radar data comprises data collected by millimeter wave radar and data collected by navigation radar; pre-processing the original environment image collected by the visual sensor to obtain the global atmospheric light value and transmittance; restoring the original environment image according to the global atmospheric light value and transmittance to obtain an enhanced image; obtaining a preliminary detection result according to the enhanced image and a pre-trained target detection model; the preliminary detection result comprises the confidence, detection frame and category information of the image estimated target; fusing the radar estimated target set, the enhanced image and the preliminary detection result to determine the position information, category information and confidence of the sea surface target. The present invention can accurately determine the sea surface target information.

Description

Technical Field

[0001] The present invention relates to the field of unmanned boats and multi-sensor fusion technology, and in particular to an unmanned boat environment adaptive multi-scale target detection method and system. Background Art

[0002] Unmanned surface vessels are mainly used to perform dangerous tasks that are not suitable for manned ships. Once equipped with advanced control systems, sensor systems, communication systems and weapon systems, surface unmanned vessels can perform a variety of war and non-war military tasks. In addition, in civilian use, surface unmanned vessels can perform tasks such as search and rescue, navigation and hydrological and geographical surveys. In particular, in the military field, surface unmanned vessels (hereinafter referred to as unmanned vessels) can operate flexibly, be deployed and maneuverable, and are easy to use. They can follow combat ships to sail and perform tasks, and can independently perform tasks in dangerous areas or areas where it is not suitable to dispatch manned ships.

[0003] Sensors that can be used on traditional unmanned boats include marine radar, laser radar, visual sensor, millimeter wave radar, etc. Generally, when performing tasks, unmanned boats will only be equipped with a single sensor according to the needs of the task and use the data of a single sensor; or equipped with multiple sensors, independently processing and using the data of different sensors. However, due to the changeable weather at sea, it is very easy to encounter rain, fog and other weather. Different sensors have their own advantages and disadvantages, and have their own adaptability to different environments. In inappropriate scenes, using a single sensor for detection will result in a considerable error. For example, visual sensors are low-cost, can provide intuitive images, and combined with various existing detection algorithms, give detailed target categories and relative position information, but are very easily affected by rain and fog; millimeter wave radars are low-cost, can also provide relatively accurate two-dimensional position and speed information, and are not easily affected by rain and fog, but have the disadvantage of a small detection range; marine radars are not expensive, are not easily affected by weather, have a wide detection range, and can also provide target position and speed information, but the error is large.

[0004] The multi-sensor fusion technology currently used in unmanned boats focuses more on the fusion of homogeneous sensors. Although it improves the detection accuracy, the information and detection distance that a single sensor can obtain are limited, and it cannot adapt well to most complex scenarios. There are also attempts to fuse the data of visual sensors and lidar for application in emergency obstacle avoidance tasks of unmanned boats, which have achieved good results, but the detection distance of lidar is limited by its strong dependence on the environment and lacks more application scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide an unmanned boat environment adaptive multi-scale target detection method and system to achieve the purpose of accurately detecting sea surface targets.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] An adaptive multi-scale target detection method for an unmanned boat environment, comprising:

[0008] Acquire radar data and preprocess the radar data to obtain a radar estimated target set; the radar data includes data collected by millimeter wave radar and data collected by navigation radar;

[0009] Acquire an original environment image collected by a visual sensor, and preprocess the original environment image to obtain a global atmospheric light value and transmittance;

[0010] Performing restoration processing on the original environment image according to the global atmospheric light value and the transmittance to obtain an enhanced image;

[0011] Obtaining preliminary detection results based on the enhanced image and the pre-trained target detection model; the preliminary detection results include confidence, detection frame and category information of the image estimated target;

[0012] The radar estimated target set, the enhanced image and the preliminary detection result are fused to determine the position information, category information and confidence of the sea surface target.

[0013] Optionally, the preprocessing of the radar data specifically includes:

[0014] Processing the radar data using a density-based clustering algorithm to obtain N clusters;

[0015] The radar data after the primary processing is subjected to clutter filtering and duplicate target elimination operations to obtain radar data after the secondary processing; wherein the radar data after the primary processing includes N clusters, and the radar data after the secondary processing includes M clusters, and N is greater than or equal to M; and the clusters represent radar estimated target information;

[0016] Based on the secondary processed radar data, a radar estimated target set is constructed.

[0017] Optionally, preprocessing the original environment image to obtain a global atmospheric light value and transmittance specifically includes:

[0018] Converting the original environment image to obtain a grayscale image;

[0019] Calculating the gradient information of the grayscale image, and dividing the original environment image based on the gradient information to obtain a sky area and a non-sky area;

[0020] A dark channel priori algorithm based on fast guided filtering is adopted to calculate the global atmospheric light value of the sky area, the transmittance of the sky area and the transmittance of the non-sky area.

[0021] Optionally, the using a dark channel a priori algorithm based on fast guided filtering to calculate the global atmospheric light value of the sky area, the transmittance of the sky area, and the transmittance of the non-sky area specifically includes:

[0022] According to dark channel prior theory, a dark channel image of the original environment image is calculated;

[0023] Screening pixels in the dark channel image whose brightness values ​​are greater than a set threshold in the sky area;

[0024] Determine and arrange the grayscale value of each pixel, and then use the maximum grayscale value as the global atmospheric light value of the sky area;

[0025] Based on the dark channel image and the global atmospheric light value, a fast guided filtering algorithm is used to calculate the transmittance of the sky area.

[0026] Optionally, the fusing the radar estimated target set, the enhanced image and the preliminary detection result to determine the position information, category information and confidence of the sea surface target specifically includes:

[0027] Drawing a radar chart based on the data in the radar estimated target set;

[0028] The enhanced image and the radar chart are spatially synchronized to achieve mapping of image coordinates to world coordinates, thereby obtaining position information of the sea surface target; the position information includes position information and speed information; wherein the speed information is determined according to the radar chart; the position information is used for world coordinate representation;

[0029] Based on the position information, calculating the distance between the sea surface target and the unmanned boat;

[0030] Calculating adaptive parameters of a visual sensor according to the distance, the transmittance of a sky area in the enhanced image, and the confidence of the image-predicted target;

[0031] According to the distance, calculating adaptive parameters of the radar; the radar includes a millimeter wave radar and a navigation radar;

[0032] Calculating the confidence of the sea surface target according to the adaptive parameters of the visual sensor and the adaptive parameters of the radar;

[0033] The category information of the image-estimated target is determined as the category information of the sea surface target.

[0034] Optionally, the step of performing spatial synchronization processing on the enhanced image and the radar chart to achieve mapping of image coordinates to world coordinates, thereby obtaining position and posture information of the sea surface target, specifically includes:

[0035] Mapping the enhanced image to the radar chart to obtain a composite image;

[0036] Establish a world coordinate system with the exact center of the initial position of the unmanned boat as the origin;

[0037] The pixel coordinates on the comprehensive image are converted to the world coordinate system to achieve the mapping from the image coordinates to the world coordinates, thereby obtaining the position and posture information of the sea surface target.

[0038] An unmanned boat environment adaptive multi-scale target detection system, comprising: a control computing component and a millimeter wave radar, a navigation radar and a visual sensor connected to the control computing component;

[0039] Wherein, the control calculation component is used to:

[0040] Acquire radar data and preprocess the radar data to obtain a radar estimated target set; the radar data includes data collected by millimeter wave radar and data collected by navigation radar;

[0041] Acquire an original environment image collected by a visual sensor, and preprocess the original environment image to obtain a global atmospheric light value and transmittance;

[0042] Performing restoration processing on the original environment image according to the global atmospheric light value and the transmittance to obtain an enhanced image;

[0043] Obtaining preliminary detection results based on the enhanced image and the pre-trained target detection model; the preliminary detection results include confidence, detection frame and category information of the image estimated target;

[0044] The radar estimated target set, the enhanced image and the preliminary detection result are fused to determine the position information, category information and confidence of the sea surface target.

[0045] Optionally, the navigation radar is installed on the top of the unmanned boat.

[0046] Optionally, the number of the millimeter-wave radars is three, and the three millimeter-wave radars are all installed on the bow of the unmanned boat; wherein the detection ranges of the three millimeter-wave radars do not overlap and cover a 180° range in front.

[0047] Optionally, there are three visual sensors, and the three visual sensors are installed directly below the navigation radar; wherein the fields of view of the three visual sensors do not overlap and cover a 180° range in front.

[0048] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] Through the technical solution provided by the present invention, the advantages of different sensors can be fully utilized. In common sunny, rainy and foggy weather conditions, the unmanned boat can relatively accurately judge the position information and category information of sea surface targets at different distance scales within a certain range, and obtain better target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 Flow chart of radar data preprocessing of the present invention

[0052] Figure 2 Schematic diagram of the original environment image and the image result after rain and fog removal using the dark channel prior theory of the present invention; (a) is the original environment image; (b) is the enhanced image;

[0053] Figure 3 This is the flow chart of the fusion algorithm of the present invention

[0054] Figure 4 This is a flow chart of a method for detecting multi-scale targets in an unmanned boat environment that is adaptive to the environment of the unmanned boat according to the present invention;

[0055] Figure 5 This is a structural diagram of an unmanned boat environment adaptive multi-scale target detection system of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] So far, the methods of applying the data fusion of millimeter-wave radar, navigation radar, and visual sensors to marine target detection are not complete, and most of them have problems such as low accuracy, poor robustness, and poor environmental adaptability. Multi-sensor information fusion technology is expected to overcome the defects of a single sensor, fully tap the complementary information between each other, and improve the reliability of marine target detection. Obviously, multi-sensor information fusion technology is the mainstream of technological development.

[0058] In order to solve the problem that the detection results of different sensors on the unmanned boat are easily affected by the environment and detection scale, this embodiment provides an unmanned boat environment adaptive multi-scale target detection method based on multi-sensor fusion after image pre-processing. The method first obtains data collected by millimeter-wave radar, marine radar, and visual sensor (such as visible light camera), and then filters the radar data by clustering method to remove clutter; then, according to the atmospheric scattering model and the dark channel prior method based on fast guided filtering, the transmittance and global ambient light (also known as the global atmospheric light value) are obtained; according to the transmittance and global ambient light, the image affected by rain and fog weather is restored to a rain-free and fog-free image, and the observation effect of the image is enhanced to improve the accuracy of the target detection algorithm; then the enhanced image (i.e., the rain-free and fog-free image) is input into the pre-trained target detection neural network to obtain the preliminary detection effect (including confidence, detection box and target category); finally, after completing the above processing, the pre-processed radar data, the enhanced image, and the target detection algorithm results are simultaneously input into the fusion module, and the weights of the fusion of multiple heterogeneous sensors are determined according to the transmittance and the estimated distance of the target, so as to obtain the final target posture information and category information; the details are as follows:

[0059] Step 1: Obtain radar data and preprocess it: Aggregate the data collected by the marine radar and the millimeter-wave radar to form radar data, and use a density-based clustering algorithm to process the radar data.

[0060] The navigation radar is installed at the top of the unmanned boat; three millimeter-wave radars are installed at the bow of the unmanned boat, and the detection ranges of the three millimeter-wave radars do not overlap and cover a 180° range in front; three visible light cameras are installed directly below the navigation radar, and the fields of view of the three visible light cameras do not overlap and cover a 180° range in front.

[0061] The exact center of the unmanned boat at its initial position is taken as the origin of the world coordinate system, and the external parameter matrix of each sensor is obtained according to the installation position of the sensor; based on the external parameter matrix, the data collected by the navigation radar and the data collected by the millimeter-wave radar are converted to the same coordinate system.

[0062] Among them, the extrinsic parameter matrix is ​​the product of the rotation matrix and the translation matrix; the rotation matrix contains the yaw angle of the sensor relative to the bow position of the unmanned boat, the pitch angle and roll angle of the sensor installation plane relative to the center plane of the unmanned boat itself, and the translation matrix contains the offset of the sensor relative to the center of the unmanned boat.

[0063] See also Figure 1 , step one mainly includes:

[0064] First, based on the external parameter matrix, the data of the marine radar and the data of the millimeter-wave radar are aggregated to obtain the radar data R raw , and use density-based clustering algorithm to cluster radar data R raw Processing (traversing all points, calculating the Euclidean distance from all points to their respective adjacent points, if the Euclidean distance is less than the set threshold, it is identified as a cluster, and the cycle is repeated until all points have their own clusters; among them, the threshold of the clustering algorithm is set as the measurement error of the sensor, and the adjacent points are defined as points with a certain point as the center and a radius within the threshold range); secondly, find a cluster with no radar tracking mark and only one point in all clusters, identify it as clutter and filter it out, and for other clusters, weighted average the points in the cluster, and finally obtain the radar estimated target set R after filtering out clutter and repeated targets. target .

[0065] The beneficial effect is that the advantages of the respective detection ranges of millimeter-wave radar and marine radar can be fully utilized to complement each other on the distance scale, and at the same time, more accurate clutter filtering and target detection can be achieved in the overlapping areas of the two.

[0066] Step 2: Obtain the original environment image collected by the visual sensor (a visible light camera in this embodiment) and pre-process it to obtain the global atmospheric light value and transmittance.

[0067] Step 2 specifically includes:

[0068] First, the original environment image is converted to obtain a grayscale image; secondly, the Canny operator is used to perform edge detection on the grayscale image, and the gradient information of the grayscale image is calculated. According to the gradient information of the grayscale image, the original image is divided into the sky area and the non-sky area; then, according to the dark channel prior theory, the dark channel image of the original image is calculated; then, the pixels with the top 0.1% brightness in the dark channel image are searched in the sky area of ​​the original image, and the maximum grayscale value of these pixels is used as the atmospheric light value of the sky area, which is also used as the global atmospheric light value; finally, after obtaining the dark channel image and the global atmospheric light value A, the transmittance t of the sky area and the non-sky area is calculated by the fast guided filtering method. s , t o Among them, the transmittance of the sky area is used as the weight for subsequent data fusion.

[0069] The transmittance calculation formula is as follows:

[0070]

[0071] Among them, I c Represents an image of a channel in the original environment image R / G / B, A c is the global atmospheric light value of the corresponding channel, ω is the adjustment coefficient; the adjustment coefficient needs to be selected according to the actual site. In this embodiment, the adjustment coefficient of the sky area is ω=0.3 and the adjustment coefficient of the non-sky area is ω=0.8.

[0072] The beneficial effect is that the calculated atmospheric transmittance is used to distinguish different environments, which is then used as a weight to determine the information confidence of different sensors, thereby better leveraging the advantages of different sensors in different environments.

[0073] Step 3: According to the global atmospheric light value of the sky area, the transmittance of the sky area and the transmittance of the non-sky area, the original environment image is restored to obtain a rain-free and defogging image to enhance the image observation effect. Then, the pre-trained target detection model is used to detect the enhanced image (rain-free and defogging image) to obtain the preliminary detection effect (including target confidence, detection box x i ,y i , w, h and target category label information). The formula for restoring the original environment image is as follows:

[0074]

[0075] Where J(x) represents the restored image, that is, the rain and fog removed image, and I(x) represents the original environment image.

[0076] Among them, the comparison effect between the original environment image and the rain and fog removal image is as follows: Figure 2 shown.

[0077] Step 4: Draw the preprocessed radar data into a radar map, and then synchronize it with the image data collected and processed by the visible light camera to complete the mapping of image coordinates to world coordinates.

[0078] Step 4 specifically includes:

[0079] First, according to the scales set by the millimeter wave radar and the navigation radar, the radar estimated target set R target The data in the radar chart I radar ; Secondly, the image I will be enhanced ca挠era Mapping to Radar Chart I radar Above; where the image coordinates (u c , v c) and radar chart coordinates (u r , v r ) is is the conversion matrix measured according to the installation position in the previous joint calibration; then the world coordinate system is established with the exact center of the initial position of the unmanned boat as the origin, and the position of the target is (x, y). The conversion matrix from the pixel coordinates on the radar chart to the world coordinates is defined as Wherein L is the width of the radar chart, and R is the detection distance of the radar. In this embodiment, L=1024.

[0080] In the overlapping detection range of the millimeter-wave radar and the marine radar, the data has been initially fused. Outside the detection range of the millimeter-wave radar and within the detection range of the marine radar, only the data pre-processed by the marine radar is used. In the present invention, the detection range of the marine radar does not exceed 10 times the detection range of the millimeter-wave radar.

[0081] The beneficial effect is that the distance (depth) information of the target in the image can be obtained, and the final output message can contain richer target information for subsequent decision-making.

[0082] Step 5: Data fusion: Convert the image coordinates into world coordinates (x, y), and calculate the Euclidean distance between the target and the unmanned boat through the world coordinates. Then, according to the relationship between the Euclidean distance, transmittance and confidence, the adaptive parameters of the camera and the radar are obtained. Finally, the target's posture information, category information and confidence are output after weighting.

[0083] See also Figure 3 , step five mainly includes:

[0084] First, the transformation matrix between radar coordinates and world coordinates can be obtained from step 4 Conversion matrix between image coordinates and radar coordinates Therefore, the conversion relationship from image coordinates to world coordinates is Finally normalized to The actual world coordinates of the target can be obtained That is, the position information of the target, where the speed information of the target is based on the radar estimated target set R target The target speed information is determined in .

[0085] Secondly, the Euclidean distance d between the target and the unmanned boat is calculated by the world coordinates, and then the adaptive parameter γ of the camera is obtained according to the relationship between the Euclidean distance d, the transmittance of the sky area, and the confidence of the sky area. c , the adaptive parameter γ of the radar is calculated based on the Euclidean distance r Among them, the adaptive parameters Adaptive parameters

[0086] Then, the category information in the preliminary test results is used to determine the category information of the target.

[0087] Finally, according to the adaptive parameters of the camera and the adaptive parameters of the radar, the confidence of the target is output after weighting. Among them, the confidence of the target output by the final system is C = γ c ·W c +γ r ·W r ; The weight of the camera is defined as W c =1, the radar weight is defined as W r = 1, and when there is a target only in the radar chart and no target in the original environment image, W c Set to 0, otherwise W r Set to 0.

[0088] In summary, the information output by the system is the detection box x i ,y i , w, h, target category label, target confidence C, radar estimated target set R corresponding to the target target Target speed information, world coordinates

[0089] Embodiment 2

[0090] To achieve the above purpose, this embodiment provides an unmanned boat environment adaptive multi-scale target detection method, see Figure 4 ,include:

[0091] Step 401: Acquire radar data and preprocess the radar data to obtain a radar estimated target set; the radar data includes data collected by millimeter wave radar and data collected by navigation radar.

[0092] Step 402: Acquire an original environment image collected by a visual sensor, and preprocess the original environment image to obtain a global atmospheric light value and transmittance.

[0093] Step 403: restore the original environment image according to the global atmospheric light value and the transmittance to obtain an enhanced image.

[0094] Step 404: Obtain a preliminary detection result based on the enhanced image and the pre-trained target detection model; the preliminary detection result includes the confidence, detection box and category information of the image estimated target.

[0095] Step 405: Fusing the radar estimated target set, the enhanced image and the preliminary detection result to determine the position information, category information and confidence level of the sea surface target.

[0096] The step of preprocessing the radar data specifically includes:

[0097] The radar data is processed using a density-based clustering algorithm to obtain N clusters.

[0098] The radar data after the primary processing is subjected to clutter filtering and repeated target elimination operations to obtain the radar data after the secondary processing; wherein the radar data after the primary processing includes N clusters, the radar data after the secondary processing includes M clusters, and N is greater than or equal to M; the clusters represent radar estimated target information.

[0099] Based on the secondary processed radar data, a radar estimated target set is constructed.

[0100] The step of preprocessing the original environment image to obtain the global atmospheric light value and transmittance specifically includes:

[0101] The original environment image is converted to obtain a grayscale image.

[0102] The gradient information of the grayscale image is calculated, and the original environment image is divided based on the gradient information to obtain a sky area and a non-sky area.

[0103] A dark channel priori algorithm based on fast guided filtering is adopted to calculate the global atmospheric light value of the sky area and the transmittance of the sky area.

[0104] The step adopts a dark channel a priori algorithm based on fast guided filtering to calculate the global atmospheric light value of the sky area, the transmittance of the sky area, and the transmittance of the non-sky area, specifically including:

[0105] According to the dark channel prior theory, a dark channel image of the original environment image is calculated.

[0106] Pixels having brightness values ​​greater than a set threshold in the dark channel image are screened in the sky area.

[0107] The grayscale value of each pixel is determined and arranged, and then the maximum grayscale value is used as the global atmospheric light value of the sky area.

[0108] Based on the dark channel image and the global atmospheric light value, a fast guided filtering algorithm is used to calculate the transmittance of the sky area.

[0109] The step of fusing the radar estimated target set, the enhanced image and the preliminary detection result to determine the position information, category information and confidence of the sea surface target specifically includes:

[0110] A radar chart is drawn based on the data in the radar estimated target set.

[0111] The enhanced image and the radar chart are spatially synchronized to achieve mapping of image coordinates to world coordinates, thereby obtaining the posture information of the sea surface target; the posture information includes position information and speed information; wherein the speed information is determined based on the radar chart; and the position information is used for world coordinate representation.

[0112] Based on the position information, the distance between the sea surface target and the unmanned boat is calculated.

[0113] The adaptive parameters of the visual sensor are calculated according to the distance, the transmittance of the sky area in the enhanced image, and the confidence of the image-predicted target.

[0114] According to the distance, adaptive parameters of the radar are calculated; the radar includes a millimeter wave radar and a navigation radar.

[0115] The confidence of the sea surface target is calculated according to the adaptive parameters of the visual sensor and the adaptive parameters of the radar.

[0116] The category information of the image-estimated target is determined as the category information of the sea surface target.

[0117] The step of performing spatial synchronization processing on the enhanced image and the radar chart to achieve mapping of image coordinates to world coordinates, and then obtaining the position and posture information of the sea surface target, specifically includes:

[0118] The enhanced image is mapped onto the radar chart to obtain a composite image.

[0119] The world coordinate system is established with the exact center of the initial position of the unmanned boat as the origin.

[0120] The pixel coordinates on the comprehensive image are converted to the world coordinate system to achieve the mapping from the image coordinates to the world coordinates, thereby obtaining the position and posture information of the sea surface target.

[0121] Embodiment 3

[0122] like Figure 5 As shown, this embodiment provides an unmanned boat environment adaptive multi-scale target detection system, including: a control computing component and a millimeter wave radar, a navigation radar and a visual sensor connected to the control computing component;

[0123] Wherein, the control calculation component is used to:

[0124] Acquire radar data and preprocess the radar data to obtain a radar estimated target set; the radar data includes data collected by millimeter wave radar and data collected by navigation radar.

[0125] An original environment image captured by a visual sensor is acquired, and the original environment image is preprocessed to obtain a global atmospheric light value and a transmittance.

[0126] The original environment image is restored according to the global atmospheric light value and the transmittance to obtain an enhanced image.

[0127] A preliminary detection result is obtained based on the enhanced image and the pre-trained target detection model; the preliminary detection result includes the confidence, detection frame and category information of the image estimated target.

[0128] The radar estimated target set, the enhanced image and the preliminary detection result are fused to determine the position information, category information and confidence of the sea surface target.

[0129] Wherein, the navigation radar is installed on the top of the unmanned boat.

[0130] The number of the millimeter-wave radars is three, and the three millimeter-wave radars are all installed on the bow of the unmanned boat; wherein the detection ranges of the three millimeter-wave radars do not overlap and cover a 180° range in front.

[0131] There are three visual sensors, and the three visual sensors are installed directly below the navigation radar; wherein the fields of view of the three visual sensors do not overlap and cover a 180° range in front.

[0132] The present invention discloses an unmanned boat environment adaptive multi-scale target detection method and system based on multi-sensor fusion after image pre-processing, and the main contents include: using multiple heterogeneous sensors to collect environmental data; pre-processing the environmental data obtained by different sensors respectively to screen out possible targets; processing the images taken by the camera according to the atmospheric scattering model and the dark channel prior theory to obtain the atmospheric transmittance; assigning different fusion weights to different sensors according to the transmittance and the target distance estimated by the radar, fusing the output results of different sensors according to the fusion weights, and obtaining the position information and category information of the target.

[0133] Compared with the prior art, the present invention has the following positive effects:

[0134] Through the technical solution provided by the present invention, the advantages of different sensors can be fully utilized. In common sunny, rainy and foggy weather conditions, the unmanned boat can relatively accurately judge the target posture information and category information at different distance scales within a certain range, and obtain better target detection results.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An adaptive multi-scale target detection method for unmanned boat environment, characterized in that: include: Acquiring radar data and preprocessing the radar data to obtain a radar estimated target set, specifically including: Processing the radar data using a density-based clustering algorithm to obtain N clusters; The radar data after the primary processing is subjected to clutter filtering and duplicate target elimination operations to obtain radar data after the secondary processing; wherein the radar data after the primary processing includes N clusters, and the radar data after the secondary processing includes M clusters, and N is greater than or equal to M; and the clusters represent radar estimated target information; Based on the radar data after the secondary processing, construct a radar estimated target set; The radar data includes data collected by millimeter wave radar and data collected by navigation radar; The original environment image collected by the visual sensor is obtained, and the original environment image is preprocessed to obtain the global atmospheric light value and transmittance, specifically including: Converting the original environment image to obtain a grayscale image; Calculating the gradient information of the grayscale image, and dividing the original environment image based on the gradient information to obtain a sky area and a non-sky area; Using a dark channel prior algorithm based on fast guided filtering, the global atmospheric light value of the sky area, the transmittance of the sky area, and the transmittance of the non-sky area are calculated; Performing restoration processing on the original environment image according to the global atmospheric light value and the transmittance to obtain an enhanced image; Obtaining preliminary detection results based on the enhanced image and the pre-trained target detection model; the preliminary detection results include confidence, detection frame and category information of the image estimated target; The radar estimated target set, the enhanced image and the preliminary detection result are fused to determine the position information, category information and confidence of the sea surface target, specifically including: converting the image coordinates into world coordinates (x, y), and calculating the Euclidean distance between the target and the unmanned boat through the world coordinates, then obtaining the adaptive parameters of the camera and the adaptive parameters of the radar according to the relationship between the Euclidean distance, transmittance and confidence, and finally outputting the position information, category information and confidence of the target after weighting.

2. The method for detecting unmanned vehicle environment-adaptive multi-scale targets according to claim 1, characterized in that: The dark channel a priori algorithm based on fast guided filtering is used to calculate the global atmospheric light value of the sky area, the transmittance of the sky area, and the transmittance of the non-sky area, specifically including: According to dark channel prior theory, a dark channel image of the original environment image is calculated; Screening pixels in the dark channel image whose brightness values ​​are greater than a set threshold in the sky area; Determine and arrange the grayscale value of each pixel, and then use the maximum grayscale value as the global atmospheric light value of the sky area; Based on the dark channel image and the global atmospheric light value, a fast guided filtering algorithm is used to calculate the transmittance of the sky area.

3. The method for detecting unmanned vehicle environment-adaptive multi-scale targets according to claim 1, characterized in that: The step of fusing the radar estimated target set, the enhanced image and the preliminary detection result to determine the position information, category information and confidence of the sea surface target specifically includes: Drawing a radar chart based on the data in the radar estimated target set; The enhanced image and the radar chart are spatially synchronized to achieve mapping of image coordinates to world coordinates, thereby obtaining position information of the sea surface target; the position information includes position information and speed information; wherein the speed information is determined according to the radar chart; the position information is used for world coordinate representation; Based on the position information, calculating the distance between the sea surface target and the unmanned boat; Calculating adaptive parameters of a visual sensor according to the distance, the transmittance of a sky area in the enhanced image, and the confidence of the image-predicted target; According to the distance, calculating adaptive parameters of the radar; the radar includes a millimeter wave radar and a navigation radar; Calculating the confidence of the sea surface target according to the adaptive parameters of the visual sensor and the adaptive parameters of the radar; The category information of the image-estimated target is determined as the category information of the sea surface target.

4. The method for detecting unmanned boat environment-adaptive multi-scale targets according to claim 3, characterized in that: The spatial synchronization processing of the enhanced image and the radar chart to achieve the mapping of image coordinates to world coordinates, and then obtain the position and posture information of the sea surface target, specifically includes: Mapping the enhanced image to the radar chart to obtain a composite image; Establish a world coordinate system with the exact center of the initial position of the unmanned boat as the origin; The pixel coordinates on the comprehensive image are converted to the world coordinate system to achieve the mapping from the image coordinates to the world coordinates, thereby obtaining the position and posture information of the sea surface target.

5. An unmanned boat environment adaptive multi-scale target detection system, characterized in that: include: A control computing component and a millimeter wave radar, a navigation radar and a visual sensor connected to the control computing component; Wherein, the control calculation component is used to: Acquire radar data and pre-process the radar data to obtain a radar estimated target set; the radar data includes data collected by millimeter wave radar and data collected by navigation radar; pre-process the radar data, specifically including: Processing the radar data using a density-based clustering algorithm to obtain N clusters; The radar data after the primary processing is subjected to clutter filtering and duplicate target elimination operations to obtain radar data after the secondary processing; wherein the radar data after the primary processing includes N clusters, and the radar data after the secondary processing includes M clusters, and N is greater than or equal to M; and the clusters represent radar estimated target information; Based on the radar data after the secondary processing, construct a radar estimated target set; The original environment image collected by the visual sensor is obtained, and the original environment image is preprocessed to obtain the global atmospheric light value and transmittance, specifically including: Converting the original environment image to obtain a grayscale image; Calculating the gradient information of the grayscale image, and dividing the original environment image based on the gradient information to obtain a sky area and a non-sky area; Using a dark channel prior algorithm based on fast guided filtering, the global atmospheric light value of the sky area, the transmittance of the sky area, and the transmittance of the non-sky area are calculated; Performing restoration processing on the original environment image according to the global atmospheric light value and the transmittance to obtain an enhanced image; Obtaining preliminary detection results based on the enhanced image and the pre-trained target detection model; the preliminary detection results include confidence, detection frame and category information of the image estimated target; The radar estimated target set, the enhanced image and the preliminary detection result are fused to determine the position information, category information and confidence of the sea surface target, specifically including: converting the image coordinates into world coordinates (x, y), and calculating the Euclidean distance between the target and the unmanned boat through the world coordinates, then obtaining the adaptive parameters of the camera and the adaptive parameters of the radar according to the relationship between the Euclidean distance, transmittance and confidence, and finally outputting the position information, category information and confidence of the target after weighting.

6. The unmanned boat environment adaptive multi-scale target detection system according to claim 5, characterized in that: The navigation radar is installed on the top of the unmanned boat.

7. The unmanned boat environment adaptive multi-scale target detection system according to claim 5, characterized in that: The number of the millimeter-wave radars is three, and the three millimeter-wave radars are all installed on the bow of the unmanned boat; wherein the detection ranges of the three millimeter-wave radars do not overlap and cover a 180° range in front.

8. The unmanned boat environment adaptive multi-scale target detection system according to claim 6, characterized in that: There are three visual sensors, and the three visual sensors are installed directly below the navigation radar; wherein the fields of view of the three visual sensors do not overlap and cover a 180° range in front.

Images