Millimeter wave radar and visual information fusion detection method for internal lake ship

Abstract

The invention provides a millimeter wave radar and visual information fusion detection method for an internal lake ship, and the method comprises the steps: obtaining the speed and azimuth information of an object in a detected range based on a millimeter wave radar, carrying out the target recognition through a photoelectric image detection module based on video frame data obtained in real time, and obtaining the position and type of a target in an image; a timestamp label is attached; according to the timestamp of each frame of the photoelectric data, selecting a radar frame with the minimum time difference as a matching frame, and mapping the two-dimensional plane coordinates of the radar to the two-dimensional plane coordinates of the photoelectric image by adopting a homography matrix, thereby mapping the radar image under the same frame into the photoelectric image; the echo points in the target rectangular frame are left, and the rest are filtered, so that the type, orientation and speed information of the target is comprehensively obtained and displayed in real time. Through timestamp synchronous matching and homography matrix mapping, the problems of missing detection and error positioning caused by the fact that a traditional single sensor is easily interfered by water surface light reflection, rain and fog shielding and the like in a complex water area environment are effectively solved.

Description

Technical Field

[0001] The invention belongs to the field of intelligent navigation technology based on computer data processing, and in particular relates to a detection method for inland lake ship millimeter wave radar and visual information fusion. Background Art

[0002] With the continuous development of the inner lake environment, the working areas of ships have become more complex, and floating objects, water piles and various small boats have had a significant impact on navigation safety and operational efficiency. Floating objects such as branches and garbage may cause collision risks, while the presence of water piles may affect the navigation path and increase the possibility of accidents. In addition, the increase in the number of small boats has made the navigation area more crowded, further increasing the risk of mutual collision. These factors not only extend the navigation time and fuel consumption, but also increase the workload of the crew. In order to meet these challenges, the introduction of an intelligent navigation perception system is particularly important. The system can monitor the surrounding environment in real time, provide navigation advice, help ships avoid obstacles, and improve the safety and efficiency of navigation. However, the current intelligent perception system often uses a single sensor, which is susceptible to missed detection and mispositioning problems caused by interference such as water surface reflection, rain and fog in complex water environments. Summary of the invention

[0003] In view of the above problems, the present invention proposes a method for detecting inner lake ships by fusion of millimeter wave radar and visual information, comprising the following steps: S1, the millimeter wave radar transmits and receives echoes to obtain the speed and direction information of the object within the measured range, wherein the direction information includes coordinate information and a timestamp tag; S2, based on the video frame data acquired in real time, target recognition is performed through the photoelectric image detection module to obtain the position and category of the target in the image, wherein the position includes the target frame and coordinates, and a timestamp label is attached; S3, according to the timestamp of each frame of photoelectric data, find the radar frame data closest in time, calculate the time difference, select the radar frame with the smallest time difference as the matching frame, and use the homography matrix to map the two-dimensional plane coordinates of the radar to the two-dimensional plane coordinates of the photoelectric image, so as to map the radar image in the same frame to the photoelectric image; S4, the echo points in the target rectangular frame are retained and the rest are filtered out, so as to fully obtain and display the type, direction and speed information of the target in real time.

[0004] Preferably, in S2, the photoelectric image detection module uses Yolov5 as a basic framework, including a backbone network, a feature fusion layer and a prediction layer; The backbone network includes a convolution layer, a slice splicing layer and multiple composite layers; after preprocessing, each frame of the video stream enters the convolution layer to extract low-level features, including edge, texture and color information, and performs region division and reconstruction through the slice splicing layer to reduce the amount of calculation and extract more detailed information; then, the data passes through multiple composite layers to achieve deep feature extraction and learn the structural information of the target object; the backbone network finally performs multi-scale feature extraction through the spatial pyramid pooling SPP layer to expand the receptive field; The feature fusion layer includes a path aggregation network; firstly, multi-scale features are extracted from the backbone network, and high-level and low-level features are fused by upsampling and cascading, so that features of different resolutions share information and enhance the robustness of detecting water targets; then, features are further extracted and optimized through convolutional layers and residual units, and finally a feature map for target detection is formed and passed to the prediction layer; The prediction layer contains a convolutional network. First, the network further refines the feature representation through multiple 3×3 convolutional layers and predicts the category of the target through a fully connected layer. The model uses an anchor box mechanism to generate an initial detection box and makes precise adjustments based on bounding box regression to adapt to the true shape and position of the target. Finally, the model applies non-maximum suppression to remove redundant boxes and ultimately outputs the location, category, and confidence of the target object.

[0005] Preferably, the specific implementation process of S3 is: S31, collecting radar target point information and target pixel information of different directions of N groups of time matching frames, respectively expressed as ( x R , y R ) and (u, v ); S32, each time 4 groups of data are selected from the N groups of data to calculate the homography matrix. For each calculation, the least squares method is used to obtain the homography matrix corresponding to each group of feature points. ; S33, for each solution Find the Euclidean distance error of all point pairs in the entire data set , and then calculate it based on the root mean square error ; S34, record the root mean square error of each calculation, and solve Minimum homography matrix; S35, based on the correspondence matrix solved in S34, maps the two-dimensional plane coordinates of the radar to the two-dimensional plane coordinates of the photoelectric image.

[0006] Preferably, the specific implementation process of S32 is as follows: Through the acquired radar photoelectric corresponding data, N groups of data are collected, and 4 groups of data are randomly selected each time to calculate the homography matrix; the coordinate transformation formula of the homography matrix is ​​as follows:

[0007] Among them, H is a 3*3 matrix, s is a constant, which is a scaling factor used to convert homogeneous coordinates back to Cartesian coordinates; then expand the formula and normalize to remove the scaling factor, and get the following formula:

[0008] Finally, the following matrix formula is formed through the linear system as follows:

[0009] The above formula is transformed into:

[0010] Q is a 2Nx9 matrix, N is the number of feature targets, h is a 9x1 matrix, that is, the homography matrix; the 9 parameters in h are solved by at least 4 non-collinear feature point pairs; the matrix solved by the least squares method is: .

[0011] Preferably, the specific implementation process of S33 is as follows: The influence of noise is removed by calculating the Euclidean distance error; for each solved homography matrix, the Euclidean distance error of all point pairs in the entire data set is calculated :

[0012] in,( ) is the point in the real image, The camera point corresponding to the radar is calculated by the homography matrix; Then, we can calculate the root mean square error For larger errors:

[0013] Where N is the N sets of data collected.

[0014] Preferably, the specific implementation process of S35 is as follows: Based on the solved homography matrix, the point position on the radar image of the corresponding timestamp is calculated by the following formula to obtain the corresponding camera point position:

[0015] in, x ri，y ri is the radar point coordinate, and the calculated point is projected onto the plane of the photoelectric image.

[0016] Preferably, S4 is specifically: According to the calculated homography matrix, all the points on the radar image corresponding to each timestamp are projected onto the camera plane through calculation. Then, according to the results of the photoelectric image detection module, only the radar projection points projected onto the detection target are retained, that is, the echo points of the target. The category, speed, distance, and azimuth information of the target during driving are obtained and fully displayed in real time on the intelligent display system.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a method for fusion detection of inner lake ships based on millimeter wave radar and visual information. This method effectively solves the problem of missed detection and mispositioning caused by the interference of water surface reflection, rain and fog in complex water environment by synchronous matching of timestamps and mapping of homography matrix. Aiming at the problem that the traditional homography matrix solution is sensitive to the collinearity of feature points, a matrix calculation method based on random sampling iterative optimization and root mean square error screening is proposed. By dynamically selecting the radar-image matching frame with the smallest time difference and introducing the Euclidean distance error evaluation mechanism, the mapping distortion caused by sensor jitter or feature noise is significantly reduced. In addition, the image blurring caused by the shaking of fishing boats is reduced by slicing splicing layers, and the composite layer fusion of residual connection and spatial pyramid pooling (SPP) is combined to achieve the synchronous detection of multi-scale targets (such as small buoys within 2 meters and 30-meter cruise ships) in the inner lake environment. At the same time, the path aggregation network (PANet) is used to optimize the feature fusion mechanism, so that the target recognition accuracy under complex backgrounds (such as bridges and dense ships) is significantly improved. Through the radar echo point screening mechanism within the target detection frame, more than 90% of invalid clutter is filtered out, realizing the dynamic correlation fusion of multi-source information. The system outputs the target category, distance, direction and speed parameters in real time, and displays them visually, providing high-confidence data support for ship collision avoidance decisions.

[0018] The experimental results show that in the test in inner lake waters, the inner lake ship fusion detection method based on millimeter-wave radar and visual information is significantly superior to the existing technology in terms of perception accuracy, environmental robustness and real-time performance, and can be widely used in scenarios such as intelligent navigation and autonomous obstacle avoidance of inner lake ships. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an overall flow chart of the detection method of the present invention.

[0020] Figure 2 It is a model structure diagram of the photoelectric target detection module YOLOV5 designed by the present invention.

[0021] Figure 3 It is a schematic diagram of the radar acquisition plane and pixel plane.

[0022] Figure 4 It is the first experimental result detection diagram generated in the embodiment of the present invention.

[0023] Figure 5 It is a second experimental result detection diagram generated in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The specific implementation process of the present invention is further described below in conjunction with specific embodiments.

[0025] The purpose of the present invention is to provide a ship intelligent perception system, which can monitor the surrounding situation of the ship in real time during navigation, automatically identify surrounding ships, bridges, docks, islands, shores, piles, etc., and obtain their speed and direction information, thereby greatly improving the safety and operation efficiency of the ship during navigation. The overall implementation process of the present invention is as follows Figure 1 As shown: S1, the millimeter wave radar transmits and receives echoes to obtain the speed and direction information of the object within the measured range, wherein the direction information includes coordinate information and a timestamp tag; S2, based on the video frame data acquired in real time, target recognition is performed through the photoelectric image detection module to obtain the position and category of the target in the image, wherein the position includes the target frame and coordinates, and a timestamp label is attached; S3, according to the timestamp of each frame of photoelectric data, find the radar frame data closest in time, calculate the time difference, select the radar frame with the smallest time difference as the matching frame, and use the homography matrix to map the two-dimensional plane coordinates of the radar to the two-dimensional plane coordinates of the photoelectric image, so as to map the radar image in the same frame to the photoelectric image; S4, the echo points in the target rectangular frame are retained and the rest are filtered out, so as to fully obtain and display the type, direction and speed information of the target in real time.

[0026] 1. Millimeter wave radar detection The millimeter-wave radar transmits and receives echoes to obtain information such as the speed and direction of objects within the measured range. The host computer receives the original echo information, performs field processing, obtains the actual speed and direction information of the detected target, and attaches a timestamp label.

[0027] 2. Photoelectric detection Photoelectric image detection uses Yolov5 as the basic framework, and carries out scenario-based optimization design for the actual needs of inland lake vessels in complex water environments, making it more distinctive and practical in target detection. The detection process is as follows: First, slice and splice each frame of the video stream, reduce the amount of calculation and extract more detail information through feature fusion, which is particularly suitable for the image acquisition needs of fishing boats in a shaking environment. Then, the image enters the composite layer, and multi-scale features are extracted using residual connections and spatial pyramid pooling (SPP) to ensure the detection capability of targets of different sizes (such as ships, bridges, docks, etc.). Then, in the feature fusion layer, the PANet module is used to integrate features of different scales, combined with upsampling and splicing operations to further retain detail information, which is particularly suitable for target recognition in complex backgrounds. Next, the prediction layer performs bounding box regression and category prediction based on the fused features. Finally, non-maximum suppression (NMS) is applied to remove redundant frames and output the final detection results, including the location, category and confidence of the target object.

[0028] The network structure of the photoelectric image detection module is as follows: Figure 2 As shown, Yolov5 is used as the basic framework, including the backbone network, feature fusion layer and prediction layer; Among them, the backbone network includes convolutional layers, slice splicing layers, and multiple composite layers. Each frame of the video stream first passes through the input preprocessing layer to perform operations such as normalization and color adjustment on the image to enhance the adaptability of the model. Subsequently, the image enters the convolutional layer to extract low-level features such as edges, textures, and color information, and is divided and reconstructed through the slice splicing layer to reduce the amount of calculation and extract more detailed information. Next, the data passes through multiple composite layers to achieve deep feature extraction, ensuring that the model can learn the structural information of the target objects (such as ships, bridges, docks, etc.). In order to further enhance the model's perception of targets of different scales, the backbone network finally performs multi-scale feature extraction through the spatial pyramid pooling (SPP) layer to expand the receptive field, so that the detection system can better cope with targets of different sizes.

[0029] The feature fusion layer includes a path aggregation network (PANet (Path Aggregation Network)). First, multi-scale features are extracted from the backbone network (Backbone). By upsampling and cascading high-level and low-level features, features of different resolutions share information and enhance the robustness of detecting targets on water. Then, features are further extracted and optimized through convolutional layers and residual units. Finally, the feature map used for target detection is formed and passed to the prediction layer.

[0030] The prediction layer contains a convolutional network. First, the network further refines the feature representation through multiple 3×3 convolutional layers and predicts the category of the target through a fully connected layer. Then, the model uses the anchor box mechanism to generate the initial detection box and makes precise adjustments based on bounding box regression to adapt to the true shape and position of the target. Finally, in order to improve the accuracy of the detection results, the model applies non-maximum suppression to remove redundant boxes and finally outputs the location, category and confidence of the target object to ensure the reliability of the detection results.

[0031] 3. Radar and Photoelectric Calibration First, time calibration is performed to extract radar and optoelectronic images at the same time, and spatial calibration is performed using a plane-to-plane method. Figure 3 According to the principle of homography transformation, the conversion relationship between the calibration radar and the camera is the mapping relationship between the target point on the calibration radar plane and the corresponding target point on the pixel plane. The specific process includes the following: 1. Collect radar target point information and target pixel information of different directions of N groups of time matching frames, which are expressed as ( x R , y R ) and (u, v ).

[0032] 2. Each time, select 4 groups (at least 4 groups) of data from N groups to calculate the homography matrix. For each calculation, use the least squares method to obtain the homography matrix corresponding to each group of feature points. ; The radar photoelectric corresponding data obtained by S31 is used to collect N groups of data, and 4 groups of data are randomly selected each time to calculate the homography matrix.

[0033] The homography matrix is ​​a mathematical tool used to describe the projective transformation between two planes, which can map points in the radar coordinate system to the camera coordinate system. Figure 3 As shown, the coordinate transformation formula is as follows:

[0034] Among them, H is a 3*3 matrix, s is a constant, which is a scale factor. Its function is to convert homogeneous coordinates back to Cartesian coordinates. Then expand the formula and normalize to remove the scale factor, and get the following formula:

[0035] Finally, the following matrix formula is formed through the linear system as follows:

[0036] The above formula can be written as:

[0037] In the above formula, Q is a 2Nx9 matrix, N is the number of feature targets, h is a 9x1 matrix, and is the homography matrix. The 9 parameters in h require at least 4 non-collinear feature point pairs to solve the homography matrix, and this overdetermined system of equations can be solved by the least squares method. The solved matrix is:

[0038] 3. For each solution Find the Euclidean distance error of all point pairs in the entire data set , and then calculate it based on the root mean square error .

[0039] The influence of noise is removed by calculating the Euclidean distance error; for each solved homography matrix, the Euclidean distance error of all point pairs in the entire data set is calculated :

[0040] in,( ) is the point in the real image, The camera point corresponding to the radar is calculated by the homography matrix; Then, we can calculate the root mean square error For larger errors:

[0041] Where N is the N sets of data collected.

[0042] 4. Record the root mean square error of each calculation and solve Minimal homography matrix.

[0043] 5. Based on the correspondence matrix solved by S34, the two-dimensional plane coordinates of the radar are mapped to the two-dimensional plane coordinates of the photoelectric image.

[0044] Based on the solved homography matrix, the point position on the radar image of the corresponding timestamp is calculated by the following formula to obtain the corresponding camera point position:

[0045] in, x ri ，y ri is the radar point coordinate, and the calculated point is projected onto the plane of the photoelectric image.

[0046] 4. Comprehensive information screening and display First, the host computer processes the original callback message information of the millimeter-wave radar to obtain the detected position, speed and other information. Then, according to the calibration results, the radar detection information at the same time is mapped to the image. Then, according to the detection results of the photoelectric image detection module, only the radar echo points in the detection frame, that is, the echo points of the target, are retained. In this way, an intelligent perception system can be formed to obtain the category, speed, distance, direction, etc. of the target during driving, and display them to improve the safety of navigation.

[0047] The specific experimental results are as follows Figure 4 and Figure 5 As shown in the figure, the left side shows the camera taking a picture, the detection result of the target is circled, the white points in the detection box are the points where the radar plane is mapped to the camera plane, and the green points are the aggregated points, which are labeled. The right side shows the millimeter wave radar detection map, the zero point is the radar position, and the green points are the detected echo signals (because the radar is a 4D radar, there are multiple points overlapped in the plane map.) The combined results are displayed in the upper right corner of the image, including: the number, type, distance and speed of the object.

[0048] The experimental results show that in the test in inner lake waters, the inner lake ship fusion detection method based on millimeter-wave radar and visual information is significantly superior to the existing technology in terms of perception accuracy, environmental robustness and real-time performance, and can be widely used in scenarios such as intelligent navigation and autonomous obstacle avoidance of inner lake ships.

[0049] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0050] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for detecting inner lake ships by fusion of millimeter wave radar and visual information, characterized in that: The following steps are involved: S1, the millimeter wave radar transmits and receives echoes to obtain the speed and direction information of the object within the measured range, wherein the direction information includes coordinate information and a timestamp tag; S2, based on the video frame data acquired in real time, target recognition is performed through the photoelectric image detection module to obtain the position and category of the target in the image, wherein the position includes the target frame and coordinates, and a timestamp label is attached; S3, according to the timestamp of each frame of photoelectric data, find the radar frame data closest in time, calculate the time difference, select the radar frame with the smallest time difference as the matching frame, and use the homography matrix to map the two-dimensional plane coordinates of the radar to the two-dimensional plane coordinates of the photoelectric image, so as to map the radar image in the same frame to the photoelectric image; S4, the echo points in the target rectangular frame are retained and the rest are filtered out, so as to fully obtain and display the type, direction and speed information of the target in real time.

2. The method for detecting inner lake ship millimeter wave radar and visual information fusion according to claim 1, characterized in that: In S2, the photoelectric image detection module uses Yolov5 as the basic framework, including a backbone network, a feature fusion layer and a prediction layer; The backbone network includes a convolution layer, a slice splicing layer and multiple composite layers; after preprocessing, each frame of the video stream enters the convolution layer to extract low-level features, including edge, texture and color information, and performs region division and reconstruction through the slice splicing layer to reduce the amount of calculation and extract more detailed information; then, the data passes through multiple composite layers to achieve deep feature extraction and learn the structural information of the target object; the backbone network finally performs multi-scale feature extraction through the spatial pyramid pooling SPP layer to expand the receptive field; The feature fusion layer includes a path aggregation network; firstly, multi-scale features are extracted from the backbone network, and high-level and low-level features are fused by upsampling and cascading, so that features of different resolutions share information and enhance the robustness of detecting water targets; then, features are further extracted and optimized through convolutional layers and residual units, and finally a feature map for target detection is formed and passed to the prediction layer; The prediction layer contains a convolutional network. First, the network further refines the feature representation through multiple 3×3 convolutional layers and predicts the category of the target through a fully connected layer. The model uses an anchor box mechanism to generate an initial detection box and makes precise adjustments based on bounding box regression to adapt to the true shape and position of the target. Finally, the model applies non-maximum suppression to remove redundant boxes and ultimately outputs the location, category, and confidence of the target object.

3. The method for detecting inner lake ship millimeter wave radar and visual information fusion as claimed in claim 1, characterized in that: The specific implementation process of S3 is as follows: S31, collecting radar target point information and target pixel information of different directions of N groups of time matching frames, respectively expressed as ( x R , y R ) and (u, v ); S32, each time 4 groups of data are selected from the N groups of data to calculate the homography matrix. For each calculation, the least squares method is used to obtain the homography matrix corresponding to each group of feature points. ; S33, for each solution Find the Euclidean distance error of all point pairs in the entire data set , and then calculate it based on the root mean square error ; S34, record the root mean square error of each calculation, and solve Minimum homography matrix; S35, based on the correspondence matrix solved in S34, maps the two-dimensional plane coordinates of the radar to the two-dimensional plane coordinates of the photoelectric image.

4. The method for detecting inner lake ship millimeter wave radar and visual information fusion as claimed in claim 3, characterized in that: The specific implementation process of S32 is as follows: Through the acquired radar photoelectric corresponding data, N groups of data are collected, and 4 groups of data are randomly selected each time to calculate the homography matrix; the coordinate transformation formula of the homography matrix is ​​as follows: Among them, H is a 3*3 matrix, s is a constant, which is a scaling factor used to convert homogeneous coordinates back to Cartesian coordinates; then expand the formula and normalize to remove the scaling factor, and get the following formula: Finally, the following matrix formula is formed through the linear system as follows: The above formula is transformed into: Q is a 2Nx9 matrix, N is the number of feature targets, h is a 9x1 matrix, that is, the homography matrix; the 9 parameters in h are solved by at least 4 non-collinear feature point pairs; the matrix solved by the least squares method is: 。 5. The method for detecting inner lake ship millimeter wave radar and visual information fusion as claimed in claim 3, characterized in that: The specific implementation process of S33 is as follows: The influence of noise is removed by calculating the Euclidean distance error; for each solved homography matrix, the Euclidean distance error of all point pairs in the entire data set is calculated : in,( ) is the point in the real image, The camera point corresponding to the radar is calculated by the homography matrix; Then, we can calculate the root mean square error For larger errors: Where N is the N sets of data collected.

6. The method for detecting inner lake ship millimeter wave radar and visual information fusion as claimed in claim 3, characterized in that: The specific implementation process of S35 is as follows: Based on the solved homography matrix, the point position on the radar image of the corresponding timestamp is calculated by the following formula to obtain the corresponding camera point position: in, x ri ，y ri is the radar point coordinate, and the calculated point is projected onto the plane of the photoelectric image.

7. The method for detecting inner lake ship millimeter wave radar and visual information fusion according to claim 1, characterized in that: The S4 is specifically: According to the calculated homography matrix, all the points on the radar image corresponding to each timestamp are projected onto the camera plane through calculation. Then, according to the results of the photoelectric image detection module, only the radar projection points projected onto the detection target are retained, that is, the echo points of the target. The category, speed, distance, and azimuth information of the target during driving are obtained and fully displayed in real time on the intelligent display system.

Images