Navigation radar assisted visual target detection method based on ship motion attitude correction

Abstract

The present application relates to the technical field of ship target detection, and particularly relates to a navigation radar auxiliary visual target detection method based on ship motion posture correction, which comprises the following steps: acquiring at least one candidate target point in a navigation radar image of a ship; correcting the at least one candidate target point, projecting three-dimensional coordinates of the acquired at least one candidate target point into a camera image, generating two-dimensional pixel coordinates of the at least one candidate target point on the camera image plane, and detecting the at least one candidate target point according to a dynamic Query formed by target features and target position coding of a region of interest determined according to a distance between the two-dimensional pixel coordinates and the ship, to generate a 2D bounding box and a target classification result of the at least one candidate target point. Thus, the problem that the navigation radar in the related art lacks three-dimensional geometric structure and continuous echo intensity information, reduces the reliability and accuracy of target detection, and cannot meet the target detection requirements of a marine scene is solved.

Description

Technical Field

[0001] This invention relates to the field of ship target detection technology, and in particular to a navigation radar-assisted visual target detection method based on ship motion attitude correction. Background Technology

[0002] Current 2D visual detection models, such as DETR and RT-DETR, demonstrate strong detection capabilities in structured scenes. However, in open water environments, such as those involving autonomous navigation of unmanned vessels at sea, detection methods relying solely on visual sensors face significant challenges. The marine environment is characterized by drastic lighting changes, highly reflective water surfaces, fog, heavy rain, and other extreme weather conditions, all of which drastically degrade visual image quality. Furthermore, compared to land-based traffic environments, targets at sea typically have greater observation distances, smaller visual scales, and stronger background interference, significantly increasing the difficulty of detecting small targets. These combined issues result in the insufficient robustness of traditional visual detection models in maritime scenarios.

[0003] To improve the stability of visual inspection, academia and industry are increasingly focusing on using sensors such as millimeter-wave radar and lidar to assist in visual inspection. However, most related technologies are based on 3D point cloud or BEV conversion methods, which are mainly applicable to lidar or imaging radar, and are difficult to apply directly to common maritime navigation radar. Navigation radar typically only provides low-resolution binary echo images in 2D polar coordinate format, lacking a clear 3D geometric structure, making it difficult to support point cloud-level or BEV-level fusion methods. In addition, the multipath effect of the sea surface environment is significant, and navigation radar images often contain a lot of noise, with targets exhibiting characteristics such as sparseness and weak reflection; in the absence of continuous echo intensity information, it is very difficult to stably extract the real target from the radar image.

[0004] Therefore, navigation radars in related technologies lack three-dimensional geometric structure and continuous echo intensity information, making it difficult to effectively support multimodal fusion detection at the point cloud level or BEV level. This reduces the reliability and accuracy of target detection and fails to meet the target detection requirements in maritime scenarios, which urgently needs to be addressed. Summary of the Invention

[0005] This invention provides a navigation radar-assisted visual target detection method based on ship motion attitude correction, which solves the problem that navigation radars in related technologies lack three-dimensional geometric structure and continuous echo intensity information, making it difficult to effectively support multimodal fusion detection at the point cloud level or BEV level, reducing the reliability and accuracy of target detection, and failing to meet the target detection requirements of maritime scenarios.

[0006] A first aspect of the present invention provides a navigation radar-assisted visual target detection method based on ship motion attitude correction, comprising the following steps: acquiring at least one candidate target point in a navigation radar image of a target ship; correcting the at least one candidate target point using a target ship attitude dynamic compensation module to obtain the target three-dimensional coordinates of the at least one candidate target point; projecting the target three-dimensional coordinates of the at least one candidate target point into a target camera image to generate two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane, determining a target region of interest based on the distance between the two-dimensional pixel coordinates and the target ship, and detecting the at least one candidate target point using a dynamic query formed by target features and target position encoding in the target region of interest to generate a 2D bounding box and target classification result for the at least one candidate target point.

[0007] Optionally, in one embodiment of the present invention, obtaining at least one candidate target point in the navigation radar image of the target vessel includes: cropping, rotating, and aligning the navigation radar image of the target vessel to obtain a processed navigation radar image; and performing target connectivity analysis on the processed navigation radar image to obtain at least one candidate target point in the navigation radar image based on the target area of ​​the target connectivity.

[0008] Optionally, in one embodiment of the present invention, the step of using the target ship attitude dynamic compensation module to correct the at least one candidate target point to obtain the target three-dimensional coordinates of the at least one candidate target point includes: obtaining the pitch angle and roll angle of the target ship based on the target ship attitude dynamic compensation module; correcting the target height of the at least one candidate target point in the target ship's own coordinate system according to the pitch angle; and rotating the coordinates of the at least one candidate target point based on the target height and the roll angle to obtain the target three-dimensional coordinates of the at least one candidate target point.

[0009] Optionally, in one embodiment of the present invention, the step of projecting the target three-dimensional coordinates of the at least one candidate target point onto the target camera image to generate the two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane includes: performing perspective projection on the target three-dimensional coordinates using the target camera intrinsic parameter matrix and the target radar-camera extrinsic parameter matrix to project the target three-dimensional coordinates of the at least one candidate target point onto the target camera image to obtain the two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane.

[0010] Optionally, in one embodiment of the present invention, the step of detecting the at least one candidate target point using a dynamic query formed by the target features and target location encoding in the target region of interest to generate a 2D bounding box and target classification result for the at least one candidate target point includes: generating two dynamic region of interest bounding boxes of different scales centered on the two-dimensional pixel coordinates corresponding to the at least one candidate target point; extracting feature vectors of the target region of interest based on the dynamic region of interest bounding boxes; flattening and performing multilayer perceptron processing on the feature vectors to generate semantic features of the dynamic query; and performing location encoding on the dynamic region of interest bounding boxes to obtain encoded location features; adding the semantic features and location features of the dynamic query to form the dynamic query; and concatenating the dynamic query with a preset number of queries to obtain a concatenated query; and performing cross-attention mechanism processing on the concatenated query to obtain a 2D bounding box and target classification result for the at least one candidate target point.

[0011] Optionally, in one embodiment of the present invention, the dynamic region of interest bounding box is represented as:

[0012]

[0013] in, and These are the width and height of the bounding box of the region of interest, respectively. and The minimum and maximum values ​​for width and height are preset. and These represent the minimum and maximum values ​​of the radial distance range, respectively.

[0014] A second aspect of the present invention provides a navigation radar-assisted visual target detection device based on ship motion attitude correction, comprising: an acquisition module for acquiring at least one candidate target point in a navigation radar image of a target ship; a correction module for correcting the at least one candidate target point using a target ship attitude dynamic compensation module to obtain the target three-dimensional coordinates of the at least one candidate target point; and a detection module for projecting the target three-dimensional coordinates of the at least one candidate target point onto a target camera image to generate two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane, determining a target region of interest based on the distance between the two-dimensional pixel coordinates and the target ship, and detecting the at least one candidate target point using a dynamic query formed by target features and target position encoding in the target region of interest to generate a 2D bounding box and target classification result for the at least one candidate target point.

[0015] Optionally, in one embodiment of the present invention, the acquisition module includes: a first acquisition unit, configured to perform cropping, rotation and alignment processing on the navigation radar image of the target ship respectively to obtain a processed navigation radar image; and an analysis unit, configured to perform target connectivity analysis on the processed navigation radar image to obtain at least one candidate target point in the navigation radar image based on the target area of ​​the target connectivity.

[0016] Optionally, in one embodiment of the present invention, the correction module includes: a second acquisition unit, configured to acquire the pitch angle and roll angle of the target ship based on the target ship attitude dynamic compensation module; a correction unit, configured to correct the target height of the at least one candidate target point in the target ship's own coordinate system according to the pitch angle; and a rotation unit, configured to rotate the coordinates of the at least one candidate target point based on the target height and the roll angle to obtain the target three-dimensional coordinates of the at least one candidate target point.

[0017] Optionally, in one embodiment of the present invention, the detection module includes: a projection unit, used to perform perspective projection on the three-dimensional coordinates of the target using the target camera intrinsic parameter matrix and the target radar-camera extrinsic parameter matrix, so as to project the target three-dimensional coordinates of the at least one candidate target point onto the target camera image, thereby obtaining the two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane.

[0018] Optionally, in one embodiment of the present invention, the detection module includes: a generation unit, configured to generate two dynamic region of interest bounding boxes of different scales centered on the two-dimensional pixel coordinates corresponding to the at least one candidate target point, and extract feature vectors of the target region of interest based on the dynamic region of interest bounding boxes; a processing unit, configured to flatten and perform multilayer perceptron processing on the feature vectors to generate semantic features of a dynamic query, and perform position encoding on the dynamic region of interest bounding boxes to obtain encoded position features; a concatenation unit, configured to add the semantic features and position features of the dynamic query to form the dynamic query, and concatenate the dynamic query with a preset number of queries to obtain a concatenated query; and a detection unit, configured to perform cross-attention mechanism processing on the concatenated query to obtain 2D bounding boxes of the at least one candidate target point and target classification results.

[0019] Optionally, in one embodiment of the present invention, the dynamic region of interest bounding box is represented as:

[0020]

[0021] in, and These are the width and height of the bounding box of the region of interest, respectively. and The minimum and maximum values ​​for width and height are preset. and These represent the minimum and maximum values ​​of the radial distance range, respectively.

[0022] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the navigation radar-assisted visual target detection method based on ship motion attitude correction as described in the above embodiments.

[0023] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described navigation radar-assisted visual target detection method based on ship motion attitude correction.

[0024] A fifth aspect of the present invention provides a computer program product, including a computer program that, when executed, is used to implement the above-described navigation radar-assisted visual target detection method based on ship motion attitude correction.

[0025] This invention first acquires candidate target points from a ship's navigation radar image. Then, it uses a ship attitude dynamic compensation module to correct these candidate target points and projects their 3D coordinates onto a camera image, generating 2D pixel coordinates on the camera image plane. Based on these 2D pixel coordinates and the distance between the candidate target points and the ship, a region of interest (ROI) is determined. A dynamic query, formed by target features and target location encoding within the ROI, is then used to detect the candidate target points, generating 2D bounding boxes and target classification results. This effectively improves the reliability and accuracy of target detection and enhances the vision-radar fusion perception system's ability to acquire distant targets. Therefore, it solves the problem in related technologies where navigation radar lacks 3D geometric structure and continuous echo intensity information, making it difficult to effectively support point cloud-level or BEV-level multimodal fusion detection, reducing the reliability and accuracy of target detection, and failing to meet the target detection requirements of maritime scenarios.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 A flowchart illustrating a navigation radar-assisted visual target detection method based on ship motion attitude correction according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of a navigation radar-assisted visual target detection method based on ship motion attitude correction according to a specific embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram of a navigation radar-assisted visual target detection device based on ship motion attitude correction according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0033] The following describes an embodiment of the navigation radar-assisted visual target detection method based on ship motion attitude correction according to the accompanying drawings. Addressing the problem mentioned in the background art that navigation radars lack three-dimensional geometric structure and continuous echo intensity information, making it difficult to effectively support point cloud-level or BEV-level multimodal fusion detection, thus reducing the reliability and accuracy of target detection and failing to meet the target detection requirements in maritime scenarios, this invention provides a navigation radar-assisted visual target detection method based on ship motion attitude correction. In this method, candidate target points in the ship's navigation radar image are first acquired. Then, the candidate target points are corrected using a ship attitude dynamic compensation module, and the acquired three-dimensional coordinates of the candidate target points are projected onto the camera image, thereby generating two-dimensional pixel coordinates of the candidate target points on the camera image plane. A region of interest is determined based on the two-dimensional pixel coordinates and the distance between the two-dimensional pixel coordinates and the ship. The candidate target points are then detected using a dynamic query formed by the target features and target position encoding within the region of interest, generating 2D bounding boxes and target classification results for the candidate target points. This effectively improves the reliability and accuracy of target detection and enhances the ability of the visual-radar fusion perception system to capture distant targets. This solves the problem that navigation radar in related technologies lacks three-dimensional geometric structure and continuous echo intensity information, making it difficult to effectively support multimodal fusion detection at the point cloud level or BEV level, reducing the reliability and accuracy of target detection, and failing to meet the target detection requirements of maritime scenarios.

[0034] Specifically, Figure 1 This is a flowchart illustrating a navigation radar-assisted visual target detection method based on ship motion attitude correction, provided in an embodiment of the present invention.

[0035] like Figure 1 As shown, the navigation radar-assisted visual target detection method based on ship motion attitude correction includes the following steps:

[0036] In step S101, at least one candidate target point is acquired from the navigation radar image of the target vessel.

[0037] In this embodiment of the invention, the target vessel is a vessel that currently requires the fusion of navigation radar and camera images for target detection.

[0038] It is understood that the embodiments of the present invention can obtain at least one candidate target point in the navigation radar image of the target ship in the following steps. Specifically, firstly, the embodiments of the present invention can crop the navigation radar image to determine the detection range, and rotate the navigation radar image according to the ship's own yaw angle so that the radar image data is aligned to the current moment. Then, connected component analysis is performed on the navigation radar image, wherein a small connected component is considered as a single candidate target point, while a large connected component may be multiple adjacent targets. The embodiments of the present invention can uniformly sample the navigation radar image at a certain angle within the field of view of the target camera in the following steps, and the first intersection of a large connected component and a ray is considered as a candidate target point, thereby improving the detection accuracy and reliability of long-distance and dense targets in complex environments.

[0039] In one embodiment of the present invention, obtaining at least one candidate target point in the navigation radar image of the target vessel includes: cropping, rotating, and aligning the navigation radar image of the target vessel to obtain a processed navigation radar image; and performing target connectivity analysis on the processed navigation radar image to obtain at least one candidate target point in the navigation radar image based on the target area of ​​the target connectivity.

[0040] In this embodiment of the invention, the target area of ​​the target connected region can be a small independent connected region or a large connected region.

[0041] In actual implementation, embodiments of the present invention perform cropping, rotation, and alignment processing on the navigation radar images of the target vessel, for example, as follows: Figure 2 As shown, the original navigation radar image of the target ship in this embodiment of the invention has a size of 2048×2048 pixels, resulting in a large effective detection range. Therefore, to reduce subsequent computational complexity and focus on the near-field region, this invention first performs region-of-interest (ROI) cropping on the navigation radar image, cropping the original image from 2048×2048 pixels to a central region of 800×800 pixels. The effective detection range of this central region is approximately 646×646 m. ​​Due to the difference in acquisition frequencies between the camera and the navigation radar (the camera frequency is 10 Hz, while the navigation radar frequency is approximately 0.8 Hz), their acquisition times are difficult to synchronize precisely. Therefore, this embodiment of the invention utilizes the ship's attitude information to perform attitude correction and alignment on the cropped navigation radar image, registering it to the current world coordinate system to obtain the processed navigation radar image, thereby ensuring the accuracy of spatial positioning.

[0042] Furthermore, embodiments of the present invention can perform target connectivity analysis on the processed navigation radar image. Since the navigation radar image is a binary image and lacks intensity information of reflection points, it is impossible to filter potential targets based on intensity thresholds. The present invention employs geometric structure-based analysis for target filtering. First, connectivity analysis is performed on the navigation radar image. Small, independent connected components are considered as single candidate target points. For large, large-area connected components, multiple closely spaced small targets may be displayed together on the navigation radar image, or they may be natural or artificial background objects, such as shorelines, bridges, or large land structures. For the aforementioned large-area connected components, the present invention uses a central ray sampling strategy for positioning. First, the ship itself, i.e., the center of the navigation radar image, is used as the sampling origin. Within the camera's field of view, uniformly distributed detection rays are emitted outward at preset angular intervals. The first intersection of each ray with the large-area connected component is considered a candidate target point.

[0043] Therefore, by combining yaw angle correction and connected component analysis, and utilizing a ray sampling strategy, the embodiments of the present invention significantly improve the ability to identify and locate potential targets in binary radar images, especially improving the detection accuracy and reliability of long-range and dense targets in complex environments.

[0044] In step S102, at least one candidate target point is corrected using the target ship attitude dynamic compensation module to obtain the target three-dimensional coordinates of at least one candidate target point.

[0045] In this embodiment of the invention, navigation radar image data only reflects the target's azimuth and distance, lacking altitude information. However, traditional projection methods use a fixed target altitude. In actual maritime navigation environments, ship platforms are not stationary; their attitude is affected by waves and wind, resulting in significant six-degree-of-freedom motion. This instability leads to systematic errors in traditional fixed-altitude projection methods. Even minute changes in ship attitude, magnified by distance, will cause severe and unacceptable deviations in the target point projected onto the camera image.

[0046] It is understandable that, such as Figure 2 As shown, the embodiment of the present invention can utilize the ship attitude dynamic compensation module in the following steps to determine the target height and rotate the target coordinates according to the ship's pitch angle and roll angle, respectively, so as to correct the coordinates of one or more candidate target points in three-dimensional space in real time, thereby coping with the turbulent sea environment and improving the accuracy of subsequent target detection and tracking.

[0047] In one embodiment of the present invention, the target ship attitude dynamic compensation module is used to correct at least one candidate target point to obtain the target three-dimensional coordinates of at least one candidate target point, including: obtaining the pitch angle and roll angle of the target ship based on the target ship attitude dynamic compensation module; correcting the target height of at least one candidate target point in the target ship's own coordinate system according to the pitch angle; and rotating the coordinates of at least one candidate target point based on the target height and roll angle to obtain the target three-dimensional coordinates of at least one candidate target point.

[0048] As one possible implementation, this embodiment of the invention introduces a ship attitude dynamic compensation module to correct the coordinates of candidate target points in three-dimensional space in real time. The ship attitude dynamic compensation module obtains the current ship pitch angle in real time from the AHRS (Attitude and Heading Reference System). and roll angle According to the ship's pitch angle Correct the radar target point, that is, the height of the candidate target point in the current ship's own coordinate system. ,Right now:

[0049]

[0050] in, The pitch angle of the ship; An assumed height for the camera can be adjusted during training. This represents the radial distance between the radar target point and the ship.

[0051] Finally, based on the ship's roll angle The coordinates of the radar target point are rotated so that the radar projection point can be accurately projected onto the target in the image, that is:

[0052]

[0053] in, For the ship's roll angle, The corrected three-dimensional coordinates of the target.

[0054] In step S103, the three-dimensional coordinates of at least one candidate target point are projected onto the target camera image to generate two-dimensional pixel coordinates of at least one candidate target point on the target camera image plane. The target region of interest is determined based on the two-dimensional pixel coordinates and the distance between the target ship. The target feature and target position encoding in the target region of interest are used to detect at least one candidate target point to generate a 2D bounding box and target classification result for at least one candidate target point.

[0055] It is understood that embodiments of the present invention can project the three-dimensional coordinates of candidate target points onto the camera image, generating two-dimensional pixel coordinates of the candidate target points on the camera image plane. Since in a camera image, distant targets typically have smaller pixels, and nearby targets typically have larger pixels, therefore, as... Figure 2 As shown, a region of interest (ROI) with smaller values ​​for distant targets and larger values ​​for nearby targets can be generated using the distance between candidate target points and the ship. Features are extracted from the pre-selected target regions using RoI Align. Finally, the extracted features are used as the semantic information of the dynamic query, and combined with the position encoding to form a dynamic query. This dynamic query is then concatenated with a fixed number of queries from RT-DETR and fed into the decoder to detect candidate target points. This yields the 2D bounding boxes of the target points and the target classification results, effectively enhancing RT-DETR's ability to detect distant, weakly textured maritime targets and significantly improving detection accuracy and robustness.

[0056] In one embodiment of the present invention, projecting the target three-dimensional coordinates of at least one candidate target point onto the target camera image to generate the two-dimensional pixel coordinates of at least one candidate target point on the target camera image plane includes: using the target camera intrinsic parameter matrix and the target radar-camera extrinsic parameter matrix to perform perspective projection on the target three-dimensional coordinates, so as to project the target three-dimensional coordinates of at least one candidate target point onto the target camera image, thereby obtaining the two-dimensional pixel coordinates of at least one candidate target point on the target camera image plane.

[0057] In some embodiments, the present invention can correct the target three-dimensional coordinates after the ship attitude dynamic compensation module in the above steps. Through the calibrated camera intrinsic parameter matrix Radar-camera extrinsic matrix Perform perspective projection to calculate the two-dimensional pixel coordinates of the candidate target points on the camera image plane. ,Right now:

[0058]

[0059] in, For the camera intrinsic parameter matrix, For the camera extrinsic matrix, The corrected three-dimensional coordinates of the target.

[0060] Optionally, in one embodiment of the present invention, at least one candidate target point is detected using a dynamic query formed by target features and target location encoding in the target region of interest to generate a 2D bounding box and target classification result for at least one candidate target point. This includes: generating two dynamic region of interest bounding boxes of different scales centered on the two-dimensional pixel coordinates corresponding to at least one candidate target point; extracting feature vectors of the target region of interest based on the dynamic region of interest bounding boxes; flattening and processing the feature vectors using a multilayer perceptron to generate semantic features of the dynamic query; encoding the location of the dynamic region of interest bounding boxes to obtain encoded location features; adding the semantic features and location features of the dynamic query to form a dynamic query; concatenating the dynamic query with a preset number of queries to obtain a concatenated query; and applying a cross-attention mechanism to the concatenated query to obtain a 2D bounding box and target classification result for at least one candidate target point.

[0061] As one possible implementation, in RT-DETR, the semantic features of the query come from the output of the encoder. For anchor points with high classification scores, the features of that point are taken as semantic features, and the anchor boxes are encoded as positional features. Despite complex attitude compensation, the two-dimensional pixel coordinates of the projected points are still affected by inherent radar positioning errors, extrinsic parameter calibration errors, and the size of the target itself. There may still be a slight error in the distance from the target center, which makes direct use... To address the insufficient accuracy of feature extraction and to cope with projection errors and target scale variations, this invention proposes an adaptive dynamic RoI generation mechanism.

[0062] Specifically, in the image, the pixel size of the target and the radial distance from the candidate target point to the ship are... There is an inverse relationship: the farther away a target is, the smaller its pixel area in the image. Therefore, an adaptive RoI bounding box is calculated and generated using the target distance provided by the navigation radar to ensure scale adaptability, where distant targets appear smaller and near targets appear larger.

[0063]

[0064] in, and These are the width and height of the bounding box of the region of interest, respectively. and The minimum and maximum values ​​for width and height are preset. and These represent the minimum and maximum values ​​of the radial distance range, respectively.

[0065] In addition, this invention generates two RoI bounding boxes of different scales for the same candidate target point. The width and height of the other bounding box are twice the standard size, in order to further enhance robustness to projection errors and ensure complete inclusion of the target.

[0066] Furthermore, after obtaining the RoI bounding box, the first-layer feature map output by the Encoder is used as input, and RoI Align is used to extract the feature vector of the target region from the feature map based on the generated dynamic RoI bounding box. Extracted After flattening and multilayer perceptron processing, the semantic features of the dynamic query are finally generated. The positional encoding of the dynamic query follows the RT-DETR positional encoding mechanism, encoding the RoI bounding box... Positional encoding is performed using a positional encoding header; secondly, the semantic and positional features are added together and concatenated with a fixed query; finally, the concatenated query is fed into the decoder, where it undergoes cross-attention processing to output the 2D bounding boxes of the candidate targets and the classification results.

[0067]

[0068]

[0069]

[0070]

[0071] in, Represents the feature vector extracted from the region corresponding to the RoI bounding box; SemEmbed represents the semantic embedding function, used to... Mapped to semantic feature vectors in the embedding space PosEmbed represents the location encoding function, which is based on the center coordinates of the RoI bounding box. and width and height Generate position embedding vectors ; For semantic features and location features The dynamic query obtained by addition; A fixed, learnable query for the original RT-DETR; The query sequence formed by concatenating fixed and dynamic queries is used as the input to the decoder.

[0072] For example, this invention uses the 00 sequence data from the Pohang Canal Dataset, a dataset specifically designed for maritime and coastal target detection, containing real navigation radar echo images and synchronously acquired visible light camera images. The dataset is divided as follows: 2500 images are randomly selected as the training set for model training, and 1500 images are used as the test set for model performance evaluation. To enhance the model's generalization ability and robustness, the images are randomly horizontally flipped, scaled, and color-enhanced during the data preprocessing stage. All input camera images are uniformly scaled to [size missing]. The model training equipment used an i7-12700K CPU and an Nvidia GTX 3060 GPU. The initial learning rate was... The batch size was 1, and a total of 36 training rounds were conducted. Table 1 shows the experimental results, as detailed below:

[0073] Table 1

[0074]

[0075] Comparison shows that the multimodal detection model fusion of navigation radar and camera proposed in this invention surpasses the benchmark model RT-DETR in all accuracy metrics, fully verifying the effectiveness and robustness of this invention. Overall average accuracy It increased by 3 percentage points. It increased by 2.2 percentage points. This improved the model's ability to acquire distant targets by 4 percentage points. Therefore, this invention enhances the model's ability to acquire distant targets through a radar target candidate module, a dynamic compensation module, and a dynamic query generation module.

[0076] The navigation radar-assisted visual target detection method based on ship motion attitude correction proposed in this invention first acquires candidate target points in the ship's navigation radar image. Then, it uses a ship attitude dynamic compensation module to correct the candidate target points and projects the acquired three-dimensional coordinates of the candidate target points onto the camera image, thereby generating two-dimensional pixel coordinates of the candidate target points on the camera image plane. Based on the two-dimensional pixel coordinates and the distance between the two-dimensional pixel coordinates and the ship, a region of interest is determined. A dynamic query formed by the target features and target position encoding within the region of interest is then used to detect the candidate target points, generating 2D bounding boxes and target classification results. This effectively improves the reliability and accuracy of target detection and enhances the ability of the vision-radar fusion perception system to capture distant targets. Therefore, it solves the problem in related technologies where navigation radar lacks three-dimensional geometric structure and continuous echo intensity information, making it difficult to effectively support point cloud-level or BEV-level multimodal fusion detection, reducing the reliability and accuracy of target detection, and failing to meet the target detection requirements of maritime scenarios.

[0077] Next, referring to the accompanying drawings, a navigation radar-assisted visual target detection device based on ship motion attitude correction according to an embodiment of the present invention is described.

[0078] Figure 3 This is a block diagram of a navigation radar-assisted visual target detection device based on ship motion attitude correction according to an embodiment of the present invention.

[0079] like Figure 3 As shown, the navigation radar-assisted visual target detection device 10 based on ship motion attitude correction includes: an acquisition module 100, a correction module 200, and a detection module 300.

[0080] Specifically, the acquisition module 100 is used to acquire at least one candidate target point in the navigation radar image of the target vessel.

[0081] The correction module 200 is used to correct at least one candidate target point using the target ship attitude dynamic compensation module to obtain the target three-dimensional coordinates of at least one candidate target point.

[0082] The detection module 300 is used to project the three-dimensional coordinates of at least one candidate target point onto the target camera image, generate two-dimensional pixel coordinates of at least one candidate target point on the target camera image plane, determine the target region of interest based on the distance between the two-dimensional pixel coordinates and the target ship, and use the target features and target position encoding in the target region of interest to detect at least one candidate target point, so as to generate a 2D bounding box and target classification result for at least one candidate target point.

[0083] Optionally, in one embodiment of the present invention, the acquisition module 100 includes: a first acquisition unit and an analysis unit.

[0084] The first acquisition unit is used to crop, rotate, and align the navigation radar image of the target ship to obtain the processed navigation radar image.

[0085] The analysis unit is used to perform target connectivity analysis on the processed navigation radar image to obtain at least one candidate target point in the navigation radar image based on the target area of ​​the target connectivity.

[0086] Optionally, in one embodiment of the present invention, the correction module 200 includes: a second acquisition unit, a correction unit, and a rotation unit.

[0087] The second acquisition unit is used to acquire the pitch angle and roll angle of the target ship based on the target ship attitude dynamic compensation module.

[0088] The correction unit is used to correct the target height of at least one candidate target point in the target ship's own coordinate system based on the pitch angle.

[0089] A rotation unit is used to rotate the coordinates of at least one candidate target point based on the target height and roll angle to obtain the target three-dimensional coordinates of at least one candidate target point.

[0090] Optionally, in one embodiment of the present invention, the detection module 300 includes a projection unit.

[0091] The projection unit is used to perform perspective projection on the three-dimensional coordinates of the target using the target camera intrinsic parameter matrix and the target radar-camera extrinsic parameter matrix, so as to project the three-dimensional coordinates of at least one candidate target point onto the target camera image, thereby obtaining the two-dimensional pixel coordinates of at least one candidate target point on the target camera image plane.

[0092] Optionally, in one embodiment of the present invention, the detection module 300 includes: a generation unit, a processing unit, a splicing unit, and a detection unit.

[0093] The generation unit is used to generate two dynamic region of interest bounding boxes of different scales, centered on the two-dimensional pixel coordinates corresponding to at least one candidate target point, and to extract the feature vector of the target region of interest based on the dynamic region of interest bounding boxes.

[0094] The processing unit is used to flatten the feature vector and perform multilayer perceptron processing to generate semantic features of the dynamic query, and to encode the bounding box of the dynamic region of interest to obtain the encoded positional features.

[0095] The concatenation unit is used to add the semantic features and positional features of the dynamic query to form a dynamic query, and to concatenate the dynamic query with a preset number of queries to obtain the concatenated query.

[0096] The detection unit is used to process the concatenated Query using a cross-attention mechanism to obtain a 2D bounding box of at least one candidate target point and a target classification result.

[0097] Optionally, in one embodiment of the present invention, the dynamic region of interest bounding box is represented as:

[0098]

[0099] in, and These are the width and height of the bounding box of the region of interest, respectively. and The minimum and maximum values ​​for width and height are preset. and These represent the minimum and maximum values ​​of the radial distance range, respectively.

[0100] It should be noted that the foregoing explanation of the embodiment of the navigation radar-assisted visual target detection method based on ship motion attitude correction also applies to the navigation radar-assisted visual target detection device based on ship motion attitude correction in this embodiment, and will not be repeated here.

[0101] The navigation radar-assisted visual target detection device based on ship motion attitude correction proposed in this invention first acquires candidate target points in the ship's navigation radar image. Then, it uses a ship attitude dynamic compensation module to correct the candidate target points and projects the acquired three-dimensional coordinates of the candidate target points onto the camera image, thereby generating two-dimensional pixel coordinates of the candidate target points on the camera image plane. Based on the two-dimensional pixel coordinates and the distance between the two-dimensional pixel coordinates and the ship, a region of interest is determined. A dynamic query formed by the target features and target position encoding within the region of interest is then used to detect the candidate target points, generating 2D bounding boxes and target classification results. This effectively improves the reliability and accuracy of target detection and enhances the vision-radar fusion perception system's ability to capture distant targets. Therefore, it solves the problem in related technologies where navigation radar lacks three-dimensional geometric structure and continuous echo intensity information, making it difficult to effectively support point cloud-level or BEV-level multimodal fusion detection, reducing the reliability and accuracy of target detection, and failing to meet the target detection requirements of maritime scenarios.

[0102] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include:

[0103] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0104] When the processor 402 executes the program, it implements the navigation radar-assisted visual target detection method based on ship motion attitude correction provided in the above embodiments.

[0105] Furthermore, electronic devices also include:

[0106] Communication interface 403 is used for communication between memory 401 and processor 402.

[0107] The memory 401 is used to store computer programs that can run on the processor 402.

[0108] Memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0109] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0110] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0111] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0112] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described navigation radar-assisted visual target detection method based on ship motion attitude correction.

[0113] This embodiment also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described navigation radar-assisted visual target detection method based on ship motion attitude correction.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0116] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0118] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0120] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0121] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A navigation radar-assisted visual target detection method based on ship motion attitude correction, characterized in that, Includes the following steps: Acquire at least one candidate target point from the navigation radar image of the target vessel; The target ship attitude dynamic compensation module is used to correct the at least one candidate target point to obtain the target three-dimensional coordinates of the at least one candidate target point; The target 3D coordinates of the at least one candidate target point are projected onto the target camera image to generate 2D pixel coordinates of the at least one candidate target point on the target camera image plane. A target region of interest (ROI) is determined based on the distance between the 2D pixel coordinates and the target ship. A dynamic query formed by the target features and target location encoding within the ROI is used to detect the at least one candidate target point, thereby generating a 2D bounding box and target classification result for the at least one candidate target point. The step of using the dynamic query formed by the target features and target location encoding within the ROI to detect the at least one candidate target point and generate a 2D bounding box and target classification result for the at least one candidate target point includes: projecting the 3D coordinates of the at least one candidate target point onto the target camera image to generate 2D pixel coordinates of the at least one candidate target point on the target camera image plane. Centered on the two-dimensional pixel coordinates corresponding to the candidate target point, two dynamic region of interest (ROI) bounding boxes of different scales are generated. Based on the dynamic ROI bounding boxes, feature vectors of the target ROI are extracted. The feature vectors are flattened and processed by a multilayer perceptron to generate semantic features of a dynamic query. The dynamic ROI bounding boxes are then positionally encoded to obtain encoded positional features. The semantic features of the dynamic query and the positional features are added together to form the dynamic query. The dynamic query is then concatenated with a preset number of queries to obtain a concatenated query. The concatenated query is then processed by a cross-attention mechanism to obtain the 2D bounding box of the at least one candidate target point and the target classification result.

2. The navigation radar-assisted visual target detection method based on ship motion attitude correction according to claim 1, characterized in that, The acquisition of at least one candidate target point in the navigation radar image of the target vessel includes: The navigation radar image of the target ship is cropped, rotated, and aligned to obtain the processed navigation radar image. The processed navigation radar image is subjected to target connectivity analysis to obtain at least one candidate target point in the navigation radar image based on the target area of ​​the target connectivity.

3. The navigation radar-assisted visual target detection method based on ship motion attitude correction according to claim 1, characterized in that, The step of using the target ship attitude dynamic compensation module to correct the at least one candidate target point to obtain the target three-dimensional coordinates of the at least one candidate target point includes: Based on the target ship attitude dynamic compensation module, the pitch angle and roll angle of the target ship are obtained; The target height of the at least one candidate target point in the target ship's own coordinate system is corrected according to the pitch angle; Based on the target height and the roll angle, the coordinates of the at least one candidate target point are rotated to obtain the target three-dimensional coordinates of the at least one candidate target point.

4. The navigation radar-assisted visual target detection method based on ship motion attitude correction according to claim 1, characterized in that, The step of projecting the target three-dimensional coordinates of the at least one candidate target point onto the target camera image to generate the two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane includes: The three-dimensional coordinates of the target are projected using the intrinsic parameter matrix of the target camera and the extrinsic parameter matrix of the target radar-camera to project the three-dimensional coordinates of the at least one candidate target point onto the target camera image, thereby obtaining the two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane.

5. The navigation radar-assisted visual target detection method based on ship motion attitude correction according to claim 1, characterized in that, The dynamically defined region of interest bounding box is represented as follows: in, and These are the width and height of the bounding box of the region of interest, respectively. and The minimum and maximum values ​​for width and height are preset. and These represent the minimum and maximum values ​​of the radial distance range, respectively.

6. A navigation radar-assisted visual target detection device based on ship motion attitude correction, characterized in that, include: The acquisition module is used to acquire at least one candidate target point in the navigation radar image of the target vessel; The correction module is used to correct the at least one candidate target point using the target ship attitude dynamic compensation module to obtain the target three-dimensional coordinates of the at least one candidate target point; The detection module is used to project the three-dimensional coordinates of the at least one candidate target point onto the target camera image, generate two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane, determine the target region of interest based on the distance between the two-dimensional pixel coordinates and the target ship, and detect the at least one candidate target point using a dynamic query formed by the target features and target location encoding in the target region of interest to generate a 2D bounding box and target classification result for the at least one candidate target point. The step of detecting the at least one candidate target point using the dynamic query formed by the target features and target location encoding in the target region of interest to generate a 2D bounding box and target classification result for the at least one candidate target point includes: projecting the three-dimensional coordinates of the at least one candidate target point onto the target camera image, generating two-dimensional pixel coordinates of the at least one candidate target point on the target camera image plane, determining the target region of interest based on the distance between the two-dimensional pixel coordinates and the target ship, and detecting the at least one candidate target point using a dynamic query formed by the target features and target location encoding in the target region of interest to generate a 2D bounding box and target classification result for the at least one candidate target point. Centered on the two-dimensional pixel coordinates corresponding to at least one candidate target point, two dynamic region of interest (ROI) bounding boxes of different scales are generated. Based on the dynamic ROI bounding boxes, feature vectors of the target ROI are extracted. The feature vectors are flattened and processed by a multilayer perceptron to generate semantic features of a dynamic query. The dynamic ROI bounding boxes are then positionally encoded to obtain encoded positional features. The semantic features and positional features of the dynamic query are added together to form the dynamic query. The dynamic query is then concatenated with a preset number of queries to obtain a concatenated query. The concatenated query is then processed by a cross-attention mechanism to obtain the 2D bounding box of the at least one candidate target point and the target classification result.

7. An electronic device, characterized in that, include: The system includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the navigation radar-assisted visual target detection method based on ship motion attitude correction as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the navigation radar-assisted visual target detection method based on ship motion attitude correction as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the navigation radar-assisted visual target detection method based on ship motion attitude correction as described in any one of claims 1-5.

Images