Pure azimuth trajectory fusion method and system based on spliced image detection target and AIS information

Abstract

The invention discloses a pure azimuth trajectory ship target fusion method based on spliced images and AIS information, and relates to the field of ship detection and information fusion. A left monocular camera, a middle monocular camera and a right monocular camera are spliced to generate a 180-degree panoramic image (the resolution ratio is 2160 * 14000), the small target recognition precision is improved by combining local image cutting and total image detection, and a visual orientation track is generated; synchronously processing the AIS information, and screening the orientation and distance track of the nearest target in front of the ship; performing secondary filtering on the visual target based on a target category, a size factor (target frame width * height * 1.0 / ordinate) and a median ordinate, and performing de-duplication and distortion compensation to obtain an optimized azimuth list; and finally, matching of the AIS and the visual target is realized through the orientation cost matrix and the Hungary algorithm. According to the method, image distortion is reduced through high-resolution splicing, distance measurement errors are avoided through pure orientation fusion, the remote target matching robustness is remarkably improved, and the method is suitable for ship collision avoidance and marine monitoring.

Description

Technical Field

[0001] The present invention relates to the field of ship detection and information fusion, and in particular to a bearing-only trajectory fusion method and system based on spliced image detection targets and AIS information. Background Art

[0002] In the research field of autonomous navigation of ships, multi-source information fusion is a crucial direction. In the autonomous navigation of ships, environmental perception is crucial for the safety of ships, navigation efficiency, and collision avoidance of other ships. Therefore, how to integrate data from different sensors to obtain more accurate and reliable information has become one of the core research topics. The fusion of AIS and radar sensors has reached a certain level of maturity, but the fusion of visual targets and AIS still faces many challenges because the visual ranging error is large. In recent years, ship target detection and tracking technology has mainly relied on the fusion of visual sensors (such as cameras) and ship automatic identification systems (AIS). However, the existing technology has the following significant defects:

[0003] Limitations of binocular-based visual inspection: While traditional binocular cameras offer a wide field of view, their resolution is low, with severe image distortion particularly evident at the edges of wide-angle viewing angles. This results in insufficient detection accuracy for small targets and significant errors in target range and orientation calculations. Furthermore, the binocular camera's stereo matching algorithm is susceptible to interference in complex sea conditions, further reducing reliability.

[0004] Limitations of deep learning-based visual inspection technology: Monocular visual inspection methods based on deep learning can directly output target distance and orientation information, but they rely on a conversion model from the image coordinate system to the world coordinate system. Because pixel changes in distant targets are less sensitive, the conversion error increases exponentially with increasing target distance, resulting in a significant decrease in positioning accuracy for distant targets. Furthermore, visual ranging is significantly affected by factors such as ambient lighting and wave reflections, making it insufficiently robust in practical applications.

[0005] Deficiencies of existing fusion methods: Existing vision and AIS fusion methods typically rely on direct matching of visual ranging data with AIS distance information. However, errors in visual ranging are directly transmitted to the fusion process, resulting in a high matching failure rate, especially when the target is far away or in areas with image distortion.

[0006] The above problems seriously restrict the practicality and reliability of ship target detection and fusion technology. There is an urgent need for an efficient fusion method that can avoid the defects of visual ranging and improve wide-angle detection accuracy. Summary of the Invention

[0007] The present invention is made to solve the above problems, and its purpose is to provide a bearing-only trajectory fusion method and system based on spliced image detection target and AIS information.

[0008] The present invention provides a pure azimuth trajectory fusion method based on spliced image detection targets and AIS information, which has the following characteristics: step 1, using a left monocular camera, a middle monocular camera, and a right monocular camera to capture images, and generating a panoramic image with a resolution of 2160×14000 covering a 180° range in front of the ship according to a label file; performing image slicing processing and full-image target detection on the panoramic image, generating and saving the azimuth trajectory of the detected target, and bypassing the reliance on visual ranging; step 2, receiving and parsing AIS information, calculating the relative azimuth of the AIS target and the ship, saving the azimuth trajectory and distance trajectory of the AIS target, and performing motion prediction on the AIS target; step 3, reading AIS targets within a range of 0-180° in front of the ship and with a distance less than a set threshold, retaining the AIS target closest to the ship in every preset distance interval, and generating ais_azimuth_list and ais_distance_list. e_list; Step 4, screen the visual detection targets based on the target category and size factor. The size factor is calculated from the width, height and ordinate of the target box to obtain nav_objs_list; perform secondary filtering based on the median ordinate median_y of nav_objs_list to generate valid_objs_list; sort the valid_objs_list in descending order of the ordinate and perform deduplication processing to remove small targets covered by large targets. At the same time, compensate for the azimuth trajectory caused by the stitching error to generate vision_azimuth_list; Step 5, construct the cost matrix of ais_azimuth_list and vision_azimuth_list, generate the matching cost based on the weighted sum of the azimuth difference and the distance difference, and complete the association matching of the AIS target and the visual target through the singular matching algorithm. The matching process only relies on the azimuth trajectory and bypasses the visual ranging data.

[0009] The bearing-only trajectory fusion method based on stitched image detection of targets and AIS information provided by the present invention may also have the following features: wherein, in step 1, the specific process of image slicing is: dividing the panoramic image into multiple sub-regions for local detection, and optimizing the target positioning accuracy in combination with the full image detection results.

[0010] The bearing-only trajectory fusion method for detecting targets and AIS information based on stitched images provided by the present invention may also have the following features: wherein, in step 1, a panoramic image with a resolution of 2160×14000 is achieved by stitching multiple cameras to improve the small target detection capability and reduce image distortion.

[0011] The bearing-only trajectory fusion method based on stitched image detection targets and AIS information provided by the present invention may also have the following features: wherein, in step 4, the bearing trajectory generation process is: determining the bearing range of each row of targets through target detection results, and generating a continuous trajectory based on target position changes, and excluding visual ranging parameters in the trajectory calculation.

[0012] The bearing-only trajectory fusion method based on spliced image detection target and AIS information provided by the present invention may also have the following characteristics: wherein, in step 4, the calculation formula of the size factor is:

[0013] distance_estimate=size_factor×y_factor

[0014] Wherein, size_factor = target box width × target box height, y_factor = 1.0 / target box vertical coordinate.

[0015] The pure bearing trajectory fusion method based on stitched image detection target and AIS information provided by the present invention may also have the following characteristics: wherein, in step 4, the bearing trajectory compensation processing process is: based on the calibration parameters of the stitched camera, the bearing values of the visual target in the left and right edge areas are linearly corrected to eliminate the influence of image distortion.

[0016] The bearing-only trajectory fusion method based on spliced image detection of targets and AIS information provided by the present invention may also have the following feature: wherein, in step 5, the singular matching algorithm is the Hungarian algorithm or the KM algorithm, which is used to minimize the total matching cost.

[0017] The present invention also provides a pure azimuth trajectory fusion system based on stitched image detection targets and AIS information, including: a wide-angle image stitching and detection module: collecting images through the left monocular camera, the middle monocular camera and the right monocular camera, and stitching together to generate a panoramic image covering a 180° range in front of the ship with a resolution of 2160×14000 according to the label file; performing image cutting processing and full-image target detection on the panoramic image, generating and saving the azimuth trajectory of the detected target, and bypassing the dependence on visual ranging; an AIS information processing module: receiving and parsing AIS information, calculating the relative azimuth of the AIS target and the ship, saving the azimuth trajectory and distance trajectory of the AIS target, and performing motion prediction on the AIS target; an AIS target screening module: reading AIS targets within the range of 0-180° in front of the ship and with a distance less than a set threshold, retaining the AIS target closest to the ship every preset distance interval, and generating ais_azimuth_list and ais_distance_l ist; Visual target filtering and optimization module: Filter visual detection targets based on target category and size factor. The size factor is calculated from the width, height and ordinate of the target box to obtain nav_objs_list; Perform secondary filtering based on the median ordinate median_y of nav_objs_list to generate valid_objs_list; Sort the valid_objs_list in descending order of ordinate and perform deduplication processing to remove small targets covered by large targets. At the same time, compensate for the azimuth track caused by stitching errors to generate vision_azimuth_list; Pure bearing track matching module: Construct the cost matrix of ais_azimuth_list and vision_azimuth_list, generate the matching cost based on the weighted sum of azimuth difference and distance difference, and complete the association matching of AIS targets and visual targets through the singular matching algorithm. The matching process only relies on the azimuth track and bypasses the visual ranging data.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the method of the present invention is implemented when the processor executes the computer program.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method of the present invention when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figures 1 to 3 4 is a flow chart of a method for fusing bearing-only tracks based on target detection using stitched images and AIS information in an embodiment of the present invention. Figures 1 to 3 Together they form a complete flowchart, in which Figure 2 Undertake Figure 1The final step, Figure 3 Undertake Figure 2 The final step. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0022] This embodiment proposes a bearing-only trajectory fusion method based on spliced image detection targets and AIS information. This method can provide a large field of view and a high target detection rate, thereby achieving a better fusion effect.

[0023] Figures 1 to 3 4 is a flow chart of a method for fusing bearing-only tracks based on target detection and AIS information in an embodiment of the present invention.

[0024] like Figures 1 to 3 As shown, the bearing-only trajectory fusion method based on spliced image detection target and AIS information involved in this embodiment includes the following steps:

[0025] In step S1, the left monocular camera, the middle monocular camera, and the right monocular camera are used to capture images. A panoramic image with a resolution of 2160 × 14000 and covering 180° in front of the ship is generated based on the label file. The panoramic image is then cropped and full-image target detection is performed on the image. The orientation trajectory of the detected target is generated and saved. This orientation trajectory generation bypasses the reliance on visual ranging.

[0026] In this embodiment, the specific process of the above-mentioned image cutting processing is: dividing the panoramic image into multiple sub-regions for local detection, and optimizing the target positioning accuracy in combination with the full image detection result.

[0027] In addition, panoramic images with a resolution of 2160×14000 are achieved through multi-camera stitching to improve small target detection capabilities and reduce image distortion.

[0028] Step S2: Receive and analyze AIS information, calculate the relative position of the AIS target and the own ship, save the position track and distance track of the AIS target, and perform motion prediction on the AIS target.

[0029] Step S3: read the AIS targets in the range of 0-180° in front of the ship and whose distance is less than the set threshold, retain the AIS targets closest to the ship in every preset distance interval, and generate ais_azimuth_list and ais_distance_list.

[0030] Step S4, screen the visual detection targets based on the target category and size factor. The size factor is calculated by the width, height and ordinate of the target box to obtain nav_objs_list; perform secondary filtering based on the median ordinate median_y of nav_objs_list to generate valid_objs_list; sort the valid_objs_list in descending order by the ordinate and perform deduplication processing to eliminate small targets covered by large targets, and compensate for the azimuth trajectory caused by the stitching error to generate vision_azimuth_list.

[0031] In this embodiment, the calculation formula of the size factor is:

[0032] distance_estimate=size_factor×y_factor

[0033] Where, size_factor = target box width × target box height, y_factor = 1.0 / target box vertical coordinate.

[0034] In addition, the generation process of the orientation trajectory is as follows: the orientation range of each row of targets is determined by the target detection results, and a continuous trajectory is generated based on the target position change. The visual ranging parameters are excluded from the trajectory calculation.

[0035] In addition, the compensation process of the orientation trajectory is as follows: according to the calibration parameters of the stitching camera, the orientation values of the visual target in the left and right edge areas are linearly corrected to eliminate the influence of image distortion.

[0036] Step S5: Construct a cost matrix for ais_azimuth_list and vision_azimuth_list, generate a matching cost based on the weighted sum of the azimuth difference and the distance difference, and complete the association matching between the AIS target and the visual target through the singular matching algorithm. The matching process relies only on the azimuth trajectory and bypasses the visual ranging data.

[0037] In this embodiment, the singular matching algorithm is the Hungarian algorithm, which is used to minimize the total matching cost.

[0038] This embodiment also provides a bearing-only trajectory fusion system based on mosaic image detection targets and AIS information, including:

[0039] The wide-angle image stitching and detection module uses the steps in S1 above to complete wide-angle image stitching and detection.

[0040] AIS information processing module: completes AIS information processing using the steps in S2 above.

[0041] AIS target screening module: Use the steps in S3 above to complete AIS target screening.

[0042] Visual target filtering and optimization module: Use the steps in S4 above to complete visual target filtering and optimization.

[0043] Pure bearings trajectory matching module: uses the steps in S5 above to complete pure bearings trajectory matching.

[0044] Compared with the existing technology, the bearing-only trajectory fusion method based on spliced image detection and AIS information provided in this embodiment can reduce the impact of camera distortion and bypass the defects of visual ranging by using bearing-only trajectory to achieve the fusion of the two. This embodiment can achieve the following effects:

[0045] 1. By stitching three monocular cameras into a wide-angle lens (180° range), the resolution can reach 2160x14000, which is more conducive to the detection of small targets and reduces the impact of image distortion.

[0046] 2. Simultaneously perform image segmentation and full-image detection on large-resolution images to improve detection accuracy.

[0047] 3. Bypassing the defects of image ranging, azimuth trajectory is used to achieve matching and fusion of visual targets and AIS information.

[0048] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A bearing-only trajectory fusion method based on spliced image detection targets and AIS information, characterized by: The following steps are involved: Step 1: Use the left monocular camera, the middle monocular camera, and the right monocular camera to capture images and stitch them together according to the label file to generate a panoramic image with a resolution of 2160×14000, covering a 180° range directly in front of the ship. Perform image cropping and full-image target detection on the panoramic image, generate and save the azimuth trajectory of the detected target, and bypass the reliance on visual ranging. Step 2: Receive and parse AIS information, calculate the relative position of the AIS target and the own ship, save the position and distance track of the AIS target, and perform motion prediction on the AIS target; Step 3: Read the AIS targets in the range of 0-180° in front of the ship and whose distance is less than the set threshold. Keep the AIS targets closest to the ship in every preset distance interval and generate ais_azimuth_list and ais_distance_list. Step 4: Filter visual detection targets based on target category and size factor, where the size factor is calculated from the width, height, and ordinate of the target box to obtain nav_objs_list; perform secondary filtering based on the median ordinate median_y of the nav_objs_list to generate valid_objs_list; sort the valid_objs_list in descending order of ordinate and perform deduplication processing to remove small targets covered by large targets, and compensate for the azimuth trajectory caused by splicing errors to generate vision_azimuth_list; Step 5: Construct the cost matrix of ais_azimuth_list and vision_azimuth_list, generate the matching cost based on the weighted sum of the azimuth difference and the distance difference, and complete the association matching of the AIS target and the visual target through the singular matching algorithm. The matching process only relies on the azimuth trajectory and bypasses the visual ranging data.

2. The method for fusion of bearing-only tracks based on target detection in mosaic images and AIS information according to claim 1, characterized in that: in, In step 1, the specific process of the image cutting process is: dividing the panoramic image into multiple sub-regions for local detection, and optimizing the target positioning accuracy in combination with the full image detection results.

3. The method for fusion of bearing-only tracks based on target detection using mosaic images and AIS information according to claim 1, characterized in that: in, In step 1, the panoramic image with a resolution of 2160×14000 is achieved by multi-camera stitching to improve the small target detection capability and reduce image distortion.

4. The method for fusion of bearing-only tracks based on target detection in mosaic images and AIS information according to claim 1, characterized in that: in, In step 4, the process of generating the orientation trajectory is as follows: determining the orientation range of each row of targets through target detection results, and generating a continuous trajectory based on the target position change, excluding visual ranging parameters in the trajectory calculation.

5. The method for fusion of bearing-only tracks based on target detection in mosaic images and AIS information according to claim 1, characterized in that: in, In step 4, the calculation formula of the size factor is: distance_estimate=size_factor×y_factor Among them, size_factor = target box width × target box height, y_factor = 1.0 / target box vertical coordinate.

6. The method for fusion of bearing-only tracks based on target detection in mosaic images and AIS information according to claim 1, characterized in that: in, In step 4, the compensation process of the orientation trajectory is: according to the calibration parameters of the stitching camera, the orientation values of the visual target in the left and right edge areas are linearly corrected to eliminate the influence of image distortion.

7. The method for fusion of bearing-only tracks based on target detection using mosaic images and AIS information according to claim 1, characterized in that: in, In step 5, the singular matching algorithm is the Hungarian algorithm or the KM algorithm, which is used to minimize the total matching cost.

8. A bearing-only trajectory fusion system based on spliced image detection targets and AIS information, characterized by: include: Wide-angle image stitching and detection module: This module uses the left monocular camera, the middle monocular camera, and the right monocular camera to capture images, and stitches them together based on the label file to generate a panoramic image covering 180° in front of the ship with a resolution of 2160×14000. The panoramic image is then sliced and full-image target detection is performed on the image, and the azimuth trajectory of the detected target is generated and saved. The generation of the azimuth trajectory bypasses the reliance on visual ranging. AIS information processing module: receives and analyzes AIS information, calculates the relative position of the AIS target and the own ship, saves the position and distance track of the AIS target, and performs motion prediction on the AIS target; AIS target screening module: reads AIS targets within the range of 0-180° in front of the ship and whose distance is less than the set threshold, retains the AIS targets closest to the ship in every preset distance interval, and generates ais_azimuth_list and ais_distance_list; Visual target filtering and optimization module: Filters visual detection targets based on target category and size factor. The size factor is calculated from the width, height and ordinate of the target box to obtain nav_objs_list; performs secondary filtering based on the median ordinate median_y of the nav_objs_list to generate valid_objs_list; sorts the valid_objs_list in descending order of ordinate and performs deduplication processing to remove small targets covered by large targets. At the same time, it compensates for the azimuth trajectory caused by splicing errors to generate vision_azimuth_list; Bearing-only track matching module: Constructs the cost matrix of ais_azimuth_list and vision_azimuth_list, generates the matching cost based on the weighted sum of the bearing difference and the distance difference, and completes the association matching of AIS targets and visual targets through the singular matching algorithm. The matching process relies only on the bearing track and bypasses the visual ranging data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images