A ship distance estimation method and device based on AIS supervision signals

Abstract

The application provides a ship distance estimation method and device based on AIS supervision signals, relates to the technical field of intelligent shipping and water traffic supervision, and solves the technical problem that the prior art relies on a single sensor or a complex multi-source fusion scheme, resulting in a supervision blind area, inaccurate distance measurement, and a complex system, and that the prior art cannot effectively estimate the distance of an AIS-free ship. The method comprises: acquiring and preprocessing AIS data and video data; obtaining a current ship predicted position through an AIS trajectory prediction model and obtaining a ship target detection frame in a current field of view angle through a target detection model; dividing the current field of view angle of the camera into equal and reliable regions and performing space-time matching; fusing the matched visual features and AIS distance information, constructing a distance regression module with the AIS distance as a supervision signal, and training the target detection model to output a ship distance estimation value. The application is used in the distance estimation process of other ships in the perspective of a shore base and a ship bridge.

Description

Technical Field

[0001] This application relates to the field of intelligent shipping and water traffic monitoring technology, and in particular to a method and apparatus for estimating ship distance based on AIS monitoring signals. Background Technology

[0002] Current waterway vessel monitoring primarily relies on sensors such as Automatic Identification System (AIS), video surveillance, and radar, but all have significant limitations. AIS systems, constrained by their event-driven mechanism, have low data update frequencies and insufficient real-time performance in densely packed or maneuvering environments. Furthermore, many non-commercial vessels lack AIS, creating monitoring blind spots. While video surveillance provides intuitive visual information, traditional analysis methods struggle to achieve accurate distance estimation, and its accuracy is easily affected by complex environmental factors such as lighting and weather, resulting in poor stability. Radar systems suffer from difficulties in identifying small targets, high equipment costs, and the inability to acquire vessel external features. Although existing research attempts to compensate for the shortcomings of single sensors through multi-source data fusion techniques, these methods generally rely on complex coordinate system transformations and precise spatiotemporal calibration, making system implementation cumbersome. In practical applications, they face multiple challenges, including sensor synchronization, data heterogeneity, and spatiotemporal alignment, and their effective distance estimation capability for vessels without AIS remains severely inadequate.

[0003] Specifically, most existing multi-source fusion methods geometrically register AIS, radar, and video images in a unified coordinate system and achieve target association and distance calculation based on accurate extrinsic parameter calibration. On the one hand, these methods require tedious joint calibration of cameras and radar / AIS before deployment, resulting in high maintenance costs and sensitivity to the construction environment. On the other hand, existing deep learning-based monocular ranging methods typically rely on manually labeled distance or disparity data for supervised training, making it difficult to collect high-quality labeled samples on a large scale in actual waterway scenarios. In particular, for a large number of small vessels without AIS equipment, current technologies cannot indirectly endow visual models with distance perception capabilities through AIS data, nor have there been any publicly available solutions for using AIS physical distance as a supervisory signal to train monocular visual ranging networks without extrinsic parameter calibration. Summary of the Invention

[0004] This application provides a method and apparatus for estimating ship distance based on AIS surveillance signals, which solves the technical problems of existing technologies that rely on a single sensor or complex multi-source fusion schemes, resulting in blind spots in supervision, inaccurate distance measurement, and system complexity, making it impossible to effectively estimate the distance to ships without AIS.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for estimating ship distance based on AIS surveillance signals is provided, including: Acquire and preprocess the ship's AIS and video data; The ship trajectory is predicted from AIS data using an AIS trajectory prediction model to obtain the predicted position of the ship at the current moment; wherein, the AIS trajectory prediction model is obtained by constructing a delay compensation model based on a Bi-LSTM network; Ship target detection is performed on video data using a target detection model to obtain ship detection boxes; The camera's field of view is divided into several equally reliable sub-regions, and the predicted ship position and the ship detection frame within the same equally reliable sub-region are spatiotemporally matched. The successfully matched ship detection boxes are spliced ​​and fused with the ship physical distances calculated from AIS data to obtain a joint feature vector. Using the joint feature vector as input and the ship's physical distance as a supervision signal, a distance regression module is constructed to supervise the training of the target detection model. The trained target detection model outputs the ship detection box and the ship distance estimate.

[0006] Based on the above technical solutions, the ship distance estimation method based on AIS surveillance signals provided in this application innovatively combines an AIS trajectory prediction model with a visual target detection model, utilizing the precise distance information provided by AIS as a surveillance signal. This enables the target detection model to perform both ship identification and distance estimation, effectively addressing the limitations of a single sensor in waterway supervision. By dividing the camera's field of view into equally reliable regions for spatiotemporal matching, complex coordinate system transformations and calibration processes are avoided, significantly reducing system complexity and deployment costs. The final trained model can accurately estimate ship distances based on visual features even without AIS signals, fundamentally solving the problem of blind spots in the supervision of ships without AIS and significantly improving the comprehensiveness, accuracy, and practicality of waterway supervision.

[0007] In conjunction with the first aspect above, in one possible implementation, the AIS trajectory prediction model is constructed by combining a Bi-LSTM network and a GRU unit, and introducing a spatiotemporal attention mechanism; wherein, the output expression of the AIS trajectory prediction model is: ; In the formula, t is time. This is a predicted value for the ship's position at the next moment; This refers to historical ship trajectory sequences in AIS data; It is a bidirectional long short-term memory network. For attention mechanisms, This is a gated loop unit.

[0008] In conjunction with the first aspect above, in one possible implementation, the object detection model is an improved single-stage object detection model, which is improved by introducing an Effective SE channel attention mechanism and a Halo local self-attention mechanism to optimize the model parameters.

[0009] In conjunction with the first aspect above, in one possible implementation, the spatiotemporal matching of the predicted ship position and the ship detection frame within the same equally reliable sub-region includes: Establish a coordinate system centered at the origin and with an initial angle of... and field of view range The equally reliable region is used to uniformly divide the camera's field of view into K sub-regions. , ;in, and These are the visible angles of the k-th and (k+1)-th sub-regions, respectively. For the angular range of each sub-region, ; Based on camera installation location The ship's latitude and longitude in the predicted position Perform coordinate transformation to obtain transformed coordinates; wherein, the transformed coordinates The expression is: ; In the formula, N represents the total number of ships; Based on the equally reliable region, the sub-region corresponding to the transformed coordinates is obtained, and the ship sub-region is obtained; A pixel coordinate system is established based on video frames, and the pixel coordinate system is divided into K pixel regions in the horizontal direction. , ;in, The starting pixel region, W represents the range of each pixel region, and W is the width of the video frame. The pixel region corresponding to the ship detection box is obtained based on the pixel coordinate system, thus obtaining the ship pixel region. Matching and verification of ship pixel regions and ship sub-regions are performed using either angle-priority or distance-priority principles.

[0010] In conjunction with the first aspect above, in one possible implementation, the matching verification of the ship pixel region and the ship sub-region includes: For ship sub-regions For several ships within the baseline, calculate the angular intervals between the ships and the baseline. ; The AIS features of several ships are classified according to the size of the angular interval. Sort the data to obtain the AIS feature sequence. The AIS features are extracted from AIS data. From left to right, the ship pixel area Several ships within the range were matched with AIS feature sequences to obtain angle matching results; For ship sub-regions Physical distance between several ships and the camera location Sort the distances to obtain the distance sequence. ; Ship pixel area from bottom to top Several ships within the range are matched with the distance sequence to obtain the distance matching results; If the angle matching result and the distance matching result are the same, the match is considered successful.

[0011] In conjunction with the first aspect above, in one possible implementation, the distance regression loss function of the distance regression module is constructed based on the mean squared error; wherein, the expression of the distance regression loss function is: ; In the formula, Estimate the distance to the ship predicted by the distance regression module. The physical distance to the ship is calculated based on AIS data.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, the distance regression module further includes a dynamic calibration mechanism; wherein the dynamic calibration mechanism uses a sliding window to smooth the estimated ship distances of consecutive frames, and updates the parameters of the target detection model online in conjunction with the latest matching results.

[0013] In conjunction with the first aspect described above, in one possible implementation, the distance regression module further includes: Freeze the parameters of the distance regression module to obtain a distance regression module with frozen weights; Acquire video data of vessels without AIS, and perform target detection on the video data to obtain the detection bounding box of vessels without AIS; Input the detection box of the vessel without AIS into the distance regression module with frozen weights to obtain the distance estimate of the vessel without AIS.

[0014] In conjunction with the first aspect above, in one possible implementation, the preprocessing step includes: Cleaning, outlier removal, and trajectory interpolation and correction of AIS data; Perform image enhancement and annotation on video data.

[0015] Secondly, a ship distance estimation device based on AIS surveillance signals is provided, comprising: The data acquisition and preprocessing module is used to simultaneously acquire and preprocess waterway monitoring video data and ship AIS data; The visual target detection module is used to extract the detection box information of ship targets from video frames based on an improved target detection model; The trajectory prediction module is used by a ship trajectory prediction model based on a hybrid attention mechanism to process historical AIS data and predict the future position of ships. The spatiotemporal matching module is used to divide the camera's field of view into equal and reliable regions and pixel regions, and to perform matching between AIS targets and video detection targets; The multimodal fusion and distance regression module is used to fuse successfully matched AIS distance information with visual features, train and apply the distance regression module to estimate and output the ship distance.

[0016] Thirdly, an electronic device is provided, comprising: a communication unit and a processing unit; the communication unit is used to acquire and preprocess AIS data and video data of a ship; the processing unit is used to predict the ship's trajectory in the AIS data using an AIS trajectory prediction model to obtain the predicted ship position at the current moment; to detect ship targets in the video data using a target detection model to obtain ship detection boxes; to divide the visible range of the camera into several equally reliable sub-regions, and to perform spatiotemporal matching of the predicted ship position and the ship detection box within the same equally reliable sub-region; to concatenate and fuse the successfully matched ship detection boxes with the ship physical distance calculated from the AIS data to obtain a joint feature vector; to construct a distance regression module using the joint feature vector as input and the ship physical distance as a supervision signal, to supervise the training of the target detection model, and to output the ship detection box and the estimated ship distance through the trained target detection model.

[0017] Fourthly, this application provides an electronic device, including: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the methods described in the first aspect and any possible implementation thereof. The electronic device may be an electronic device or a chip within an electronic device.

[0018] Fifthly, this application provides a ship distance estimation system based on AIS surveillance signals, comprising: a data acquisition and preprocessing module, a visual target detection module, a trajectory prediction module, a spatiotemporal matching module, and a multimodal fusion and distance regression module; wherein, the data acquisition and preprocessing module is used to simultaneously acquire and preprocess waterway monitoring video data and ship AIS data; the visual target detection module is used to extract the detection box information of ship targets from video frames based on an improved target detection model; the trajectory prediction module is used to process historical AIS data based on a ship trajectory prediction model with a hybrid attention mechanism to predict the future position of the ship; the spatiotemporal matching module is used to divide the camera's field of view into an equal and reliable region and a pixel region, and to perform matching between AIS targets and video detection targets; the multimodal fusion and distance regression module is used to fuse the successfully matched AIS distance information with visual features, train and apply the distance regression module, and realize the estimation and output of ship distance.

[0019] In a sixth aspect, this application provides a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0020] In a seventh aspect, this application provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0021] This application provides a method and apparatus for ship distance estimation based on AIS surveillance signals. By combining an AIS trajectory prediction model with a visual target detection model, it innovatively utilizes the precise distance information provided by AIS as a surveillance signal. This enables the target detection model to perform distance estimation while simultaneously identifying ships, effectively overcoming the limitations of single sensors in waterway monitoring. By dividing the camera's field of view into equally reliable regions for spatiotemporal matching, complex coordinate system transformations and calibration processes are avoided, significantly reducing system complexity and deployment costs. The final trained model can accurately estimate ship distances based on visual features even without AIS signals, fundamentally solving the problem of blind spots in the monitoring of ships without AIS and significantly improving the comprehensiveness, accuracy, and practicality of waterway monitoring.

[0022] Compared with the prior art, this application has at least the following technical innovations and beneficial effects: (1) A monocular visual ranging method is proposed, which uses the ship physical distance provided by AIS as the supervision signal. By constructing a multimodal joint feature vector, the AIS distance information is explicitly introduced into the training process of the target detection network, so that the monocular camera has absolute distance perception capability without relying on additional ranging sensors, thus avoiding the cost of a large amount of manually labeled distance data.

[0023] (2) A calibration-free time-space matching strategy based on the fan-shaped division of the camera field of view and the division of the pixel region is proposed. Through the dual sorting and matching of angle priority and distance priority, the accurate alignment of multi-source heterogeneous data can be achieved without the need for external parameter calibration between the camera and the AIS receiving device. This simplifies the system deployment process and reduces the dependence on on-site construction accuracy.

[0024] (3) A dynamic calibration mechanism including sliding window smoothing and online fine-tuning is constructed. The distance regression module is adaptively corrected by using the prediction results of continuous frames and the latest AIS matching samples, which improves the stability and robustness of distance measurement under complex hydrological and meteorological conditions and long-term operation scenarios.

[0025] (4) By introducing AIS distance supervision during the training phase and freezing the weights of the distance regression module during the operation phase, the distance perception knowledge transfer from “ships with AIS” to “ships without AIS” was realized, enabling the system to effectively measure the distance of small fishing boats, work boats and other targets that have not installed AIS equipment, and significantly reducing the blind spots of waterway supervision. Attached Figure Description

[0026] Figure 1 A system architecture diagram of a ship distance estimation system based on AIS surveillance signals is provided for embodiments of this application; Figure 2 A flowchart illustrating a ship distance estimation method based on AIS surveillance signals provided in this application embodiment; Figure 3 The image shows the ship target detection results provided in the embodiments of this application; Figure 4 A flowchart illustrating the AIS data and video data fusion method provided in this application embodiment; Figure 5 This is a schematic diagram of the ship ranging effect provided in the embodiments of this application; Figure 6 A network architecture diagram of a multimodal hybrid neural network for ship distance estimation based on AIS surveillance signals provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The ship distance estimation method based on AIS surveillance signals provided in this application embodiment can be applied to, for example... Figure 1 In the ship distance estimation system 100 based on AIS surveillance signals shown, such as Figure 1 As shown, the communication system includes: a data acquisition and preprocessing module 10, a visual target detection module 20, a trajectory prediction module 30, a spatiotemporal matching module 40, and a multimodal fusion and distance regression module 50.

[0028] Among them, the data acquisition and preprocessing module 10 is used to simultaneously acquire and preprocess waterway monitoring video data and ship AIS data; The visual target detection module 20 is used to extract the detection box information of ship targets from video frames based on an improved target detection model; The trajectory prediction module 30 is used to process historical AIS data and predict the future position of ships based on a hybrid attention mechanism ship trajectory prediction model. The spatiotemporal matching module 40 is used to divide the camera's field of view into an equal and reliable region and a pixel region, and to perform matching between the AIS target and the video detection target. The multimodal fusion and distance regression module 50 is used to fuse successfully matched AIS distance information with visual features, train and apply the distance regression module to estimate and output the ship distance.

[0029] To address the technical problems of existing technologies that rely on single sensors or complex multi-source fusion schemes, resulting in regulatory blind spots, inaccurate ranging, and system complexity, thus failing to effectively estimate the distance to vessels without AIS, this application provides a vessel distance estimation method based on AIS surveillance signals. This method includes: Acquire and preprocess the ship's AIS and video data; The ship trajectory is predicted from AIS data using an AIS trajectory prediction model to obtain the predicted position of the ship at the current moment; wherein, the AIS trajectory prediction model is obtained by constructing a delay compensation model based on a Bi-LSTM network; Ship target detection is performed on video data using a target detection model to obtain ship detection boxes; The camera's field of view is divided into several equally reliable sub-regions, and the predicted ship position and the ship detection frame within the same equally reliable sub-region are spatiotemporally matched. The successfully matched ship detection boxes are spliced ​​and fused with the ship physical distances calculated from AIS data to obtain a joint feature vector. Using the joint feature vector as input and the ship's physical distance as a supervision signal, a distance regression module is constructed to supervise the training of the target detection model. The trained target detection model outputs the ship detection box and the ship distance estimate.

[0030] It should be noted that the entire training process of the target detection model does not rely on the external geometric calibration parameters between the camera and the AIS receiving device. After training, for target ships that do not carry AIS devices or whose AIS data is missing, only the ship detection box is input into the distance regression module, and the corresponding ship distance estimate is output. The parameters of the distance regression module are frozen after training and are used to realize the mapping process from the visual features extracted from the ship detection box to the physical distance of the ship.

[0031] Based on this, the technical problems of existing technologies relying on a single sensor or complex multi-source fusion schemes, resulting in regulatory blind spots, inaccurate ranging, and system complexity, are solved, making it impossible to effectively estimate the distance to vessels without AIS.

[0032] like Figure 2 As shown in the embodiments of this application, the ship distance estimation method based on AIS surveillance signals includes: S201. Acquire the ship's AIS data and video data and perform preprocessing.

[0033] The AIS data includes MMSI, vessel position, speed, and heading information; the video data is acquired in real time via a camera; and the vessel detection frame includes position coordinates and category labels.

[0034] In some implementations, the preprocessing step includes: Cleaning, outlier removal, and trajectory interpolation and correction of AIS data; Perform image enhancement and annotation on video data.

[0035] It should be noted that the AIS data has a clear target number and global spatiotemporal coordinates, and the video data contains real-time images of the waterway. It has high resolution and a fast update frequency, and can provide continuous visual information. The system synchronously collects AIS data and video data and records them according to a unified timestamp to form multi-source perception raw input. The preprocessing of AIS data and video data is for format standardization and quality improvement to ensure the validity and consistency of the data.

[0036] For example, in a certain location, AIS devices and webcams are deployed to collect waterway video datasets and synchronized AIS datasets. The video data camera system uses wide-angle high-definition cameras with a resolution of 1920×1080 and transmits the data to the local machine via an SSH remote tunnel; AIS data is obtained from an external API via HTTP requests, and the retrieved results are saved as JSON format files, further converted into CSV files, and the collected AIS data is preprocessed, including AIS data cleaning, outlier removal, trajectory correction, and interpolation. The corrected AIS data is shown in Table 1.

[0037] Table 1

[0038] S202. Predict the ship's trajectory using the AIS trajectory prediction model to obtain the ship's predicted position at the current moment.

[0039] The AIS trajectory prediction model is obtained by constructing a delay compensation model based on a Bi-LSTM network.

[0040] In some implementations, the AIS trajectory prediction model is constructed by combining a Bi-LSTM network and a GRU unit, and a spatiotemporal attention mechanism is introduced; wherein, the output expression of the AIS trajectory prediction model is: ; In the formula, t is time. This is a predicted value for the ship's position at the next moment; This refers to historical ship trajectory sequences in AIS data; It is a bidirectional long short-term memory network. For attention mechanisms, This is a gated loop unit.

[0041] It should be noted that this model combines a Bi-LSTM network to capture the global spatiotemporal dependence of ship trajectories, introduces GRU units to optimize the gradient propagation process, and uses a spatiotemporal attention mechanism to assign weights according to the importance of different time steps in the trajectory, dynamically focusing on key trajectory points. This effectively overcomes the prediction instability of traditional LSTM in handling complex scenarios such as inland waterway bends, and can significantly improve the continuous prediction accuracy of ship trajectories.

[0042] For example, the operating system used was Linux version 5.4.0-200-generic, the CPU version was E5-2698 v4 @2.20GHz, the GPU was Tesla V100-DGXS-32GB with 32GB of video memory, the CPU was Intel(R) Xeon(R), the compiler was Python 3.10.15, the deep learning framework was PyTorch 2.5.0, and the CUDA version was 12.4. During the training of the AIS trajectory prediction model, the sliding window length was set to 6, the step size was 1, and the dataset was divided into training and test sets in an 8:2 ratio. The final comparison between the predicted and actual ship trajectories is shown in Table 2.

[0043] Table 2

[0044] S203. Detect ship targets in the video data using a target detection model to obtain ship detection boxes.

[0045] In some implementations, the target detection model is an improved single-stage target detection model, which is improved by introducing an Effective SE channel attention mechanism and a Halo local self-attention mechanism to optimize the model parameters.

[0046] It should be noted that the target detection model introduces the Effective SE channel attention mechanism and the Halo local self-attention mechanism to optimize model parameters, improve the detection accuracy of small and low-resolution targets, and reduce the number of model parameters, making it more suitable for deployment on edge devices and adapting to the needs of waterway supervision in different environments and scenarios.

[0047] For example, when training the ship target detection model, the number of iterations is set to 300, the input batch size is 16, the initial learning rate is 0.01, and the output ship target detection result is shown in the figure below. Figure 3 As shown, the data includes vessel type (cargo ships and tankers, etc.) and confidence level. Video data is detected using an improved YOLO single-stage object detection model, automatically extracting the coordinate information of the detection boxes for each vessel. Each detection box contains the center coordinates of the i-th detected target. The width and height of the i-th detection bounding box This provides support for subsequent matching and ranging.

[0048] S204. Divide the camera's field of view into several equally reliable sub-regions, and perform spatiotemporal matching of the predicted ship position and the ship detection frame within the same equally reliable sub-region.

[0049] In some implementations, the spatiotemporal matching of the predicted ship position and the ship detection frame within the same equally reliable sub-region includes: Establish a coordinate system centered at the origin and with an initial angle of... and field of view range The equally reliable region is used to uniformly divide the camera's field of view into K sub-regions. , ;in, and These are the visible angles of the k-th and (k+1)-th sub-regions, respectively. For the angular range of each sub-region, ; Based on camera installation location The ship's latitude and longitude in the predicted position Perform coordinate transformation to obtain transformed coordinates; wherein, the transformed coordinates The expression is: ; In the formula, N represents the total number of ships; Based on the equally reliable region, the sub-region corresponding to the transformed coordinates is obtained, and the ship sub-region is obtained; A pixel coordinate system is established based on video frames, and the pixel coordinate system is divided into K pixel regions in the horizontal direction. , ;in, The starting pixel region, W represents the range of each pixel region, and W is the width of the video frame. The pixel region corresponding to the ship detection box is obtained based on the pixel coordinate system, thus obtaining the ship pixel region. Matching and verification of ship pixel regions and ship sub-regions are performed using either angle-priority or distance-priority principles.

[0050] It should be noted that by adopting an isoangular and equidistant reliable region division based on the camera's field of view, and combining AIS trajectory prediction with video detection results for logical spatiotemporal matching, and by improving Kalman filtering and dual verification mechanisms, the differences in format, frequency and spatial resolution of multi-source heterogeneous data and the problem of temporal conflicts are effectively solved. There is no need for cumbersome coordinate system transformation and external calibration process, which significantly reduces system complexity and deployment cost.

[0051] For example, an equally reliable monitoring area is established with the camera installation location (118.35° longitude, 31.28° latitude) as the coordinate origin, a starting angle of 30°, and a field of view of 120°. This area is then evenly divided into 6 sub-regions, each covering a 20° viewing angle. Simultaneously, a 1920×1080 resolution video frame is horizontally divided into 6 pixel regions, each 320 pixels wide. The ship's predicted latitude and longitude (e.g., MMSI) is then transformed using coordinate transformation. The predicted location of 413879988 (longitude 118.32°, latitude 31.29°) was converted to polar coordinates, and the relative distance to the ship was calculated to be 310.056 meters and the azimuth angle to be 45.3°. Based on this, it was determined that the ship belongs to the first equally reliable sub-region. At the same time, through target detection, it was determined that the ship is located in the third pixel region in the pixel coordinate system (horizontal coordinate interval [640, 960]). When multiple targets appear in the same equally reliable sub-region, the angle priority principle is used to match the target sequence with the smallest polar coordinate angle difference with the detection box sequence from left to right in the pixel region, and the distance priority principle is used for double verification. Finally, the matching result is optimized by an improved Kalman filter algorithm.

[0052] In some implementations, the matching verification of the ship pixel region and the ship sub-region includes: For ship sub-regions For several ships within the baseline, calculate the angular intervals between the ships and the baseline. ; The AIS features of several ships are classified according to the size of the angular interval. Sort the data to obtain the AIS feature sequence. The AIS features are extracted from AIS data. From left to right, the ship pixel area Several ships within the range were matched with AIS feature sequences to obtain angle matching results; For ship sub-regions Physical distance between several ships and the camera location Sort the distances to obtain the distance sequence. ; Ship pixel area from bottom to top Several ships within the range are matched with the distance sequence to obtain the distance matching results; If the angle matching result and the distance matching result are the same, the match is considered successful.

[0053] For example, when three ships appear simultaneously in the third equally reliable sub-region, the system first calculates the angular intervals between each ship and the baseline (60°), which are 2.1°, 3.5°, and 5.2°, respectively. Based on this, the AIS features are sorted in ascending order to obtain the sequence [ship A, ship B, ship C]. At the same time, three target boxes are detected from left to right in the corresponding pixel area (horizontal coordinate 640-960 pixels). After initially establishing the correspondence through angle-first matching, the system further sorts them according to the distance from the ship to the camera to obtain [312m, 328m, 345m], and verifies the consistency of spatial distribution from bottom to top in the pixel area. When both matching results show the same correspondence (ship A - target in the lower left, ship B - target in the middle, ship C - target in the upper right), the matching is determined to be successful.

[0054] S205. The successfully matched ship detection boxes are spliced ​​and fused with the ship physical distances calculated from the AIS data to obtain a joint feature vector.

[0055] As an example, such as Figure 4 As shown, the process of fusing AIS data and video data is systematically demonstrated, constructing an end-to-end processing framework from multi-source data input to regulatory application output. The diagram clearly presents six core stages using a layered architecture: First, the perception input layer synchronously receives AIS data and video data, establishing a basic data stream with a unified timestamp; then, the preprocessing layer performs interpolation and completion on the AIS data and enhanced target detection on the video data; next, the region segmentation and prediction layer models the camera's field of view as an equally reliable region and uses the AIS ship prediction model to achieve spatiotemporal prediction of ship trajectories; in the multi-source matching layer, a dual strategy of initial screening based on region overlap and fine matching based on attribute constraints is adopted to effectively solve the problems of target occlusion and temporal differences; the distance perception layer innovatively integrates monocular visual ranging and AIS coordinate conversion results to generate highly reliable distance estimates; finally, the application output layer uploads the results to the waterway regulatory platform to realize dynamic visual monitoring and collision warning functions for ships.

[0056] For example, after a ship detection bounding box is successfully matched with AIS data, the system performs deep fusion of the visual features of the bounding box and the AIS physical distance: taking ship target 1 as an example, firstly, the system extracts the 128-dimensional visual feature vector of its bounding box [0.174844, 0.415556, 0.095104, 0.030074] from the improved YOLO detection model, and at the same time obtains the AIS physical distance of 310.056 meters calculated through coordinate transformation; then, after normalization, this distance value is concatenated with the visual feature vector to form a 129-dimensional joint feature vector; this joint feature vector contains both the ship's external scale information and the precise spatial distance information, providing rich multimodal input for the subsequent distance regression module.

[0057] S206. Using the joint feature vector as input and the ship's physical distance as a supervision signal, construct a distance regression module to supervise the training of the target detection model. The trained target detection model outputs the ship detection box and the ship distance estimate.

[0058] It should be noted that during the multimodal input stage, the system concatenates video frames, target detection box coordinates, and the physical distance information between the ship and the navigation mark provided by AIS into a cross-modal joint feature vector, which is then input into the supervised distance regression module. The mean squared error (MSE) loss function is used for training, and a sliding window dynamic calibration mechanism is designed to smooth the predicted values ​​of consecutive frames, improving output stability in dynamic scenes. After sufficient training, even when encountering ships in new scenes that do not carry or have temporarily disabled AIS equipment, the model can automatically estimate the relative distance between the ship and the navigation mark or shore-based base station based solely on the video detection results.

[0059] For example, the schematic diagram of the ship ranging effect output by this solution is as follows: Figure 5 As shown in Table 3, the corresponding ship distance information is also presented. Furthermore, to verify the effectiveness of the distance measurement model in this application, the actual distance from the ship to the navigation mark at the current moment was calculated using the Earth coordinate system and the Haversine semi-versus formula, and compared with the distance measurement results output by the system. The verification results are shown in Table 4. The results show that the average percentage error remains below 5%, fully demonstrating the adaptability and practicality of this application for ship distance measurement in complex waterway scenarios.

[0060] Table 3

[0061] Table 4

[0062] In some implementations, the distance regression loss function of the distance regression module is constructed based on the mean squared error; wherein, the expression of the distance regression loss function is: ; In the formula, Estimate the distance to the ship predicted by the distance regression module. The physical distance to the ship is calculated based on AIS data.

[0063] It should be noted that the distance regression module makes full use of the spatiotemporal consistency and polar coordinate information of the previous AIS data and video detection results to achieve high-precision regression of the distance from the pixel position of the ship to the actual distance, effectively enhancing the distance perception capability of ship target detection.

[0064] In some implementations, the distance regression module further includes a dynamic calibration mechanism; wherein the dynamic calibration mechanism uses a sliding window to smooth the estimated ship distances of consecutive frames and updates the parameters of the target detection model online in conjunction with the latest matching results.

[0065] It should be noted that the sliding window dynamic calibration mechanism is designed to smooth continuous frames, thereby achieving adaptive optimization of the real-time performance and stability of ship ranging output in dynamic scenarios. This provides technical support for accurate ranging and positioning of ships in complex waters and enhances the scalability and scalability of the waterway monitoring system.

[0066] It should be noted that the length of the sliding window and the weight of each frame are adaptively adjusted according to the ship traffic flow in the channel and the degree of scene change: the length of the sliding window is increased during periods of dense ship traffic or frequent target obstruction to improve output stability, and the length of the sliding window is decreased during periods of sparse ship traffic or drastic environmental changes to improve response speed.

[0067] For example, the dynamic calibration mechanism uses a sliding window of size 5 to perform weighted averaging of ship distance estimates in consecutive video frames, where the current frame has a weight of 0.4 and historical frames have exponentially decaying weights. When the system detects that the distance fluctuation exceeds a threshold (e.g., 8 meters) for three consecutive frames, it automatically triggers a parameter update process: the latest successfully matched AIS-visual data pair is added to the training queue, and the distance regression network is fine-tuned online with a learning rate of 0.0001.

[0068] In some implementations, the distance regression module further includes: Freeze the parameters of the distance regression module to obtain a distance regression module with frozen weights; Acquire video data of vessels without AIS, and perform target detection on the video data to obtain the detection bounding box of vessels without AIS; Input the detection box of the vessel without AIS into the distance regression module with frozen weights to obtain the distance estimate of the vessel without AIS.

[0069] It should be noted that the physical distance of ships in AIS data serves as a supervisory signal, guiding the target detection model to learn the nonlinear mapping relationship between pixel coordinates and actual distances. This enables the model to automatically estimate the physical distance of similar ships based on visual information when there is no AIS signal, achieving continuous and effective monitoring of ships in scenarios where AIS data is missing or unavailable, and significantly improving the accuracy and timeliness of waterway monitoring. After training the target detection model with matching samples containing AIS distance supervisory information, the parameters of the distance regression module are frozen to achieve distance estimation for ships without AIS.

[0070] It should be noted that the quality of the received AIS data is evaluated, and the credibility weight of the AIS distance information is determined based on the update time interval of the AIS messages, the signal-to-noise ratio, and the smoothness of the trajectory. When constructing the loss function of the distance regression module, the credibility weight is multiplied by the distance regression error of the corresponding sample to reduce the impact of abnormal AIS data on the training of the distance regression module.

[0071] For example, the trained distance regression module achieves accurate distance measurement of vessels without AIS through its internally learned visual feature-physical distance mapping function. In practice, the model is first trained using 4132 sets of AIS-visual matching data in a certain waterway, and then all network weight parameters are frozen. In actual deployment, when the system detects a fishing vessel without AIS (such as a target with missing MMSI), the visual features of its vessel detection box [0.511198, 0.469907, 0.018229, 0.041556] are directly input into the frozen model, and the forward inference outputs a distance estimate of 409.313 meters.

[0072] This application proposes a waterway vessel distance perception technology based on multi-source data fusion. Its core idea is to combine the vessel's AIS information with a target detection model. The specific implementation process is as follows: Figure 6 As shown. Figure 6This paper reveals the internal architecture of a multimodal hybrid neural network for ship distance estimation based on AIS supervision signals, detailing the algorithmic implementation mechanism from feature extraction to distance regression. The network structure includes a dual-channel input interface for parallel processing. The AIS data stream on the left is processed by a Bi-LSTM+GRU hybrid network to extract spatiotemporal features, while the video data stream on the right is processed by a CNN network integrating Effective SE channel attention and Halo local self-attention to extract visual features. The cross-attention fusion module in the middle of the diagram achieves deep interaction of multimodal features through an adaptive weight allocation mechanism, effectively capturing the correlation between visual features and location information. Finally, under a supervised learning framework, the true distance provided by AIS is used as the supervision signal, and the output accuracy of the distance regression module is optimized through the MSE loss function. The innovation of this network design lies in transforming the traditional object detection task into a transferable distance estimation problem, enabling the model to apply the learned visual-distance mapping relationship to ships without AIS, truly achieving a breakthrough in knowledge transfer.

[0073] Based on the above technical solutions, the ship distance estimation method based on AIS surveillance signals provided in this application innovatively utilizes the precise distance information provided by AIS as a surveillance signal by combining an AIS trajectory prediction model with a visual target detection model. This enables the target detection model to perform distance estimation while simultaneously identifying ships, effectively overcoming the limitations of a single sensor in waterway supervision. By dividing the camera's field of view into equally reliable regions for spatiotemporal matching, complex coordinate system transformation and calibration processes are avoided, significantly reducing system complexity and deployment costs. The final trained model can accurately estimate ship distances based on visual features even without AIS signals, fundamentally solving the problem of blind spots in the supervision of ships without AIS and significantly improving the comprehensiveness, accuracy, and practicality of waterway supervision.

[0074] In one possible implementation, this application embodiment also provides a ship distance estimation device based on AIS surveillance signals, comprising: The data acquisition and preprocessing module is used to simultaneously acquire and preprocess waterway monitoring video data and ship AIS data; The visual target detection module is used to extract the detection box information of ship targets from video frames based on an improved target detection model; The trajectory prediction module is used by a ship trajectory prediction model based on a hybrid attention mechanism to process historical AIS data and predict the future position of ships. The spatiotemporal matching module is used to divide the camera's field of view into equal and reliable regions and pixel regions, and to perform matching between AIS targets and video detection targets; The multimodal fusion and distance regression module is used to fuse successfully matched AIS distance information with visual features, train and apply the distance regression module to estimate and output the ship distance.

[0075] It should be noted that through a highly integrated modular design, the entire process from data acquisition to distance estimation is automated. This device innovatively combines trajectory prediction, visual inspection, and multimodal fusion modules, effectively solving the regulatory blind spots inherent in single sensors. Furthermore, guided by AIS surveillance signals, the system possesses autonomous distance sensing capabilities. This device can achieve accurate distance estimation for various types of vessels, both with and without AIS signals, without relying on complex calibration processes. This significantly improves the automation level and all-weather monitoring capabilities of waterway supervision. Simultaneously, its modular architecture facilitates maintenance and upgrades, making it suitable for deployment on edge computing devices.

[0076] For example, the device is deployed in a smart navigation aid system for a waterway. Its data acquisition and preprocessing module acquires data synchronously through a shore-based high-definition camera (1920×1080 resolution) and an AIS receiver. The visual target detection module uses an improved YOLOv5 model (introducing an Effective SE attention mechanism) to extract ship detection boxes in real time. The trajectory prediction module uses a Bi-LSTM and attention mechanism-based AIS trajectory prediction model to predict future positions using six historical trajectory points as input. The spatiotemporal matching module divides the 120-degree field of view into eight equally reliable 22.5-degree regions and adopts an angle-distance dual verification strategy. The multimodal fusion and distance regression module fuses successfully matched visual features with AIS distances into a 129-dimensional feature vector and uses a fully connected network with a dynamic calibration mechanism to estimate ship distances.

[0077] The beneficial effects of the technical solution in this application are as follows: (1) By combining AIS data with a target detection model, the present invention enables the target detection model to have distance estimation capabilities, which can more accurately obtain the positioning information of ships. Even without AIS data, ships can be effectively monitored, thus improving the comprehensiveness and accuracy of waterway supervision.

[0078] (2) The improved single-stage target detection model introduces the Effective SE channel attention mechanism and the Halo local self-attention mechanism, optimizes the model parameters, improves the detection accuracy of small targets and low-resolution targets, and reduces the number of model parameters, making it more suitable for deployment on edge devices and adapting to the needs of waterway supervision in different environments and scenarios.

[0079] (3) The proposed dynamic fusion strategy for uncalibrated multi-source data adopts a logical spatiotemporal matching method based on regional division and an improved Kalman filter algorithm, which solves the geometric constraints and spatiotemporal conflicts in traditional data fusion methods, improves the accuracy and real-time performance of heterogeneous data fusion, and provides strong support for the refined monitoring of waterway traffic conditions.

[0080] (4) This invention does not rely on complex coordinate system transformation and calibration processes, simplifies the data processing flow, reduces the complexity and computational cost of the system, improves the scalability and maintainability of the system, and is conducive to its widespread application in actual waterway supervision.

[0081] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0082] When using integrated units, Figure 7 A possible structural schematic diagram of the electronic device (denoted as electronic device 50) involved in the above embodiments is shown. The electronic device 50 includes a processing unit 501 and a communication unit 502, and may also include a storage unit 503. Figure 7 The structural diagram shown can be used to illustrate the structure of the electronic device involved in the above embodiments.

[0083] when Figure 7 The schematic diagram shown is used to illustrate the structure of the electronic device involved in the above embodiments. The processing unit 501 is used to control and manage the operation of the electronic device, the communication unit 502 is used for the electronic device to communicate with other devices, and the storage unit 503 is used to store the program code and data of the electronic device.

[0084] For example, communication unit 502 is used to acquire and preprocess the ship's AIS data and video data; The processing unit 501 is used to predict ship trajectories from AIS data using an AIS trajectory prediction model to obtain the predicted ship position at the current moment; to detect ship targets from video data using a target detection model to obtain ship detection boxes; to divide the camera's field of view into several equally reliable sub-regions and to perform spatiotemporal matching of the predicted ship positions and ship detection boxes within the same equally reliable sub-region; to concatenate and fuse the successfully matched ship detection boxes with the ship physical distances calculated from the AIS data to obtain a joint feature vector; to construct a distance regression module using the joint feature vector as input and the ship physical distance as a supervision signal to supervise the training of the target detection model; and to output ship detection boxes and ship distance estimates through the trained target detection model.

[0085] The processing unit 501 can be a processor or a controller, and the communication unit 502 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 503 can be a memory. When the electronic device 50 is a chip, the processing unit 501 can be a processor or a controller, and the communication unit 502 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 503 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0086] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the electronic device 50 can be considered as the communication unit 502 of the electronic device 50, and the processor with processing functions can be considered as the processing unit 501 of the electronic device 50. Optionally, the device in the communication unit 502 used to implement the receiving function can be considered as the communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 502 used to implement the transmitting function can be considered as the transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0087] Figure 7If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0088] Figure 7 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0089] This application also provides a hardware structure diagram of an electronic device (denoted as electronic device 60), see [link to diagram]. Figure 8 The electronic device 60 includes a processor 601, and optionally, a memory 602 connected to the processor 601.

[0090] In the first possible implementation, see Figure 8 The electronic device 60 also includes a transceiver 603. The processor 601, memory 602, and transceiver 603 are connected via a bus. The transceiver 603 is used to communicate with other devices or communication networks. Optionally, the transceiver 603 may include a transmitter and a receiver. The device in the transceiver 603 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 603 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0091] Based on the first possible implementation method Figure 8 The structural diagram shown can be used to illustrate the structure of the electronic device involved in the above embodiments.

[0092] in, Figure 8 This can also be illustrated by a system chip in an electronic device. In this case, the actions performed by the aforementioned electronic device can be implemented by this system chip; the specific actions performed can be found above and will not be repeated here.

[0093] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0094] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0095] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

Claims

1. A method for estimating ship distance based on AIS surveillance signals, characterized in that, include: Acquire and preprocess the ship's AIS and video data; The ship trajectory is predicted from AIS data using an AIS trajectory prediction model to obtain the predicted position of the ship at the current moment; wherein, the AIS trajectory prediction model is obtained by constructing a delay compensation model based on a Bi-LSTM network; Ship target detection is performed on video data using a target detection model to obtain ship detection boxes; The camera's field of view is divided into several equally reliable sub-regions, and the predicted ship position and the ship detection frame within the same equally reliable sub-region are spatiotemporally matched. The successfully matched ship detection boxes are spliced ​​and fused with the ship physical distances calculated from AIS data to obtain a joint feature vector. Using the joint feature vector as input and the ship's physical distance as a supervision signal, a distance regression module is constructed to supervise the training of the target detection model. The trained target detection model outputs the ship detection box and the ship distance estimate. The spatiotemporal matching of predicted ship positions and ship detection frames within the same equally reliable sub-region includes: Establish a coordinate system centered at the origin and with an initial angle of... and field of view range The equally reliable region is used to uniformly divide the camera's field of view into K sub-regions. , ;in, and These are the visible angles of the k-th and (k+1)-th sub-regions, respectively. For the angular range of each sub-region, ; Based on camera installation location The ship's latitude and longitude in the predicted position Perform coordinate transformation to obtain transformed coordinates; wherein, the transformed coordinates The expression is: ； In the formula, N represents the total number of ships; Based on the equally reliable region, the sub-region corresponding to the transformed coordinates is obtained, and the ship sub-region is obtained; A pixel coordinate system is established based on video frames, and the pixel coordinate system is divided into K pixel regions in the horizontal direction. , ;in, The starting pixel region, W represents the range of each pixel region, and W is the width of the video frame. The pixel region corresponding to the ship detection box is obtained based on the pixel coordinate system, thus obtaining the ship pixel region. Matching and verification of ship pixel regions and ship sub-regions are performed using either angle-priority or distance-priority principles. The matching and verification of the ship pixel region and ship sub-region includes: For ship sub-regions For several ships within the baseline, calculate the angular intervals between the ships and the baseline. ; The AIS features of several ships are classified according to the size of the angular interval. Sort the data to obtain the AIS feature sequence. The AIS features are extracted from AIS data. From left to right, the ship pixel area Several ships within the range were matched with AIS feature sequences to obtain angle matching results; For ship sub-regions Physical distance between several ships and the camera location Sort the distances to obtain the distance sequence. ; Ship pixel area from bottom to top Several ships within the range are matched with the distance sequence to obtain the distance matching results; If the angle matching result and the distance matching result are the same, the match is considered successful.

2. The ship distance estimation method based on AIS surveillance signals according to claim 1, characterized in that, The AIS trajectory prediction model is constructed by combining a Bi-LSTM network and a GRU unit, and introduces a spatiotemporal attention mechanism; the output expression of the AIS trajectory prediction model is: ； In the formula, t is time. This is a predicted value for the ship's position at the next moment; This refers to historical ship trajectory sequences in AIS data; It is a bidirectional long short-term memory network. For attention mechanisms, This is a gated loop unit.

3. The ship distance estimation method based on AIS surveillance signals according to claim 1, characterized in that, The object detection model is an improved single-stage object detection model. The improvement of the single-stage object detection model is achieved by introducing an Effective SE channel attention mechanism and a Halo local self-attention mechanism to optimize the model parameters.

4. The ship distance estimation method based on AIS surveillance signals according to claim 1, characterized in that, The distance regression loss function of the distance regression module is constructed based on the mean squared error; wherein, the expression of the distance regression loss function is: ； In the formula, Estimate the distance to the ship predicted by the distance regression module. The physical distance to the ship is calculated based on AIS data.

5. The ship distance estimation method based on AIS surveillance signals according to claim 4, characterized in that, The distance regression module also includes a dynamic calibration mechanism; wherein, the dynamic calibration mechanism uses a sliding window to smooth the estimated ship distances of consecutive frames, and updates the parameters of the target detection model online in combination with the latest matching results.

6. The ship distance estimation method based on AIS surveillance signals according to claim 1, characterized in that, The distance regression module further includes: Freeze the parameters of the distance regression module to obtain a distance regression module with frozen weights; Acquire video data of vessels without AIS, and perform target detection on the video data to obtain the detection bounding box of vessels without AIS; Input the detection box of the vessel without AIS into the distance regression module with frozen weights to obtain the distance estimate of the vessel without AIS.

7. The ship distance estimation method based on AIS surveillance signals according to claim 1, characterized in that, The preprocessing steps include: Cleaning, outlier removal, and trajectory interpolation and correction of AIS data; Perform image enhancement and annotation on video data.

8. A ship distance estimation device based on AIS surveillance signals according to claim 1, characterized in that, include: The data acquisition and preprocessing module is used to simultaneously acquire and preprocess waterway monitoring video data and ship AIS data; The visual target detection module is used to extract the detection box information of ship targets from video frames based on an improved target detection model; The trajectory prediction module is used by a ship trajectory prediction model based on a hybrid attention mechanism to process historical AIS data and predict the future position of ships. The spatiotemporal matching module is used to divide the camera's field of view into equal and reliable regions and pixel regions, and to perform matching between AIS targets and video detection targets; The multimodal fusion and distance regression module is used to fuse successfully matched AIS distance information with visual features, train and apply the distance regression module to estimate and output the ship distance.

Images