Intelligent inspection method for ship traffic

Abstract

The invention provides an intelligent inspection method for ship traffic, which comprises the following steps: acquiring a visible light image and an infrared image of a target water area to be inspected, and respectively carrying out time and space marking on the visible light image and the infrared image to obtain the marked visible light image and infrared image; fusing the marked visible light image and the infrared image to obtain a fused image; inputting the fused image into a YOLOv10 detection framework for target detection to obtain a target category, a bounding box and a confidence coefficient; screening out a key frame comprising a ship from the fused image according to the output target category; performing time sequence embedding processing on the key frame and the detected AIS signal of the ship, and inputting the processed key frame and AIS signal and a region rule of a region where the target water area to be inspected is located into an expert model to obtain a research and judgment result; and the expert model is obtained by performing fine adjustment on the large language model by utilizing LoRA.

Description

Technical Field

[0001] This invention relates to the field of ship inspection technology, and in particular to an intelligent ship traffic inspection method. Background Technology

[0002] Currently, common methods of water patrol mainly rely on manual duty and on-site inspections by patrol vessels, or fixed-point monitoring by shore-based cameras and recorded by duty personnel. These methods have limited coverage, insufficient mobility, are greatly affected by terrain and hydrological conditions, and require a large number of people for long-term patrols and recording. They are labor-intensive, inefficient, lack real-time performance, have high labor costs, and are easily affected by subjective factors.

[0003] To improve coverage, shore-based radar, Automatic Identification System (AIS) receivers, and fixed video surveillance have been deployed in some waters. However, AIS receivers have insufficient coverage for small fishing boats and targets without or with AIS not activated. Shore-based radar and fixed video surveillance suffer from blind spots and decreased accuracy under obstructed conditions, with small targets at long distances, and complex weather conditions. Furthermore, they can only provide limited target information and are insufficient to accurately characterize target behavior and intent, resulting in significant false alarms and missed alarms.

[0004] With the development of drone technology, drones can be used for motorized inspections of key waterways at a lower cost. Current solutions mostly involve drones taking pictures / videos of waterways and shorelines, which are then manually observed and interpreted frame by frame by regulatory personnel. This method requires processing massive amounts of images, is time-consuming, inefficient, lacks real-time performance, and is prone to missed or misjudgments, making it difficult to meet the needs for rapid detection and handling of maritime traffic safety incidents.

[0005] While current AI-assisted inspection technologies have made some progress in target detection and preliminary classification, most solutions employ lightweight embedded models, capable of only basic target recognition. They struggle to integrate with tracking data, Automatic Identification System (AIS) information, and regional rules for comprehensive analysis, and are even less able to determine high-level semantic intent. For complex scenarios such as illegal fishing, unauthorized berthing, and speeding, traditional algorithms remain limited to simple image pattern matching, lacking a deep understanding of scene context, temporal features, and rules, resulting in insufficient intelligence. Summary of the Invention

[0006] Purpose of the invention: To address the problems of insufficient intelligence, frequent omissions and misjudgments, insufficient coverage, and high reliance on manual labor in existing ship traffic inspection technologies, this invention proposes an intelligent ship traffic inspection method.

[0007] Technical solution: A method for intelligent inspection of ship traffic, comprising the following steps:

[0008] Step 1: Acquire visible light and infrared images of the target water area to be inspected, and perform temporal and spatial annotations on the visible light and infrared images respectively to obtain annotated visible light and infrared images;

[0009] Step 2: Fuse the labeled visible light image with the infrared image to obtain the fused image;

[0010] Step 3: Input the fused image into the YOLOv10 detection framework for target detection to obtain the target category, bounding box, and confidence score. The target category includes ships. Based on the output target category, select keyframes that include ships from the fused image.

[0011] Step 4: Perform time-series embedding processing on the keyframes and the AIS signals of the vessels detected in Step 3. Input the processed keyframes and AIS signals, as well as the regional rules of the area where the target waters to be inspected are located, into the expert model to obtain the judgment results. The judgment results include: event category, risk level, and handling recommendations.

[0012] The expert model is obtained by fine-tuning a large language model using LoRA.

[0013] Furthermore, the process of fusing the labeled visible light image with the infrared image to obtain a fused image specifically includes:

[0014] The labeled visible light image and infrared image are merged to obtain the feature map. , is represented as:

[0015]

[0016] Here, concat represents the merge operation. , These represent the labeled visible light image and infrared image, respectively;

[0017] The labeled visible light and infrared images are processed separately using a multilayer perceptron (MLP) and represented as follows:

[0018]

[0019]

[0020] For feature maps Perform a linear transformation, expressed as:

[0021]

[0022] In the formula, The weight matrix is ​​represented as a linear transformation. Indicates bias;

[0023] The result of the linear transformation The class probability distribution is obtained by transforming the data using the Softmax activation function, as follows:

[0024]

[0025] The fusion operation is performed according to the following formula to obtain the fused image F:

[0026]

[0027]

[0028] .

[0029] In the formula, ⊙ represents element-wise multiplication, and ⊙ represents channel weighting with a broadcast mechanism. This represents tensor addition.

[0030] Furthermore, the YOLOv10 detection framework is obtained by adding an attention module to the backbone network of the existing YOLOv10 detection framework. The attention module is used to perform global semantic mining and local detail enhancement on the input.

[0031] Furthermore, the input to the attention module is the output of the backbone network of the existing YOLOv10 detection framework.

[0032] Furthermore, the attention module includes the following steps:

[0033] Assume the output of the backbone network of the existing YOLOv10 detection framework is represented as follows: ,Will Decomposed along the channel dimension into two sub-feature maps with the same spatial size but half the number of channels. Including the first sub-feature map Second sub-feature map In the formula, H represents the height, W represents the width, and C represents the number of channels;

[0034] Using the first sub-feature map The Q, K, and V values ​​required to construct the attention mechanism are represented as follows:

[0035]

[0036]

[0037]

[0038] SPP stands for Spatial Pyramid Pooling Layer;

[0039] Based on Q, K, and V, scaled dot product attention is calculated to obtain the output of the attention branch, represented as:

[0040]

[0041] in, This is the scaling factor;

[0042] The output of the attention branch is compared with the second sub-feature map. The components are fused together to obtain the output. :

[0043]

[0044]

[0045] Where Conv represents the convolution operation, Y1 represents the intermediate parameters, and CBS represents the composite convolutional module consisting of a convolutional layer, a batch normalization layer, and an activation function.

[0046] Furthermore, the YOLOv10 detection framework is obtained by introducing a fusion module into the neck of the existing YOLOv10 detection framework. The fusion module is used to merge feature maps of different sizes. Based on the merged feature maps, feature maps of different resolutions are obtained. The feature maps of different resolutions are then sent to the detection head of the existing YOLOv10 detection framework for processing.

[0047] Furthermore, the fusion module is used to merge feature maps of different sizes, and the specific operations include:

[0048] Based on resolution from low to high, the multi-layer feature maps of the input fusion module are defined as the first feature map. Second feature map Third feature map ;

[0049] For the first feature map An upsampling operation is performed to obtain the upsampled feature map. ;

[0050] For the third feature map The feature map is obtained by using depthwise separable convolution. ;

[0051] Upsampled feature map Second feature map Feature maps after depthwise separable convolution The fusion process is performed to obtain the merged feature map output by the fusion module.

[0052] Furthermore, the upsampled feature map... Second feature map Feature maps after depthwise separable convolution The fusion process is performed to obtain the merged feature map output by the fusion module, specifically including:

[0053] Upsampled feature map Second feature map Feature maps after depthwise separable convolution Perform a merging operation to obtain a fused feature map. ;

[0054] For fused feature maps Spatial segmentation is performed to obtain 8 sub-feature maps with the same channels, each of which is denoted as . ,in, ;

[0055] For each sub-feature map Perform the same parallel enhancement processing, including:

[0056]

[0057]

[0058]

[0059] Wherein, Conv convolution is a 1×1 convolution. , Indicates the result of an intermediate process. Represents the enhanced sub-feature map ; This indicates the average pooling operation. Represents the Sigmoid function;

[0060] All enhanced sub-feature maps are merged to obtain the merged feature map output by the fusion module.

[0061] Furthermore, the YOLOv10 detection framework also includes: a backbone network, a neck, and a detection head;

[0062] The backbone network includes a first convolutional Conv module, a second convolutional Conv module, a third convolutional Conv module, a first depthwise separable convolutional DsConv module, a second depthwise separable convolutional DsConv module, a first C2f module, a second C2f module, a third C2f module, a first downsampling SCDown module, a second downsampling SCDown module, a first C2fCIB module, and an attention module;

[0063] The fused image is sequentially passed through a first convolutional Conv module, a second convolutional Conv module, and a first depthwise separable convolutional DsConv module to obtain feature map F2;

[0064] The feature map F2 then passes through the first C2f module, the third convolutional Conv module, and the second C2f module in sequence to obtain the feature map F3;

[0065] The feature map F3 is sequentially processed by the first downsampling SCDown module and the third C2f module to obtain the feature map F4;

[0066] The feature map F4 is sequentially processed by the second downsampling SCDown module, the second depthwise separable convolution DsConv module, the first C2fCIB module, and the attention module to obtain the feature map F5;

[0067] The neck includes a first fusion module, a second fusion module, a first C3k2 module, a second C3k2 module, a fourth C2f module, a fifth C2f module, a sixth C2f module, a third downsampling SCDown module, a fourth downsampling SCDown module, a fifth downsampling SCDown module, and a second C2fCIB module;

[0068] The feature maps F5, F4 and F3 are merged by the first fusion module to obtain a merged feature map, and then the merged feature map is passed through the first C3k2 module to obtain feature map P1.

[0069] The feature maps P1, F3, and F2 are merged by the second fusion module to obtain another merged feature map, which is then processed by the second C3k2 module to obtain feature map P2.

[0070] The feature map P2 is merged with the feature map F2 through the upsampling channel. The merging result is input to the fourth C2f module to obtain the feature map D1. The feature map D1 is then sent to the detection head for processing.

[0071] The feature map D1 is merged with the feature map P2 by the third downsampling SCDown module, and then input into the fifth C2f module to obtain the feature map D2, which is then sent into the detection head.

[0072] The feature map D2 is merged with the feature map P1 by the fourth downsampling SCDown module. The merged result is input to the sixth C2f module to obtain the feature map D3, which is then sent to the detection head.

[0073] The feature map D3 is merged with the feature map F4 by the fifth downsampling SCDown module. The merged result is input into the second C2fCIB module to obtain the feature map D4, which is then sent into the detection head.

[0074] In the detection head, target detection is performed based on feature maps D1, D2, D3, and D4, and the bounding box, target category, and confidence score are output.

[0075] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0076] (1) This invention combines the flexible perception capabilities of UAVs with the powerful reasoning capabilities of cloud-based large models to achieve all-weather, all-element, and intelligent inspection of waterways, anchorages, and key waters. It can automatically identify and understand abnormal behavior, provide risk classification and handling strategies, and form a closed-loop system of UAV-cloud-monitoring terminal, which significantly improves the efficiency of discovering, analyzing, and handling water traffic safety incidents.

[0077] (2) The method of the present invention can significantly reduce regulatory costs, improve inspection efficiency and quality, and provide strong protection for waterway safety and water area management. Attached Figure Description

[0078] Figure 1 This is a flowchart of a ship traffic intelligent inspection method proposed in an embodiment of the present invention;

[0079] Figure 2 This is a schematic diagram of the DMA fusion module proposed in an embodiment of the present invention;

[0080] Figure 3 This is a schematic diagram of the attention mechanism proposed in an embodiment of the present invention;

[0081] Figure 4 This is a schematic diagram of the FMR fusion module proposed in an embodiment of the present invention;

[0082] Figure 5 This is a diagram of the improved YOLOv10 detection framework proposed in an embodiment of the present invention;

[0083] Figure 6 This is a schematic diagram illustrating the use of a cloud-based large language model for abnormal behavior recognition and intent analysis, as proposed in an embodiment of the present invention. Detailed Implementation

[0084] The technical solution of this embodiment will now be further described in conjunction with the accompanying drawings and examples.

[0085] This invention proposes an intelligent inspection method for ship traffic, such as... Figure 1 As shown, the main steps include:

[0086] Step 1: The ground monitoring terminal creates an inspection task and sends it to the drone. After the drone completes the self-test of its embedded AI processing module and communication module, it flies along a preset route. The inspection task includes a flight mission and an image acquisition mission.

[0087] Step 2: The drone is equipped with a visible light camera, an infrared imaging unit, a positioning and timing module, an embedded AI processing module, and a communication module.

[0088] Visible light and infrared images of waterways, anchorages and shorelines are acquired simultaneously using visible light cameras and infrared imaging units. Combined with latitude and longitude and timestamps generated by the positioning and timing module, the visible light and infrared images are labeled in time and space respectively, realizing the integrated recording of multimodal images and temporal geographic information. The labeled visible light and infrared images are then input into the embedded AI processing module.

[0089] Step 3: The embedded AI processing module first performs Gaussian filtering on the input labeled visible light and infrared images to suppress noise. Then, the filtered visible light and infrared images are input into the DMA fusion module. In the DMA fusion module, the filtered visible light and infrared images are fused to obtain a fused image. The fused image is then input into the improved YOLOv10 detection framework, which outputs the target category, bounding box, and confidence score. Based on the output target category, keyframes including ships are obtained.

[0090] The purpose of the DMA fusion module used in this embodiment of the invention is to fuse visible light images with infrared images to enhance the detectability of distant, low-light and small-scale targets, and improve the robustness and accuracy of detection.

[0091] Figure 2 The structure of the DMA fusion module is shown, specifically including:

[0092] The filtered and labeled visible light image and infrared image are then merged, i.e. Figure 2 The feature map is obtained by splicing the data in the middle. , is represented as:

[0093]

[0094] Here, concat represents the merge operation. Represents visible light images. Indicates infrared image.

[0095] The labeled visible light and infrared images are processed separately using a multilayer perceptron (MLP) and represented as follows:

[0096]

[0097]

[0098] For feature maps Perform a linear transformation, expressed as:

[0099]

[0100] In the formula, The weight matrix is ​​represented as a linear transformation. Indicates bias;

[0101] The result of the linear transformation The class probability distribution is obtained by transforming the data using the Softmax activation function, as follows:

[0102]

[0103] The fusion operation is performed according to the following formula:

[0104]

[0105]

[0106] .

[0107] In the formula, the symbol " " " indicates element-wise multiplication, and the symbol "⊙" indicates channel weighting with a broadcast mechanism, aligning its size with the feature map to achieve channel-wise feature recalibration. This represents tensor addition, and F represents the fused image.

[0108] The fused image F serves as the input to the improved YOLOv10 detection framework.

[0109] When detecting small objects from the perspective of a drone, traditional algorithms often encounter challenges such as background noise, missing information, and complex multi-object interactions. These challenges affect detection performance. To balance efficiency and accuracy requirements, meet the limitations of edge deployment, and adapt to computing chips, this invention improves the existing YOLOv10 detection framework. The existing YOLOv10 detection framework mainly includes: a backbone network, a neck, and a head. The backbone network is used for feature extraction, the neck is used to fuse feature maps of different scales, and the head is the final detection part, generating bounding boxes, object categories, and confidence scores.

[0110] The improved YOLOv10 detection framework used in this embodiment of the invention is obtained by adding an attention module to the backbone network of the existing YOLOv10 detection framework. This attention module is the PRA (Pixel Aggregation Attention Module), which is used to perform global semantic mining and local detail enhancement on the input feature map.

[0111] like Figure 5 As shown, the backbone network includes a first convolutional Conv module, a second convolutional Conv module, a third convolutional Conv module, a first depthwise separable convolutional DsConv module, a second depthwise separable convolutional DsConv module, a first C2f module, a second C2f module, a third C2f module, a first downsampling SCDown module, a second downsampling SCDown module, a first C2fCIB module, and a PRA attention module.

[0112] The fused image F is sequentially passed through the first convolutional Conv module, the second convolutional Conv module, and the first depthwise separable convolutional DsConv module to obtain the feature map F2.

[0113] Feature map F2 then passes through the first C2f module, the third convolutional Conv module, and the second C2f module in sequence to obtain feature map F3.

[0114] Feature map F3 is passed through the first downsampling SCDown module and the third C2f module in sequence to obtain feature map F4.

[0115] Feature map F4 is sequentially processed by the second downsampling SCDown module, the second depthwise separable convolution DsConv module, the first C2fCIB module, and the PRA attention module to obtain feature map F5. The PRA attention module, as a lightweight attention enhancement module, is located as the last module in the backbone network. It solves the problems of background noise interference and small target feature obscuring in UAV images by performing global semantic mining and local detail enhancement on the feature map output by the previous layer.

[0116] Overall, the backbone network gradually downsamples the input fused image frames, eventually obtaining feature maps F2, F3, F4 and F5 with gradually reduced scale.

[0117] Furthermore, the improved YOLOv10 detection framework used in this embodiment of the invention introduces a fusion module into the neck of the existing YOLOv10 detection framework. This fusion module is used to merge feature maps of different sizes. Based on the merged feature maps, feature maps of different resolutions are obtained and sent to the detection head of the existing YOLOv10 detection framework for processing.

[0118] like Figure 5 As shown, specifically in the neck, there are upsampling channels, downsampling channels, and cascaded fusion branches, including: a first FMR fusion module, a second FMR fusion module, a first C3k2 module, a second C3k2 module, an upsampling channel UpSample, a fourth C2f module, a fifth C2f module, a sixth C2f module, a third downsampling SCDown module, a fourth downsampling SCDown module, a fifth downsampling SCDown module, and a second C2fCIB module.

[0119] First, feature maps F5, F4, and F3 are merged by the first FMR fusion module to obtain a multi-layer feature map. This multi-layer feature map is then processed by the first C3k2 module to obtain feature map P1. Next, feature maps P1, F3, and F2 are merged by the second FMR fusion module to obtain another multi-layer feature map. This multi-layer feature map is then processed by the second C3k2 module to obtain feature map P2. Feature map P2 is then merged with feature map F2 via the upsampling channel UpSample and input into the fourth C2f module to obtain feature map D1. This D1 is used as a low-resolution feature map and sent to the detection head for processing.

[0120] Next, feature map D1 is merged with feature map P2 through the third downsampling SCDown module (Concat), and then input into the fifth C2f module to obtain feature map D2, which is used as a medium-to-low resolution feature map and sent to the detection head.

[0121] Feature map D2 is merged with feature map P1 by the fourth downsampling SCDown module (Concat) and then input into the sixth C2f module to obtain feature map D3, which is used as a medium-resolution feature map and sent to the detection head.

[0122] Finally, feature map D3 is merged with feature map F4 through the fifth downsampling SCDown module (Concat) and input into the second C2fCIB module to obtain feature map D4, which is then sent to the detection head as a high-resolution feature map.

[0123] In this embodiment of the invention, the detection head includes a one-to-many detection head and a one-to-one detection head.

[0124] The detection head employs the YOLOv10 model, which performs object detection based on multi-scale feature maps. By processing feature maps layer by layer from low to high resolution, the detection head can efficiently capture target information of different sizes. Specifically, the detection head generates bounding boxes, target categories, and confidence scores through multiple convolutional layers, and combines this with the previous feature map information to perform final target localization and classification.

[0125] In other words, the improved YOLOv10 detection framework introduces a PRA (Pixel Aggregation Attention Module) and multiple FMR fusion modules into the existing YOLOv10 detection framework. The PRA attention module is used to mine the global semantics and local details of the input fused image frames to suppress background interference, improve the detection performance and feature representation ability of small targets, and achieve the identification of ships and personnel targets. The FMR fusion modules are used to merge multi-layer feature maps to obtain the merged feature map.

[0126] The following is a further explanation of each module in conjunction with the accompanying drawings.

[0127] Figure 3 The structure of the PRA attention module is shown, specifically including:

[0128] The output of the backbone network in the YOLOv10 detection framework is the input to the attention module. Assuming the input feature map to the attention module... To reduce computational complexity, a channel separation operation is first performed on the input feature map, separating the input feature map... Perform a decomposition operation, breaking it down along the channel dimension into two spaces of the same size but with half the number of channels. The sub-feature map, including the first sub-feature map Second sub-feature map H represents height, W represents width, and C represents the number of channels; represented as:

[0129]

[0130] Subsequently, the first sub-feature map was used. The Q, K, and V values ​​required to construct the attention mechanism are represented as follows:

[0131]

[0132]

[0133]

[0134] SPP stands for Spatial Pyramid Pooling Layer.

[0135] Based on Q, K, and V, scaled dot product attention is calculated to obtain the output of the attention branch, represented as:

[0136]

[0137] in, This is the scaling factor.

[0138] The output of the attention branch is compared with the second sub-feature map. The components are fused together to obtain the output. :

[0139]

[0140]

[0141] Where Conv represents the convolution operation, Y1 represents the intermediate parameters, and CBS represents the composite convolutional module consisting of a convolutional layer, a batch normalization layer, and an activation function.

[0142] Figure 4 The structure of the FMR fusion module is shown. In this embodiment of the invention, an FMR fusion module is introduced into the neck to merge multi-layer feature maps. The FMR fusion module can effectively integrate low-resolution features with strong semantic information and high-resolution features with detailed information. For high-resolution features, upsampling is used to improve resolution and retain details such as edges and textures of small targets. For low-resolution features, depthwise separable convolution is used for lightweight processing to extract high-level semantic information. After fusing features at three different scales, the fused feature map is spatially segmented, and each sub-feature map is enhanced in parallel to perform global semantic guidance and local detail enhancement. Finally, the sub-feature maps are fused. This preserves the details of small targets and enhances semantic discriminative power.

[0143] like Figure 4 As shown, it specifically includes:

[0144] For each FMR fusion module, define feature maps at different levels of input. ,in:

[0145] It is a low-resolution feature map that contains detailed information, such as the edges and textures of small objects.

[0146] It is a medium-resolution feature map that contains general semantic information.

[0147] It is a high-resolution feature map that contains high-level semantic information.

[0148] In this embodiment of the invention, the first FMR fusion module processes feature map F5, feature map F4, and feature map F3, which respectively correspond to , and The second FMR fusion module processes feature map P1, feature map F3, and feature map F2, which can also correspond to... , and .

[0149] For low-resolution feature maps An upsampling operation is performed to improve the resolution of the feature map. The upsampled feature map is as follows: , is represented as:

[0150]

[0151] For high-resolution feature maps Lightweight processing using depthwise separable convolution (DSConv) is employed to extract high-level semantic information. The depthwise separable convolution operation is as follows:

[0152]

[0153] in, and These are the feature maps after upsampling and after depthwise separable convolution, respectively.

[0154] Upsampled feature map Medium resolution feature map Feature maps after depthwise separable convolution To integrate. Specifically, this includes:

[0155] First, perform a feature merging (Concat) operation:

[0156]

[0157] In this way, a fused feature map containing features at different scales is obtained. .

[0158] Next Spatial segmentation is performed, decomposing it into 8 sub-feature maps of the same size and with the same number of channels. Each sub-feature map is denoted as... ,in, .

[0159] For each sub-feature map Perform the same parallel enhancement processing, including:

[0160]

[0161]

[0162]

[0163] Wherein, Conv convolution is a 1×1 convolution. , Indicates the result of an intermediate process. Represents the enhanced sub-feature map ; This indicates the average pooling operation. This represents the Sigmoid function.

[0164] Finally, all enhanced sub-feature maps are merged to form the final highly expressive feature output, which is the merged feature map output by the fusion module, represented as:

[0165]

[0166] In this way, the improved YOLOv10 detection framework retains the details of small targets while enhancing the final fused feature map with semantic discriminative power.

[0167] Step 4: Send the target category, bounding box, and keyframes including the ship to the cloud and ground monitoring terminals via the communication module.

[0168] Step 5: Connect the cloud to the Automatic Identification System (AIS) deployed in the target waters to obtain AIS signals from the AIS system, which include information such as the ship's speed and heading.

[0169] Step 6: Perform time-series embedding processing on the keyframes of the vessel and the detected AIS signals. Input the processed keyframes and AIS signals, along with the regional rules of the target waterway to be inspected, into the cloud-based large language model to achieve abnormal behavior identification and intent analysis, obtaining the analysis results. These results include event category, risk level, and handling recommendations, and are then transmitted back to the ground monitoring terminal. Event categories include, but are not limited to, speeding, illegal mooring, illegal fishing, entering prohibited navigation areas, and suspected pollution emissions.

[0170] The cloud-based large language model used in this step was obtained according to the following steps:

[0171] Time series data is embedded in the prompts of the cloud-based large language model. Then, LoRA is used to fine-tune the cloud-based large language model. The model is trained first and then inferred to achieve efficient adjustment of model parameters. After embedding and fine-tuning the time series data, an expert model specifically for the river basin environment is formed. This expert model can accurately understand and predict the river basin in real time under different environmental changes, and provide more accurate decision support.

[0172] In the embodiments of the present invention, when fine-tuning, existing data is used to adjust the large language model, so that the cloud-based large language model can complete the identification of abnormal behavior and the judgment of intent, and thus obtain the judgment result.

[0173] The time series data embedded in the embodiments of the present invention will now be further explained.

[0174] By using temporal embedding technology, time-series data such as keyframes of ships and AIS signals are embedded and processed to extract time-related feature representations:

[0175]

[0176] in, This indicates a text prompt word with timing information, which includes an AIS signal. This indicates a keyframe sequence containing the ship.

[0177] Based on the temporally correlated feature representation, combined with the regional rules of the area where the target water area to be inspected is located, the cloud-based big language model can integrate key frame sequences and AIS signals through a multimodal reasoning framework to predict violations.

[0178] By predicting violations, the cloud-based big data model assigns a risk level to each vessel's behavior and outputs specific handling suggestions. Based on the reasoning results of the cloud-based big data model, it can generate appropriate risk levels (low, medium, and high) and corresponding response strategies according to the vessel's behavior patterns (such as illegal mooring and speeding).

[0179] All detection results and reasoning processes are recorded and fed back to the cloud for incremental training and optimization of the cloud-based large language model. This ensures that the cloud-based large language model can adapt to changes in ship behavior under different scenarios and continuously improve its adaptability and accuracy in complex scenarios.

[0180] This invention utilizes text prompts with temporal information and keyframes containing ship information, combined with regional rules governing the area where the target waterway to be inspected is located, to perform cross-modal feature fusion using the Qwen2.5VL model, enabling intelligent detection and prediction of violations such as ship overtaking, illegal U-turns, and low-speed navigation. Figure 6 As shown, the text prompts with time-series information not only include the domain definition (maritime supervision and ship violation detection) and task introduction, but also the time-series information described in text, specifically the ship's speed sequence and position coordinate information, as well as keyframes containing the ship.

[0181] Step 7: The ground monitoring terminal receives images from the drone and the analysis results from the cloud-based large language model. These are then visualized on the front-end interface, and tiered audio-visual alarms are triggered based on the risk level. Simultaneously, based on the handling recommendations, the bytetrack algorithm is used to issue linkage commands to control the drone to perform follow-up shooting, switch infrared, lock recording, or trigger audio-visual alarms, forming a closed-loop architecture of lightweight edge detection + deep inference in the cloud.

[0182] Compared to traditional inspections that rely on manual monitoring or a single video stream, this invention emphasizes the combination of multimodal perception and cross-frame temporal processing. It also introduces AIS signals and regional rules of the area where the target water body is located to participate in reasoning, thereby improving the robustness of identifying continuous and concealed abnormal behaviors. The UAV end uses lightweight detection and feature extraction to ensure real-time performance, while the cloud uses a cloud-based large language model to perform high-level semantics, intent, and risk assessment, in order to achieve a collaborative mechanism of "fast detection on the edge + deep reasoning in the cloud".

Claims

1. A method for intelligent inspection of ship traffic, characterized by: The method comprises the following steps: Step 1: obtaining visible light images and infrared images of a target water area to be inspected, and respectively performing time and space labeling on the visible light images and the infrared images to obtain labeled visible light images and infrared images; Step 2: fusing the labeled visible light images and infrared images to obtain fused images; Step 3: inputting the fused images into a YOLOv10 detection framework for target detection to obtain a target category, a bounding box and a confidence, wherein the target category comprises a ship; and filtering key frames comprising the ship from the fused images according to the output target category; Step 4: performing time sequence embedding processing on the key frames and AIS signals of the ship detected in step 3, inputting the processed key frames and AIS signals and regional rules of a region where the target water area to be inspected is located into an expert model to obtain a judgment result; The judgment result comprises an event category, a risk level and a disposal suggestion. The expert model is obtained by fine-tuning a large language model using LoRA.

2. The intelligent inspection method of ship traffic according to claim 1, characterized in that: The fusing of the labeled visible light images and infrared images to obtain the fused images comprises: The labeled visible light image and the infrared image are combined to obtain a feature map is represented as: ； wherein concat denotes a merge operation, , respectively denote the labeled visible light image and the infrared image. respectively processing the labeled visible light images and infrared images through a multi-layer perception (MLP) network, represented as: ； ； To the feature map A linear transformation is made, denoted as: ； wherein the weight matrix representing the linear transformation, denotes a bias; The result of the linear transformation The class probability distribution is obtained by transforming the data using the Softmax activation function, as follows: ； performing fusion operation according to the following formula to obtain the fused images F: ； ； ； wherein denotes element-wise multiplication, and denotes channel-wise weighting with broadcasting mechanism, denotes tensor addition.

3. The intelligent inspection method of ship traffic according to claim 1, characterized in that: The YOLOv10 detection framework is obtained by adding an attention module to a backbone network of an existing YOLOv10 detection framework, and the attention module is used for global semantic mining and local detail enhancement of input.

4. The intelligent inspection method of ship traffic according to claim 3, characterized in that: The input of the attention module is the output of the backbone network of the existing YOLOv10 detection framework.

5. The intelligent inspection method of ship traffic according to claim 4, characterized in that: In the attention module, the following steps are included: Assuming that the output of the backbone network of the existing YOLOv10 detection framework is represented as , which is decomposed into two sub-feature maps with the same spatial size but half the number of channels along the channel dimension , including a first sub-feature map and a second sub-feature map ; in the formula, H represents height, W represents width, and C represents the number of channels.​ Utilize the first sub-feature map Q, K, V required for constructing the attention mechanism are represented as: ； ； ； wherein SPP is a spatial pyramid pooling layer; based on Q, K and V, performing scaled dot-product attention calculation to obtain the output of the attention branch, represented as: ； wherein is a scaling factor; output of the attention branch is fused with the second sub-feature map fusion is performed to obtain an output : ； ； wherein Conv represents a convolution operation, Y1 represents an intermediate parameter, and CBS represents a composite convolution module composed of a convolution layer, a batch normalization layer and an activation function.

6. The intelligent inspection method of ship traffic according to claim 5, characterized in that: The YOLOv10 detection framework is obtained by introducing a fusion module into a neck of an existing YOLOv10 detection framework, the fusion module is used for merging feature maps of different sizes, and based on the merged feature maps, feature maps of different resolutions are obtained, and the feature maps of different resolutions are sent into a detection head of the existing YOLOv10 detection framework for processing.

7. The intelligent inspection method of ship traffic according to claim 6, characterized in that: The fusion module is used for merging feature maps of different sizes, and the specific operation comprises: According to the resolution from low to high, the multi-layer feature maps of the input fusion module are defined as a first feature map , a second feature map , and a third feature map ; For the first feature map , an up-sampling operation is performed to obtain an up-sampled feature map ; For the third feature map , a depth separable convolution is used for processing to obtain a feature map after the depth separable convolution ; The up-sampled feature map , the second feature map and the feature map after the depth separable convolution are fused to obtain a merged feature map output by the fusion module.

8. The intelligent inspection method of ship traffic according to claim 7, characterized in that: The up-sampled feature map , the second feature map , and the feature map after the depth separable convolution are fused to obtain a merged feature map output by a fusion module, specifically including: The up-sampled feature map , the second feature map and the feature map after the depth separable convolution are subjected to a merging operation to obtain a fused feature map ; To the fusion feature map Spatial segmentation is performed to obtain 8 sub-feature maps with the same channel, and each of the sub-feature maps is denoted as Wherein, ; The same parallel enhancement process is performed for each sub-feature map including: ； ； ； wherein, Conv convolution is 1x1 convolution, , denotes an intermediate process result, denotes an enhanced sub-feature map ; denotes an average pooling operation, denotes a Sigmoid function; merging all enhanced sub-feature maps to obtain a merged feature map output by the fusion module.

9. The intelligent inspection method of ship traffic according to claim 8, characterized in that: The YOLOv10 detection framework further comprises a backbone network, a neck and a detection head. In the backbone network, a first convolutional Conv module, a second convolutional Conv module, a third convolutional Conv module, a first depth separable convolutional DsConv module, a second depth separable convolutional DsConv module, a first C2f module, a second C2f module, a third C2f module, a first subsampling SCDown module, a second subsampling SCDown module, a first C2fCIB module, and an attention module are included. The fused image sequentially passes through the first convolutional Conv module, the second convolutional Conv module, and the first depth separable convolutional DsConv module to obtain a feature map F2. The feature map F2 sequentially passes through the first C2f module, the third convolutional Conv module, and the second C2f module to obtain a feature map F3. The feature map F3 sequentially passes through the first subsampling SCDown module and the third C2f module to obtain a feature map F4. The feature map F4 sequentially passes through the second subsampling SCDown module, the second depth separable convolutional DsConv module, the first C2fCIB module, and the attention module to obtain a feature map F5. In the neck, a first fusion module, a second fusion module, a first C3k2 module, a second C3k2 module, a fourth C2f module, a fifth C2f module, a sixth C2f module, a third subsampling SCDown module, a fourth subsampling SCDown module, a fifth subsampling SCDown module, and a second C2fCIB module are included. The feature map F5, the feature map F4, and the feature map F3 are merged through the first fusion module to obtain a merged feature map, and then the merged feature map is passed through the first C3k2 module to obtain a feature map P1. The feature map P1, the feature map F3, and the feature map F2 are merged through the second fusion module to obtain another merged feature map, and the another merged feature map is passed through the second C3k2 module to obtain a feature map P2. The feature map P2 is merged with the feature map F2 through an upsampling channel, and the merging result is input to the fourth C2f module to obtain a feature map D1, and the feature map D1 is sent to a detection head for processing. The feature map D1 is merged with the feature map P2 through the third subsampling SCDown module, and then input to the fifth C2f module to obtain a feature map D2, and the feature map D2 is sent to the detection head. The feature map D2 is merged with the feature map P1 through the fourth subsampling SCDown module, and the merging result is input to the sixth C2f module to obtain a feature map D3, and the feature map D3 is sent to the detection head. The feature map D3 is merged with the feature map F4 through the fifth subsampling SCDown module, and the merging result is input to the second C2fCIB module to obtain a feature map D4, and the feature map D4 is sent to the detection head. In the detection head, target detection is performed based on the feature map D1, the feature map D2, the feature map D3, and the feature map D4, and a bounding box, a target class, and a confidence are output.

Images