Inland waterway ship detection method and system based on AIS-video, electronic equipment and readable storage medium

Abstract

The invention provides an AIS-video-based inland waterway ship detection method and system, and the system comprises the steps: collecting the ship identity information of a ship lock entrance region through an AIS information receiving device on the basis that a ship uses a ship reservation system to submit the ship lock passing date, the ship driving destination, the ship model and other information for reservation lock passing in advance; the identified ship identity information is matched with the ship lockage reservation information, and a lockage gate is planned for the ship in real time, so that the problem of low ship lockage reservation speed is solved, and the accuracy of inland ship lockage detection is improved; through pre-identification and secondary identification processes, the problem of inaccurate identification of ship lockage information is solved, pre-identification uses an AIS technology to identify ship information in an entrance area, the problem of untimely acquisition of the ship information is solved, secondary identification uses a video detection technology to identify the ship information in an approach channel, and through further matching of the ship identity information, the ship lockage information can be accurately identified. The problem that the ship AIS information is tampered is solved, and the accuracy of inland ship entry detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to an inland waterway ship detection method, system, electronic equipment and readable storage medium based on AIS-video. Background Art

[0002] Inland waterway target detection technology can improve shipping safety and lock operation efficiency when ships enter the lock, and plays an indispensable role in promoting the transformation of the entire inland waterway shipping system towards modernization and intelligence.

[0003] In recent years, identity detection technology has been widely used in vehicles and ships. The Automatic Identification System (AIS) is widely used in ship identification technology. AIS can provide static and dynamic information of ships, including ship name, ZMO number, ship length, ship width, ship type and electronic positioning. AIS can be used directly to identify the ship's identity, but the captain can arbitrarily tamper with the ship information stored in AIS. Therefore, the single AIS technology still has the problem that the ship information security is not guaranteed and the ship's safety through the lock is not guaranteed. At present, various video detection technologies are used in target detection of vehicles and ships, including two detection methods based on anchor frames and anchor-free frames. The detection algorithm based on anchor frames is divided into one-stage target detection and two-stage target detection. The one-stage target detection algorithm, such as the yolo series and the SSD series, applies a single neural network to the entire image. The network divides the image into multiple regions and predicts the bounding box and probability of each region at the same time; the two-stage target detection algorithm is mainly the RCNN series, which first extracts a series of target candidate frames, then uses the convolutional neural network to extract features, and finally uses the classifier to identify and locate the target in each region. Anchor-free target detection algorithm The anchor-free algorithm does not require a preset anchor frame, but determines the target position by connecting points. However, due to the complexity of the inland waterway environment, including light changes, obstructions, and the diversity of ship types, the single video detection technology still has the problem of low accuracy and difficulty in predicting complex scenes or situations with many obstructions. In addition, it is rare to apply a single AIS technology or AIS-video dual detection technology to ship entry detection. Therefore, the use of AIS-video dual detection technology is of great significance to improving the accuracy of inland ship lock entry detection and improving ship safety. Summary of the invention

[0004] In order to solve the technical problems raised in the above background technology, a multi-scale ship detection method and system for inland waterways based on AIS-video is provided to improve the accuracy of the inland ship entry detection process and ensure the accuracy of the ship entry identity.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] An inland waterway ship detection method based on AIS-video, comprising the following steps:

[0007] S1, using a shore-based AIS base station to receive AIS information sent by a ship to pass through the lock, and performing pre-identification of the ship's identity;

[0008] S2, matching the ship identity information sent by the received AIS base station with the ship information for passing through the lock reserved in the lock reservation system. If the information matches successfully, the lock reservation system schedules the ship's passing through the lock, enabling the ship to enter the approach channel of the lock to wait for passing through the lock. Otherwise, the ship's passing through the lock request is rejected;

[0009] S3, obtaining the ship image in the approach channel and identifying the ship identity information in the image;

[0010] S4, matching the identified ship identity information with the ship identity information sent by the AIS base station for secondary identity identification. If the information matches successfully, the ship starts to pass through the lock. Otherwise, the ship's passing through the lock request is rejected.

[0011] Further, in step S3, the YOLOv7 algorithm is used to identify the ship identity information in the image.

[0012] Further, step S3 includes the following steps: First, in the original Backbone of YOLOv7, ShuffleNetv2 is used as the backbone network of YOLOv7, and the ShuffleNetv2 backbone network is used for feature extraction. During the feature extraction process, a lightweight self-attention mechanism CA module is added to enhance the spatial coordinate information of the feature map, obtaining a feature map with attention weights; the extracted feature map with attention weights undergoes Neck module feature fusion processing to obtain three sizes of features: large, medium, and small. Based on the extraction of the three sizes of features, the PAFPN structure is used to gradually fuse the features of adjacent layers through a top-down path, a bottom-up path, and horizontal connections to obtain the final feature map; finally, the detection head of YOLOv7 compares each cell on the final feature map with a predefined anchor box, and optimizes the position of the prediction box through a loss function, thereby predicting the position and category of the ship, and finally obtaining the ship identity information.

[0013] Further, the ship identity information includes ship identification, ship length, ship width, and ship type information.

[0014] The present invention also provides an inland waterway ship detection system based on AIS-video, comprising:

[0015] AIS Information Transmission and Processing Module: It is used to receive the ship identity information of the ship to pass through the lock currently through the shore-based AIS base station and perform pre-identification of the ship identity.

[0016] Video Information Processing Module: It is used to obtain the ship video images of the ships to pass through the lock in the approach channel and identify the ship identity information in the video images.

[0017] Information Matching Module: It is used to match the ship identity information sent by the received AIS base station with the ship lock reservation system's reserved ship information for passing through the lock. If the information matches successfully, the ship lock reservation system is used to schedule the ship lock gate, so that the ship to pass through the lock enters the approach channel of the lock gate to wait for passing through the lock. Otherwise, the ship lock passing request is rejected; and to match the identified ship identity information with the ship identity information sent by the AIS base station. If the information matches successfully, the ship passing through the lock starts, otherwise the ship lock passing request is rejected.

[0018] Further, the Information Matching Module includes an AIS Information and Reservation Information Matching Sub-module and an AIS Information and Video Information Matching Sub-module. The AIS Information and Reservation Information Matching Sub-module is used to match the ship identity information sent by the received AIS base station with the ship lock reservation system's reserved ship information for passing through the lock. If the information matches successfully, the ship lock reservation system is used to schedule the ship lock gate, so that the ship to pass through the lock enters the approach channel of the lock gate to wait for passing through the lock. Otherwise, the ship lock passing request is rejected; the AIS Information and Video Information Matching Sub-module is used to match the identified ship identity information with the ship identity information sent by the AIS base station. If the information matches successfully, the ship passing through the lock starts, otherwise the ship lock passing request is rejected.

[0019] Further, the Video Information Processing Module adopts the YOLOv7 algorithm framework, integrates the ShuffleNetv2 network architecture in the backbone network part of the YOLOv7 algorithm framework, integrates the CA module in the ELAN module of the YOLOv7's Backbone part, and integrates the PAFPN structure in the Neck part of the YOLOv7.

[0020] Furthermore, the information matching module uses the representational text matching CDSSM method for information matching. First, one-hot is used as the input to map the input words into a vector space. The convolutional neural network is used to capture the local patterns in the text, identify the text semantics, and pool the features. Then, the features are mapped into a low-dimensional and compact vector. In the matching layer, the cosine similarity is used to measure the semantic similarity R(Q, D) between the query Query and the document Doc. The semantic similarity between the Query and the positive sample of the document Doc is transformed into a posterior probability distribution P(D|Q) through the softmax function to reflect the similarity between the query and all possible documents, so as to query the most similar document. According to the posterior probability, the document with the highest probability is selected as the matching result. If the highest probability value is lower than 0.8, it is considered that there is no matching document.

[0021] The present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the above methods are implemented.

[0022] The present invention further provides a readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of any one of the above methods are executed.

[0023] Due to the above technical solutions, the present invention has the following beneficial effects:

[0024] Based on the AIS-video inland waterway ship detection system and method of the present invention, on the basis that ships use the ship reservation system in advance to submit information such as the ship lockage date, the ship's travel destination, and the ship model for lockage reservation, the AIS information receiving device is used to collect the ship identity information in the ship lock gate area, match the identified ship identity information with the ship lockage reservation information, and plan the lock gate for the ship in real time, effectively solving the problem of slow ship lockage reservation speed, improving the ship lockage efficiency, and enhancing the accuracy of inland ship entry lock detection; through the pre-identification and secondary identification processes, the problem of inaccurate identification of ship lockage information is solved. The pre-identification uses AIS technology to identify ship information in the lock gate area, solving the problem of untimely acquisition of ship information. The secondary identification uses video detection technology to identify ship information in the approach channel. By further matching the ship identity information, the problem of tampering with ship AIS information is solved, and the accuracy of inland ship entry lock detection is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of the AIS-video-based inland waterway ship detection method according to a preferred embodiment of the present invention;

[0026] Figure 2 It is a module diagram of an inland waterway ship detection system based on AIS-video in a preferred embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of the working state of the ship AIS in a preferred embodiment of the present invention;

[0028] Figure 4 It is a schematic diagram of the working state of the video information processing module in an inland waterway ship detection system based on AIS-video in a preferred embodiment of the present invention. Specific embodiments

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0031] Please refer to Figures 1 to 4 together. In view of the problems of slow speed of ship passing through the lock reservation and mismatch between ship reservation information and ship information, the present invention provides an inland waterway ship detection system based on AIS-video, including an AIS information transmission and processing module, a video information processing module and an information matching module.

[0032] The AIS information transmission and processing module of this embodiment is used to receive the AIS information sent by the AIS of the ship to pass through the lock by using the shore-based AIS base station for pre-identification of the ship identity. The AIS system receives the AIS information sent by the ship AIS by accessing instruments such as an electronic chart system, a gyrocompass, a log, a VHF transceiver, etc., and constructing an AIS communication model based on the Open System Interconnection Reference Model, thereby forming an AIS information transmission and processing module, and then obtaining information such as ship identification, ship length, ship width, ship type, cargo carried, ship position and ship destination, realizing the efficient upload and processing of ship AIS information, and providing comprehensive ship dynamic and static information.

[0033] The Automatic Identification System (AIS) for ships usually consists of on-board equipment and shore-based facilities. The on-board equipment includes an AIS transmitter, receiver, transceiver, and information processor, etc. The shore-based facilities include an AIS base station and an AIS center. The base station is used to receive ship information and transmit it to the AIS center, which is responsible for processing and distributing the information. The AIS base station automatically broadcasts and receives dynamic information and static information sent by the on-board equipment on the ship through the VHF radio frequency. The dynamic information includes position, speed, course, etc., and the static information includes the ship's call sign, IMO number, identification number, etc. At the same time, the AIS system obtains accurate global positioning data through a GNSS receiver, including the longitude, latitude, altitude of the ship and possibly time information. The gyrocompass provides the ship's course information, and the log measures the ship's speed and voyage, providing the necessary speed and distance data for the ship's dynamic positioning. These data are integrated into the AIS system through the input interface and processed and coordinated by the communication processor. The AIS system also includes an alarm to monitor and warn of potential navigation risks, a steering gear to automatically adjust the ship's course according to navigation instructions, a simple internal test module for system self-check to ensure the normal operation of the equipment. The operator can interact with the system through the keyboard and other equipment interfaces for manual control and data input.

[0034] When the ship arrives at the designated area, the ship's AIS equipment will automatically broadcast a signal containing the ship's detailed information, including the ship's identification, ship length, ship width, ship type, cargo carried, ship position, and destination information. These signals are transmitted through the VHF radio band, captured by the shore-based receiving base station, and stored in the server. The base station is equipped with a high-performance receiver and antenna system, which can ensure stable signal reception under various weather and sea conditions. The server removes invalid or incorrect information through data cleaning, and then parses and stores this data for easy analysis and retrieval.

[0035] The video information processing module is used to obtain the video images of the ships waiting to pass through the lock in the approach channel and identify the identity information of the ships in the video images.

[0036] In this embodiment, by installing a camera or a video sensor in the approach channel to capture the ship video, after the video information processing module obtains the captured ship video, the YOLOv7 network model is used to identify the identity information of the ship in the video, and information such as the ship number, ship length, ship width, and ship model is obtained. Specifically, the video information processing module adopts the YOLOv7 algorithm framework, and integrates the ShuffleNetv2 network architecture into the backbone network part (Backbone) of the YOLOv7 algorithm framework. The CA module is integrated into the ELAN module in the Backbone part of YOLOv7, and the PAFPN structure is integrated into the Neck part of YOLOv7. During the recognition process, first, the ShuffleNetv2 backbone network is used for feature extraction. During the feature extraction process, the added lightweight self-attention mechanism CA module enhances the feature representation by considering the spatial coordinate information of the feature map, and obtains a feature map with attention weights to extract ship features more accurately and quickly. The extracted feature map with attention weights undergoes feature fusion processing through the Neck module to obtain features of three sizes: large, medium, and small. Based on the extraction of features of these three sizes, the PAFPN structure uses a top-down path, a bottom-up path, and lateral connections to gradually fuse the features of adjacent layers to obtain the final feature map. Finally, the detection head of YOLOv7 compares each cell at each position on the final feature map with a predefined anchor box, and optimizes the position of the prediction box through a loss function, thereby predicting the position and category of the ship, and finally obtaining the identity information of the ship.

[0037] In view of the problem that the data processing volume of ship target detection is large and the detection speed is slow in this embodiment, the Sufflenetv2 network architecture is introduced into the YOLOv7 backbone network. Pointwise grouped convolution is adopted to reduce the number of parameters and the amount of calculation by grouping. The channel shuffle operation is used to rearrange the channels of the feature map after grouped convolution to enhance the information exchange between different groups, and by establishing connections between different feature points in the channels, the model's ability to identify ship features is enhanced. In view of the problem of low video detection accuracy, the CA attention mechanism is added to the YOLOv7 framework. Horizontally, the CA module performs max pooling by using a pooling window with a width equal to the width of the entire feature map and a height of 1. Vertically, a pooling window with a height equal to the height of the entire feature map and a width of 1 is used to capture the spatial direction information. Then, a 1*1 convolution is used for feature map transformation to encode the spatial information, obtaining an attention weight map and performing normalization. The two generated attention weight maps are weighted and fused with the original feature map respectively, and finally the original feature map after attention weighting is output, providing richer and more discriminative features and improving the performance of target detection. The general expression of Attention is:

[0038]

[0039] Where Q, K, and V are the Query vector, Key vector, and Value vector respectively, and softmax(*) represents the normalization function. represents the square root of the dimension of the key vector K.

[0040] The CA module embeds coordinate information by decomposing the global pooling operation into two one-dimensional operations and captures long-range spatial interactions in the input feature map. This method not only preserves spatial location information but also improves the efficiency of the model because it reduces the number of required parameters and computations. Specifically, the CA module first performs average pooling on each channel of the input feature map X using a pooling kernel of size (H,1) along the horizontal direction and a pooling kernel of size (1,W) along the vertical direction. For an input feature map with height h and width w, the output of the c-th channel after this process can be expressed as:

[0041]

[0042] Where represents the average pooling in the vertical direction for the c-th channel; represents the average pooling in the horizontal direction for the c-th channel, and X c (h,i) represents the pixel value of the c-th channel at position (h,i) in the feature map X; X c (j,W) represents the pixel value of the c-th channel at position (j,w) in the feature map X.

[0043] The CA attention mechanism generates an attention feature map g by concatenating the average pooling feature maps in the horizontal and vertical directions, followed by 1×1 convolution for dimensionality reduction and batch normalization, and finally through the Sigmoid activation function:

[0044] g = δ(G1([zh,zw])) (4)

[0045] Where [z h ,z w represents the concatenation operation of z h and z w , G1 represents the 1*1 convolution operation, and δ represents the normalization and non-linearization process.

[0046] The generated attention feature map g is processed separately in the height and width directions through 1×1 convolution operations to obtain two sets of feature maps g h , g w , and through the Sigmoid activation function, the attention weights corresponding to the height and width directions are obtained respectively from these two sets of feature maps:

[0047] f h = σ(G h (g h )) (5)

[0048] f w = σ(G w (g w )) (6)

[0049] Among them, σ is the activation function, G h 、G w represent 1*1 convolution operations, f h is the attention weight in the height direction, and f w is the attention weight in the width direction.

[0050] Finally, the final feature map y with attention weights in the width and height directions is calculated through the attention weight matrix multiplication c :

[0051]

[0052] Among them, i and j are the indexes of height and width, and X c (i, j) is the pixel value of the c-th channel in the feature map X at the position (i, j); is the attention weight in the height direction of the c-th channel, is the attention weight in the width direction of the c-th channel.

[0053] To improve the recognition accuracy of YOLOv7, a loss function is set in YOLO to reduce the detection error. The loss function generally consists of three parts: class loss, object confidence loss, and coordinate regression loss. In this embodiment, a CIOU loss function is established based on IOU, as follows:

[0054] Loss = L giou + L obj + L cls (8)

[0055] In the formula, L giou represents the coordinate loss; L obj represents the object confidence loss; L cls represents the classification loss.

[0056] The coordinate loss is used to measure the spatial distance between the predicted box and the ground truth box. In the embodiment of the present invention, the CLOULoss loss calculation function is used, considering three geometric parameters: the overlapping area, the center point distance, and the aspect ratio of the predicted box and the ground truth box:

[0057]

[0058] Among them, ρ(b, b gt ) is the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c is the diagonal length of the minimum bounding rectangle; w and h are the width and height of the predicted bounding box respectively; w gt and h gt are the width and height of the ground truth bounding box respectively; CIOU represents the improved intersection over union; IOU represents the intersection over union of the predicted bounding box and the ground truth bounding box, A and B represent the areas of the predicted bounding box and the ground truth bounding box respectively; α represents the weight coefficient, and v is the shape consistency term, which is used to measure the difference in aspect ratio between the predicted bounding box and the ground truth bounding box; L CIOU represents the CIOU loss function.

[0059] The object confidence loss L obj is used to optimize the model's confidence judgment on whether the predicted bounding box contains an object, and usually the binary cross-entropy loss is used for calculation:

[0060]

[0061] Among them, N is the number of positive samples; t obji is the true label indicating whether the i-th predicted bounding box contains an object; p obji is the confidence that the model predicts the i-th predicted bounding box contains an object.

[0062] The classification loss L cls calculates the cross-entropy loss between the predicted class probability and the true class, and sums up the classification losses of all predicted bounding boxes to ensure that the model can correctly identify the class to which the object in the image belongs:

[0063]

[0064] Among them, N is the number of positive samples; P i (c) and respectively represent the class probability prediction and the actual class label of the model for the i-th predicted bounding box.

[0065] The information matching module is used to match the ship identity information sent by the AIS base station received with the information of the ships reserved for passing through the lock in the lock reservation system. If the information matching is successful, the lock reservation system is used to schedule the ship passing through the lock, so that the ship to pass through the lock enters the approach channel of the lock gate to wait for passing through the lock. Otherwise, the ship passing through the lock request is rejected; and to match the identified ship identity information with the ship identity information sent by the AIS base station. If the information matching is successful, the ship passing through the lock starts. Otherwise, the ship passing through the lock request is rejected.

[0066] In this embodiment, the information matching module includes an AIS information and reservation information matching sub-module and an AIS information and video information matching sub-module. The AIS information and reservation information matching sub-module is used to match the vessel identity information sent by the AIS base station received with the information of the vessels reserved for passing through the lock in the lock reservation system. If the information matches successfully, the lock reservation system schedules the lock gate for the vessel, enabling the vessel to enter the approach channel of the lock gate and wait for lock passage. Otherwise, the lock passage request of the vessel is rejected. The AIS information and video information matching sub-module is used to match the identified vessel identity information with the vessel identity information sent by the AIS base station. If the information matches successfully, the lock passage of the vessel begins. Otherwise, the lock passage request of the vessel is rejected.

[0067] In this embodiment, the inland waterway vessel detection system based on AIS-video further includes a storage module, which is used to store the information of the vessels reserved for passing through the lock in the lock reservation system and the vessel identity information obtained by the video information processing module from the images captured by the video sensor. Specifically, the information of the vessels reserved for passing through the lock in the lock reservation system is the lock passage reservation information filled in by the vessel in the reservation lock passage APP. The vessel first fills in the vessel basic information and lock passage reservation information in the reservation lock passage APP in advance. The reservation lock passage APP collects this information and uploads it to the storage module of the inland waterway vessel detection system based on AIS-video. When the vessel sails to the designated location, the inland waterway vessel detection system receives the vessel's AIS information. The AIS information and reservation information matching sub-module automatically matches it with the existing vessel basic information and lock passage information in the storage module. If the match is successful, it indicates that the vessel identity is correct, and then the vessel is scheduled. The reservation lock passage APP belongs to the prior art. To save space, it will not be elaborated here. When the vessel is waiting in the approach channel, the video information processing module uploads the vessel identity information obtained from the images captured by the video sensor to the storage module of the inland waterway vessel detection system based on AIS-video. The AIS information and video information matching sub-module matches this information with the vessel's AIS information in the storage module. If the match is successful, it indicates that the vessel identity is correct, and then the lock passage of the vessel begins.

[0068] Since it is necessary to quickly and accurately match ship information, and considering that the information provided by AIS for one-time identification is relatively comprehensive, while the secondary video identification may have limitations due to environmental factors, and there are differences in the information obtained from the two identifications. To solve this problem, the embodiment of the present invention adopts the representational text matching CDSSM (Convolutional Deep Structured Semantic Model) method to perform information matching. This method first uses one-hot as the input, maps the input words to the vector space, captures the local patterns in the text through a convolutional neural network, identifies the text semantics, and pools the features. Then, the features are mapped into a low-dimensional and compact vector. In the matching layer, the cosine similarity is used to measure the semantic similarity R(Q, D) between the query (Query) and the document (Doc):

[0069]

[0070] In the formula, y Q and y D respectively represent the semantic vectors of the query and the document. This method is commonly used for text matching in high-dimensional spaces, only paying attention to the direction of the vector and not considering its magnitude, and can effectively measure the semantic similarity;

[0071] Through the softmax function, the semantic similarity between the Query and the positive sample Doc can be transformed into a posterior probability distribution P(D|Q), which reflects the similarity between the query and all possible documents, and is used to query the most similar document, as shown in formula (17):

[0072]

[0073] In the formula, γ represents the scaling factor.

[0074] In the training stage, the loss function is minimized through maximum likelihood estimation to improve the accuracy and robustness of the model. The minimized loss function L is:

[0075]

[0076] In the formula, D + represents the positive sample document.

[0077] The embodiment of the present invention also provides an inland waterway ship detection method based on AIS-video, including the following steps:

[0078] S1, using the shore-based AIS base station to receive the AIS information sent by the ship to pass through the lock, and performing pre-identification of the ship's identity. Specifically, the ship's identity information includes information such as ship identification, ship length, ship width, and ship type.

[0079] S2. Match the vessel identity information sent by the received AIS base station with the information of the vessels reserved for passing through the lock in the lock reservation system. If the information matches successfully, schedule the lock gate for the vessel through the lock reservation system, so that the vessel to pass through the lock enters the approach channel of the lock gate and waits to pass through the lock. Otherwise, reject the vessel's request to pass through the lock.

[0080] S3. Obtain the images of the vessels waiting to pass through the lock in the approach channel and identify the vessel identity information in the images.

[0081] In this embodiment, in step S3, the YOLOv7 algorithm is used to identify the vessel identity information in the video image. Specifically: First, in the original Backbone of YOLOv7, ShuffleNetv2 is used as the backbone network of YOLOv7, and the ShuffleNetv2 backbone network is used for feature extraction. During the feature extraction process, the lightweight self-attention mechanism CA module is added to enhance the feature representation by considering the spatial coordinate information of the feature map, and a feature map with attention weights is obtained; the extracted feature map with attention weights undergoes feature fusion processing through the Neck module to obtain features of three sizes: large, medium, and small. On the basis of extracting features of the three sizes of large, medium, and small, the PAFPN structure is used to gradually fuse the features of adjacent layers through the top-down path, bottom-up path, and lateral connections to obtain the final feature map; finally, the detection head of YOLOv7 compares each cell on the final feature map with the predefined anchor boxes, and optimizes the position of the prediction box through the loss function, thereby predicting the position and category of the vessel, and finally obtaining the vessel identity information.

[0082] S4. Match the identified vessel identity information with the vessel identity information sent by the AIS base station for secondary identity verification. If the information matches successfully, start the process of the vessel passing through the lock. Otherwise, reject the vessel's request to pass through the lock.

[0083] The method and system for detecting inland waterway vessels based on AIS-video according to the embodiments of the present invention combines AIS information reception and processing with YOLOv7 recognition technology, realizes double identification of vessel information, ensures the legality and safety of vessel lock passage, and improves the lock passage efficiency and speed at the same time. Specifically, the embodiments of the present invention use shore-based AIS base stations to collect vessel information in the lock gate area, including vessel length, vessel width, vessel type, electronic positioning and other information, match the vessel lock passage reservation information, and plan the lock gate for the vessel in real time, solving the problem of slow vessel reservation lock passage speed, reducing the waiting time for vessels to pass through the lock, and increasing the speed of vessels passing through the lock.

[0084] To solve the problem that the AIS information of ships is easily tampered with, resulting in the lack of guarantee for the safety of ships passing through the lock, the embodiment of the present invention introduces secondary recognition on the basis of single-ship AIS information matching. After the ship travels to the approach channel, the YOLOv7 technology is used for secondary ship recognition to avoid the mismatch between the ship reservation information and the ship information, ensure the legal navigation of the ship, and improve the safety of ships passing through the lock. In addition, in the information matching step, the embodiment of the present invention adopts the representative text matching CDSSM method. Through semantic embedding and feature fusion technologies, high-precision matching of corresponding information in the first AIS recognition and the second video recognition is achieved. By converting text information into semantic vectors in the vector space and using cosine similarity to measure the semantic similarity between the query and the document, the CDSSM method can quickly and accurately match ship information. This precise matching mechanism not only improves the efficiency of ships passing through the lock but also enhances the safety of the entire lock-passing process.

[0085] The embodiment of the present invention also improves the YOLOv7 model. By integrating the SuffleNetv2 network architecture and the CA attention mechanism, it not only reduces the computational amount but also enhances the feature recognition ability, effectively distinguishing ships of different sizes and shapes. Specifically, the embodiment of the present invention improves the traditional YOLOv7 and proposes to add the sufflenetv2 network architecture to YOLOv7, which reduces the number of model parameters and the computational amount while enhancing the information exchange between different groups, improving the feature recognition ability, and optimizing the detection speed and efficiency. Secondly, due to the large range of ship size changes, it is considered to add the CA attention mechanism on the basis of the FPN pyramid structure. While using the FPN structure to obtain ships of different sizes, it outputs a feature map with weights to improve the feature recognition ability, and thus better distinguish ships of different sizes and shapes, thereby significantly improving the accuracy of ship target detection.

[0086] The embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method are implemented.

[0087] The memory may include a high-speed RAM memory, or may also include non-volatile memory, such as at least one disk memory. The memory may also be a memory array. The memory may also be partitioned, and the partitions may be combined into virtual volumes according to certain rules. The processor may be a central processing unit CPU, or a GPU, or may be an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiment of the present invention.

[0088] Embodiments of the present invention also provide a readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the method are executed.

[0089] Those skilled in the art should understand that embodiments of the present invention can be provided as a method, an apparatus, or a computer program product. Therefore, the present invention can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified function in one flow Figure 1 or multiple flows and / or one block or multiple blocks in the block diagram.

[0091] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified function in one flow Figure 1 or multiple flows and / or one block or multiple blocks in the block diagram.

[0092] The above description is a detailed description of the preferred and feasible embodiments of the present invention, but the embodiments are not used to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit disclosed by the present invention should fall within the scope of the patent covered by the present invention.

Claims

1. An inland waterway ship detection method based on AIS-video, characterized in that, It includes the following steps: S1. Use the shore-based AIS base station to receive the AIS information sent by the ship to pass through the lock, and perform pre-identification of the ship's identity; S2. Match the ship's identity information sent by the received AIS base station with the information of the ship to pass through the lock reserved in the lock reservation system. If the information matches successfully, the lock reservation system will schedule the ship's lock passage, so that the ship to pass through the lock enters the approach channel of the lock gate and waits to pass through the lock. Otherwise, the ship's lock passage request will be rejected; S3. Obtain the image of the ship waiting to pass through the lock in the approach channel, and identify the identity information of the ship in the image; S4. Match the identified ship's identity information with the ship's identity information sent by the AIS base station for secondary identity identification. If the information matches successfully, the ship will start to pass through the lock. Otherwise, the ship's lock passage request will be rejected.

2. The AIS-video-based inland waterway ship detection method according to claim 1, wherein, In step S3, the YOLOv7 algorithm is used to identify the identity information of the ship in the image.

3. The method for detecting inland waterway vessels based on AIS-video according to claim 2, characterized in that, Step S3 includes the following steps: First, in the original Backbone of YOLOv7, ShuffleNetv2 is used as the backbone network of YOLOv7, and the ShuffleNetv2 backbone network is used for feature extraction. During the feature extraction process, the lightweight self-attention mechanism CA module is added to enhance the spatial coordinate information of the feature map, and a feature map with attention weights is obtained; the extracted feature map with attention weights undergoes feature fusion processing through the Neck module to obtain features of three sizes: large, medium, and small. On the basis of extracting features of these three sizes, the PAFPN structure is used to gradually fuse the features of adjacent layers through the top-down path, bottom-up path, and lateral connections to obtain the final feature map; finally, the detection head of YOLOv7 compares each cell on the final feature map with the predefined anchor boxes, and optimizes the position of the prediction box through the loss function, thereby predicting the position and category of the ship, and finally obtaining the identity information of the ship.

4. The method for detecting inland waterway vessels based on AIS-video according to claim 1, wherein, The ship's identity information includes ship identification, ship length, ship width, and ship type information.

5. An inland waterway ship detection system based on AIS-video, characterized in that, It includes: AIS information transmission and processing module: used to receive the ship's identity information of the current ship to pass through the lock through the shore-based AIS base station and perform pre-identification of the ship's identity; Video information processing module: used to obtain the video image of the ship waiting to pass through the lock in the approach channel and identify the identity information of the ship in the video image; Information matching module: used to match the ship's identity information sent by the received AIS base station with the information of the ship to pass through the lock reserved in the lock reservation system. If the information matches successfully, the lock reservation system will schedule the ship's lock passage, so that the ship to pass through the lock enters the approach channel of the lock gate and waits to pass through the lock. Otherwise, the ship's lock passage request will be rejected; And match the identified ship's identity information with the ship's identity information sent by the AIS base station. If the information matches successfully, the ship will start to pass through the lock. Otherwise, the ship's lock passage request will be rejected.

6. The AIS-video-based inland waterway ship detection system according to claim 5, characterized in that, The information matching module includes an AIS information and reservation information matching sub-module and an AIS information and video information matching sub-module. The AIS information and reservation information matching sub-module is used to match the ship identity information sent by the received AIS base station with the information of the ships reserved for passing through the lock in the lock reservation system. If the information matches successfully, the lock reservation system schedules the ship passing through the lock, so that the ship to pass through the lock enters the approach channel of the lock gate to wait for passing through the lock. Otherwise, the ship passing through the lock request is rejected. The AIS information and video information matching sub-module is used to match the identified ship identity information with the ship identity information sent by the AIS base station. If the information matches successfully, the ship starts to pass through the lock. Otherwise, the ship passing through the lock request is rejected.

7. The AIS-video-based inland waterway ship detection system according to claim 5, wherein The video information processing module adopts the YOLOv7 algorithm framework, integrates the ShuffleNetv2 network architecture in the backbone network part of the YOLOv7 algorithm framework, integrates the CA module in the ELAN module of the Backbone part of YOLOv7, and integrates the PAFPN structure in the Neck part of YOLOv7.

8. The AIS-video-based inland waterway ship detection system according to claim 5, wherein The information matching module uses the representational text matching CDSSM method for information matching. First, it uses one-hot as the input to map the input words into a vector space. It captures the local patterns in the text through a convolutional neural network, recognizes the text semantics, and pools the features. Then, it maps the features into a low-dimensional and compact vector. In the matching layer, it uses cosine similarity to measure the semantic similarity R(Q, D) between the query Query and the document Doc. It converts the semantic similarity between the Query and the positive sample of the document Doc into a posterior probability distribution through the softmax function P ( D |Q), to reflect the similarity between the query and all possible documents, and is used to query the most similar document; according to the posterior probability, it selects the document with the highest probability as the matching result. If the highest probability value is lower than 0.8, it is considered that there is no matching document.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A readable storage medium, characterized in that, A computer program is stored on the readable storage medium. When the computer program is run by the processor, the steps of the method according to any one of claims 1 to 4 are executed.

Images