A method, system, device and medium for automatic identification of mooring behavior

Abstract

The application relates to a mooring behavior automatic identification method, system, device and medium, which comprises the following steps: acquiring complete data of ship lock passing; using a neural network algorithm to simulate learning of historical illegal behavior data of unmooring or unmooring in place, and outputting a weighted average score and a weighted proportion of image definition in different environments; using a YOLO8 algorithm to detect a ship in a ship picture and real-time video acquired, and identify a crew position; based on ship basic information and the identified crew position, using a CNN model to identify mooring behavior of the crew; extracting the weighted average score and the weighted proportion to calculate a definition influence factor, constructing a SIFT algorithm deformation formula based on the definition influence factor, calculating the similarity between the identified mooring behavior of the crew and the unmooring or unmooring in place behavior obtained through simulation learning, and identifying the unmooring or unmooring in place behavior. Compared with the prior art, the application has the advantages of high identification precision.

Description

Technical Field

[0001] This invention relates to the field of ship lock mooring identification technology, and in particular to an automatic identification method, system, device and medium for mooring behavior. Background Technology

[0002] Mooring is a crucial step in ship passage through locks. After a ship enters the lock chamber through the gate, the chamber adjusts the water level by controlling pumps or sluice gates to match the ship's draft with the water level outside the lock chamber, ensuring the ship's safety and the smooth flow of traffic. Before adjusting the water level in the lock chamber, the crew entering the lock must perform mooring operations to secure the ship firmly, ensuring its stability while berthed and preventing movement or collisions caused by external forces such as water flow and wind. If a ship is not moored or is improperly moored while berthed, it may collide with the dock, dry dock, or other berthed vessels and deviate from its berth position when affected by external forces, causing significant losses and potentially endangering the lives of the crew, while also exacerbating traffic congestion. Currently, most locks nationwide rely on manual judgment of mooring behavior through monitoring screens, which is inefficient, costly, and prone to errors, failing to completely eliminate the risks associated with mooring. Current research on mooring behavior recognition mainly focuses on equipping specialized mooring recognition devices and conducting video unit analysis. For example, patent CN111967409A discloses a ship lock mooring image recognition device and recognition method. It rarely uses the current mainstream target tracking technology and machine learning algorithms. There is a lack of effective research on solving the problem of ships not being moored or not being moored properly. The recognition accuracy needs to be improved. At the same time, the alarm for the behavior of not being moored or not being moored properly still adopts the traditional sound and light alarm, which is simple and inefficient. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an automatic mooring behavior identification method, system, device, and medium. Based on neural network machine learning methods and feature extraction algorithms, the system automatically learns from a large number of data samples of unmoored or improperly moored behaviors in the database, then extracts the behavioral characteristics of personnel, improving the accuracy of unmoored or improperly moored behavior identification. Simultaneously, AR panoramic technology is used to integrate ship information with video surveillance, supporting the display of ship information and annotation of unmoored or improperly moored behaviors on the real-time monitoring video screen, and capturing images and videos for preservation as evidence for punishment. For unmoored or improperly moored behaviors, the system, based on actual conditions, supports alarms via voice, text, and SMS, expanding the alarm reception range.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] According to a first aspect of the present invention, an automatic method for identifying mooring behavior is provided, the method comprising the following steps:

[0006] Obtain complete data on vessel passage through the lock, including basic vessel information, vessel images, real-time video, and images of historical violations;

[0007] Considering the impact of the environment on image clarity, a neural network algorithm is used to simulate and learn from historical violation data of unmoored or inadequate mooring, and outputs the weighted average score and weighting ratio of image clarity under different environments.

[0008] The YOLOv8 algorithm is used to detect targets in the acquired ship images and real-time videos and identify the positions of the crew.

[0009] Based on basic ship information and the identified crew positions, a CNN model is used to identify crew mooring behavior.

[0010] The weighted average score and weighted ratio obtained from the simulation learning are extracted to calculate the sharpness impact factor. Based on the sharpness impact factor, a modified formula of the SIFT algorithm is constructed. The SIFT algorithm modified formula is used to calculate the similarity between the identified mooring behavior of the crew and the unmoored or inadequate mooring behavior obtained from the simulation learning, thereby identifying the unmoored or inadequate mooring behavior.

[0011] As a preferred technical solution, the different environments include daytime (excluding rainy / foggy days), nighttime, foggy days, and rainy days.

[0012] As a preferred technical solution, the method for calculating the weighted average score and weighting ratio of image sharpness under different environments is as follows:

[0013] The image sharpness was calculated separately for daytime (non-rainy / foggy), nighttime, foggy, and rainy days, using the following method:

[0014]

[0015] Where f(x,y) represents the gray value of the pixel (x,y) corresponding to image f, and D(f) is the image sharpness;

[0016] The image sharpness of the images taken during the day, night, foggy, and rainy days were weighted and averaged to obtain the weighted average scores W(f1), W(f2), W(f3), and W(f4) for each environment.

[0017] Calculate the weighted proportion based on the weighted average score:

[0018] U(f i )=W(f i ) / W(f1), i=1,2,3,4

[0019] Where U(f) i The weighted ratio is calculated based on daytime sharpness.

[0020] As a preferred technical solution, the YOLOv8 algorithm consists of three parts: a backbone feature extraction layer, a feature fusion module, and a detection head. The backbone feature extraction layer transmits real-time video, images, basic ship information, and identified crew position data to the feature fusion module. The feature fusion module then transmits the fused data to the detection head. The backbone feature extraction layer incorporates the SCConv module, which includes a Spatial Reconstruction Unit (SRU) and a Channel Reconstruction Unit (CRU). The SRU utilizes the spatial redundancy of features and, through a separation operation, separates feature maps with rich spatial content from corresponding feature maps with less information. It uses the scaling factor in the group normalization layer to evaluate the information content of different feature maps and combines two different information features through a cross-reconstruction operation. The CRU utilizes the channel redundancy characteristic and adopts a segmentation-transformation-fusion strategy, introducing a compression ratio to control the feature channels.

[0021] As a preferred technical solution, the YOLOv8 algorithm performs the following steps:

[0022] S301, Determine an intermediate feature map X∈R N×C×H×W Where N is the batch processing axis, C is the channel axis, and H and W are the spatial height and width axes, respectively. The input feature X is standardized using a group normalization layer, as shown below:

[0023]

[0024] Where μ and σ are the mean and standard deviation of X, ℇ is a positive number added to stabilize the division, and γ and β are trainable affine transformations;

[0025] S302, Introducing trainable parameters The variance of spatial pixels for each batch and channel is measured by the normalized correlation weight W. γ It is obtained through the following equation:

[0026]

[0027] S303, through the sigmoid function, will be converted from W γ The weights of the reweighted feature maps are mapped to the range (0,1) and gated using a threshold. Weights above the threshold are set to 1 to obtain information-rich weights W1, while weights below the threshold are set to 0 to obtain non-information-rich weights W2. This process yields W... i The process is represented by the following equation:

[0028]

[0029] S304, multiply the input feature X by W1 and W2 respectively to obtain two weighted features: information-rich features. and features with less information The cross-reconstruction operation fully combines the features of two different information richness levels with weighted values. and And enhance the information flow between them, and transform the features after cross-reconstruction and Connect them to obtain a spatially refined feature map The process is represented as:

[0030]

[0031] in This indicates element-wise multiplication. This indicates element-wise addition. Indicates splicing;

[0032] S306, adopts This represents a k\timesk convolution kernel. A standard convolution is defined as... , X , Y ∈R c*k*k Representing the input and output features after convolution, standard convolution is replaced by Channel Reconstruction Units (CRUs), which include segmentation, transformation, and fusion operations:

[0033] Segmentation: For a given spatial refinement feature ,Will The channels are divided into two parts, one part with αC channels and the other part with (1-α)C channels, where 0≤a≤1, representing the segmentation ratio. The feature map channels are compressed using 1×1 convolution, and a compression ratio r is introduced to control the number of CRU feature channels to balance computational cost. Spatial refinement of the features is achieved after segmentation and compression operations. Divided into upper part and the lower part ;

[0034] Conversion: The input is fed into the upconversion stage, where it is processed using efficient convolutional operations GWC and PWC, in the same... Perform k×k GWC and 1x1 PWC operations, and sum the outputs to form a merged representative feature map. The upconversion stage is represented as:

[0035]

[0036] in, , These are the learnable weight matrices of GWC and PWC. and These are the input and output feature maps for the upconversion stage, respectively.

[0037] The input is fed into the down-transformation stage, where a 1x1 PWC operation is applied to generate a feature map with shallow hidden details, and the features are reused. To obtain more feature maps without incurring additional costs, the generated and reused features are concatenated to form the output of the bottom stage. :

[0038]

[0039] in, It is the learnable weight matrix of PwC. It's a connection operation. and These are the input and output feature maps for the downconversion stage, respectively.

[0040] Fusion: After the transformation, a simplified SKNet method is used to adaptively merge the output features from the up-conversion and down-conversion stages. and First, global average pooling is used to collect global spatial information to obtain channel statistics. :

[0041]

[0042] The global channel descriptors S1 and S2 from the upconversion and downconversion stages are stacked together, and a channel attention operation is used to generate a feature importance vector. As shown below:

[0043]

[0044] In the feature importance vector Under guidance, features are merged in a channel-based manner. and Obtain channel refinement features Y As shown below:

[0045] .

[0046] As a preferred technical solution, the SIFT algorithm deformation formula calculates the distance between feature vectors in two-dimensional space using Euclidean distance, and then adds a sharpness influence factor to calculate the similarity:

[0047] V=sqrt((X1-X2) 2 +(Y1-Y2)2 )-p(f i )

[0048] Where X1 and X2 are the x-coordinates of two random vectors, Y1 and Y2 are the y-coordinates of two random vectors, and p(f i ) represents the clarity factor, V represents the similarity, and sqrt represents the square root.

[0049] As a preferred technical solution, for daytime images taken on non-rainy or foggy days, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f1)];

[0050] For nighttime images, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f2)] / W(f2)*U(f2);

[0051] For images taken on foggy days, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f3)] / W(f3)*U(f3);

[0052] For rainy day images, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f4)] / W(f4)*U(f4);

[0053] Wherein, D(f) i W(f) represents image sharpness. i U(f) is the weighted average score corresponding to environment i. i ) represents the weighted ratio corresponding to environment i, where i=1 represents daytime without rain or fog, i=2 represents nighttime, i=3 represents foggy days, and i=4 represents rainy days.

[0054] According to a second aspect of the present invention, an automatic mooring behavior recognition system is provided for implementing the method, the system comprising:

[0055] The neural network algorithm module obtains complete data on vessel passage from the lock control center database, including basic vessel information, vessel images, real-time video, and historical violation images. Considering the impact of the environment on image clarity, it uses a neural network algorithm to simulate and learn from historical data on unmoored or improperly moored behaviors, and outputs the weighted average score and weighting ratio of image clarity under different environments. The simulated violations, along with the weighted average score and weighting ratio, are sent to the data storage module for storage.

[0056] The target detection algorithm module obtains basic information, images, and real-time videos of ships passing through the lock from the database of the lock control center. It uses the YOLOv8 algorithm to perform target detection on the ships in the obtained images and real-time videos, identifies the positions of the crew members, and sends the identified position data and basic ship information to the data storage module for storage.

[0057] The deep learning module obtains basic ship information and identified crew positions from the data storage module, uses a CNN model to identify crew mooring behavior, and extracts the weighted average score and weighting ratio obtained from simulation learning from the data storage module to calculate the sharpness impact factor. Based on the sharpness impact factor, a modified SIFT algorithm formula is constructed. The modified SIFT algorithm formula is used to calculate the similarity between the identified crew mooring behavior and the unmoored or inadequate mooring behavior obtained from simulation learning, identify the unmoored or inadequate mooring behavior, and send the identification results to the data storage module for storage.

[0058] The AR panoramic module receives basic ship information and the identified unmoored or improperly moored behavior from the data storage module when the deep learning module detects such behavior. It then displays the location of the offending crew member and issues voice, text, and SMS alerts.

[0059] The data storage module is used to store the processing results of the neural network algorithm module, the object detection algorithm module, and the deep learning module.

[0060] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0061] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) This invention uses a neural network learning algorithm to learn about the behavior of not mooring or improper mooring, and to gain a deeper understanding of the characteristics of violations. Even violations not found in the database can be easily identified. When detecting violations, a neural network algorithm is used for simulation learning instead of directly comparing the identified crew behavior with the data in the lock control center database. First, it is difficult for crew behavior characteristics to be completely consistent, which limits the accuracy of comparison. Second, after learning by the algorithm, it can identify violations that have not been detected before, accurately grasp the identification loopholes, improve the efficiency of lock passage, and protect the safety of ships passing through the lock.

[0064] (2) This invention uses mainstream deep learning models and SIFT algorithms for feature extraction and similarity comparison, and performs behavioral analysis, feature extraction and similarity comparison on crew members. The accuracy and efficiency have been greatly improved compared with the original level.

[0065] (3) The system of the present invention can be well integrated with the existing lock chamber video monitoring system, realize the reuse of the system, and greatly reduce the construction cost. Further improvements are made to the existing video monitoring system in the lock chamber, avoiding the waste of the original system, reducing the construction cost of the new system, and realizing the upgrade of ship mooring behavior recognition technology. It has the two major characteristics of low cost and high efficiency.

[0066] (4) This invention breaks the one-way early warning mechanism and adds SMS early warning. For the behavior of not mooring or mooring inadequate, it not only alerts the regulatory authorities but also notifies the ship owner at the same time, so as to achieve information symmetry between the two parties. It not only reminds the ship owner to make timely rectifications but also facilitates the regulatory authorities to carry out subsequent punishments. Attached Figure Description

[0067] Figure 1 This is a flowchart of the method of the present invention;

[0068] Figure 2 This is a schematic diagram of the YOLOv8 model structure;

[0069] Figure 3 This is a schematic diagram of the SCConv model structure;

[0070] Figure 4 This is a schematic diagram of the SRU model structure;

[0071] Figure 5 This is a schematic diagram of the CRU model structure;

[0072] Figure 6 This is a schematic diagram of the system structure of the present invention;

[0073] Figure 7 This is a comparison chart of recognition results in one embodiment. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0075] Example 1

[0076] This embodiment provides an automatic method for identifying mooring behavior, such as... Figure 1 As shown, the method includes the following steps:

[0077] S1 acquires complete data on vessel passage through the lock, including basic vessel information, vessel images, real-time video, and images of historical violations.

[0078] S2. Considering the impact of the environment on image clarity, a neural network algorithm is used to simulate and learn from historical violation data of unmoored or improperly moored vehicles, and outputs the weighted average score and weighting ratio of image clarity under different environments.

[0079] Since the clarity of camera captures is related to the lighting conditions during the day (excluding rain and fog), night, foggy days, and rainy days, poor lighting conditions leading to decreased image clarity can affect subsequent similarity calculations, thus causing bias in the judgment of cable-carrying behavior. Therefore, this embodiment will incorporate this factor into the similarity calculation. The different environments considered include daytime (excluding rain and fog), nighttime, foggy days, and rainy days.

[0080] The weighted average score and weighting ratio of image sharpness under different environments are calculated as follows:

[0081] The image sharpness was calculated separately for daytime (non-rainy / foggy), nighttime, foggy, and rainy days, using the following method:

[0082]

[0083] Where f(x,y) represents the gray value of the pixel (x,y) corresponding to image f, and D(f) is the image sharpness;

[0084] The image sharpness of the images taken during the day, night, foggy, and rainy days were weighted and averaged to obtain the weighted average scores W(f1), W(f2), W(f3), and W(f4) for each environment.

[0085] Calculate the weighted proportion based on the weighted average score:

[0086] U(f i )=W(f i ) / W(f1), i=1,2,3,4

[0087] Where U(f) i The weighted ratio is calculated based on daytime sharpness.

[0088] S3 uses the YOLOv8 algorithm to perform target detection on ships in the acquired ship images and real-time videos, and identifies the positions of crew members.

[0089] like Figure 2As shown, the YOLOv8 algorithm consists of three parts: a backbone feature extraction layer, a neck feature fusion module, and a head detection head. The backbone feature extraction layer transmits real-time video, images, basic ship information, and identified crew positions to the neck feature fusion module. The neck feature fusion module then fuses the data and transmits it to the head detection head. For example, ... Figure 3 As shown, the backbone feature extraction layer integrates the SCConv module, which includes a spatial reconstruction unit (SRU) and a channel reconstruction unit (CRU). PreviousConvblock represents the previous convolutional block, Input Feature represents the input feature, Spatial-Refined Feature represents the spatial refinement feature, Channel-Refined Feature represents the channel refinement feature, and NextConvblock represents the next convolutional block.

[0090] like Figure 4 As shown, SRU utilizes the spatial redundancy of features. Through a separation operation, it separates feature maps with rich spatial content from corresponding feature maps with less information. It uses the scaling factor in the Group Normalization (GN) layer to evaluate the information content of different feature maps. Through a cross-reconstruction operation, it combines two different information features with weights to enhance the information flow between them and reduce storage costs. Here, Spatial Reconstruction Unit represents the spatial reconstruction unit, InputFeature represents the input feature, Spatial-Refined Feature represents the spatial refinement feature, GroupNormalization represents group normalization, Sigmoid represents the logistic function, Threshold represents the threshold, Multiplication represents multiplication, Summation represents summation, Concatenation represents merging, Separate represents separation, and Reconstruct represents reconstruction.

[0091] like Figure 5 As shown, CRU utilizes channel redundancy and employs a segmentation-transformation-fusion strategy, introducing compression ratio control for feature channels to reduce computational costs. Here, Channel Reconstruction Unit represents the channel reconstruction unit, Spatial-Refined Feature represents the spatial refinement feature, Split represents splitting, Transform represents transformation, Fuse represents fusion, Pooling represents pooling, Soft Max represents the maximum soft value, Element-wise summation represents element-wise summation, Concatenation represents merging, and Element-wise multiplication represents element-wise multiplication.

[0092] Specifically, the YOLOv8 algorithm performs the following steps:

[0093] S301, Determine an intermediate feature map X∈R N×C×H×W Where N is the batch processing axis, C is the channel axis, and H and W are the spatial height and width axes, respectively. The input feature X is standardized using a group normalization layer, as shown below:

[0094]

[0095] Where μ and σ are the mean and standard deviation of X, ℇ is a positive number added to stabilize division, and γ and β are trainable affine transformations.

[0096] S302, Introducing trainable parameters The variance of spatial pixels for each batch and channel is measured by the normalized correlation weight W. γ It is obtained through the following equation:

[0097]

[0098] S303, through the sigmoid function, will be converted from W γ The weights of the reweighted feature maps are mapped to the range (0,1) and gated using a threshold. Weights above the threshold are set to 1 to obtain information-rich weights W1, while weights below the threshold are set to 0 to obtain non-information-rich weights W2. This process yields W... i The process is represented by the following equation:

[0099]

[0100] S304, multiply the input feature X by W1 and W2 respectively to obtain two weighted features: information-rich features. and features with less information In this way, we have successfully divided the input features into two parts: It possesses spatial content that is rich in information and highly expressive, and There is almost no information, and it is considered redundant.

[0101] The cross-reconstruction operation fully combines the features of two different information richness levels in a weighted manner. and And enhance the information flow between them, and transform the features after cross-reconstruction and Connect them to obtain a spatially refined feature map The process is represented as:

[0102]

[0103] in This indicates element-wise multiplication. This indicates element-wise addition. This represents concatenation. Applying SRU to the intermediate input feature X separates information-rich features from less information-rich features, and reconstructs them to enhance representative features and suppress redundant features in the spatial dimension. However, this spatial refinement of the feature map... There is still redundancy in the channel dimension.

[0104] S306, to utilize the channel redundancy of features, introduces a Channel Reconstruction Unit (CRU), which employs a "segmentation-transformation-fusion" strategy. Typically, repeated standard k×k convolutions are used to extract features, resulting in some relatively redundant feature maps along the channel dimension. This embodiment adopts... This represents a k\timesk convolution kernel. A standard convolution is defined as... , X , Y ∈R c*k*k This represents the input and output features after convolution, replacing the standard convolution with a Channel Reconstruction Unit (CRU), which is implemented through three operators: Split, Transform, and Fuse.

[0105] Segmentation: For a given spatial refinement feature ,Will The channels are divided into two parts, one with αC channels and the other with (1-α)C channels, where 0≤a≤1, representing the segmentation ratio. 1×1 convolution is used to compress the channels of the feature map to improve computational efficiency. A compression ratio r is introduced to control the number of feature channels in the CRU to balance computational cost (typically r is set to 2 in experiments). Spatial refinement of the features is performed after segmentation and compression operations. Divided into upper part and the lower part .

[0106] Conversion: The input is fed into the upconversion stage, acting as a "rich feature extractor." Efficient convolutional operations GWC and PWC are used instead of standard k×k convolutions to extract high-level representative information while reducing computational cost. Due to sparse convolutional connections, GWC reduces parameters and computational cost but disrupts information flow between channel groups. PWC, on the other hand, compensates for this information loss and facilitates information flow between feature channels. Therefore, in the same... Perform k×k GWC and 1x1 PWC operations, and sum the outputs to form a merged representative feature map. The upconversion stage is represented as:

[0107]

[0108] in, , These are the learnable weight matrices of GWC and PWC. and These are the input and output feature maps for the upconversion stage, respectively. The upconversion stage utilizes GWC and PWC on the same feature maps. The above combinations are used to extract rich, representative features with less computational cost. .

[0109] The input is fed into the down-transformation stage, where a 1x1 PWC operation is applied to generate a feature map with shallow hidden details, which complements the rich feature extractor and reuses the features. To obtain more feature maps without incurring additional costs, the generated and reused features are concatenated to form the output of the bottom stage. :

[0110]

[0111] in, It is the learnable weight matrix of PwC. It's a connection operation. and These are the input and output feature maps for the downconversion stage, respectively.

[0112] Fusion: After the transformation, a simplified SKNet method is used to adaptively merge the output features from the up-conversion and down-conversion stages. and First, global average pooling is used to collect global spatial information to obtain channel statistics. :

[0113]

[0114] The global channel descriptors S1 and S2 from the upconversion and downconversion stages are stacked together, and a channel attention operation is used to generate a feature importance vector. As shown below:

[0115]

[0116] In the feature importance vector Under guidance, features are merged in a channel-based manner. and Obtain channel refinement features Y As shown below:

[0117]

[0118] S4 uses a CNN model to identify crew mooring behavior based on basic ship information and the identified crew positions.

[0119] This part uses an existing model, and will not be described in detail here.

[0120] S5. Extract the weighted average score and weighted ratio obtained from the simulation learning to calculate the sharpness impact factor. Based on the sharpness impact factor, construct the SIFT algorithm transformation formula. Use the SIFT algorithm transformation formula to calculate the similarity between the identified mooring behavior of the crew and the unmoored or inadequate mooring behavior obtained from the simulation learning, and identify the unmoored or inadequate mooring behavior.

[0121] The modified SIFT algorithm calculates the distance between feature vectors in two-dimensional space using Euclidean distance and adds a sharpness factor to calculate similarity.

[0122] V=sqrt((X1-X2) 2 +(Y1-Y2) 2 )-p(f i )

[0123] Where X1 and X2 are the x-coordinates of two random vectors, Y1 and Y2 are the y-coordinates of two random vectors, and p(f i ) represents the clarity factor, V represents the similarity, and sqrt represents the square root.

[0124] Among them, the clearer the image, the greater the clarity factor.

[0125] For daytime images taken on non-rainy or foggy days, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f1)];

[0126] For nighttime images, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f2)] / W(f2)*U(f2);

[0127] For images taken on foggy days, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f3)] / W(f3)*U(f3);

[0128] For rainy day images, the sharpness impact factor is calculated as follows: p(f i )=[D(f i)-W(f4)] / W(f4)*U(f4);

[0129] Wherein, D(f) i W(f) represents image sharpness. i U(f) is the weighted average score corresponding to environment i. i ) represents the weighted ratio corresponding to environment i, where i=1 represents daytime without rain or fog, i=2 represents nighttime, i=3 represents foggy days, and i=4 represents rainy days.

[0130] The smaller the calculated similarity value V, the more similar the two targets are. In this embodiment, a similarity V < 0.75 is defined as a relatively high target similarity, while V ≥ 0.75 indicates that the targets are not very similar. The specific value can be adjusted according to the actual situation. Therefore, when V < 0.75, the identified crew mooring behavior is similar to the simulated unmoored or improperly moored behavior, and it is considered that the crew has engaged in unmoored or improperly moored behavior.

[0131] Example 2

[0132] The above is an introduction to the method embodiments. The following system embodiments will further illustrate the solution of the present invention.

[0133] like Figure 6 As shown, the system includes:

[0134] The neural network algorithm module obtains complete data on vessel passage from the lock control center database, including basic vessel information, vessel images, real-time video, and historical violation images. Considering the impact of the environment on image clarity, it uses a neural network algorithm to simulate and learn from historical data on unmoored or improperly moored behaviors, and outputs the weighted average score and weighting ratio of image clarity under different environments. The simulated violations, along with the weighted average score and weighting ratio, are sent to the data storage module for storage.

[0135] The target detection algorithm module obtains basic information, images, and real-time videos of ships passing through the lock from the database of the lock control center. It uses the YOLOv8 algorithm to perform target detection on the ships in the obtained images and real-time videos, identifies the positions of the crew members, and sends the identified position data and basic ship information to the data storage module for storage.

[0136] The deep learning module obtains basic ship information and identified crew positions from the data storage module, uses a CNN model to identify crew mooring behavior, and extracts the weighted average score and weighting ratio obtained from simulation learning from the data storage module to calculate the sharpness impact factor. Based on the sharpness impact factor, a modified SIFT algorithm formula is constructed. The modified SIFT algorithm formula is used to calculate the similarity between the identified crew mooring behavior and the unmoored or inadequate mooring behavior obtained from simulation learning, identify the unmoored or inadequate mooring behavior, and send the identification results to the data storage module for storage.

[0137] The AR panoramic module receives basic ship information and the identified unmoored or improperly moored behavior from the data storage module when the deep learning module detects such behavior. It then displays the location of the offending crew member and issues voice, text, and SMS alerts.

[0138] The data storage module is used to store the processing results of the neural network algorithm module, the object detection algorithm module, and the deep learning module.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0140] Example 3

[0141] This embodiment conducts simulation experiments on the method of Embodiment 1, and the performance comparison results of the present invention and YOLOv8l-pose are shown in Table 1.

[0142] Table 1 Comparison of Indicator Performance

[0143]

[0144] Where P represents precision, Precision = TP / (TP+FP), which is the proportion of correctly predicted results among all results where the prediction is positive; R represents recall, recall = TP / (TP+FN), which is the proportion of correctly predicted results among all results where the true value is positive; Box represents a detection box; pose represents a keypoint; mAP represents Mean Average Precision; mAP50 represents the average precision at an IOU threshold greater than 0.5; mAP50-95 represents the average mAP at different IOU thresholds (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

[0145] Image recognition results of the two methods are as follows Figure 7As shown, (a1)-(a9) are the recognition results of YOLOv8l-pose, and (b1)-(b9) are the recognition results of the present invention. It can be seen that the recognition accuracy of the present invention is higher and the recognition results are more accurate.

[0146] Example 4

[0147] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0148] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0149] The processing unit executes the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 by any other suitable means (e.g., by means of firmware).

[0150] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0151] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0153] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An automatic method for identifying mooring behavior, characterized in that, The method includes the following steps: Obtain complete data on vessel passage through the lock, including basic vessel information, vessel images, real-time video, and images of historical violations; Considering the impact of the environment on image clarity, a neural network algorithm is used to simulate and learn from historical violation data of unmoored or improperly moored lines, and outputs the weighted average score and weighting ratio of image clarity under different environments; the different environments include daytime, nighttime, foggy days, and rainy days (excluding rainy and foggy days). The YOLOv8 algorithm is used to detect targets in the acquired ship images and real-time videos and identify the positions of the crew. Based on basic ship information and the identified crew positions, a CNN model is used to identify crew mooring behavior. The weighted average score and weighted ratio obtained from simulation learning are extracted to calculate the sharpness impact factor. Based on the sharpness impact factor, a modified SIFT algorithm formula is constructed. This modified SIFT algorithm formula is used to calculate the similarity between the identified mooring behavior of crew members and the unmoored or improperly moored behavior obtained from simulation learning, thus identifying unmoored or improperly moored behaviors. Specifically, the modified SIFT algorithm formula calculates the distance between feature vectors in two-dimensional space using Euclidean distance and superimposes the sharpness impact factor to calculate the similarity. V=sqrt((X1-X2) 2 +(Y1-Y2) 2 )-p(f i ) Where X1 and X2 are the x-coordinates of two random vectors, Y1 and Y2 are the y-coordinates of two random vectors, and p(f i ) represents the clarity impact factor, V represents the similarity, and sqrt represents the square root. For daytime images taken on non-rainy or foggy days, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f1)]; For nighttime images, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f2)] / W(f2)*U(f2); For images taken on foggy days, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f3)] / W(f3)*U(f3); For rainy day images, the sharpness impact factor is calculated as follows: p(f i )=[D(f i )-W(f4)] / W(f4)*U(f4); Wherein, D(f) i W(f) represents image sharpness. i U(f) is the weighted average score corresponding to environment i. i ) represents the weighted ratio corresponding to environment i, where i=1 represents daytime without rain or fog, i=2 represents nighttime, i=3 represents foggy days, and i=4 represents rainy days.

2. The automatic identification method for mooring behavior according to claim 1, characterized in that, The calculation methods for the weighted average score and weighting ratio of image sharpness under different environments are as follows: The image sharpness was calculated separately for daytime (non-rainy / foggy), nighttime, foggy, and rainy days, using the following method: Where f(x,y) represents the gray value of the pixel (x,y) corresponding to image f, and D(f) is the image sharpness; The image sharpness of the images taken during the day, night, foggy, and rainy days were weighted and averaged to obtain the weighted average scores W(f1), W(f2), W(f3), and W(f4) for each environment. Calculate the weighted proportion based on the weighted average score: U(f i )=W(f i ) / W(f1),i=1,2,3,4 Where U(f) i The weighted ratio is calculated based on daytime sharpness.

3. The automatic identification method for mooring behavior according to claim 1, characterized in that, The YOLOv8 algorithm consists of three parts: a backbone feature extraction layer, a feature fusion module, and a detection head. The backbone feature extraction layer transmits real-time video, images, basic ship information, and identified crew position data to the feature fusion module. The feature fusion module then transmits the fused data to the detection head. The backbone feature extraction layer incorporates the SCConv module, which includes a Spatial Reconstruction Unit (SRU) and a Channel Reconstruction Unit (CRU). The SRU utilizes the spatial redundancy of features and separates feature maps with rich spatial content from their corresponding less information-rich feature maps through a separation operation. It uses a scaling factor in the group normalization layer to evaluate the information content of different feature maps and combines two different information features through a cross-reconstruction operation. The CRU utilizes the channel redundancy characteristic and employs a segmentation-transformation-fusion measurement, introducing a compression ratio to control the feature channels.

4. The automatic identification method for mooring behavior according to claim 3, characterized in that, The YOLOv8 algorithm described above performs the following steps: S301, Determine an intermediate feature map X∈R N×C×H×W Where N is the batch processing axis, C is the channel axis, and H and W are the spatial height and width axes, respectively. The input feature X is standardized using a group normalization layer, as shown below: Where μ and σ are the mean and standard deviation of X, ℇ is a positive number added to stabilize the division, and γ and β are trainable affine transformations; S302, Introducing trainable parameters The variance of spatial pixels for each batch and channel is measured by the normalized correlation weight W. γ It is obtained through the following equation: S303, through the sigmoid function, will be converted from W γ The weights of the reweighted feature maps are mapped to the range (0,1) and gated using a threshold. Weights above the threshold are set to 1 to obtain information-rich weights W1, while weights below the threshold are set to 0 to obtain non-information-rich weights W2. This process yields W... i The process is represented by the following equation: S304, multiply the input feature X by W1 and W2 respectively to obtain two weighted features: information-rich features. and features with less information The cross-reconstruction operation fully combines the features of two different information richness levels with weighted values. and And enhance the information flow between them, and transform the features after cross-reconstruction and Connect them to obtain a spatially refined feature map The process is represented as: in This indicates element-wise multiplication. This indicates element-wise addition. Indicates splicing; S306, adopts This represents a k\timesk convolution kernel. A standard convolution is defined as... , X , Y ∈R C*k*k Representing the input and output features after convolution, standard convolution is replaced by Channel Reconstruction Units (CRUs), which include segmentation, transformation, and fusion operations: Segmentation: For a given spatial refinement feature ,Will The channels are divided into two parts, one part with αC channels and the other part with (1-α)C channels, where 0≤α≤1, representing the segmentation ratio. The feature map channels are compressed using 1×1 convolution, and a compression ratio r is introduced to control the number of CRU feature channels to balance computational cost. Spatial feature refinement is achieved after segmentation and compression operations. Divided into upper part and the lower part ; Conversion: The input is fed into the upconversion stage, where it is processed using efficient convolutional operations GWC and PWC, in the same... Perform k×k GWC and 1x1 PWC operations, and sum the outputs to form a merged representative feature map. The upconversion stage is represented as: in, , These are the learnable weight matrices of GWC and PWC. and These are the input and output feature maps for the upconversion stage, respectively. The input is fed into the down-transformation stage, where a 1x1 PWC operation is applied to generate a feature map with shallow hidden details, and the features are reused. To obtain more feature maps without incurring additional costs, the generated and reused features are concatenated to form the output of the bottom stage. : in, It is the learnable weight matrix of PwC. It's a connection operation. and These are the input and output feature maps for the downconversion stage, respectively. Fusion: After the transformation, a simplified SKNet method is used to adaptively merge the output features from the up-conversion and down-conversion stages. and First, global average pooling is used to collect global spatial information to obtain channel statistics. : The global channel descriptors S1 and S2 from the upconversion and downconversion stages are stacked together, and a channel attention operation is used to generate a feature importance vector. As shown below: In the feature importance vector Under guidance, features are merged in a channel-based manner. and Obtain channel refinement features Y As shown below: 。 5. An automatic mooring behavior recognition system, characterized in that, For implementing the method as described in any one of claims 1 to 4, the system comprises: The neural network algorithm module obtains complete data on vessel passage from the lock control center database, including basic vessel information, vessel images, real-time video, and historical violation images. Considering the impact of the environment on image clarity, it uses a neural network algorithm to simulate and learn from historical data on unmoored or improperly moored behaviors, and outputs the weighted average score and weighting ratio of image clarity under different environments. The simulated violations, along with the weighted average score and weighting ratio, are sent to the data storage module for storage. The target detection algorithm module obtains basic information, images, and real-time videos of ships passing through the lock from the database of the lock control center. It uses the YOLOv8 algorithm to perform target detection on the ships in the obtained images and real-time videos, identifies the positions of the crew members, and sends the identified position data and basic ship information to the data storage module for storage. The deep learning module obtains basic ship information and identified crew positions from the data storage module, uses a CNN model to identify crew mooring behavior, and extracts the weighted average score and weighting ratio obtained from simulation learning from the data storage module to calculate the sharpness impact factor. Based on the sharpness impact factor, a modified SIFT algorithm formula is constructed. The modified SIFT algorithm formula is used to calculate the similarity between the identified crew mooring behavior and the unmoored or inadequate mooring behavior obtained from simulation learning, identify the unmoored or inadequate mooring behavior, and send the identification results to the data storage module for storage. The AR panoramic module receives basic ship information and the identified unmoored or improperly moored behavior from the data storage module when the deep learning module detects such behavior. It then displays the location of the offending crew member and issues voice, text, and SMS alerts. The data storage module is used to store the processing results of the neural network algorithm module, the object detection algorithm module, and the deep learning module.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images