Mooring behavior automatic identification method, system, equipment and medium

Abstract

The invention relates to a mooring behavior automatic identification method, system and device, and a medium. The method comprises the following steps: obtaining complete data of ship lockage; performing simulation learning on historical violation behavior data of non-mooring or non-in-place mooring by using a neural network algorithm, and outputting a weighted average score and a weighted proportion of image definition in different environments; a YOLO8 algorithm is adopted to carry out target detection on the ship in the obtained ship picture and the real-time video, and the position of a crew is identified; based on the basic information of the ship and the identified position of the sailor, identifying the mooring behavior of the sailor by adopting a CNN (Convolutional Neural Network) model; and extracting a weighted average score and a weighted proportion to calculate a definition influence factor, constructing an SIFT algorithm deformation formula based on the definition influence factor, calculating the similarity between the identified mooring behavior of the sailors and the non-mooring behavior or the non-in-place mooring behavior obtained by simulation learning, and identifying the non-mooring behavior or the non-in-place mooring behavior. Compared with the prior art, the method has the advantages of high recognition precision and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of mooring identification for ships passing through locks, and in particular to a method, system, equipment and medium for automatically identifying mooring behavior. Background Art

[0002] Ship mooring is an important part of ship passing through the lock. After the ship passes through the lock gate and enters the lock chamber, the lock chamber needs to adjust the water level in the lock by controlling the water pump or the sluice gate so that the draft of the ship is adapted to the water level outside the lock chamber to ensure the safety of the ship and the smooth flow of the waterway. Before the lock chamber adjusts the water level, the crew entering the lock must perform the mooring operation to firmly fix the ship to ensure that the ship can remain stable when docked to prevent the ship from moving or colliding due to external forces such as water flow and wind. When the ship is docked, if the mooring is not done or the mooring is improper, when affected by external forces, the ship may collide with the dock, dock or other docked ships and deviate from the docking position, causing huge losses, and is very likely to endanger the lives of the crew and aggravate the congestion of the waterway. At present, most of the locks in the country manually judge whether the ship has moored through monitoring images, which is inefficient, costly, and has a high error rate, and it is impossible to completely eliminate the risk of mooring. Current research on mooring behavior recognition is mainly focused on equipping special mooring recognition devices and performing video unit analysis. For example, patent CN111967409A discloses a mooring image recognition device and recognition method for ships passing through locks. It rarely uses the current mainstream target tracking technology and machine learning algorithms. There is a lack of effective research on solving the problem of ships not mooring or mooring inadequately. The recognition accuracy needs to be improved. At the same time, the traditional sound and light alarm is still used for the alarm of not mooring or mooring inadequately. The alarm method is single and inefficient. Summary of the invention

[0003] The purpose of the present invention is to provide a method, system, device and medium for automatically identifying mooring behavior in order to overcome the defects of the above-mentioned prior art. Based on the machine learning method and feature extraction algorithm of neural network, the system automatically learns a large number of data samples of unmoored or inadequate mooring behaviors in the database, and then extracts the behavioral characteristics of the personnel to enhance the accuracy of identifying unmoored or inadequate mooring behaviors. At the same time, AR panoramic technology is used to integrate ship information with video surveillance, support the display of ship information and the marking of unmoored or inadequate mooring behaviors on the screen of real-time monitoring video, and capture pictures and videos for preservation as punishment evidence. For unmoored or inadequate mooring behaviors, the system supports alarms in three ways: voice, text and text messages based on actual conditions, expanding the alarm receiving range.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] According to a first aspect of the present invention, there is provided a method for automatically identifying mooring behavior, the method comprising the following steps:

[0006] Obtain complete data on ships passing through the locks, including basic ship information, ship pictures, real-time videos and images of historical violations;

[0007] Considering the impact of the environment on image clarity, a neural network algorithm is used to simulate and learn historical violation data of not tying the cable or not tying the cable in place, and output the weighted average score and weighted ratio of image clarity under different environments;

[0008] The YOLO8 algorithm is used to detect the ships in the acquired ship images and real-time videos and identify the positions of the crew members;

[0009] Based on the basic information of the ship and the identified crew positions, the CNN model is used to identify the crew's mooring behavior;

[0010] The weighted average score and weighted ratio obtained by simulation learning are extracted to calculate the clarity influencing factor. The SIFT algorithm deformation formula is constructed based on the clarity influencing factor. The SIFT algorithm deformation formula is used to calculate the similarity between the identified crew member's mooring behavior and the unmoored or inadequate mooring behavior obtained by simulation learning, and the unmoored or inadequate mooring behavior is identified.

[0011] As a preferred technical solution, the different environments include daytime (excluding rainy and foggy days), nighttime, foggy days and rainy days.

[0012] As a preferred technical solution, the calculation method of the weighted average score and weighted ratio of image clarity under different environments is:

[0013] The clarity of the images during the daytime, at night, on foggy days, and on rainy days are calculated separately. The calculation method is as follows:

[0014]

[0015] Where f(x,y) represents the gray value of the pixel (x,y) corresponding to image f, and D(f) is the image clarity;

[0016] The clarity of the images in the daytime, nighttime, foggy day, and rainy day are weighted averaged to obtain the weighted average score W(f 1 )、W(f 2 )、W(f 3 ) and W(f 4 );

[0017] Calculate the weighted proportion based on the weighted average score:

[0018] U(f i )=W(fi ) / W(f 1 ),i=1,2,3,4

[0019] Where U(f i ) is the weighted ratio calculated based on the daytime clarity.

[0020] As an optimal technical solution, the YOLO8 algorithm consists of three parts: a backbone feature extraction layer, a feature fusion module, and a detection head. The backbone feature extraction layer transmits real-time video, pictures, basic ship information, and identified crew position data to the feature fusion module, and the feature fusion module transmits the fused data to the detection head. The backbone feature extraction layer integrates the SCConv module, including a spatial reconstruction unit SRU and a channel reconstruction unit CRU. The SRU utilizes the spatial redundancy of the features and separates the feature map with rich information of spatial content from the corresponding feature map with less information through a separation operation. The scaling factor in the group normalization layer is used to evaluate the information content of different feature maps. Through a cross-reconstruction operation, two different information features are combined and weighted. The CRU utilizes the channel redundancy characteristics, adopts a segmentation-conversion-fusion strategy, and introduces a compression ratio to control the feature channel.

[0021] As a preferred technical solution, the YOLO8 algorithm performs the following steps:

[0022] S301, determine an intermediate feature map X∈R N×C×H×W , where N is the batch axis, C is the channel axis, H and W are the spatial height and width axes, respectively, and the input feature X is standardized using the group normalization layer as follows:

[0023]

[0024] Where μ and σ are the mean and standard deviation of X, ε is a positive number added to stabilize the division, and γ and β are trainable affine transformations;

[0025] S302, introduce trainable parameter γ∈R C , used to measure the variance of spatial pixels of each batch and channel, the normalized correlation weight W γ Obtained by the following equation:

[0026]

[0027] S303, through the sigmoid function, W γ The weight values ​​of the reweighted feature map are mapped to the range (0, 1) and gated by a threshold, and weights above the threshold are set to 1 to obtain the informative weights W 1 , set the weights below the threshold to 0 to obtain the non-informative weights W 2, get W i The process is represented by the following equation:

[0028] W i = Gate(Sigmoid(W γ (Gn(X))))

[0029] S304, multiply the input feature X by W 1 and W 2 , we get two weighted features: informative features and less informative features A cross-reconstruction operation is used to fully combine the weighted features of two different information richness levels. and And strengthen the information flow between them, and reconstruct the cross-reconstructed feature X w1 and X w2 Concatenate to obtain the spatially refined feature map X w , the process is expressed as:

[0030]

[0031] in represents element-wise multiplication, represents element-by-element addition, ∪ represents concatenation;

[0032] S306, using M k ∈R c*k*k represents a k\times k convolution kernel, a standard convolution is defined as Y = M k X,X,Y∈R c*k*k Representing the input and output features after convolution, the standard convolution is replaced by the channel reconstruction unit CRU, which includes segmentation, conversion and fusion operations:

[0033] Segmentation: Refine the feature X for a given space w εR c*k*k , X w The channels of CRU are divided into two parts, one of which has αC channels and the other has (1-α)C channels, where 0≤a≤1 represents the segmentation ratio; 1\times1 convolution is used to compress the channels of the feature map, and the compression ratio r is introduced to control the feature channels of CRU to balance the computational cost. After the segmentation and compression operations, the spatial refinement feature X w Divided into upper part X up and the lower part X low ;

[0034] Conversion: X up It is input to the up-conversion stage and processed by efficient convolution operations GWC and PWC.up Perform k×k GWC and 1x1 PWC operations on the vector, and add the outputs to form a combined representative feature map Y 1 , the up-conversion stage is expressed as:

[0035]

[0036] in, is the learnable weight matrix of GWC and PWC, and They are the input and output feature maps of the up-conversion stage, respectively;

[0037] X low is input to the down-conversion stage, where a 1x1 PWC operation is applied to generate a feature map with shallow hidden details and reuse the feature X low To obtain more feature maps without adding extra cost, the generated and reused features are concatenated to form the output Y of the bottom stage. 2 :

[0038]

[0039] in, is the learnable weight matrix of PWC, ∪ is the connection operation, and are the input and output feature maps of the down-conversion stage, respectively;

[0040] Fusion: After conversion, the output features Y of the up-conversion stage and the down-conversion stage are adaptively merged using a simplified SKNet method 1 and Y 2 First, use global average pooling to collect global spatial information and obtain channel statistics S m εR c *1*1 :

[0041]

[0042] The global channel descriptors S1 and S2 of the up-conversion stage and the down-conversion stage are stacked together, and the channel attention operation is used to generate the feature importance vector β 1 , As shown below:

[0043]

[0044] In the feature importance vector β 1 ,β 2 Under the guidance of 1 and Y 2 Get the channel refined feature Y as follows:

[0045] Y=β 1 Y 1 +β 2 Y 2 .

[0046] As a preferred technical solution, the SIFT algorithm deformation formula calculates the distance of feature vectors in two-dimensional space through Euclidean distance, and superimposes the clarity influencing factor to calculate the similarity:

[0047] V = sqrt((X 1 -X 2 ) 2 +(Y 1 -Y 2 ) 2 )-p(f i )

[0048] Among them, X 1 , X 2 is the horizontal coordinate of two random vectors, Y 1 , Y 2 are the ordinates of two random vectors, p(f i ) is the clarity influencing factor, V is the similarity, and sqrt is the square root.

[0049] As a preferred technical solution, for daytime images on non-rainy and foggy days, the calculation method of the clarity impact factor is: p(f i )=[D(f i )-W(f 1 )];

[0050] For nighttime images, the clarity factor is calculated as follows: p(f i )=[D(f i )-W(f 2 )] / W(f 2 )*U(f 2 );

[0051] For foggy day pictures, the calculation method of clarity influencing factor is: p(f i )=[D(f i )-W(f 3 )] / W(f 3 )*U(f 3 );

[0052] For rainy day pictures, the calculation method of the clarity factor is: p(f i )=[D(f i )-W(f 4 )] / W(f 4 )*U(f 4 );

[0053] Among them, D(f i ) is the image clarity, W(f i ) is the weighted average score corresponding to environment i, U(f i ) is the weighted proportion corresponding to environment i, i=1 represents daytime without rain or fog, i=2 represents nighttime, i=3 represents foggy day, and i=4 represents rainy day.

[0054] According to a second aspect of the present invention, there is provided a system for automatically identifying mooring behavior, for implementing the method described, the system comprising:

[0055] The neural network algorithm module obtains complete data of ships passing through the lock from the lock chamber control center database, including basic information of the ship, ship pictures, real-time videos and images of historical violations. Considering the impact of the environment on image clarity, the neural network algorithm is used to simulate and learn the historical data of unmoored or inadequate mooring behaviors, and outputs the weighted average score and weighted ratio of image clarity under different environments. The simulated violations, weighted average score and weighted ratio are sent to the data storage module for storage;

[0056] The target detection algorithm module obtains the basic information of ships passing through the lock, ship pictures and real-time videos from the lock chamber control center database, uses the YOLO8 algorithm to detect the ships in the acquired ship pictures and real-time videos, identifies the positions of the crew members, and sends the identified position data and basic ship information to the data storage module for storage;

[0057] The deep learning module obtains the basic information of the ship and the identified crew positions from the data storage module, uses the CNN model to identify the crew's mooring behavior, and extracts the weighted average score and weighted ratio obtained by simulation learning from the data storage module to calculate the clarity influencing factor, constructs the SIFT algorithm deformation formula based on the clarity influencing factor, uses the SIFT algorithm deformation formula to calculate the similarity between the identified crew's mooring behavior and the unmoored or inadequate mooring behavior obtained by simulation learning, identifies the unmoored or inadequate mooring behavior, and sends the identification result to the data storage module for storage;

[0058] AR panoramic module: when the deep learning module identifies that the ship is not moored or the mooring is not in place, it receives the basic information of the ship and the identified behavior of not mooring or mooring in place sent by the data storage module, displays the position of the offending crew member, and issues voice, text and SMS alarms;

[0059] The data storage module is used to store the processing results of the neural network algorithm module, the target detection algorithm module and the deep learning module.

[0060] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0061] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The present invention uses a neural network learning algorithm to learn the behavior of not mooring or mooring in place, and deeply understands the characteristics of illegal behaviors. Even illegal behaviors that are not in the database can be easily identified. When detecting illegal behaviors, a neural network algorithm is used for simulation learning instead of directly comparing the identified crew behavior with the data in the lock chamber control center database. First, the crew behavior characteristics are difficult to be completely consistent, and the comparison accuracy is limited. Second, after algorithm learning, it can identify illegal behaviors that have not been detected before, accurately grasp the identification loopholes, improve the efficiency of passing the lock, and protect the safety of ships passing the lock.

[0064] (2) The present invention adopts the mainstream deep learning model and SIFT algorithm to perform feature extraction and similarity comparison, and conducts behavioral analysis, feature extraction and similarity comparison on crew members, which greatly improves the accuracy and efficiency compared with the original level.

[0065] (3) The system of the present invention can be well integrated with the existing video surveillance system of the ship lock chamber, realizing system reuse and significantly reducing construction costs. Further improvements are made to the existing video surveillance system of the ship lock chamber, avoiding the waste of reports from the original system, reducing the construction cost of the new system, and realizing the upgrade of the ship mooring behavior recognition technology, which has the two characteristics of low cost and high efficiency.

[0066] (4) The present invention breaks the one-way warning mechanism and adds SMS warning. For the behavior of not mooring or mooring in place, not only the regulatory department is alerted, but also the ship owner is notified at the same time, so as to achieve information symmetry between the two parties, reminding the ship owner to make timely rectification and facilitating the subsequent punishment by the regulatory department. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of the method of the present invention;

[0068] Figure 2 This is a schematic diagram of the YOLO8 model structure;

[0069] Figure 3 Schematic diagram of the SCConv model structure;

[0070] Figure 4 It is a schematic diagram of the SRU model structure;

[0071] Figure 5 It is a schematic diagram of the CRU model structure;

[0072] Figure 6 It is a schematic diagram of the system structure of the present invention;

[0073] Figure 7 A comparison chart of recognition results in an embodiment. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0075] Example 1

[0076] This embodiment provides a method for automatically identifying mooring behavior. Figure 1 As shown, the method comprises the following steps:

[0077] S1, obtain complete data of ships passing through the locks, including basic ship information, ship pictures, real-time videos and historical violation images.

[0078] S2, considering the impact of the environment on image clarity, uses a neural network algorithm to simulate and learn historical violation data of not tying the cable or not tying the cable in place, and outputs the weighted average score and weighted ratio of image clarity under different environments.

[0079] Since the camera capture clarity is related to the light during the day (non-rainy and foggy days), night, foggy days and rainy days, poor light conditions will cause a decrease in image clarity, which will affect the subsequent similarity calculation, thereby causing deviations in the judgment of the cable behavior. Therefore, this embodiment will take this factor into consideration in the similarity calculation. The different environments considered include daytime, nighttime, foggy days and rainy days.

[0080] The calculation method of the weighted average score and weighted ratio of image clarity under different environments is:

[0081] The clarity of the images during the daytime, at night, on foggy days, and on rainy days are calculated separately. The calculation method is as follows:

[0082]

[0083] Where f(x,y) represents the gray value of the pixel (x,y) corresponding to image f, and D(f) is the image clarity;

[0084] The clarity of the images in the daytime, nighttime, foggy day, and rainy day are weighted averaged to obtain the weighted average score W(f 1 )、W(f 2 )、W(f 3 ) and W(f 4 );

[0085] Calculate the weighted proportion based on the weighted average score:

[0086] U(f i )=W(f i ) / W(f 1 ),i=1,2,3,4

[0087] Where U(f i ) is the weighted ratio calculated based on the daytime clarity.

[0088] S3, uses the YOLO8 algorithm to detect the ships in the acquired ship images and real-time videos and identify the crew positions.

[0089] like Figure 2 As shown in Figure 1, the YOLO8 algorithm consists of three parts: the backbone feature extraction layer Backbone, the feature fusion module Neck, and the detection head Head. The backbone feature extraction layer transmits real-time video, pictures, basic ship information and identified crew position data to the feature fusion module, and the feature fusion module transmits the fused data to the detection head. Figure 3 As shown in the figure, the backbone feature extraction layer integrates the SCConv module, including the spatial reconstruction unit SRU and the channel reconstruction unit CRU, PreviousConvblock represents the previous convolution block, Input Feature represents the input feature, Spatial-Refined Feature represents the spatial refinement feature, Channel-Refined Feature represents the channel refinement feature, and Next Convblock represents the next convolution block.

[0090] like Figure 4As shown in the figure, SRU utilizes the spatial redundancy of features, separates the feature maps with rich information of spatial content from the corresponding feature maps with less information through separation operation, evaluates the information content of different feature maps by using the scaling factor in the group normalization (GN) layer, and combines two different information features by cross reconstruction operation to strengthen the information flow between them and reduce the storage cost; wherein, Spatial Reconstruction Unit represents spatial reconstruction unit, InputFeature represents input feature, Spatial-Refined Feature represents spatial refined feature, GroupNormalization represents group normalization, Sigmoid represents logical function, Threshold represents threshold value, Multiplication represents multiplication, Summation represents addition, Concatenation represents merging, Separate represents separation, and Reconstruct represents reconstruction.

[0091] like Figure 5 As shown in the figure, CRU uses the redundancy characteristics of channels, adopts the split-conversion-fusion strategy, introduces the compression ratio to control the feature channel, and reduces the computational cost. Among them, Channel Reconstruction Unit represents the channel reconstruction unit, Spatial-Refined Feature represents the spatial refinement feature, Split represents splitting, Transform represents transformation, Fuse represents fusion, Pooling represents pooling, Soft Max represents the maximum soft value, Element-wise summation represents element-wise summation, Concatenation represents merging, and Element-wise multiplication represents element-wise multiplication.

[0092] Specifically, the YOLO8 algorithm performs the following steps:

[0093] S301, determine an intermediate feature map X∈R N×C×H×W , where N is the batch axis, C is the channel axis, H and W are the spatial height and width axes, respectively, and the input feature X is standardized using the group normalization layer as follows:

[0094]

[0095] where μ and σ are the mean and standard deviation of X, ε is a positive number added to stabilize the division, and γ and β are trainable affine transformations.

[0096] S302, introduce trainable parameter γ∈R C , used to measure the variance of spatial pixels of each batch and channel, the normalized correlation weight Wγ Obtained by the following equation:

[0097]

[0098] S303, through the sigmoid function, W γ The weight values ​​of the reweighted feature map are mapped to the range (0, 1) and gated by a threshold, and weights above the threshold are set to 1 to obtain the informative weights W 1 , set the weights below the threshold to 0 to obtain the non-informative weights W 2 , get W i The process is represented by the following equation:

[0099] W=Gate(Sigmoid(W γ (GN(X))))

[0100] S304, multiply the input feature X by W 1 and W 2 , we get two weighted features: informative features and less informative features In this way, we successfully divide the input features into two parts: With rich and expressive spatial content, There is almost no information and it is considered redundant.

[0101] A cross-reconstruction operation is used to fully combine the weighted features of two different information richness levels. and And strengthen the information flow between them, and reconstruct the cross-reconstructed feature X w1 and X w2 Concatenate to obtain the spatially refined feature map X w . The process is expressed as:

[0102]

[0103] in represents element-wise multiplication, represents element-by-element addition, and ∪ represents concatenation. After applying SRU to the intermediate input feature X, the features with rich information are separated from the features with less information, and they are reconstructed to enhance the representative features and suppress the redundant features in the spatial dimension. However, the spatial refinement feature map X w There is still redundancy in the channel dimension.

[0104] S306, in order to utilize the channel redundancy of features, a channel reconstruction unit (CRU) is introduced, which utilizes the “segmentation-conversion-fusion” strategy. Generally speaking, repeated standard k\times k convolutions are used to extract features, resulting in some relatively redundant feature maps along the channel dimension. This embodiment uses M k ∈R c*k*k represents a k\times k convolution kernel, a standard convolution is defined as Y = M k X,X,Y∈R c*k*k Represents the input and output features after convolution, replacing the standard convolution with the channel reconstruction unit CRU, which is implemented through three operators - split Split, transform Transform and fusion Fuse.

[0105] Segmentation: Refine the feature X for a given space w εR c*k*k , X w The channels of CRU are divided into two parts, one of which has αC channels and the other has (1-α)C channels, where 0≤a≤1 represents the segmentation ratio. 1\times1 convolution is used to compress the channels of the feature map to improve computational efficiency. The compression ratio r is introduced to control the feature channels of CRU to balance the computational cost (r is typically set to 2 in the experiment). After the segmentation and compression operations, the spatially refined features X w Divided into upper part X up and the lower part X low .

[0106] Conversion: X up is input to the up-conversion stage as a "rich feature extractor" that uses efficient convolution operations GWC and PWC to replace standard k\times k convolutions to extract high-level representative information while reducing computational cost. Due to sparse convolutional connections, GWC reduces parameters and computation, but cuts off the information flow between channel groups. PWC compensates for the loss of information and helps information flow between feature channels. Therefore, for the same X up Perform k×k GWC and 1x1 PWC operations on the vector, and add the outputs to form a combined representative feature map Y 1 , the up-conversion stage is expressed as:

[0107]

[0108] in, is the learnable weight matrix of GWC and PWC, and The up-conversion stage uses GWC and PWC on the same feature map X. upThe combination of the above can extract rich representative features Y with less computational cost. 1 .

[0109] X low is input to the down-conversion stage, where a 1x1 PWC operation is applied to generate a feature map with shallow hidden details as a complement to the rich feature extractor and reuse the feature X low To obtain more feature maps without adding extra cost, the generated and reused features are concatenated to form the output Y of the bottom stage. 2 :

[0110]

[0111] in, is the learnable weight matrix of PWC, ∪ is the connection operation, and are the input and output feature maps of the down-conversion stage, respectively;

[0112] Fusion: After conversion, the output features Y of the up-conversion stage and the down-conversion stage are adaptively merged using a simplified SKNet method 1 and Y 2 First, use global average pooling to collect global spatial information and obtain channel statistics S m εR c*1*1 :

[0113]

[0114] The global channel descriptors S1 and S2 of the up-conversion stage and the down-conversion stage are stacked together, and the channel attention operation is used to generate the feature importance vector β 1 , As shown below:

[0115]

[0116] In the feature importance vector β 1 ,β 2 Under the guidance of 1 and Y 2 Get the channel refined feature Y as follows:

[0117] Y=β 1 Y 1 +β 2 Y 2

[0118] S4, based on the basic information of the ship and the identified crew positions, the CNN model is used to identify the crew’s mooring behavior.

[0119] This part adopts the existing model, and will not be described in detail in this embodiment.

[0120] S5, extract the weighted average score and weighted ratio obtained by simulation learning to calculate the clarity influencing factor, construct the SIFT algorithm deformation formula based on the clarity influencing factor, use the SIFT algorithm deformation formula to calculate the similarity between the identified crew member's mooring behavior and the unmoored or inadequate mooring behavior obtained by simulation learning, and identify the unmoored or inadequate mooring behavior.

[0121] The SIFT algorithm deformation formula calculates the distance of feature vectors in two-dimensional space through Euclidean distance, and superimposes the clarity influencing factor to calculate the similarity:

[0122] V = sqrt((X 1 -X 2 ) 2 +(Y 1 -Y 2 ) 2 )-p(f i )

[0123] Among them, X 1 , X 2 is the horizontal coordinate of two random vectors, Y 1 , Y 2 are the ordinates of two random vectors, p(f i ) is the clarity influencing factor, V is the similarity, and sqrt is the square root.

[0124] Among them, the clearer the image, the greater the clarity impact factor.

[0125] For daytime images on non-rainy and foggy days, the calculation method of the clarity impact factor is: p(f i )=[D(f i )-W(f 1 )];

[0126] For nighttime images, the clarity factor is calculated as follows: p(f i )=[D(f i )-W(f 2 )] / W(f 2 )*U(f 2 );

[0127] For foggy day pictures, the calculation method of clarity influencing factor is: p(f i )=[D(f i )-W(f 3 )] / W(f 3 )*U(f 3 );

[0128] For rainy day pictures, the calculation method of the clarity factor is: p(f i )=[D(f i )-W(f 4 )] / W(f 4 )*U(f 4 );

[0129] Among them, D(f i ) is the image clarity, W(f i ) is the weighted average score corresponding to environment i, U(f i ) is the weighted proportion corresponding to environment i, i=1 represents daytime without rain or fog, i=2 represents nighttime, i=3 represents foggy day, and i=4 represents rainy day.

[0130] The smaller the value of the calculated similarity V is, the more similar the two targets are. In this embodiment, when the similarity V is defined as < 0.75, the target similarity is relatively large, and when V ≥ 0.75, the targets are not very similar. The setting of the specific value can be changed according to the actual situation. Then, when V < 0.75, the identified crew mooring behavior is similar to the simulated learning behavior of not mooring or not mooring in place, and it is considered that the crew has not moored or moored in place behavior.

[0131] Example 2

[0132] The above is an introduction to the method embodiment. The following is a further explanation of the solution of the present invention through a system embodiment.

[0133] like Figure 6 As shown, the system includes:

[0134] The neural network algorithm module obtains complete data of ships passing through the lock from the lock chamber control center database, including basic information of the ship, ship pictures, real-time videos and images of historical violations. Considering the impact of the environment on image clarity, the neural network algorithm is used to simulate and learn the historical data of unmoored or inadequate mooring behaviors, and outputs the weighted average score and weighted ratio of image clarity under different environments. The simulated violations, weighted average score and weighted ratio are sent to the data storage module for storage;

[0135] The target detection algorithm module obtains the basic information of ships passing through the lock, ship pictures and real-time videos from the lock chamber control center database, uses the YOLO8 algorithm to detect the ships in the acquired ship pictures and real-time videos, identifies the positions of the crew members, and sends the identified position data and basic ship information to the data storage module for storage;

[0136] The deep learning module obtains the basic information of the ship and the identified crew positions from the data storage module, uses the CNN model to identify the crew's mooring behavior, and extracts the weighted average score and weighted ratio obtained by simulation learning from the data storage module to calculate the clarity influencing factor, constructs the SIFT algorithm deformation formula based on the clarity influencing factor, uses the SIFT algorithm deformation formula to calculate the similarity between the identified crew's mooring behavior and the unmoored or inadequate mooring behavior obtained by simulation learning, identifies the unmoored or inadequate mooring behavior, and sends the identification result to the data storage module for storage;

[0137] AR panoramic module: when the deep learning module identifies that the ship is not moored or the mooring is not in place, it receives the basic information of the ship and the identified behavior of not mooring or mooring in place sent by the data storage module, displays the position of the offending crew member, and issues voice, text and SMS alarms;

[0138] The data storage module is used to store the processing results of the neural network algorithm module, the target detection algorithm module and the deep learning module.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0140] Example 3

[0141] This embodiment conducts a simulation experiment on the method of Embodiment 1, and obtains the index performance comparison results of the present invention and Yolov81-pose as shown in Table 1.

[0142] Table 1 Index performance comparison

[0143]

[0144] Among them, P stands for precision, Precision = TP / (TP+FP), which is the proportion of correct predictions among all results whose predictions are Positive; R stands for recall, recall = TP / (TP+FN), which is the proportion of correct predictions among all results whose true values ​​are Positive; Box stands for detection box; pose stands for key points; mAP stands for Mean Average Precision; mAP50 represents the average precision when the IOU threshold is greater than 0.5; mAP50-95: represents the average mAP at different IOU thresholds (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

[0145] The image recognition results of the two methods are shown in Figure 7As shown, (a1)-(a9) are the recognition results of Yolov8l-pose, and (b1)-(b9) are the recognition results of the present invention. It can be seen that the recognition accuracy of the present invention is higher and the recognition result is more accurate.

[0146] Example 4

[0147] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0148] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0149] The processing unit performs the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 in any other appropriate manner (e.g., by means of firmware).

[0150] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0151] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for automatically identifying mooring behavior, characterized in that: The method comprises the following steps: Obtain complete data on ships passing through the locks, including basic ship information, ship pictures, real-time videos and images of historical violations; Considering the impact of the environment on image clarity, a neural network algorithm is used to simulate and learn historical violation data of not tying the cable or not tying the cable in place, and output the weighted average score and weighted ratio of image clarity under different environments; The YOLO8 algorithm is used to detect the ships in the acquired ship images and real-time videos and identify the positions of the crew members; Based on the basic information of the ship and the identified crew positions, the CNN model is used to identify the crew's mooring behavior; The weighted average score and weighted ratio obtained by simulation learning are extracted to calculate the clarity influencing factor. The SIFT algorithm deformation formula is constructed based on the clarity influencing factor. The SIFT algorithm deformation formula is used to calculate the similarity between the identified crew member's mooring behavior and the unmoored or inadequate mooring behavior obtained by simulation learning, and the unmoored or inadequate mooring behavior is identified.

2. The method for automatically identifying mooring behavior according to claim 1, characterized in that: The different environments include daytime (excluding rainy and foggy days), nighttime, heavy foggy days and rainy days.

3. The method for automatically identifying mooring behavior according to claim 2, characterized in that: The weighted average score and weighted ratio of the image clarity under different environments are calculated as follows: The clarity of the images during the daytime, at night, on foggy days, and on rainy days are calculated separately. The calculation method is as follows: Where f(x,y) represents the gray value of the pixel (x,y) corresponding to image f, and D(f) is the image clarity; The clarity of the images in the daytime, nighttime, foggy and rainy days are weighted averaged to obtain the weighted average scores W(f1), W(f2), W(f3) and W(f4) in each environment. Calculate the weighted proportion based on the weighted average score: U(f i )=W(f i ) / W(f1),i=1,2,3,4 Where U(f i ) is the weighted ratio calculated based on the daytime clarity.

4. The method for automatically identifying mooring behavior according to claim 1, characterized in that: The YOLO8 algorithm is composed of three parts: a backbone feature extraction layer, a feature fusion module, and a detection head. The backbone feature extraction layer transmits real-time video, pictures, basic ship information, and identified crew position data to the feature fusion module. The feature fusion module transmits the fused data to the detection head. The backbone feature extraction layer integrates the SCConv module, which includes a spatial reconstruction unit SRU and a channel reconstruction unit CRU. The SRU utilizes the spatial redundancy of features and separates feature maps with rich information of spatial content from corresponding feature maps with less information through separation operations. The scaling factor in the group normalization layer is used to evaluate the information content of different feature maps. Through a cross-reconstruction operation, two different information features are combined and weighted. The CRU utilizes the channel redundancy characteristics, adopts segmentation-conversion-fusion measurement, and introduces compression ratio to control feature channels.

5. The method for automatically identifying mooring behavior according to claim 4, characterized in that: The YOLO8 algorithm performs the following steps: S301, determine an intermediate feature map X∈R N×C×H×W , where N is the batch axis, C is the channel axis, H and W are the spatial height and width axes, respectively, and the input feature X is standardized using the group normalization layer as follows: Where μ and σ are the mean and standard deviation of X, ε is a positive number added to stabilize the division, and γ and β are trainable affine transformations; S302, introduce trainable parameter γ∈R C , used to measure the variance of spatial pixels of each batch and channel, the normalized correlation weight W γ Obtained by the following equation: S303, through the sigmoid function, W γ The weight values ​​of the reweighted feature map are mapped to the range (0, 1) and gated by a threshold. The weights above the threshold are set to 1 to obtain informative weights W1, and the weights below the threshold are set to 0 to obtain non-informative weights W2. i The process is represented by the following equation: W i =Gate(Sigmoid(W γ (GN(X)))) S304, multiply the input feature X by W1 and W2 respectively to obtain two weighted features: information-rich features and less informative features A cross-reconstruction operation is used to fully combine the weighted features of two different information richness levels. and And strengthen the information flow between them, and reconstruct the cross-reconstructed feature X w1 and X w2 Concatenate to obtain the spatially refined feature map X w , the process is expressed as: in represents element-wise multiplication, represents element-by-element addition, ∪ represents concatenation; S306, using M k ∈R c*k*k represents a k\times k convolution kernel, a standard convolution is defined as Y = M k X,X,Y∈R c *k*k Representing the input and output features after convolution, the standard convolution is replaced by the channel reconstruction unit CRU, which includes segmentation, conversion and fusion operations: Segmentation: Refine the feature X for a given space w εR c*k*k , X w The channels of CRU are divided into two parts, one of which has αC channels and the other has (1-α)C channels, where 0≤a≤1 represents the segmentation ratio; 1\times1 convolution is used to compress the channels of the feature map, and the compression ratio r is introduced to control the feature channels of CRU to balance the computational cost. After the segmentation and compression operations, the spatial refinement feature X w Divided into upper part X up and the lower part X low ; Conversion: X up It is input to the up-conversion stage and processed by efficient convolution operations GWC and PWC. up Perform k×k GWC and 1x1 PWC operations on the vector, and add the outputs to form a combined representative feature map Y1. The up-conversion stage is expressed as: in, is the learnable weight matrix of GWC and PWC, and They are the input and output feature maps of the up-conversion stage, respectively; X low is input to the down-conversion stage, where a 1x1 PWC operation is applied to generate a feature map with shallow hidden details and reuse the feature X low To obtain more feature maps without incurring additional cost, the generated and reused features are concatenated to form the output Y2 of the bottom stage: in, is the learnable weight matrix of PWC, ∪ is the connection operation, and are the input and output feature maps of the down-conversion stage, respectively; Fusion: After the conversion, the simplified SKNet method is used to adaptively merge the output features Y1 and Y2 of the up-conversion stage and the down-conversion stage. First, global average pooling is used to collect global spatial information to obtain channel statistics. The global channel descriptors S1 and S2 of the up-conversion stage and the down-conversion stage are stacked together, and the channel attention operation is used to generate the feature importance vector As shown below: Under the guidance of feature importance vectors β1, β2, the channel-refined feature Y is obtained by merging features Y1 and Y2 in a channel-wise manner, as shown below: Y=β1Y1+β2Y2.

6. The method for automatically identifying mooring behavior according to claim 2, characterized in that: The SIFT algorithm deformation formula calculates the distance of feature vectors in two-dimensional space through Euclidean distance, and superimposes the clarity influencing factor to calculate the similarity: V=sqrt((X1-X2) 2 +(Y1-Y2) 2 )-p(f i ) Among them, X1 and X2 are the horizontal coordinates of two random vectors, Y1 and Y2 are the vertical coordinates of two random vectors, and p(f i ) is the clarity influencing factor, V is the similarity, and sqrt is the square root.

7. The method for automatically identifying mooring behavior according to claim 6, characterized in that: For daytime images on non-rainy and foggy days, the calculation method of the clarity impact factor is: p(f i )=[D(f i )-W(f1)]; For nighttime images, the clarity factor is calculated as follows: p(f i )=[D(f i )-W(f2)] / W(f2)*U(f2); For foggy day pictures, the calculation method of clarity influencing factor is: p(f i )=[D(f i )-W(f3)] / W(f3)*U(f3); For rainy day pictures, the calculation method of the clarity factor is: p(f i )=[D(f i )-W(f4)] / W(f4)*U(f4); Among them, D(f i ) is the image clarity, W(f i ) is the weighted average score corresponding to environment i, U(f i ) is the weighted proportion corresponding to environment i, i=1 represents daytime without rain or fog, i=2 represents nighttime, i=3 represents foggy day, and i=4 represents rainy day.

8. A system for automatically identifying mooring behavior, characterized in that: For implementing the method according to any one of claims 1 to 7, the system comprises: The neural network algorithm module obtains complete data of ships passing through the lock from the lock chamber control center database, including basic information of the ship, ship pictures, real-time videos and images of historical violations. Considering the impact of the environment on image clarity, the neural network algorithm is used to simulate and learn the historical data of unmoored or inadequate mooring behaviors, and outputs the weighted average score and weighted ratio of image clarity under different environments. The simulated violations, weighted average score and weighted ratio are sent to the data storage module for storage; The target detection algorithm module obtains the basic information of ships passing through the lock, ship pictures and real-time videos from the lock chamber control center database, uses the YOLO8 algorithm to detect the ships in the acquired ship pictures and real-time videos, identifies the positions of the crew members, and sends the identified position data and basic ship information to the data storage module for storage; The deep learning module obtains the basic information of the ship and the identified crew positions from the data storage module, uses the CNN model to identify the crew's mooring behavior, and extracts the weighted average score and weighted ratio obtained by simulation learning from the data storage module to calculate the clarity influencing factor, constructs the SIFT algorithm deformation formula based on the clarity influencing factor, uses the SIFT algorithm deformation formula to calculate the similarity between the identified crew's mooring behavior and the unmoored or inadequate mooring behavior obtained by simulation learning, identifies the unmoored or inadequate mooring behavior, and sends the identification result to the data storage module for storage; AR panoramic module: when the deep learning module identifies that the ship is not moored or the mooring is not in place, it receives the basic information of the ship and the identified behavior of not mooring or mooring in place sent by the data storage module, displays the position of the offending crew member, and issues voice, text and SMS alarms; The data storage module is used to store the processing results of the neural network algorithm module, the target detection algorithm module and the deep learning module.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images