A water gauge intelligent observation method and system based on multi-unmanned equipment collaboration

Abstract

The present invention relates to the field of water gauge observation technology, and discloses a water gauge intelligent observation method and system based on the collaboration of multiple unmanned equipment. The observation method includes: collecting ship long-range and short-range images and ship draft data; identifying water level data and water gauge symbol data corresponding to the water level data in the ship long-range and short-range images based on an improved YOLOV5 neural network; using a clustering algorithm to optimize the error of the water level data and the water gauge symbol data to obtain preliminary water gauge observation data; using a Kalman filter network to dynamically assign weights to the ship draft data and the preliminary water gauge observation data, and weightedly fuse the ship draft data and the preliminary water gauge observation data to obtain fused water gauge observation data; the Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network. The present invention can solve the problems of low efficiency and high risk of traditional manual water gauge observation, as well as the weak anti-interference ability of existing single-sensor water gauge observation and limited observation in harsh environments.

Description

Technical Field

[0001] The present invention relates to the technical field of water gauge observation, and in particular to a water gauge intelligent observation method and system based on the collaboration of multiple unmanned equipment. Background Art

[0002] Draft weight measurement is currently the primary method for measuring cargo ship loading in port shipping. Accurate measurement of cargo ship draft is crucial for safe navigation and port cargo management. Traditional manual draft gauge observation methods are inefficient, and observation accuracy is significantly affected by subjective factors. In severe weather conditions, wind and waves cause the waterline to fluctuate greatly, making manual observation dangerous and difficult to ensure accuracy. To address the shortcomings of manual observation, early solutions were mostly based on high-precision sensors for remote, real-time observation. The draft depth of a cargo ship is measured using radar, sonar, pressure sensors, and other technologies. Pressure sensors are the most widely used. By performing proportional calculations on the data collected by pressure sensors installed on the bottom of the cargo ship, the effects of water density and gravitational acceleration are eliminated, resulting in relatively accurate draft data.

[0003] However, these sensors remain underwater for extended periods, requiring regular maintenance. Repairing these devices is difficult and expensive, and single sensors are susceptible to the complex underwater environment. Consequently, unmanned, low-cost visual inspection solutions have been proposed. These use cameras, drones, and other equipment to capture real-time video and images. These solutions then use image processing algorithms like edge detection or YOLO neural network vision algorithms to automatically identify water gauge readings. However, these methods are often limited by port environments. In inclement weather, capturing ship images is difficult, image quality is difficult to guarantee, and algorithm accuracy is limited.

[0004] In summary, the existing water level observation technology has problems such as low manual observation efficiency, low precision of single-modality sensors, high maintenance cost, and visual algorithms that are easily affected by the environment. Summary of the Invention

[0005] The present invention aims to provide a method and system for intelligent water gauge observation based on the collaboration of multiple unmanned devices. This method uses drones and unmanned boats to collect distant images of ships and close-up images of water gauges. Water level data and water gauge symbol data are detected and identified using an improved YOLOv5 neural network. An underwater robot equipped with a pressure sensor is also used to collect ship draft data. Water gauge observation data is then acquired using a real-time state estimation network and a dynamic weight fusion network. This system, which utilizes multiple unmanned devices, addresses the low efficiency and high risk of manual water gauge data observation, as well as the weak anti-interference capabilities and difficulty in application in harsh environments of existing water gauge observation methods using a single sensor.

[0006] In order to achieve the above-mentioned purpose, the technical solutions provided by the present invention are as follows:

[0007] In a first aspect, the present invention provides a water gauge intelligent observation method based on the collaboration of multiple unmanned equipment, comprising:

[0008] S1. Collecting ship's near and far images and ship's draft data;

[0009] S2. Identify the water level data in the ship's near and far view images, as well as the corresponding water gauge symbol data, based on the improved YOLOV5 neural network;

[0010] S3. Use a clustering algorithm to optimize the errors of the water level data and water gauge symbol data, filter out invalid prediction box information, and calculate and obtain preliminary water gauge observation data;

[0011] S4. Use a Kalman filter network to dynamically assign weights to the ship draft data and the preliminary draft observation data, and weightedly fuse the ship draft data and the preliminary draft observation data to obtain fused draft observation data; the Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network.

[0012] As a possible implementation method, the real-time state estimation network is built based on the traditional Kalman filter process. The noise of the ship draft data and the preliminary water gauge observation data is eliminated by learning the Kalman gain. The prediction process includes the following steps:

[0013]

[0014] P k|k―1 =FP k―1|k―1 F T +Q;

[0015] The update process is as follows:

[0016] K k =P k|l―1 H k T (H k P k|k―1 H k T +R) ―1 ,

[0017]

[0018] P k|k =(I-K k H k )P k|k―1 ;

[0019] Where, represents the prior estimate of the fused water gauge observation data at time k, represents the posterior estimate of the fused water gauge observation data at time k, z k is the observation value at time k, namely the ship draft data and preliminary water gauge observation data, F represents the state transfer matrix, H k represents the measurement matrix at time k, Q and R represent the Kalman filter noise, P k|k represents the posterior estimated covariance matrix, P k|k―1 represents the prior estimated covariance matrix, K k represents the Kalman gain, and I represents the identity matrix.

[0020] As a possible implementation method, the dynamic weight fusion network is constructed by setting the real-time state estimation network measurement matrix H k Implementing multi-source data fusion and weight distribution includes the following sub-steps:

[0021] S40. Initialize the measurement matrix H k =I;

[0022] S41. Use the prior estimate to multiply the measurement matrix to obtain the predicted observation value Use the ship draft data at time k and the preliminary water gauge observation data to obtain the observation value z k Subtract the predicted observations The difference obtained As network input features;

[0023] S42. Use GRU network to measure the matrix H k To update, follow these steps:

[0024] The difference Δz k After feature extraction by the fully connected layer, it is sent to the GRU network:

[0025] out FC =FC([Δz k ]);

[0026] The calculation process of the GRU network is as follows:

[0027] r t =σ(W r [h t―1 ,out FC ]+b r );

[0028] z t =σ(W z [h t―1 ,out FC ]+b z );

[0029]

[0030] Where r t Represents the reset gate output, z t represents the update gate output, and h t Denote candidate hidden states and final hidden states respectively, W r and W z Represent the weight of the reset gate output and the weight of the update gate output, b r and b z Represents the bias of the reset gate output and the bias of the update gate output, ⊙ represents the dot product operation, W h represents the weight of the candidate hidden state, b h Represents the bias of the candidate hidden state.

[0031] GRU obtains the final hidden state by resetting the gate output and updating the gate output to control the storage and forgetting of information. After the final hidden state is scaled by the fully connected layer, the measurement matrix Hk to be updated is obtained as follows:

[0032] H k =FC(h t );

[0033] Among them, FC represents the fully connected layer.

[0034] As a possible implementation method, the improved YOLOV5 neural network specifically uses the adaptive activation function ACON instead of the default nonlinear activation function ReLU; uses the weighted bidirectional feature pyramid network structure BiFPN instead of the path aggregation neural network PANet; and adds a plug-and-play attention module CBAM.

[0035] As a possible implementation method, the adaptive activation function ACON is obtained using the following formula:

[0036] f(x)=p1·ReLU(x)+p2·ReLU(―x);

[0037] Among them, p1 and p2 are adaptive parameters, and the ReLU(x) function is expressed as:

[0038]

[0039] As a possible implementation method, the following formula is used to obtain the weighted bidirectional feature pyramid network structure BiFPN:

[0040]

[0041] in, and They are top-down and top-up feature maps respectively, w1 and w2 are self-learning weights.

[0042] As a possible implementation method, the plug-and-play attention module CBAM includes a channel attention module and a spatial attention module. The calculation process of the channel attention module is as follows:

[0043]

[0044] f max =max i,j F c,i,j , c=1,2,…,C;

[0045]

[0046] Where, f avg represents the global average pooling result, f max represents the global maximum pooling result, f represents the channel descriptor obtained by vector splicing, F represents the input image feature map, F represents the number of channels, H and W represent the height and width of the feature map respectively, represents a two-dimensional real vector space of dimension 2C;

[0047] The channel weights are generated by a shared multi-layer perceptron MLP as follows:

[0048] m c =σ(W2·δ(W1·f+b1)+b2),

[0049] F out =m⊙F;

[0050] Where m c represents the channel weight, F out represents the feature map weighted by the channel weight, W1 and W2 represent the weight matrices of MLP, b1 and b2 represent the bias terms, ⊙ represents the dot product operation, and σ represents the activation function;

[0051] The calculation process of the spatial attention module is as follows:

[0052]

[0053] f′ max =max c F out,c ,

[0054] F s =Concat(f′ avg ,f′ max );

[0055] Where f′ avg represents the channel average pooling result, f′ max Indicates the channel maximum pooling result, F sRepresents the two-dimensional feature map obtained by channel splicing;

[0056] F s The spatial attention map is obtained through a convolutional layer, and the spatial attention map is combined with the feature map weighted by the channel weight to obtain the final output F final :

[0057] M spatial =σ(Conv(F s )),

[0058] F final =m spatial ⊙F out ;

[0059] Where, F final Represents the final output, m spatial Represents the two-dimensional feature map F s The spatial attention map F is obtained through a convolutional layer. out Represents the feature map weighted by channel weights.

[0060] As a possible implementation method, the ship's long and short-range images are collected by drones and unmanned boats; and / or the ship's draft depth data is collected by an underwater robot attached to the bottom of the ship.

[0061] In a second aspect, the present invention provides a water gauge intelligent observation system based on the collaboration of multiple unmanned equipment, comprising: a data acquisition and preprocessing module, an image detection module and a dynamic weight fusion module;

[0062] The data acquisition and pre-processing module is used to obtain the ship's long-range and short-range images and the ship's draft depth data;

[0063] The image detection module is used to identify the water level data in the ship's long and short-range images, as well as the water gauge symbol data corresponding to the water level data, and to optimize the errors of the water level data and water gauge symbol data to obtain preliminary water gauge observation data;

[0064] The dynamic weight fusion module is used to dynamically assign weights to the ship's draft data and the preliminary draft observation data, and to weightedly fuse the ship's draft data and the preliminary draft observation data to obtain fused draft observation data.

[0065] As a possible implementation method, the data acquisition and preprocessing module includes a ship long-range and short-range image acquisition unit and a ship draft depth data acquisition unit. The ship long-range and short-range image acquisition unit includes a drone and an unmanned boat for acquiring long-range and short-range images of the ship. The ship draft depth data acquisition unit includes an underwater robot attached to the bottom of the ship for outputting ship draft depth data.

[0066] The image detection module includes an image recognition unit and a water gauge symbol recognition unit. The image recognition unit is used to recognize the water level line data in the long and short-range images of the ship, and the water gauge symbol recognition unit is used to output the water gauge symbol data corresponding to the water level line.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The intelligent water gauge observation system based on the collaboration of multiple unmanned equipment provided by the present invention is coordinated by multiple unmanned equipment, fully utilizing the unmanned equipment in the port area, using drones and unmanned boats to collect distant and near-sighted images of ships and close-up images of water gauges, and using underwater robots equipped with pressure sensors to collect ship draft data. It is then combined with improved YOLOV5 neural networks, clustering algorithms, Kalman filter networks and other intelligent means for fusion observation. There is no need to install additional high-precision sensors, solving the problem of difficult maintenance and high maintenance costs of fixed sensors. In addition, the underwater robots are flexible and maneuverable and can adapt to various environments and hulls, solving the problem of limited observation in harsh environments and saving deployment and maintenance costs.

[0069] 2. The intelligent water gauge observation method based on the collaboration of multiple unmanned equipment provided by the present invention proposes multi-source data fusion based on the Kalman filter network, combines the advantages of sensor measurement and visual algorithm detection, overcomes the problem of limited accuracy of single-modal perception, and can obtain more accurate and interference-resistant observation results compared with traditional single-modal-based technical solutions and solutions relying on visual algorithms.

[0070] 3. The intelligent water level gauge observation method based on the collaboration of multiple unmanned equipment provided by the present invention performs pre-processing such as cropping and affine transformation on the collected images, and uses the improved YOLOV5 neural network to train the image data, so that the neural network can better extract image features and achieve more accurate water level line data and water level gauge symbol data recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0072] Figure 1 This is a flow chart of a water gauge intelligent observation method based on collaboration of multiple unmanned equipment in an embodiment of the present invention;

[0073] Figure 2 Schematic diagram of a water gauge intelligent observation system based on collaboration of multiple unmanned equipment in an embodiment of the present invention;

[0074] Figure 3This is the recognition result of the preliminary water gauge observation data based on the improved YOLOV5 neural network in an embodiment of the present invention;

[0075] Figure 4 This is the fused water gauge observation data result finally obtained in the embodiment of the present invention.

[0076] Reference numerals

[0077] 1-Data acquisition and preprocessing module, 10-Close-range image acquisition unit, 100-UAV, 101-Unmanned boat, 11-Ship draft depth data acquisition unit, 110-Underwater robot;

[0078] 2-image detection module, 20-image recognition unit, 21-water gauge symbol recognition unit;

[0079] 3-Dynamic weight fusion module. DETAILED DESCRIPTION

[0080] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the words "first" and "second" are used in the embodiments of the present invention to distinguish between identical or similar items with substantially the same functions and effects. For example, the first threshold and the second threshold are merely used to distinguish between different thresholds and do not limit their order. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0081] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0082] In the present invention, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The following at least one item (item) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (item) of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0083] This invention aims to address the low efficiency and high risk of traditional manual observation during ship draft gauge observation, as well as the weak anti-interference capability of single sensors and the limited observation capabilities in harsh environments. By doing so, it proposes an intelligent draft gauge observation method and system based on the collaboration of multiple unmanned devices. These devices include drones, unmanned boats, and underwater robots equipped with pressure sensors. The drones and unmanned boats collect images of the ship's long and short ranges and the draft gauge's close-up, while the underwater robot equipped with pressure sensors collects ship draft data. Combined with intelligent methods such as an improved YOLOV5 neural network, clustering algorithms, and Kalman filtering networks, this method achieves intelligent draft gauge observation with higher accuracy, lower cost, and greater resistance to environmental interference.

[0084] In the first aspect, the present invention provides a water gauge intelligent observation method based on the cooperation of multiple unmanned equipment, see Figure 1 ,include:

[0085] S1. Collecting ship's near and far images and ship's draft data;

[0086] As a possible implementation method, the ship's near- and far-sighted images are acquired by drones and unmanned boats; and / or the ship's draft data is acquired by an underwater robot attached to the bottom of the ship.

[0087] As an example, a drone and an unmanned boat simultaneously capture the real-time water level and water gauge scale at a ship's location from multiple angles, obtaining a series of images. The drone primarily captures distant images of the ship's location, i.e., water level images; the unmanned boat primarily captures close-up images of the ship's location, i.e., water gauge scale images. These images are then pre-processed using cropping and affine transformation to obtain high-precision near- and distant images of the ship:

[0088] For example, the formula for affine transformation is:

[0089]

[0090] Where (x, y) is the original coordinate, (x′, y′) is the new coordinate after affine transformation, a, b, c, d are the elements of linear transformation, t x , t y Represents the translation amount. Through affine transformation, the global water level line and water scale can be obtained.

[0091] As an example, the pressure sensor equipped with an underwater robot attached to the bottom of the ship is used to obtain the current draft data of the ship. The current draft data is denoised based on the autoencoder, and the encoding method is as follows:

[0092] z=f(x)=σ(W e x+b e ),

[0093] The decoding method is as follows:

[0094]

[0095] Where W e and W d is the weight matrix of the autoencoder-decoder, b e and b d is its corresponding bias, σ is the activation function, and the autoencoder represents the input data in a low-dimensional form through effective encoding and achieves noise reduction through decoding and reconstruction.

[0096] The present invention involves the coordinated cooperation of multiple unmanned equipment, making full use of the unmanned equipment in the port area, using drones and unmanned boats to collect distant and near-sighted images of ships and close-up images of water gauges, and using underwater robots equipped with pressure sensors to collect ship draft data. There is no need to install additional high-precision sensors, which solves the problems of difficult and costly maintenance of fixed sensors. In addition, the underwater robots are flexible and maneuverable, and can adapt to various environments and hulls, which solves the problem of limited observation in harsh environments and saves deployment and maintenance costs.

[0097] S2. Identify the water level data in the ship's near and far view images, as well as the corresponding water gauge symbol data, based on the improved YOLOV5 neural network;

[0098] As a possible implementation method, the improved YOLOV5 neural network specifically uses the adaptive activation function ACON instead of the default nonlinear activation function ReLU, which can adaptively choose whether to activate neurons and improve network accuracy; uses the weighted bidirectional feature pyramid network structure BiFPN instead of the path aggregation neural network PANet, making feature extraction more accurate; and adds a plug-and-play attention module CBAM.

[0099] As a possible implementation method, the adaptive activation function ACON is obtained using the following formula:

[0100] f(x)=p1·ReLU(x)+p2·ReLU(―x);

[0101] Among them, p1 and p2 are adaptive parameters, and the ReLU(x) function is expressed as:

[0102]

[0103] As a possible implementation method, the following formula is used to obtain the weighted bidirectional feature pyramid network structure BiFPN:

[0104]

[0105] in, and They are top-down and top-up feature maps respectively, w1 and w2 are self-learning weights.

[0106] As a possible implementation method, the plug-and-play attention module CBAM includes a channel attention module and a spatial attention module. The calculation process of the channel attention module is as follows:

[0107]

[0108] f max =max i,j F c,i,j , c=1,2,…,C;

[0109]

[0110] Where, f avg represents the global average pooling result, f max represents the global maximum pooling result, f represents the channel descriptor obtained by vector splicing, F represents the input image feature map, C represents the number of channels, H and W represent the height and width of the feature map respectively, represents a two-dimensional real vector space of dimension 2C;

[0111] The channel weights are generated by a shared multi-layer perceptron MLP as follows:

[0112] m c =σ(W2·δ(W1·f+b1)+b2),

[0113] F out =m⊙F;

[0114] Where m c represents the channel weight, F out represents the feature map weighted by the channel weight, W1 and W2 represent the weight matrices of MLP, b1 and b2 represent the bias terms, ⊙ represents the dot product operation, and σ represents the activation function;

[0115] The calculation process of the spatial attention module is as follows:

[0116]

[0117] f′ max =max c F out,c ,

[0118] F s =Concat(f′ avg ,f′ max );

[0119] Where f′ avg represents the channel average pooling result, f′ max Indicates the channel maximum pooling result, F s Represents the two-dimensional feature map obtained by channel splicing;

[0120] F s The spatial attention map is obtained through a convolutional layer, and the spatial attention map is combined with the feature map weighted by the channel weight to obtain the final output F final :

[0121] m spatial =σ(Conv(F s )),

[0122] F final =m spatial ⊙F out ;

[0123] Where, F final Represents the final output, m spatial Represents the two-dimensional feature map F s The spatial attention map F is obtained through a convolutional layer. out Represents a feature map weighted by channel weights. By adding spatial attention modules and channel attention modules to extract features, the feature extraction accuracy is further improved.

[0124] S3. Use a clustering algorithm to optimize the errors of the water level data and water gauge symbol data, filter out invalid prediction box information, and calculate and obtain preliminary water gauge observation data;

[0125] As an example, a random search clustering algorithm (CLARANS algorithm) is used to optimize the error of water level data and water gauge symbol data. The process is as follows:

[0126] Use Euclidean distance to measure the similarity or difference between data points:

[0127]

[0128] The clustering cost of each core point is calculated as the sum of the distances between the core point and all the data points assigned to it. Assume that the current core point set is M = {m1,m2,...,m k}, the total cost C of clustering can be expressed as:

[0129]

[0130] Where C j is assigned to the core point m j A set of data points. d(p,m j ) is the data point p and the core point mj When performing a neighborhood search, for each candidate core point m', the new clustering cost C(M') is compared with the current clustering cost C(M). If the new cost is lower, the current core point set M is replaced with the new core point set M' until the iteration condition is met.

[0131] The water level line data and water gauge symbol data after error optimization are calculated to obtain the preliminary water gauge observation data. The specific calculation is as follows:

[0132]

[0133] Where D represents the preliminary water gauge observation data, L w Indicates the distance between the water level line and the bottom border of the character above it, L n Indicates character height, L d It represents the distance between characters.

[0134] The present invention performs preprocessing such as cropping and affine transformation on the collected images, and then uses the improved YOLOV5 neural network to train the image data, so that the neural network can better extract image features. Thereafter, a clustering algorithm is used to optimize the errors of the water level line data and the water gauge symbol data, and invalid prediction box information is filtered out, thereby achieving more accurate recognition of the water level line data and the water gauge symbol data.

[0135] S4. A Kalman filter network is used to dynamically assign weights to the ship's draft data and preliminary draft observation data, and weightedly fuse the ship's draft data and preliminary draft observation data to obtain fused draft observation data. The Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network.

[0136] As a possible implementation method, the real-time state estimation network is built based on the traditional Kalman filter process. The noise of the ship draft data and the preliminary water gauge observation data is eliminated by learning the Kalman gain. The prediction process includes the following steps:

[0137]

[0138] P k|k―1 =FP k―1|k―1 F T +Q;

[0139] The update process is as follows:

[0140] K k =P k|k―1 H k T (H k P k|k―1 H kT +R) ―1 ,

[0141]

[0142] P k|k =(I-K k H k )P k|k―1 ;

[0143] Where, represents the prior estimate of the fused water gauge observation data at time k, represents the posterior estimate of the fused water gauge observation data at time k, z k is the observation value at time k, namely the ship draft data and preliminary water gauge observation data, F represents the state transfer matrix, H k represents the measurement matrix at time k, Q and R represent the Kalman filter noise, P k|k represents the posterior estimated covariance matrix, P k|k―1 represents the prior estimated covariance matrix, K k represents the Kalman gain, and I represents the identity matrix.

[0144] As a possible implementation method, the dynamic weight fusion network is constructed by setting the real-time state estimation network measurement matrix H k Implementing multi-source data fusion and weight distribution includes the following sub-steps:

[0145] S40. Initialize the measurement matrix H k =I;

[0146] S41. Use the prior estimate to multiply the measurement matrix to obtain the predicted observation value Use the ship draft data at time k and the preliminary water gauge observation data to obtain the observation value z k Subtract the predicted observations The difference obtained As network input features;

[0147] S42. Use GRU network to measure the matrix H k To update, follow these steps:

[0148] The difference Δz k After feature extraction by the fully connected layer, it is sent to the GRU network:

[0149] out FC =FC([Δz k ]);

[0150] The calculation process of the GRU network is as follows:

[0151] r t=σ(W r [h t―1 ,out FC ]+b r );

[0152] z t =σ(W z [h t―1 ,out FC ]+b z );

[0153]

[0154] Where r t Represents the reset gate output, z t represents the update gate output, and h t Denote candidate hidden states and final hidden states respectively, W r and W z Represent the weight of the reset gate output and the weight of the update gate output, b r and b z Represents the bias of the reset gate output and the bias of the update gate output, ⊙ represents the dot product operation, W h represents the weight of the candidate hidden state, b h represents the bias of the candidate hidden state;

[0155] GRU obtains the final hidden state by resetting the gate output and updating the gate output to control the storage and forgetting of information. After the final hidden state is scaled by the fully connected layer, the measurement matrix H to be updated is obtained. k as follows:

[0156] H k =FC(h t );

[0157] Among them, FC represents the fully connected layer.

[0158] See also Figures 3 and 4 , which is the recognition result obtained in the actual application of the present invention. Specifically, UAVs and unmanned boats are used to collect long-range and short-range images of ships. The improved YOLOV5 neural network is used to recognize the water level data and water gauge symbol data of the ships in the images, and the preliminary water gauge observation data is calculated. Figure 3 The recognition results of the preliminary water gauge observation data are shown. The fusion observation results based on the Kalman filter network were further evaluated. Specifically, the ship draft data collected by the underwater robot and the preliminary water gauge observation data were dynamically weighted and fused to obtain fused water gauge observation data. The fused water gauge observation data was then mapped back into the image space. See Table 1 for the comparison results of the fused water gauge observation data and the manual observation data for 40 frames of images:

[0159] Table 1 Comparison results of fused water gauge observation data and manual observation data in 40 frames of images

[0160]

[0161]

[0162] Figure 4 This is the final result of the fused water gauge observation data. By comparison, it can be seen that, compared with the original YOLOV5 neural network, the data recognition accuracy based on the improved YOLOV5 neural network is improved from 90% to 99%, the average accuracy is improved from 77% to 85%, the recall rate is improved from 94% to 98%, and the number of prediction boxes with a confidence score above 0.5 is significantly increased. By taking manual observations as the true value, the average absolute error of the original YOLOV5 neural network is calculated to be 0.0532m, and the average absolute error of the improved YOLOV5 neural network is 0.0345m. In addition, the present invention reduces the fluctuation phenomenon of the water gauge observation data caused by wind and waves. The calculated absolute error of the ship draft depth data is 0.0232m, and the average absolute value error of the fused water gauge observation data is 0.0134m. The experimental results confirm that the present invention overcomes the problem of limited accuracy of single modal data and achieves better performance than a single visual solution or sensor measurement solution.

[0163] The present invention proposes a multi-source data fusion based on a Kalman filter network, which combines the advantages of sensor measurement and visual algorithm detection, overcomes the problem of limited accuracy of single-modal perception, and can obtain more accurate and interference-resistant observation results compared with traditional single-modal-based technical solutions and solutions relying on visual algorithms.

[0164] In the second aspect, the present invention provides a water gauge intelligent observation system based on the cooperation of multiple unmanned equipment, see Figure 2 , including: data acquisition and preprocessing module 1, image detection module 2 and dynamic weight fusion module 3;

[0165] The data acquisition and preprocessing module 1 is used to obtain the ship's long and short-range images and the ship's draft depth data;

[0166] The image detection module 2 is used to identify the water level data in the ship's long and short-range images, as well as the water gauge symbol data corresponding to the water level data, and to optimize the errors of the water level data and the water gauge symbol data to obtain preliminary water gauge observation data;

[0167] The dynamic weight fusion module 3 is used to dynamically assign weights to the ship's draft data and the preliminary draft observation data, and weightedly fuse the ship's draft data and the preliminary draft observation data to obtain fused draft observation data.

[0168] As a possible implementation, see Figure 2 The data acquisition and preprocessing module 1 includes a ship near- and far-field image acquisition unit 10 and a ship draft depth data acquisition unit 11. The ship near- and far-field image acquisition unit 10 includes a drone 100 and an unmanned boat 101 for acquiring near- and far-field images of the ship. The ship draft depth data acquisition unit 11 includes an underwater robot 110 attached to the bottom of the ship for acquiring ship draft depth data.

[0169] The image detection module 2 includes an image recognition unit 20 and a water gauge symbol recognition unit 21. The image recognition unit 20 is used to recognize water level line data in the long and short view images of the ship, and the water gauge symbol recognition unit 21 is used to output water gauge symbol data corresponding to the water level line.

[0170] The present invention cooperates with multiple unmanned equipment, fully utilizes the unmanned equipment in the port area, adopts drones and unmanned boats to collect distant and near-sighted images of ships and close-up images of water gauges, adopts underwater robots equipped with pressure sensors to collect ship draft data, and then combines improved YOLOV5 neural network, clustering algorithm, Kalman filter network and other intelligent means to perform fusion observation. There is no need to install additional high-precision sensors, which solves the problem of difficult maintenance and high maintenance cost of fixed sensors. In addition, the underwater robot is flexible and maneuverable, can adapt to various environments and hulls, solves the problem of limited observation in harsh environments, and saves deployment and maintenance costs.

[0171] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0172] Although the present invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof may be made without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. It will be apparent that various modifications and variations of the present invention may be made by those skilled in the art without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such modifications and variations as fall within the scope of the invention and its equivalents.

Claims

1. A water gauge intelligent observation method based on the collaboration of multiple unmanned equipment, characterized in that: include: S1. Collecting ship's near- and far-field images and ship's draft depth data; S2. Identify the water level data in the vessel's near- and far-field images, as well as the corresponding water gauge symbol data, based on an improved YOLOV5 neural network. Specifically, the improved YOLOV5 neural network employs the adaptive activation function ACON instead of the default nonlinear activation function ReLU; employs the weighted bidirectional feature pyramid network structure BiFPN instead of the path aggregation neural network PANet; and incorporates a plug-and-play attention module (CBAM). The CBAM includes a channel attention module and a spatial attention module. The computational process of the channel attention module is as follows: Where, represents the global average pooling result, Represents the global maximum pooling result, Represents the channel descriptor obtained by vector concatenation, represents the input image feature map, Indicates the number of channels, and Represent the height and width of the feature map respectively, represents a two-dimensional real vector space of dimension 2C; The channel weights are generated by a shared multi-layer perceptron MLP as follows: Where, represents the channel weight, represents the feature map weighted by channel weights, and represents the weight matrix of MLP, and represents the bias term, represents the dot product operation, represents the activation function; The calculation process of the spatial attention module is as follows: Where, represents the channel average pooling result, Represents the channel maximum pooling result, Represents the two-dimensional feature map obtained by channel splicing; Will A spatial attention map is obtained through a convolutional layer, and the spatial attention map is combined with the feature map weighted by the channel weights to obtain the final output : Where, Represents the final output, Represents a two-dimensional feature map A spatial attention map is obtained through a convolutional layer, Represents the feature map weighted by channel weights; S3. Using a clustering algorithm to optimize the water level data and water gauge symbol data, filter out invalid prediction box information, and calculate preliminary water gauge observation data; S4. Using a Kalman filter network to dynamically assign weights to the ship's draft data and preliminary water gauge observation data, weighted fusion of the ship's draft data and preliminary water gauge observation data is performed to obtain fused water gauge observation data; The Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network.

2. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The real-time state estimation network is built based on the traditional Kalman filtering process, and eliminates the noise of the ship draft data and the preliminary water gauge observation data by learning the Kalman gain, including the prediction process and the update process. The prediction process is specifically as follows: ， ； The update process is specifically as follows: Where, express k The prior estimation of water gauge observation data is integrated at all times. express k The posterior estimation of the water gauge observation data is integrated at all times. for k The observation values at the moment, namely the ship's draft data and preliminary water gauge observation data, represents the state transition matrix, express k The measurement matrix at time, and represents the Kalman filter noise, represents the posterior estimated covariance matrix, represents the prior estimated covariance matrix, represents the Kalman gain, Represents the identity matrix.

3. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 2 is characterized in that: The dynamic weight fusion network is constructed by setting the real-time state estimation network measurement matrix Implementing multi-source data fusion and weight distribution includes the following sub-steps: S40. Initialize the measurement matrix ; S41. Use the prior estimate to multiply the measurement matrix to obtain the predicted observation value ,use k Observation values of ship draft data and preliminary water gauge observation data at the moment Subtract the predicted observations The difference obtained As network input features; S42. Use GRU network to measure the matrix To update, follow these steps: The difference After feature extraction by the fully connected layer, it is sent to the GRU network: The calculation process of the GRU network is as follows: ； ； ； ； Where, Represents the reset gate output, represents the update gate output, and represent the candidate hidden state and the final hidden state respectively, and Represent the weight of the reset gate output and the weight of the update gate output, respectively. and Represent the bias of the reset gate output and the bias of the update gate output, respectively. represents the dot product operation, represents the weight of the candidate hidden state, represents the bias of the candidate hidden state; GRU obtains the final hidden state by resetting the gate output and updating the gate output to control the storage and forgetting of information. After the final hidden state is scaled by the fully connected layer, the measurement matrix to be updated is obtained. as follows: in, represents a fully connected layer.

4. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The adaptive activation function ACON is obtained using the following formula: in, and is the adaptive parameter, The function is expressed as:

5. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The weighted bidirectional feature pyramid network structure BiFPN is obtained using the following formula: in, and are top-down and top-up feature maps respectively, and is the self-learning weight.

6. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The ship's near and far-field images are acquired by a drone and an unmanned boat; and / or the ship's draft data are acquired by an underwater robot attached to the bottom of the ship.

7. A water gauge intelligent observation system based on the collaboration of multiple unmanned equipment, characterized in that: The water gauge intelligent observation system applies the water gauge intelligent observation method based on collaboration of multiple unmanned equipment as described in any one of claims 1 to 6.

Images