Water gauge intelligent observation method and system based on cooperation of multiple unmanned devices

Abstract

The invention relates to the technical field of water gauge observation, and discloses a water gauge intelligent observation method and system based on cooperation of multiple unmanned devices. The observation method comprises the following steps: acquiring ship far and near scene images and ship draft data; identifying water line data in the far and near scene images of the ship and water gauge symbol data corresponding to the water line data based on an improved YOLOV5 neural network; performing error optimization on the water line data and the water gauge symbol data by adopting a clustering algorithm to obtain initial water gauge observation data; dynamically allocating weights to the ship draft data and the preliminary water gauge observation data by adopting a Kalman filtering network, and performing weighted fusion on the ship draft data and the preliminary water gauge observation data to obtain fused water gauge observation data; the Kalman filtering network comprises a real-time state estimation network and a dynamic weight fusion network. The problems that a traditional manual water gauge is low in observation efficiency and high in risk, and an existing single-sensor water gauge is poor in anti-interference capacity, limited in severe environment observation and the like can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of water gauge observation, and in particular to an intelligent water gauge observation method and system based on the cooperation of multiple unmanned devices. Background Art

[0002] Water gauge weighing is currently the main method for measuring the loading capacity of cargo ships in port shipping. Accurate measurement of the draft of cargo ships is very important for safe navigation and port cargo management. The traditional manual water gauge observation method has low efficiency, and the observation accuracy is greatly affected by human subjective factors. Under bad weather conditions, the draft line fluctuates greatly due to wind and waves. Manual observation is dangerous to some extent and the accuracy is difficult to guarantee. In order to address the defects of manual observation, most early solutions were based on high-precision sensors to achieve remote real-time observation. The draft depth of cargo ships was read through radar, sonar, pressure sensors, etc. Among them, pressure sensors were the most widely used. By performing proportional operations on the data collected by the pressure sensors installed at the bottom of the cargo ship, the influence caused by water density and gravitational acceleration was eliminated, and relatively accurate draft data was obtained.

[0003] However, these sensors are located underwater for a long time and need to be maintained regularly. Equipment maintenance is difficult and the cost is relatively high. Moreover, a single sensor is easily affected by the complex underwater environment. Therefore, unmanned and low-cost vision detection solutions have been proposed. Through equipment such as cameras and drones, on-site videos and images are collected in real time, and then the automatic recognition of water gauge readings can be realized through image processing algorithms such as edge detection or vision algorithms such as YOLO neural network. However, these methods are often limited by the port environment. Under bad weather, it is difficult to collect ship images, the image quality is difficult to guarantee, and the algorithm accuracy is limited.

[0004] In summary, the existing water gauge observation technologies have problems such as low efficiency of manual observation, low accuracy of single-modal sensors, high maintenance costs, and vision algorithms being easily affected by the environment. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent water gauge observation method and system based on the cooperation of multiple unmanned devices. An unmanned aerial vehicle and an unmanned boat are used to collect the long-distance view of the ship and the close-up view of the water gauge. Based on the improved YOLOv5 neural network, the water level line data and water gauge symbol data are detected and recognized. At the same time, an underwater robot equipped with a pressure sensor is combined to collect the draft depth data of the ship, and a real-time state estimation network and a dynamic weight fusion network are used to obtain the water gauge observation data. The present invention is jointly cooperated by multiple unmanned devices, which can solve the problems of low efficiency and high risk of manually observing water gauge data, as well as the weak anti-interference ability of the existing water gauge observation methods using single sensors and the difficulty in applying them to harsh environments.

[0006] To achieve the above object, the technical solution provided by the present invention is as follows:

[0007] In a first aspect, the present invention provides an intelligent water gauge observation method based on multi-unmanned equipment cooperation, including:

[0008] S1. Collect the near and far view images of the ship and the ship draft depth data;

[0009] S2. Based on the improved YOLOV5 neural network, identify the water level line data in the near and far view images of the ship and the water gauge symbol data corresponding to the water level line data;

[0010] S3. Use a clustering algorithm to optimize the errors of the water level line data and the water gauge symbol data, filter out invalid prediction box information, and calculate and obtain preliminary water gauge observation data;

[0011] S4. Use a Kalman filter network to dynamically assign weights to the ship draft depth data and the preliminary water gauge observation data, and perform weighted fusion of the ship draft depth data and the preliminary water gauge observation data to obtain fused water gauge observation data; the Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network.

[0012] As a possible implementation, the real-time state estimation network is built based on the traditional Kalman filter process, and eliminates the noise of the ship draft depth data and the preliminary water gauge observation data by learning the Kalman gain, including a prediction process and an update process. The prediction process is specifically:

[0013]

[0014] P k|k―1 = FP k―1|k―1 F T + Q;

[0015] The update process is specifically:

[0016] K k = P k|l―1 H k T (H k P k|k―1 H k T + R) ―1 ,

[0017]

[0018] P k|k = (I ― K k H k )P k|k―1 ;

[0019] In the formula, represents the prior estimate of the fused water gauge observation data at time k, Denote the posterior estimate of the fused water gauge observation data at time k as z k is the observation value at time k, i.e., the ship draft data and the preliminary water gauge observation data. F represents the state transition matrix, and H k represents the measurement matrix at time k. Q and R represent the Kalman filter noise, and P k|k represents the posterior estimate covariance matrix, and P k|k―1 represents the prior estimate covariance matrix, and K k represents the Kalman gain, and I represents the identity matrix.

[0020] As a possible implementation, the dynamic weight fusion network realizes multi-source data fusion and weight allocation by setting the measurement matrix H of the real-time state estimation network k and includes the following sub-steps:

[0021] S40. Initialize the measurement matrix H k = I;

[0022] S41. Multiply the prior estimate by the measurement matrix to obtain the predicted observation value Use the ship draft data and the preliminary water gauge observation data at time k to obtain the observation value z k Subtract the predicted observation value The obtained difference is used as the network input feature;

[0023] S42. Update the measurement matrix H using a GRU network k The update method is as follows:

[0024] Send the difference Δz k through the fully connected layer for feature extraction and then into the GRU network:

[0025] out FC = FC([Δz k );

[0026] The calculation process of the GRU network is as follows:

[0027] r t = σ(W r [h t―1 , out FC + b r );

[0028] z t = σ(W z [h t―1 , out FC + b z );

[0029]

[0030] In the formula, r t represents the reset gate output, z t represents the update gate output, and h t represent the candidate hidden state and the final hidden state respectively, W r and W z represent the weights of the reset gate output and the update gate output respectively, b r and b z represent the biases of the reset gate output and the update gate output respectively, ⊙ represents the dot product operation, W h represents the weight of the candidate hidden state, b h represents the bias of the candidate hidden state.

[0031] The GRU controls the storage and forgetting of information through the reset gate output and the update gate output, obtains the final hidden state, and after scaling the final hidden state through the fully connected layer, obtains the measurement matrix Hk to be updated as follows:

[0032] H k = FC(h t );

[0033] where FC represents the fully connected layer.

[0034] As a possible implementation, the improved YOLOV5 neural network is specifically: using the adaptive activation function ACON to replace the default non-linear activation function ReLU; using the weighted bidirectional feature pyramid network structure BiFPN to replace the path aggregation neural network PANet; adding the plug-and-play attention module CBAM.

[0035] As a possible implementation, the adaptive activation function ACON is obtained by the following formula:

[0036] f(x) = p 1 ·ReLU(x) + p 2 ·ReLU(―x);

[0037] where p 1 and p 2 are adaptive parameters, and the ReLU(x) function is expressed as:

[0038]

[0039] As a possible implementation, the weighted bidirectional feature pyramid network structure BiFPN is obtained by the following formula:

[0040]

[0041] where, and Feature maps that are top - down and top - up respectively, w 1 and w 2 are self - learning weights.

[0042] As a possible implementation, the plug - and - play attention module CBAM includes a channel attention module and a spatial attention module. The calculation process of the channel attention module is as follows:

[0043]

[0044] f max = max i,j F c,i,j , c = 1, 2, …, C;

[0045]

[0046] In the formula, f avg represents the global average pooling result, f max represents the global maximum pooling result, f represents the channel descriptor obtained by vector concatenation, F represents the input image feature map, F represents the number of channels, H and W represent the height and width of the feature map respectively, represents a two - dimensional real vector space of dimension 2C;

[0047] Generate the channel weights through a shared multi - layer perceptron MLP as follows:

[0048] m c = σ(W 2 ·δ(W 1 ·f + b 1 ) + b 2 ),

[0049] F out = m ⊙ F;

[0050] In the formula, m c represents the channel weights, F out represents the feature map weighted by the channel weights, W 1 and W 2 represent the weight matrices of the MLP, b 1 and b 2 represent the bias terms, ⊙ represents the dot - product operation, and σ represents the activation function;

[0051] The calculation process of the spatial attention module is as follows:

[0052]

[0053] f′ max = max c F out,c ,

[0054] F s = Concat(f′ avg , f′ max );

[0055] Wherein, f′ avg represents the channel average pooling result, f′ max represents the channel maximum pooling result, and F s represents the two-dimensional feature map obtained through channel concatenation;

[0056] The F s is passed through a convolutional layer to obtain a spatial attention map, and the spatial attention map is combined with the feature map weighted by the channel weights to obtain the final output F final :

[0057] M spatial = σ(Conv(F s ))

[0058] F final = m spatial ⊙ F out ;

[0059] Wherein, F final represents the final output, and m spatial represents the two-dimensional feature map F s is passed through a convolutional layer to obtain a spatial attention map, and F out represents the feature map weighted by the channel weights.

[0060] As a possible implementation, the ship's far and near view images are obtained by drones and unmanned boats; and / or, the ship's draft depth data is obtained by underwater robots adsorbed on the bottom of the ship.

[0061] In a second aspect, the present invention provides a water gauge intelligent observation system based on multi-unmanned equipment cooperation, including: a data acquisition and preprocessing module, an image detection module, and a dynamic weight fusion module;

[0062] The data acquisition and preprocessing module is used to obtain the ship's far and near view images and the ship's draft depth data;

[0063] The image detection module is used to identify the water level line data in the ship's far and near view images, as well as the water gauge symbol data corresponding to the water level line data, and optimize the errors of the water level line data and the water gauge symbol data to obtain preliminary water gauge observation data;

[0064] The dynamic weight fusion module is used to dynamically allocate weights to the ship's draft depth data and the preliminary water gauge observation data, and weighted-fuse the ship's draft depth data and the preliminary water gauge observation data to obtain the fused water gauge observation data.

[0065] As a possible implementation, the data acquisition and preprocessing module includes a ship long-range and short-range image acquisition unit and a ship draft depth data acquisition unit. The ship long-range and short-range image acquisition unit includes an unmanned aerial vehicle and an unmanned boat, which are used to acquire ship long-range and short-range images. The ship draft depth data acquisition unit includes an underwater robot adsorbed on the bottom of the ship, which is used to output ship draft depth data;

[0066] The image detection module includes an image recognition unit and a water gauge symbol recognition unit. The image recognition unit is used to recognize the water level line data in the ship long-range and short-range images, and the water gauge symbol recognition unit is used to output the water gauge symbol data corresponding to the water level line.

[0067] Compared with the prior art, the beneficial effects produced by the present invention are as follows:

[0068] 1. The intelligent water gauge observation system based on multi-unmanned equipment cooperation provided by the present invention is cooperated by multiple unmanned equipment, makes full use of the unmanned equipment in the port area, uses an unmanned aerial vehicle and an unmanned boat to acquire ship long-range and short-range and water gauge close-range images, uses an underwater robot equipped with a pressure sensor to acquire ship draft depth data, and then combines intelligent means such as an improved YOLOV5 neural network, a clustering algorithm, and a Kalman filter network for fusion observation. There is no need to install high-precision sensors additionally, which solves the problems of difficult maintenance and high maintenance cost of fixed sensors; moreover, the underwater robot is flexible and mobile, can adapt to various environments and hulls, solves the problem of limited observation under harsh environments, and also saves deployment and maintenance costs.

[0069] 2. The intelligent water gauge observation method based on multi-unmanned equipment cooperation provided by the present invention proposes multi-source data fusion based on the Kalman filter network, combines the advantages of sensor measurement and visual algorithm detection, overcomes the problem of limited accuracy of single-modal perception, and can obtain more accurate and stronger anti-interference observation results compared with traditional single-modal-based technical solutions and solutions relying on visual algorithms.

[0070] 3. The intelligent water gauge observation method based on multi-unmanned equipment cooperation provided by the present invention preprocesses the acquired images such as cropping and affine transformation, and trains the image data using an improved YOLOV5 neural network, enabling the neural network to better extract image features and realizing more accurate recognition of water level line data and water gauge symbol data. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0072] Figure 1Flowchart of the intelligent water gauge observation method based on multi-unmanned equipment cooperation in the embodiments of the present invention;

[0073] Figure 2 Schematic diagram of the intelligent water gauge observation system based on multi-unmanned equipment cooperation in the embodiments of the present invention;

[0074] Figure 3 Recognition result of the preliminary water gauge observation data based on the improved YOLOV5 neural network in the embodiments of the present invention;

[0075] Figure 4 Final fusion water gauge observation data result obtained in the embodiments of the present invention.

[0076] Reference numerals

[0077] 1 - Data acquisition and preprocessing module, 10 - Close-range image acquisition unit, 100 - UAV, 101 - Unmanned boat, 11 - Ship draft depth data acquisition unit, 110 - Underwater robot;

[0078] 2 - Image detection module, 20 - Image recognition unit, 21 - Water gauge symbol recognition unit;

[0079] 3 - Dynamic weight fusion module. Detailed implementation manners

[0080] For the convenience of clearly describing the technical solutions in the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and no limitation is imposed on their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0081] It should be noted that in the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0082] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the relationship between related objects and indicates that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. The following at least one (item) or its similar expression refers to any combination of these items, including any combination of single (item) or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b, and c, where a, b, and c can be single or multiple.

[0083] The present invention aims to solve the problems in the process of ship draft observation, such as low efficiency and high risk of traditional manual observation, weak anti-interference ability of a single sensor, and limited observation in harsh environments. A method and system for intelligent draft observation based on the cooperation of multiple unmanned equipment are proposed. The multiple unmanned equipment includes unmanned aerial vehicles, unmanned boats, and underwater robots equipped with pressure sensors. The unmanned aerial vehicle and the unmanned boat collect the long-distance and short-distance views of the ship and the close-up image of the draft, and the underwater robot equipped with a pressure sensor collects the ship's draft depth data. Combining intelligent means such as the improved YOLOV5 neural network, clustering algorithm, and Kalman filter network, intelligent draft observation with higher accuracy, lower cost, and stronger anti-environment interference ability is realized.

[0084] In a first aspect, the present invention provides a method for intelligent draft observation based on the cooperation of multiple unmanned equipment, see Figure 1 , including:

[0085] S1. Collect the long-distance and short-distance views of the ship and the ship's draft depth data;

[0086] As a possible implementation, the long-distance and short-distance views of the ship are collected by the unmanned aerial vehicle and the unmanned boat; and / or, the ship's draft depth data is collected by the underwater robot adsorbed on the bottom of the ship;

[0087] As an example, the real-time water level and draft scale of the ship position at the same time are photographed from multiple angles by the unmanned aerial vehicle and the unmanned boat to obtain a series of photographed images. The unmanned aerial vehicle mainly photographs the long-distance view of the ship position, that is, the water level line image; the unmanned boat mainly photographs the close-up image of the ship position, that is, the draft scale image. The obtained series of photographed images are preprocessed, such as cropping and affine transformation, to obtain high-precision long-distance and short-distance views of the ship:

[0088] Exemplarily, the formula for affine transformation is:

[0089]

[0090] Wherein, (x, y) are the original coordinates, (x′, y′) are the new coordinates after affine transformation, a, b, c, d are the elements of the linear transformation, and t x and t y represent the translation amount. Through affine transformation, the global water level line and the water gauge scale can be obtained.

[0091] As an example, a pressure sensor equipped on an underwater robot adsorbed to the bottom of the ship is used to obtain the current draft depth data of the ship, and the current draft depth data is denoised based on an autoencoder. The encoding method is as follows:

[0092] z = f(x) = σ(W e x + b e ),

[0093] The decoding method is as follows:

[0094]

[0095] where W e and W d are the weight matrices of the autoencoder, b e and b d are the corresponding biases, σ is the activation function, and the autoencoder performs low-dimensional representation of the input data through effective encoding and realizes noise reduction through decoding and reconstruction.

[0096] The present invention is cooperated by multiple unmanned equipment, makes full use of the unmanned equipment in the port area, uses drones and unmanned boats to collect the long-distance and short-distance views of the ship and the close-up view of the water gauge, and uses an underwater robot equipped with a pressure sensor to collect the draft depth data of the ship. There is no need to install high-precision sensors additionally, which solves the problems of difficult maintenance and high maintenance cost of fixed sensors; moreover, the underwater robot is flexible and mobile, can adapt to various environments and hulls, solves the problem of limited observation under harsh environments, and also saves the deployment and maintenance costs.

[0097] S2. Based on the improved YOLOV5 neural network, identify the water level line data in the long-distance and short-distance views of the ship and the water gauge symbol data corresponding to the water level line data;

[0098] As a possible implementation manner, the improved YOLOV5 neural network is specifically: using the adaptive activation function ACON to replace the default non-linear activation function ReLU, which can adaptively select whether to activate neurons and improve the network accuracy; using the weighted bidirectional feature pyramid network structure BiFPN to replace the path aggregation neural network PANet, making the accuracy of feature extraction higher; adding the plug-and-play attention module CBAM.

[0099] As a possible implementation manner, the adaptive activation function ACON is obtained by the following formula:

[0100] f(x) = p 1 ·ReLU(x) + p 2 ·ReLU(−x);

[0101] Wherein, p 1 and p 2 are adaptive parameters, and the ReLU(x) function is expressed as:

[0102]

[0103] As a possible implementation, the weighted bidirectional feature pyramid network structure BiFPN is obtained by using the following formula:

[0104]

[0105] Wherein, and are the top-down and bottom-up feature maps respectively, and w 1 and w 2 are self-learning weights.

[0106] As a possible implementation, the plug-and-play attention module CBAM includes a channel attention module and a spatial attention module. The calculation process of the channel attention module is as follows:

[0107]

[0108] f max = max i,j F c,i,j , c = 1, 2, …, C;

[0109]

[0110] In the formula, f avg represents the global average pooling result, f max represents the global maximum pooling result, f represents the channel descriptor obtained by vector concatenation, F represents the input image feature map, C represents the number of channels, H and W represent the height and width of the feature map respectively, represents a two-dimensional real vector space of dimension 2C;

[0111] The channel weights are generated by a shared multi-layer perceptron MLP as follows:

[0112] m c = σ(W 2 ·δ(W 1 ·f + b 1 ) + b 2 ),

[0113] Fout = m⊙F;

[0114] In the formula, m c represents the channel weight, and F out represents the feature map weighted by the channel weight. W 1 and W 2 represent the weight matrices of the MLP, and b 1 and b 2 represent the bias terms. ⊙ represents the dot product operation, and σ represents the activation function;

[0115] The calculation process of the spatial attention module is as follows:

[0116]

[0117] f′ max = max c F out,c ,

[0118] F s = Concat(f′ avg , f′ max );

[0119] In the formula, f′ avg represents the result of channel average pooling, f′ max represents the result of channel max pooling, and F s represents the two-dimensional feature map obtained by channel concatenation;

[0120] Pass F s through a convolutional layer to obtain the spatial attention map, and combine the spatial attention map with the feature map weighted by the channel weight to obtain the final output F final :

[0121] m spatial = σ(Conv(F s ))

[0122] F final = m spatial ⊙F out ;

[0123] In the formula, F final represents the final output, and m spatial represents that the two-dimensional feature map F s passes through a convolutional layer to obtain the spatial attention map, and F out represents the feature map weighted by the channel weight. By adding the spatial attention module and the channel attention module to extract features, the feature extraction accuracy is further improved.

[0124] S3. Use a clustering algorithm to optimize the errors of the water level line data and the water gauge symbol data, filter out the invalid prediction box information, and calculate and obtain the preliminary water gauge observation data;

[0125] As an example, use the CLARANS algorithm, a clustering algorithm of random search, to optimize the errors of the water level line data and the water gauge symbol data. The process is as follows:

[0126] Use the Euclidean distance to measure the similarity or difference between data points:

[0127]

[0128] The clustering cost of each core point is calculated as: the sum of the distances between this core point and all the data points assigned to it. Assume that the current set of core points is M = {m 1 , m 2 ,..., m k}, then the total clustering cost C can be expressed as:

[0129]

[0130] In the formula, C j is the set of data points assigned to the core point m j . d(p, m j ) is the distance between the data point p and the core point m j . When performing neighborhood search, for each candidate core point m', it is necessary to compare the new clustering cost C(M') and the current clustering cost C(M). If the new cost is lower, then replace the current set of core points M with the new set of core points M' until the iteration condition is met.

[0131] Calculate the preliminary water gauge observation data from the water level line data and the water gauge symbol data after error optimization. The specific calculation is as follows:

[0132]

[0133] Among them, D represents the preliminary water gauge observation data, L w represents the distance from the water level line to the lower border of the nearest character above it, L n represents the character height, and L d represents the distance between characters.

[0134] The present invention performs preprocessing such as cropping and affine transformation on the collected images, then uses an improved YOLOV5 neural network to train the image data to enable the neural network to better extract image features, and then uses a clustering algorithm to optimize the errors of the water level line data and the water gauge symbol data, filtering out the invalid prediction box information, and can realize more accurate recognition of the water level line data and the water gauge symbol data.

[0135] S4. Dynamically allocate weights to the ship draft depth data and the preliminary water gauge observation data using a Kalman filter network, and perform weighted fusion on the ship draft depth data and the preliminary water gauge observation data to obtain fused water gauge observation data; the Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network.

[0136] As a possible implementation, the real-time state estimation network is built based on the traditional Kalman filter process, and eliminates the noise of the ship draft depth data and the preliminary water gauge observation data by learning the Kalman gain, including a prediction process and an update process. The prediction process is specifically:

[0137]

[0138] P k|k―1 = FP k―1|k―1 F T + Q;

[0139] The update process is specifically:

[0140] K k = P k|k―1 H k T (H k P k|k―1 H k T + R) ―1 ,

[0141]

[0142] P k|k = (I - K k H k )P k|k―1 ;

[0143] In the formula, represents the prior estimate of the fused water gauge observation data at time k, represents the posterior estimate of the fused water gauge observation data at time k, z k is the observed value at time k, that is, the ship draft depth data and the preliminary water gauge observation data, F represents the state transition matrix, H k represents the measurement matrix at time k, Q and R represent the Kalman filter noise, P k|k represents the posterior estimate covariance matrix, P k|k―1 represents the prior estimate covariance matrix, K k represents the Kalman gain, and I represents the identity matrix.

[0144] As a possible implementation, the dynamic weight fusion network sets the measurement matrix H of the real-time state estimation networkk Implement multi-source data fusion and weight allocation, including the following sub-steps:

[0145] S40. Initialize the measurement matrix H k = I;

[0146] S41. Multiply the prior estimate by the measurement matrix to obtain the predicted observation value Use the draft depth data of the ship at time k and the preliminary water gauge observation data to obtain the observation value z k Subtract the predicted observation value The obtained difference is used as the network input feature;

[0147] S42. Use the GRU network to update the measurement matrix H k The update method is as follows:

[0148] Send the difference Δz k through the fully connected layer for feature extraction and then into the GRU network:

[0149] out FC = FC([Δz k )

[0150] The calculation process of the GRU network is as follows:

[0151] r t = σ(W r [h t―1 , out FC + b r )

[0152] z t = σ(W z [h t―1 , out FC + b z )

[0153]

[0154] In the formula, r t represents the output of the reset gate, z t represents the output of the update gate, and h t represent the candidate hidden state and the final hidden state respectively, W r and W z represent the weights of the reset gate output and the update gate output respectively, b r and b z represent the biases of the reset gate output and the update gate output respectively, ⊙ represents the dot product operation, W h represents the weight of the candidate hidden state, b hRepresents the bias of the candidate hidden state;

[0155] The GRU controls the storage and forgetting of information through the reset gate output and the update gate output, obtains the final hidden state, and after scaling the final hidden state through a fully connected layer, obtains the measurement matrix H to be updated k as follows:

[0156] H k = FC(h t );

[0157] where FC represents the fully connected layer.

[0158] See Figures 3 to 4 , which is the recognition result obtained in the actual application of the present invention. Specifically, a drone and an unmanned boat are used to collect the near and far view images of the ship, and based on the improved YOLOV5 neural network, the waterline data and water gauge symbol data of the ship in the image are recognized, and the preliminary water gauge observation data is calculated. Figure 3 shows the recognition result of the preliminary water gauge observation data. Further evaluate the fusion observation result based on the Kalman filter network. Specifically, the draft depth data of the ship collected by the underwater robot and the preliminary water gauge observation data are dynamically weighted and fused to obtain the fused water gauge observation data, and the fused water gauge observation data is mapped back to the image space. See Table 1, which records the comparison results of the fused water gauge observation data and the manual observation data in 40 frames of images:

[0159] Table 1 Comparison results of fused water gauge observation data and manual observation data in 40 frames of images

[0160]

[0161]

[0162] Figure 4 is the finally obtained result of the fused water gauge observation data. By comparison, compared with the original YOLOV5 neural network, the data recognition accuracy of the improved YOLOV5 neural network has been improved from 90% to 99%, the average precision has been improved from 77% to 85%, the recall rate has been improved from 94% to 98%, and the number of prediction boxes with a confidence score above 0.5 has increased significantly. By taking the manual observation as the ground truth, the average absolute error of the original YOLOV5 neural network is calculated to be 0.0532m, and the average absolute error of the improved YOLOV5 neural network is 0.0345m. Moreover, the present invention reduces the fluctuation phenomenon of the water gauge observation data caused by wind and waves. The absolute error of the draft depth data of the ship is calculated to be 0.0232m, and the average absolute value error of the fused water gauge observation data is 0.0134m. The experimental results confirm that the present invention overcomes the problem of limited accuracy of single-modal data and achieves better performance than single-vision or sensor measurement solutions.

[0163] The present invention proposes multi-source data fusion based on a Kalman filter network, which combines the advantages of sensor measurement and visual algorithm detection, overcomes the problem of limited accuracy of single-modal perception, and can obtain more accurate and stronger anti-interference observation results compared with traditional single-modal-based technical solutions and solutions relying on visual algorithms.

[0164] In a second aspect, the present invention provides a water gauge intelligent observation system based on the cooperation of multiple unmanned equipment. Refer to Figure 2 , including: a data acquisition and preprocessing module 1, an image detection module 2, and a dynamic weight fusion module 3;

[0165] The data acquisition and preprocessing module 1 is used to obtain the far and near view images of the ship and the ship draft depth data;

[0166] The image detection module 2 is used to identify the water level line data in the far and near view images of the ship, as well as the water gauge symbol data corresponding to the water level line data, and optimize the errors of the water level line data and the water gauge symbol data to obtain preliminary water gauge observation data;

[0167] The dynamic weight fusion module 3 is used to dynamically assign weights to the ship draft depth data and the preliminary water gauge observation data, and weight-fuse the ship draft depth data and the preliminary water gauge observation data to obtain fused water gauge observation data.

[0168] As a possible implementation, refer to Figure 2 , the data acquisition and preprocessing module 1 includes a far and near view image acquisition unit 10 of the ship and a ship draft depth data acquisition unit 11. The far and near view image acquisition unit 10 of the ship includes a drone 100 and an unmanned boat 101, which are used to acquire the far and near view images of the ship. The ship draft depth data acquisition unit 11 includes an underwater robot 110 adsorbed on the bottom of the ship, which is used to acquire the ship draft depth data;

[0169] The image detection module 2 includes an image recognition unit 20 and a water gauge symbol recognition unit 21. The image recognition unit 20 is used to identify the water level line data in the far and near view images of the ship, and the water gauge symbol recognition unit 21 is used to output the water gauge symbol data corresponding to the water level line.

[0170] The present invention is coordinated by multiple unmanned devices, making full use of the unmanned devices in the port area. It uses drones and unmanned boats to collect the long-distance and close-up images of ships and the close-up images of draft marks, and uses an underwater robot equipped with a pressure sensor to collect the ship's draft depth data. Then, through intelligent means such as an improved YOLOV5 neural network, clustering algorithm, and Kalman filter network for fusion observation, there is no need to install additional high-precision sensors, solving the problems of difficult maintenance and high maintenance costs of fixed sensors. Moreover, the underwater robot is flexible and maneuverable, can adapt to various environments and hulls, solves the problem of limited observation in harsh environments, and also saves deployment and maintenance costs.

[0171] Although the present invention has been described in connection with various embodiments, however, in the process of implementing the claimed invention, those skilled in the art can understand and achieve other variations of the disclosed embodiments by viewing the drawings, the disclosure content, and the like. In the specification, the term "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality of cases. A single processor or other unit can implement several functions listed in the specification. Certain measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0172] Although the present invention has been described in connection with specific features and their embodiments, it is obvious that various modifications and combinations can be made without departing from the spirit and scope of the present invention. Accordingly, the present specification and the drawings are only exemplary descriptions of the present invention and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A water gauge intelligent observation method based on the collaboration of multiple unmanned equipment, characterized in that: include: S1. Collect ship's near and far images and ship's draft depth data; S2. Identify the water level data in the long and short view images of the ship based on the improved YOLOV5 neural network, and the water gauge symbol data corresponding to the water level data; S3. Using a clustering algorithm to optimize the error of the water level data and the water gauge symbol data, filtering out invalid prediction box information, and calculating and obtaining preliminary water gauge observation data; S4. Using a Kalman filter network to dynamically assign weights to the ship draft data and preliminary water gauge observation data, weighted fusion of the ship draft data and preliminary water gauge observation data to obtain fused water gauge observation data; The Kalman filter network includes: a real-time state estimation network and a dynamic weight fusion network.

2. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The real-time state estimation network is built based on the traditional Kalman filtering process, and the noise of the ship draft data and the preliminary water gauge observation data is eliminated by learning the Kalman gain, including a prediction process and an update process. The prediction process is specifically as follows: P k|k―1 =FP k―1|k―1 F T +Q; The update process is specifically as follows: K k =P k|k―1 H k T (H k P k|k―1 H k T +R) ―1 , P k|k =(I―K k H k )P k|k―1 ; In the formula, represents the prior estimate of the fused water gauge observation data at time k, represents the posterior estimate of the fused water gauge observation data at time k, z k is the observation value at time k, i.e., the ship draft depth data and the preliminary water gauge observation data, F represents the state transfer matrix, H k represents the measurement matrix at time k, Q and R represent the Kalman filter noise, P k|k represents the posterior estimated covariance matrix, P k|k―1 represents the prior estimated covariance matrix, K k represents the Kalman gain and I represents the identity matrix.

3. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 2 is characterized in that: The dynamic weight fusion network is constructed by setting the real-time state estimation network measurement matrix H k The realization of multi-source data fusion and weight distribution includes the following sub-steps: S40. Initialize the measurement matrix H k =I; S41. Use the prior estimate to multiply the measurement matrix to obtain the predicted observation value Use the ship draft data at time k and the preliminary water gauge observation data to obtain the observation value z k Subtract the predicted observations The difference obtained As network input features; S42. Use GRU network to measure the matrix H k To update, the update method is as follows: The difference Δz k After feature extraction by the fully connected layer, it is sent to the GRU network: out FC =FC([Δz k ]); The calculation process of the GRU network is as follows: r t =σ(W r [h t―1 ,out FC ]+b r ); z t =σ(W z [h t―1 ,out FC ]+b z ); In the formula, r t Represents the reset gate output, z t represents the update gate output, and h t represent the candidate hidden state and the final hidden state respectively, W r and W z Represent the weight of the reset gate output and the weight of the update gate output, respectively, r and b z Represent the bias of the reset gate output and the bias of the update gate output, ⊙ represents the dot product operation, W h represents the weight of the candidate hidden state, b h represents the bias of the candidate hidden state; GRU obtains the final hidden state by resetting the gate output and updating the gate output control information storage and forgetting, and obtains the measurement matrix H to be updated after the final hidden state is scaled by the fully connected layer k as follows: H k =FC(h t ); Among them, FC represents the fully connected layer.

4. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The improved YOLOV5 neural network specifically adopts the adaptive activation function ACON instead of the default nonlinear activation function ReLU; adopts the weighted bidirectional feature pyramid network structure BiFPN instead of the path aggregation neural network PANet; and adds a plug-and-play attention module CBAM.

5. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 4 is characterized in that: The adaptive activation function ACON is obtained by the following formula: f(x)=p1·ReLU(x)+p2·ReLU(-x); Among them, p1 and p2 are adaptive parameters, and the ReLU(x) function is expressed as:

6. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 4 is characterized in that: The weighted bidirectional feature pyramid network structure BiFPN is obtained using the following formula: in, and They are the top-down and top-up feature maps respectively, and w1 and w2 are the self-learning weights.

7. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 4 is characterized in that: The plug-and-play attention module CBAM includes a channel attention module and a spatial attention module. The calculation process of the channel attention module is as follows: f max =max i,j F c,i,j ,c=1,2,…,C; In the formula, f avg represents the global average pooling result, f max represents the global maximum pooling result, f represents the channel descriptor obtained by vector concatenation, F represents the input image feature map, C represents the number of channels, H and W represent the height and width of the feature map respectively, represents a two-dimensional real vector space of dimension 2C; The channel weights are generated through a shared multi-layer perceptron MLP as follows: m c =σ(W2·δ(W1·f+b1)+b2), F out =m⊙F; In the formula, m c represents the channel weight, F out represents the feature map weighted by the channel weight, W1 and W2 represent the weight matrix of MLP, b1 and b2 represent the bias terms, ⊙ represents the dot product operation, and σ represents the activation function; The calculation process of the spatial attention module is as follows: f′ max =max c F out,c , F s =Convat(f′ avg ,f′ max ); In the formula, f′ avg represents the channel average pooling result, f′ max represents the channel maximum pooling result, F s Represents the two-dimensional feature map obtained by channel concatenation; F s A spatial attention map is obtained through a convolutional layer, and the spatial attention map is combined with the feature map weighted by the channel weights to obtain the final output F final : m spatial =σ(Conv(F s )), F final =m spatial ⊙F out ; In the formula, F final Represents the final output, m spatial Represents the two-dimensional feature map F s The spatial attention map F is obtained through a convolutional layer. out Represents a feature map weighted by channel weights.

8. The water gauge intelligent observation method based on multi-unmanned equipment collaboration according to claim 1 is characterized in that: The long and short-range images of the ship are acquired by collecting by unmanned aerial vehicles and unmanned boats; and / or, the draft data of the ship is acquired by collecting by an underwater robot attached to the bottom of the ship.

9. A water gauge intelligent observation system based on the collaboration of multiple unmanned equipment, characterized in that: include: Data acquisition and preprocessing module, image detection module and dynamic weight fusion module; The data acquisition and preprocessing module is used to obtain the ship's long and short-range images and the ship's draft depth data; The image detection module is used to identify the water level data in the long and short view images of the ship, and the water gauge symbol data corresponding to the water level data, and to optimize the errors of the water level data and the water gauge symbol data to obtain preliminary water gauge observation data; The dynamic weight fusion module is used to dynamically assign weights to the ship's draft data and preliminary water gauge observation data, and weightedly fuse the ship's draft data and preliminary water gauge observation data to obtain fused water gauge observation data.

10. The water gauge intelligent observation system based on multi-unmanned equipment collaboration according to claim 9 is characterized in that: The data acquisition and preprocessing module includes a ship near- and far-view image acquisition unit and a ship draft depth data acquisition unit. The ship near- and far-view image acquisition unit includes a drone and an unmanned boat for acquiring near- and far-view images of the ship. The ship draft depth data acquisition unit includes an underwater robot attached to the bottom of the ship for outputting the ship draft depth data. The image detection module includes an image recognition unit and a water gauge symbol recognition unit. The image recognition unit is used to recognize water level line data in the long and short view images of the ship, and the water gauge symbol recognition unit is used to output water gauge symbol data corresponding to the water level line.

Images