An intelligent universal water gauge intensive reading method and system integrating multimodal information

Abstract

The present invention provides an intelligent universal water gauge intensive reading method and system for fusing multimodal information, including: preprocessing RGB images and DN values; extracting feature points through a threshold-driven orb algorithm, locating feature point sparse areas, and generating virtual feature point pairs; realizing registration alignment through affine transformation, then performing channel superposition, generating a four-channel image fusing multispectral data, and using a deep learning model to obtain a waterline segmentation mask map and character recognition data in each four-channel image; using a microwave rangefinder camera to determine the waterline overhead angle, fitting the fitting relationship between the actual tonnage and the waterline overhead angle under different waterline overhead angles, and then obtaining the actual tonnage of the ship as the first reading of the waterline; using the designed equidistant water gauge map, through the corresponding affine transformation, realizing water gauge fitting, and obtaining the second reading of the waterline; averaging the first and second readings of the waterline to obtain the accurate reading of the waterline. The present invention can perform intensive reading on the universal water gauge.

Description

Technical Field

[0001] The present invention relates to the technical field of water gauge weighing, and in particular to an intelligent universal water gauge intensive reading method and system integrating multi-modal information. Background Art

[0002] Draft weighing is one of the most widely used methods for ship cargo trading. It measures the average draft data before and after loading and unloading cargo by observing the draft of the ship, and then calculates the actual displacement of the ship and the weight of the ship's materials, thereby calculating the ship's deadweight. At present, domestic ports generally use manual observation and manual calculation to carry out weight appraisal when carrying out draft weighing. However, due to factors such as the operating time and environment being unstable, and the differences in the ability level of the observers, the data obtained in this way is very subjective, and is affected by sea waves, the tilting state of the bulk carrier and other reasons. The read draft data has certain errors and poor reproducibility, which seriously affects the accurate value of the ship's draft.

[0003] With the rapid development and widespread application of image processing technology, more and more algorithms for water gauge recognition and reading are emerging. As for the existing water gauge reading methods, most of them are based on scale lines for numerical reading. By identifying the pixel distance between the water line and the scale line in the image, the specific value of the actual water line position is derived. This method relies on the relative relationship between the scale line and the water line to a large extent. In the recognition process, it is necessary not only to accurately segment the position of the water surface line, but also to identify and locate the corresponding position of the scale line to detect the relative distance between the two. The degree of limitation is large. Once a water gauge without scale lines is encountered, this method will be greatly hindered; at the same time, affected by factors such as seawater immersion and natural environment, the water gauge may be severely corroded, resulting in unclear display of the scale value, and ultimately causing a large deviation in the water level reading, making it impossible to correctly monitor the water level. Summary of the invention

[0004] The present invention provides an intelligent universal water gauge intensive reading method and system integrating multimodal information, which is used to intensively read the universal water gauge. The technical solution is as follows:

[0005] On the one hand, an intelligent universal water level reading method integrating multimodal information is provided, including:

[0006] S1, obtaining a multispectral water level image taken on site, the multispectral water level image including an RGB image and a remote sensing image pixel brightness value DN value, forming a DN value matrix with the DN values, normalizing the DN value matrix to form a corresponding multispectral grayscale image, and scaling the RGB image to the same size as the multispectral image;

[0007] S2, extracting feature points by using a threshold-driven orb algorithm, obtaining matching feature point pairs in the scaled RGB image and the multispectral grayscale image, locating a sparse feature point area in the scaled RGB image, adding an appropriate number of virtual feature points in the sparse feature point area in the scaled RGB image, adding an equal number of virtual feature points at the corresponding same positions in the multispectral grayscale image, and forming virtual feature point pairs together with the virtual feature points of the scaled RGB image, calculating an affine transformation matrix based on the matching feature point pairs and the virtual feature point pairs, and performing registration and alignment of the scaled RGB image to the multispectral grayscale image;

[0008] S3, superimposing the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data, and using a deep learning model to obtain the corresponding waterline segmentation mask image and recognized character data in each four-channel image;

[0009] S4. Use a microwave ranging camera to measure the distance between the camera and the hull and the water surface respectively to obtain a corresponding waterline bird's-eye view angle. Combine the waterline bird's-eye view angle and other data obtained according to the deep learning model to fit the fitting relationship between the actual tonnage reading corresponding to each image at different waterline bird's-eye view angles and the waterline bird's-eye view angle, and then obtain the actual tonnage of the ship corresponding to the different waterline bird's-eye view angles as the first reading of the waterline;

[0010] S5, using the designed equidistant water ruler diagram, by matching corresponding coordinate point pairs, generating a corresponding affine transformation matrix, attaching the equidistant water ruler to the corresponding position in the four-channel image, and obtaining a second reading of the waterline;

[0011] S6. Calculate the average value of the first reading and the second reading of the waterline to obtain an accurate reading of the waterline.

[0012] On the other hand, an intelligent universal water level reading system integrating multimodal information is provided, comprising:

[0013] An acquisition module is used to acquire a multispectral water level image taken on site, wherein the multispectral water level image includes an RGB image and a remote sensing image pixel brightness value DN value, the DN values ​​are combined into a DN value matrix, the DN value matrix is ​​normalized to form a corresponding multispectral grayscale image, and the RGB image is scaled to the same size as the multispectral image;

[0014] A registration module, for extracting feature points by using a threshold-driven orb algorithm, obtaining matching feature point pairs in the scaled RGB image and the multispectral grayscale image, locating a sparse feature point region in the scaled RGB image, adding an appropriate number of virtual feature points in the sparse feature point region in the scaled RGB image, adding an equal number of virtual feature points at corresponding identical positions in the multispectral grayscale image, and forming virtual feature point pairs together with the virtual feature points of the scaled RGB image, calculating an affine transformation matrix based on the matching feature point pairs and the virtual feature point pairs, and performing registration and alignment of the scaled RGB image to the multispectral grayscale image;

[0015] The first processing module is used to perform channel superposition of the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data, and use a deep learning model to obtain a corresponding waterline segmentation mask image and recognized character data in each four-channel image;

[0016] A fitting module is used to use a microwave ranging camera to measure the distance between the camera and the hull and the water surface respectively, obtain a corresponding waterline bird's-eye view, and combine the waterline bird's-eye view and other data obtained according to the deep learning model to fit the actual tonnage reading corresponding to each image at different waterline bird's-eye view and the fitting relationship between the waterline bird's-eye view, thereby obtaining the actual tonnage of the ship corresponding to the different waterline bird's-eye view as the first reading of the waterline;

[0017] A fitting module is used to use the designed equidistant water ruler diagram to match corresponding coordinate point pairs to generate a corresponding affine transformation matrix, fit the equidistant water ruler to the corresponding position in the four-channel image, and obtain a second reading of the waterline;

[0018] The second processing module is used to calculate the average value of the first reading and the second reading of the waterline to obtain an accurate reading of the waterline.

[0019] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned intelligent universal water level intensive reading method integrating multimodal information.

[0020] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the above-mentioned intelligent universal water level intensive reading method integrating multimodal information.

[0021] Compared with the prior art, the above technical solution has at least the following beneficial effects:

[0022] 1) Introduce multispectral information dimensions and combine them with RGB images to form a richer and more comprehensive image data source. This makes the data source not only contain the corresponding color features and texture features, but also introduces the spectral characteristics of the object, which can significantly reduce the influence of environmental factors such as light reflection and sea foam, and achieve accurate reading of the waterline from multiple dimensions.

[0023] 2) Improve the image registration method, add virtual feature point pairs in the sparse image feature area, and form the final feature point pairs together with the existing matching feature point pairs, so as to realize the registration alignment of the scaled RGB image to the multi-spectral grayscale image. This method can greatly reduce the registration failure problem caused by uneven distribution of feature points and a small number of feature points, thereby improving the registration error tolerance rate.

[0024] 3) The waterline value is read simultaneously by hardware fitting and software fitting, and the waterline reading is accurately read from multiple levels. At the same time, the above two methods can greatly solve the reading error problem caused by crooked water level measurement and overhead shooting of water level measurement, so as to make the final waterline reading more accurate.

[0025] 4) Integrate multi-modal information such as multi-spectral grayscale images, RGB images, microwave ranging radar, etc. to jointly accurately read the waterline, use the rich feature information of different types of data, make joint decisions, and accurately read the final reading, thereby improving the overall robustness and accuracy of the system and having strong anti-interference ability.

[0026] 5) It can intelligently realize accurate readings under universal water gauges without relying on the position information of scale lines to obtain the corresponding readings of the water surface line, and has high flexibility and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flow chart of an intelligent universal water level intensive reading method integrating multimodal information provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of image preprocessing provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of an improved registration method provided by an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of threshold-driven feature matching provided by an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of generating virtual feature points provided by an embodiment of the present invention;

[0033] Figure 6 A schematic diagram of generating virtual feature point pairs provided by an embodiment of the present invention;

[0034] Figure 7 is a schematic diagram of generating a four-channel image provided by an embodiment of the present invention;

[0035] Figure 8 is a schematic diagram of deep learning model recognition provided by an embodiment of the present invention;

[0036] Fig. 9 is a waterline segmentation mask image provided by an embodiment of the present invention;

[0037] Fig.10 is a schematic diagram of microwave ranging camera measurement provided by an embodiment of the present invention;

[0038] Fig.11 A schematic diagram of a microwave camera-assisted measurement fitting relationship provided in an embodiment of the present invention;

[0039] Fig.12 A schematic diagram of an equidistant water ruler providing an embodiment of the present invention to assist in generating fitting engraved lines;

[0040] Fig.13 A schematic diagram of equidistant water ruler fitting when the image is taken directly in the embodiment of the present invention;

[0041] Fig.14 Schematic diagram of equidistant water ruler fitting when image shooting is skewed provided by an embodiment of the present invention

[0042] Fig.15 A block diagram of an intelligent universal water gauge intensive reading system integrating multimodal information provided by an embodiment of the present invention;

[0043] Fig.16 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The embodiment of the present invention provides an intelligent universal water gauge intensive reading method integrating multimodal information, aiming to reduce the influence of factors such as scale lines on water gauge readings. By introducing multispectral information data and relying on an improved image registration algorithm to accurately extract image features, the multi-dimensional, multi-level, and high-tolerance accurate reading of any water gauge can be achieved from the two perspectives of hardware fitting and algorithm fitting, thereby improving the reading efficiency and accuracy. Figure 1-14 , describes in detail an intelligent universal water level intensive reading method that integrates multimodal information provided by an embodiment of the present invention.

[0046] like Figure 1 As shown, an embodiment of the present invention provides an intelligent universal water level intensive reading method integrating multimodal information, comprising:

[0047] S1. Obtain a multispectral water level image taken on site, wherein the multispectral water level image includes an RGB image and a remote sensing image pixel brightness value DN value, the DN values ​​are combined into a DN value matrix, the DN value matrix is ​​normalized to form a corresponding multispectral grayscale image, and the RGB image is scaled to the same size as the multispectral image. These steps can be used as image preprocessing steps before subsequent steps, such as Figure 2 As shown;

[0048] The step of normalizing the DN value matrix to form a corresponding multispectral grayscale image includes:

[0049] Extract the maximum and minimum values ​​in the DN value matrix, and map the original DN value to the digital range of 0-255 through the normalization formula to generate the corresponding multispectral grayscale image. In addition, the generated multispectral grayscale image can be contrast enhanced to improve the accuracy of subsequent feature point recognition, and scale the RGB image to the same size as the multispectral grayscale image. The normalization formula is as follows:

[0050]

[0051] S2, extracting feature points by using a threshold-driven orb algorithm, obtaining matching feature point pairs in the scaled RGB image and the multispectral grayscale image, locating a sparse feature point area in the scaled RGB image, adding an appropriate number of virtual feature points in the sparse feature point area in the scaled RGB image, adding an equal number of virtual feature points at the corresponding same positions in the multispectral grayscale image, and forming virtual feature point pairs together with the virtual feature points of the scaled RGB image, calculating an affine transformation matrix based on the matching feature point pairs and the virtual feature point pairs, and performing registration and alignment of the scaled RGB image to the multispectral grayscale image. This improved registration method is also as shown in Figure 3 As shown;

[0052] Before registering the scaled RGB image with the multispectral grayscale image, feature point extraction and matching are required.

[0053] Optionally, the step S2 extracts feature points by using a threshold-driven orb algorithm to obtain matching feature point pairs in the scaled RGB image and the multispectral grayscale image, specifically including:

[0054] Get the image height value of the scaled RGB image and set the row range threshold to Image height value, then the row range of each feature point is the total area of ​​the upper and lower extended row range thresholds centered on the current height of each feature point;

[0055] The feature points of the corresponding multispectral grayscale image within the row range of each scaled RGB image feature point are screened, matched and formed into matching feature point pairs in the two images, including:

[0056] The row range area of ​​each scaled RGB image feature point is obtained, the row range area is migrated to the multispectral grayscale image at the same position, and possible matching feature points are screened in the row range area to generate matching feature point pairs of the two images, such as Figure 4 As shown, the registration rate can be accelerated and the accuracy of matching feature point pairs between two images can be improved (preventing matching to other places that are far apart).

[0057] Optionally, the step of locating a sparse feature point area in the scaled RGB image, adding an appropriate number of virtual feature points in the sparse feature point area in the scaled RGB image, and adding an equal number of virtual feature points at corresponding positions of the multispectral grayscale image to form virtual feature point pairs together with the virtual feature points of the scaled RGB image, specifically includes:

[0058] Locating a feature point sparse area in the scaled RGB image according to the matched feature point pairs includes:

[0059] Generate an all-zero matrix of the same size as the scaled RGB image, and set 1 at the corresponding position of the all-zero matrix according to the existing feature point position in the scaled RGB image;

[0060] Set a matrix with all zero length, width and step length A convolution kernel of all 1s of size is applied to the all-zero matrix. The convolution kernel divides the all-zero matrix after setting to 1 into 25 regions. A convolution operation is performed on each region. The result of the regional convolution is used as the density value of the corresponding region. The density values ​​of all regions are compared, the largest density value is selected, and the threshold is set to the maximum density value. The area whose area density value is less than the threshold value is regarded as a sparse area of ​​feature points, and the sparse area of ​​feature points in the scaled RGB image is obtained;

[0061] For each sparse area, the Gaussian distribution principle is used to determine the distribution of virtual feature points. The mean of the Gaussian distribution is determined by the boundary of the sparse area. The mean of the horizontal Gaussian distribution is the median of the horizontal coordinates of the upper left corner and the upper right corner of the boundary (for example, if the horizontal coordinates of the upper left corner and the upper right corner are 100 and 200 respectively, the median is 150), and the mean of the vertical Gaussian distribution is the median of the vertical coordinates of the upper left corner and the lower left corner of the boundary. The standard deviation of the horizontal Gaussian distribution and the standard deviation of the vertical Gaussian distribution are both the regional density values ​​corresponding to this sparse area. However, if the corresponding regional density value is 0, the standard deviation is 1. A series of discrete values ​​are generated in the horizontal and vertical directions of each area according to the Gaussian distribution. These values ​​will guide the horizontal and vertical coordinates of the virtual feature points inside the area, including:

[0062] In the generated horizontal discrete value set and vertical discrete value set, one value is taken out for pairing respectively. Each horizontal value is paired with all vertical values, and each vertical value is also paired with all horizontal values, so as to cover all possible pairing combinations to form all possible point pairs. These point pairs will be used as the coordinates of virtual feature points in this sparse area. Let the horizontal Gaussian distribution discrete value set be recorded as (x1, x2, …, x n-1 ,x n ), the vertical Gaussian distribution discrete value set is recorded as (y1,y2,…,y m-1 ,y m ), then the coordinates of the virtual feature points are: (x1,y1), (x1,y2)...(x1,y m )、(x2,y1)、(x2,y2)...(x2,y m ),...,(x n ,y1)、(x n ,y2)...(x n ,y m ),like Figure 5 As shown;

[0063] Map these coordinates to the multispectral grayscale image, set the same virtual feature points at the corresponding coordinate positions, and form virtual feature point pairs together with the virtual feature points in the scaled RGB image, such as Figure 6 shown.

[0064] S3, superimposing the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data, and using a deep learning model to obtain the corresponding waterline segmentation mask image and recognized character data in each four-channel image;

[0065] Optionally, the step of performing channel superposition of the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data in S3 specifically includes:

[0066] Generate a blank four-channel image;

[0067] Insert the R, G, and B channels of the registered RGB image into the first, second, and third channels of the blank four-channel image, and insert the DN value matrix into the fourth channel of the blank four-channel image to complete the channel superposition between the registered RGB image and the DN value matrix, thereby generating a four-channel image fused with multispectral data, such as Figure 7 shown.

[0068] The deep learning model is used to obtain the corresponding waterline segmentation mask map and recognized character data in each four-channel image, such as Figure 8 As shown, the specific steps of how to use the deep learning model to obtain the corresponding waterline segmentation mask image and the recognized character data in each four-channel image are prior art and will not be repeated here.

[0069] Optionally, the deep learning model includes but is not limited to these base models: U-net, Transformer, YoloV5;

[0070] like Fig. 9 As shown, the waterline segmentation mask image is an image with binary characteristics, in which water pixels are marked as 1 and non-water pixels are marked as 0. By analyzing the waterline segmentation mask image, the entire image is traversed column by column, and the first non-zero pixel point in each column is identified, thereby obtaining the segmentation line points on all waterline segmentation mask images, that is, the precise coordinate information of each point on the waterline;

[0071] The character data includes the character category, the coordinates of the center point of the character detection frame, and the height and width values ​​of the detection frame.

[0072] S4. Use a microwave ranging camera to measure the distance between the camera and the hull and the water surface respectively to obtain a corresponding waterline bird's-eye view angle. Combine the waterline bird's-eye view angle and other data obtained according to the deep learning model to fit the fitting relationship between the actual tonnage reading corresponding to each image at different waterline bird's-eye view angles and the waterline bird's-eye view angle, and then obtain the actual tonnage of the ship corresponding to the different waterline bird's-eye view angles as the first reading of the waterline;

[0073] Optionally, the S4 specifically includes:

[0074] A microwave ranging camera is used to measure the distance between the camera and the hull and the water surface respectively. The angle between the hypotenuse formed between the microwave ranging camera and the water surface line and the vertical direction of the microwave ranging camera is obtained by using the triangle tangent. The angle is defined as the camera's top-down angle, as follows: Fig.10As shown;

[0075] The 0-90° overhead viewing angle interval is divided into 9 regions of equal angles, and 50 groups of multispectral water gauge images are collected in the corresponding regions using a multispectral camera, and the 50 groups of multispectral water gauge images in each region are converted into 50 four-channel images, and the deep learning model is used to obtain the corresponding waterline segmentation mask map and the recognized character data in the 50 four-channel images in each region, respectively, and the character data includes the character category, the coordinates of the center point of the character detection frame, and the height and width values ​​of the detection frame;

[0076] Taking the center point of the near-water character detection frame as the center, take 25 pixel position areas on the left and right, find all the segmentation line points on the waterline segmentation mask map in this area, and take the ordinates of all the segmentation line points to calculate the average value as the corresponding point of the waterline segmentation mask map under the center point of the near-water character detection frame;

[0077] The pixel distance between the center point of the near-water character detection frame and the corresponding point of the waterline segmentation mask map is calculated as a, and the pixel distance between the center point of the near-water character detection frame and the center point of the second near-water character detection frame is b. Assume that the camera's downward angle is θ, the near-water character reading is m, and the actual tonnage of the ship is Y. In the fitting process, the Y value is regarded as prior knowledge. Experienced ship workers give the actual tonnage reading corresponding to each image as the Y value based on 50 four-channel images of each area taken. The Y value is used as known data, and a, b, m, and θ are used to fit the appropriate k1, k2, and k3. The fitting formula is as follows:

[0078] |mY|=k1θ 3 +k2a 2 +k3b+20

[0079] In the 9 regions, k1, k2, and k3 in the corresponding region are fitted in each region and substituted into the fitting formula to form the fitting relationship between Y and θ in different regions;

[0080] In actual prediction, the corresponding overhead angle θ is obtained according to the multispectral water level image taken on site, and then different fitting functions are selected to obtain the actual tonnage Y of the ship at different angles as the first reading of the waterline, such as Fig.11 shown.

[0081] S5, using the designed equidistant water ruler diagram, by matching corresponding coordinate point pairs, generating a corresponding affine transformation matrix, attaching the equidistant water ruler to the corresponding position in the four-channel image, and obtaining a second reading of the waterline;

[0082] Alternatively, if Fig.12 As shown, the S5 specifically includes:

[0083] like Fig.13 As shown in (Schematic diagram of equidistant water ruler fitting when the image is taken straight) and 14 (Schematic diagram of equidistant water ruler fitting when the image is taken crooked), an equidistant water ruler diagram including 21 equidistant points is designed. These 21 points will be used to fit the center points of the two character detection frames in the real hull water ruler;

[0084] With the right direction as the horizontal coordinate and the downward direction as the vertical coordinate, all line segments in the equidistant water gauge map are detected using Hough transform (the specific steps are prior art and will not be repeated here), and the coordinates of the two end points of the line segment are returned, and the difference between the horizontal coordinate and the vertical coordinate between the coordinates of the two end points of each line segment is calculated, and the maximum value of the two types of differences is taken as the length information of each line segment, and the length information of all line segments is traversed, and the maximum value is taken as the vertical marking line of the equidistant water gauge map, and the remaining line segments are taken as the horizontal marking lines of the equidistant water gauge map;

[0085] Take the point with the smallest ordinate in the coordinates of the two end points corresponding to the vertical marking line as point 1A in the candidate point pair, rearrange the left end points of all horizontal marking lines (the point with the smallest abscissa in the coordinates of each line segment end point) according to the size of the ordinate, record the coordinates of the rearranged points and the corresponding position numbers, and select the 1st and 11th points from top to bottom as candidate points, among which point 1 and point 1A are repeated points, but if the coordinate values ​​are different, point 1A is the main coordinate, and point 11 is the 2A point in the candidate point pair;

[0086] The coordinates of the center point of the near-water character detection frame and the center point of the second near-water character detection frame are obtained by the deep learning model, the center point of the near-water character detection frame and the candidate point 2A constitute a coordinate point pair, and the center point of the second near-water character detection frame and the coordinates of the point 1A constitute a coordinate point pair;

[0087] Through these two coordinate point pairs, relying on affine transformation, the overall equidistant water gauge is scaled and corresponds to the corresponding position of the real water gauge photo, and the coordinates of all points corresponding to numbers 11 to 21 after transformation are obtained, as well as the coordinates of the segmentation line points in the waterline segmentation mask map corresponding to the horizontal coordinates of these points;

[0088] Compare the ordinates of the numbered points after point 11 with the ordinates of each dividing line point, find the first numbered point below the corresponding dividing line point, and find the coordinates and numbers of the numbered points above this numbered point;

[0089] The scale line values ​​on the equidistant water ruler corresponding to the two numbers (the scale line values ​​are the readings of the scale lines of the equidistant water ruler, such as scale line value 1.6 and scale line value 1.7) are averaged and used as the reading of the equidistant water ruler;

[0090] The integer digits of the equidistant water ruler are 0, 1, and 2, so the scale interval of the equidistant water ruler is 1. By identifying the character results of the near-water character and the second-near-water character, the difference between the two is obtained, which is the scale interval of the real water ruler. The scale intervals of the two types of water rulers are combined to obtain the interval multiples (for example, the near-water character recognition result is 4, and the second-near-water character recognition result is 6, then the scale interval of the real water ruler is 2, and the interval multiple between it and the equidistant water ruler is 2), and the second reading result of the waterline is obtained as follows:

[0091] The second reading of the waterline = the value of the second nearest water character - the interval multiple × the reading of the manual water gauge.

[0092] S6. Calculate the average value of the first reading and the second reading of the waterline to obtain an accurate reading of the waterline.

[0093] like Fig.15 As shown, the embodiment of the present invention also provides an intelligent universal water level reading system integrating multimodal information, including:

[0094] The acquisition module 1510 is used to acquire a multispectral water level image taken on site, wherein the multispectral water level image includes an RGB image and a remote sensing image pixel brightness value DN value, the DN values ​​are combined into a DN value matrix, the DN value matrix is ​​normalized to form a corresponding multispectral grayscale image, and the RGB image is scaled to the same size as the multispectral image;

[0095] A registration module 1520 is used to extract feature points by using a threshold-driven orb algorithm, obtain matching feature point pairs in the scaled RGB image and the multispectral grayscale image, locate a sparse feature point area in the scaled RGB image, add an appropriate number of virtual feature points in the sparse feature point area in the scaled RGB image, add an equal number of virtual feature points at the corresponding same positions in the multispectral grayscale image, and form virtual feature point pairs together with the virtual feature points of the scaled RGB image, calculate an affine transformation matrix based on the matching feature point pairs and the virtual feature point pairs, and perform registration and alignment of the scaled RGB image to the multispectral grayscale image;

[0096] The first processing module 1530 is used to perform channel superposition on the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data, and obtain a corresponding waterline segmentation mask image and recognized character data in each four-channel image using a deep learning model;

[0097] The fitting module 1540 is used to use a microwave ranging camera to measure the distance between the camera and the hull and the water surface respectively, obtain a corresponding waterline top-down angle, and fit the fitting relationship between the actual tonnage reading corresponding to each image at different waterline top-down angles and the waterline top-down angle in combination with the waterline top-down angle and other data obtained according to the deep learning model, so as to obtain the actual tonnage of the ship corresponding to the different waterline top-down angles as the first reading of the waterline;

[0098] The fitting module 1550 is used to use the designed isometric water ruler diagram to match corresponding coordinate point pairs to generate a corresponding affine transformation matrix, fit the isometric water ruler to the corresponding position in the four-channel image, and obtain a second reading of the waterline;

[0099] The second processing module 1560 is used to calculate the average of the first reading value and the second reading value of the waterline to obtain an accurate reading value of the waterline.

[0100] An intelligent universal water level gauge intensive reading system integrating multimodal information provided in an embodiment of the present invention has a functional structure corresponding to an intelligent universal water level gauge intensive reading method integrating multimodal information provided in an embodiment of the present invention, which will not be described in detail here.

[0101] Fig.16 It is a structural diagram of an electronic device 1600 provided in an embodiment of the present invention. The electronic device 1600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1601 and one or more memories 1602, wherein at least one instruction is stored in the memory 1602, and the at least one instruction is loaded and executed by the processor 1601 to implement the steps of the above-mentioned intelligent universal water level intensive reading method that integrates multimodal information.

[0102] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the intelligent universal water gauge intensive reading method integrating multimodal information. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0103] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent universal water gauge intensive reading method integrating multimodal information, characterized in that: include: S1, obtaining a multispectral water level image taken on site, the multispectral water level image including an RGB image and a remote sensing image pixel brightness value DN value, forming a DN value matrix with the DN values, normalizing the DN value matrix to form a corresponding multispectral grayscale image, and scaling the RGB image to the same size as the multispectral image; S2, extracting feature points by using a threshold-driven orb algorithm, obtaining matching feature point pairs in the scaled RGB image and the multispectral grayscale image, locating a sparse feature point area in the scaled RGB image, adding an appropriate number of virtual feature points in the sparse feature point area in the scaled RGB image, adding an equal number of virtual feature points at the corresponding same positions in the multispectral grayscale image, and forming virtual feature point pairs together with the virtual feature points of the scaled RGB image, calculating an affine transformation matrix based on the matching feature point pairs and the virtual feature point pairs, and performing registration and alignment of the scaled RGB image to the multispectral grayscale image; S3, superimposing the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data, and using a deep learning model to obtain the corresponding waterline segmentation mask image and recognized character data in each four-channel image; S4. Use a microwave ranging camera to measure the distances between the camera and the hull and the water surface respectively to obtain a corresponding waterline bird's-eye view angle, and combine the waterline bird's-eye view angle and the corresponding waterline segmentation mask map and recognized character data in each four-channel image obtained according to the deep learning model to fit the fitting relationship between the actual tonnage reading corresponding to each image at different waterline bird's-eye view angles and the waterline bird's-eye view angle, thereby obtaining the actual tonnage of the ship corresponding to the different waterline bird's-eye view angles as the first reading of the waterline; S5, using the designed equidistant water ruler diagram, by matching corresponding coordinate point pairs, generating a corresponding affine transformation matrix, attaching the equidistant water ruler to the corresponding position in the four-channel image, and obtaining a second reading of the waterline; S6. Calculate the average value of the first reading and the second reading of the waterline to obtain an accurate reading of the waterline.

2. The method according to claim 1, characterized in that In S2, feature point extraction is performed by using a threshold-driven orb algorithm to obtain matching feature point pairs in the scaled RGB image and the multispectral grayscale image, specifically including: Get the image height value of the scaled RGB image and set the row range threshold to Image height value, then the row range of each feature point is the total area of ​​the upper and lower extended row range thresholds centered on the current height of each feature point; The feature points of the corresponding multispectral grayscale image within the row range of each scaled RGB image feature point are screened, matched and formed into matching feature point pairs in the two images, including: The row range area of ​​each scaled RGB image feature point is obtained, the row range area is migrated to the multispectral grayscale image at the same position, and possible matching feature points are screened in the row range area, so as to generate matching feature point pairs of the two images.

3. The method according to claim 1, characterized in that The step S2 locates a sparse feature point area in the scaled RGB image, adds an appropriate number of virtual feature points in the sparse feature point area in the scaled RGB image, and adds an equal number of virtual feature points at the corresponding same positions in the multispectral grayscale image to form virtual feature point pairs together with the virtual feature points of the scaled RGB image, specifically including: Locating a feature point sparse area in the scaled RGB image according to the matched feature point pairs includes: Generate an all-zero matrix of the same size as the scaled RGB image, and set 1 at the corresponding position of the all-zero matrix according to the existing feature point position in the scaled RGB image; Set a matrix with all zero length, width and step length A convolution kernel of all 1s of size is applied to the all-zero matrix. The convolution kernel divides the all-zero matrix after setting to 1 into 25 regions. A convolution operation is performed on each region. The result of the regional convolution is used as the density value of the corresponding region. The density values ​​of all regions are compared, the largest density value is selected, and the threshold is set to the maximum density value. The area whose area density value is less than the threshold value is regarded as a sparse area of ​​feature points, and the sparse area of ​​feature points in the scaled RGB image is obtained; For each sparse area, the Gaussian distribution principle is used to determine the distribution of virtual feature points. The mean of the Gaussian distribution is determined by the boundary of the sparse area. The mean of the horizontal Gaussian distribution is the median of the horizontal coordinates of the upper left corner and the upper right corner of the boundary. The mean of the vertical Gaussian distribution is the median of the vertical coordinates of the upper left corner and the lower left corner of the boundary. The standard deviation of the horizontal Gaussian distribution and the standard deviation of the vertical Gaussian distribution are both the regional density values ​​corresponding to this sparse area. However, if the corresponding regional density value is 0, the standard deviation is 1. A series of discrete values ​​are generated in the horizontal and vertical directions of each area according to the Gaussian distribution. These values ​​will guide the horizontal and vertical coordinates of the virtual feature points inside the area, including: In the generated horizontal discrete value set and vertical discrete value set, one value is taken out for pairing respectively. Each horizontal value is paired with all vertical values, and each vertical value is also paired with all horizontal values, so as to cover all possible pairing combinations to form all possible point pairs. These point pairs will be used as the coordinates of virtual feature points in this sparse area. Let the horizontal Gaussian distribution discrete value set be recorded as (x1, x2, ..., x n-1 , x n ), the longitudinal Gaussian distribution discrete value set is recorded as (y1, y2, ..., y m-1 ,y m ), then the virtual feature point coordinates are: (x1, y1), (x1, y2)…(x1, y m ), (x2, y1), (x2, y2)…(x2, y m ),…,(x n ,y1),(x n , y2)…(x n ,y m ); These coordinates are mapped to the multispectral grayscale image, and the same virtual feature points are set at the corresponding coordinate positions to form virtual feature point pairs together with the virtual feature points in the scaled RGB image.

4. The method according to claim 1, characterized in that: In S3, the registered RGB image and the DN value matrix are channel-superimposed to generate a four-channel image fused with multispectral data, which specifically includes: Generate a blank four-channel image; The R, G, and B channels of the registered RGB image are inserted into the first, second, and third channels of the blank four-channel image, and the DN value matrix is ​​inserted into the fourth channel of the blank four-channel image to complete the channel superposition between the registered RGB image and the DN value matrix, thereby generating a four-channel image fused with multispectral data.

5. The method according to claim 1, characterized in that The deep learning models include but are not limited to the following base models: U-net, Transformer, YoloV5; The waterline segmentation mask is an image with binary characteristics, in which water pixels are marked as 1 and non-water pixels are marked as 0. By analyzing the waterline segmentation mask, the entire image is traversed column by column, and the first non-zero pixel in each column is identified, thereby obtaining the segmentation line points on all waterline segmentation masks, that is, the precise coordinate information of each point on the waterline; The character data includes the character category, the coordinates of the center point of the character detection frame, and the height and width values ​​of the detection frame.

6. The method according to claim 1, characterized in that The S4 specifically includes: A microwave ranging camera is used to measure the distance between the camera and the hull and the water surface respectively, and the angle between the hypotenuse formed between the microwave ranging camera and the water surface line and the vertical direction of the microwave ranging camera is obtained by using the triangle tangent, and the angle is defined as the waterline top-down angle; The interval of 0-90° is divided into 9 regions of equal angles, and 50 groups of multispectral water gauge images are collected in the corresponding regions using a multispectral camera, and the 50 groups of multispectral water gauge images in each region are converted into 50 four-channel images, and the deep learning model is used to obtain the corresponding waterline segmentation mask map and the recognized character data in the 50 four-channel images in each region, respectively, and the character data includes the character category, the coordinates of the center point of the character detection frame, and the height and width values ​​of the detection frame; Taking the center point of the near-water character detection frame as the center, take 25 pixel position areas on the left and right, find all the segmentation line points on the waterline segmentation mask map in this area, and take the ordinates of all the segmentation line points to calculate the average value as the corresponding point of the waterline segmentation mask map under the center point of the near-water character detection frame; The pixel distance between the center point of the near-water character detection frame and the corresponding point of the waterline segmentation mask map is calculated as a, and the pixel distance between the center point of the near-water character detection frame and the center point of the second near-water character detection frame is b. Assume that the waterline overlooking angle is θ, the near-water character reading is m, and the actual tonnage of the ship is Y. In the fitting process, the Y value is regarded as prior knowledge. Experienced ship workers give the actual tonnage reading corresponding to each image as the Y value based on 50 four-channel images of each area taken. The Y value is used as known data, and a, b, m, and θ are used to fit the appropriate k1, k2, and k3. The fitting formula is as follows: |m-Y|=k1θ 3 +k2a 2 +k3b+20 In the 9 regions, k1, k2, and k3 in the corresponding region are fitted in each region and substituted into the fitting formula to form the fitting relationship between Y and θ in different regions; In the actual prediction, different fitting functions are selected according to the waterline bird's-eye view θ measured by the microwave ranging camera and the multispectral water level image taken on site to obtain the actual tonnage Y of the ship corresponding to different angles as the first reading of the waterline.

7. The method according to claim 1, characterized in that The S5 specifically includes: Design an equidistant water gauge diagram consisting of 21 equidistant points. These 21 points will be used to fit the center points of the two character detection frames in the real hull water gauge. With the right direction as the horizontal coordinate and the downward direction as the vertical coordinate, all line segments in the equidistant water gauge map are detected using Hough transform, and the coordinates of the two end points of the line segments are returned. The difference between the horizontal coordinate and the vertical coordinate between the coordinates of the two end points of each line segment is calculated, and the maximum value of the two types of differences is taken as the length information of each line segment. The length information of all line segments is traversed, and the maximum value is taken as the vertical marking line of the equidistant water gauge map, and the remaining line segments are taken as the horizontal marking lines of the equidistant water gauge map; Take the point with the smallest ordinate among the two end points of the vertical marking line as point 1A in the candidate point pair, rearrange the left end points of all horizontal marking lines according to the size of the ordinate, record the rearranged point coordinates and the corresponding position numbers, and select the 1st and 11th points from top to bottom as candidate points, where point 1 and point 1A are repeated points, but if the coordinate values ​​are different, point 1A is the main coordinate, and point 11 is the 2A point in the candidate point pair; The coordinates of the center point of the near-water character detection frame and the center point of the second near-water character detection frame are obtained by the deep learning model, the center point of the near-water character detection frame and the candidate point 2A constitute a coordinate point pair, and the center point of the second near-water character detection frame and the coordinates of the point 1A constitute a coordinate point pair; Through these two coordinate point pairs, relying on affine transformation, the overall equidistant water gauge is scaled and corresponds to the corresponding position of the real water gauge photo, and the coordinates of all points corresponding to numbers 11 to 21 after transformation are obtained, as well as the coordinates of the segmentation line points in the waterline segmentation mask map corresponding to the horizontal coordinates of these points; Compare the ordinates of the numbered points after point 11 with the ordinates of each dividing line point, find the first numbered point below the corresponding dividing line point, and find the coordinates and numbers of the numbered points above this numbered point; The scale line values ​​on the equidistant water ruler corresponding to the two numbers are averaged and used as the reading of the equidistant water ruler; The integer digits of the equidistant water ruler are 0, 1, and 2, so the scale interval of the equidistant water ruler is 1. By identifying the character results of the near water character and the second near water character, the difference between the two is obtained, which is the scale interval of the real water ruler. Combining the scale intervals of the two types of water rulers, the interval multiples are obtained, and the second reading result of the waterline is obtained as follows: The second reading of the waterline = the value of the second nearest water character - the interval multiple × the reading of the manual water gauge.

8. An intelligent universal water gauge reading system integrating multimodal information, characterized in that: include: An acquisition module is used to acquire a multispectral water level image taken on site, wherein the multispectral water level image includes an RGB image and a remote sensing image pixel brightness value DN value, the DN values ​​are combined into a DN value matrix, the DN value matrix is ​​normalized to form a corresponding multispectral grayscale image, and the RGB image is scaled to the same size as the multispectral image; A registration module, for extracting feature points by using a threshold-driven orb algorithm, obtaining matching feature point pairs in the scaled RGB image and the multispectral grayscale image, locating a sparse feature point region in the scaled RGB image, adding an appropriate number of virtual feature points in the sparse feature point region in the scaled RGB image, adding an equal number of virtual feature points at corresponding identical positions in the multispectral grayscale image, and forming virtual feature point pairs together with the virtual feature points of the scaled RGB image, calculating an affine transformation matrix based on the matching feature point pairs and the virtual feature point pairs, and performing registration and alignment of the scaled RGB image to the multispectral grayscale image; The first processing module is used to perform channel superposition of the registered RGB image and the DN value matrix to generate a four-channel image fused with multispectral data, and use a deep learning model to obtain a corresponding waterline segmentation mask image and recognized character data in each four-channel image; A fitting module is used to use a microwave ranging camera to measure the distance between the camera and the hull and the water surface respectively, obtain a corresponding waterline bird's-eye view, combine the waterline bird's-eye view and the corresponding waterline segmentation mask map and recognized character data in each four-channel image obtained according to the deep learning model, fit the fitting relationship between the actual tonnage reading corresponding to each image at different waterline bird's-eye view and the waterline bird's-eye view, and then obtain the actual tonnage of the ship corresponding to the different waterline bird's-eye view as the first reading of the waterline; A fitting module is used to use the designed equidistant water ruler diagram to match corresponding coordinate point pairs to generate a corresponding affine transformation matrix, fit the equidistant water ruler to the corresponding position in the four-channel image, and obtain a second reading of the waterline; The second processing module is used to calculate the average value of the first reading and the second reading of the waterline to obtain an accurate reading of the waterline.

9. An electronic device, comprising a processor and a memory, wherein instructions are stored in the memory, characterized in that: The instructions are loaded and executed by the processor to implement the intelligent universal water level reading method integrating multimodal information as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein instructions are stored in the storage medium, characterized in that: The instructions are loaded and executed by the processor to implement the intelligent universal water level reading method integrating multimodal information as described in any one of claims 1-7.

Images