Shipborne monitoring system for marine engineering water area

Abstract

The invention discloses a marine engineering water area-oriented shipborne monitoring system, which comprises a sensing acquisition system, a target setting module, a preprocessing system, an intelligent network identification system and a shooting and identification precision real-time regulation and control system, and is characterized in that the sensing acquisition system comprises an image acquisition and acquisition system, a ship GPS module and a storage module; the wave image shooting range, the monitoring target distance M and the recognition precision are determined in real time according to the size L, the navigation parameter direction and the speed of the ship, image shooting is carried out on the sea surface of a key area, the effectiveness of ship-borne image acquisition is achieved, and a foundation is laid for effective recognition of wave parameters in the industry.

Description

[0001] This application is a divisional application of the application with the application date of July 3, 2024, application number 202410891177.9, and invention title "A Shipborne Image Acquisition Method for Synchronous Monitoring in Waters of Offshore Engineering". Technical Field

[0002] This application relates to mobile image recognition technology and is applied to shipborne, specifically a shipborne monitoring system for waters of offshore engineering. Background Art

[0003] Waves are key influencing factors in the implementation of offshore engineering. The safety of engineering ships, construction efficiency, and operation of key equipment are all significantly affected by the waves in the surrounding waters. With the continuous development of China's dredging industry in the overseas market, the operating waters have expanded from inshore to offshore, and the wave conditions have become more complex, significantly affecting the construction efficiency, and the safety of engineering ships also faces great challenges.

[0004] In the process of developing new-generation dredging ships, new technologies such as big data and artificial intelligence are more and more widely used. As the cornerstone of the development of intelligent dredging, realizing the synchronous monitoring of the basic data of the hydrological environment in the operating waters is an extremely crucial part of realizing the digitization of the dredging process. However, currently, the acquisition of wave data in the sea area mainly relies on specialized hydrological, meteorological, and maritime management agencies, and the obtained data is also limited to large-scale regional forecasts and the monitoring of individual relatively fixed stations. For engineering ships in navigation during construction in different waters, the existing hydrological technologies or acquisition methods have poor real-time performance, low integration, and low accuracy, and cannot achieve real-time monitoring synchronized with construction, which does not meet the requirements of safe and efficient construction of deep-sea and far-sea projects and the requirements of intelligent dredging technology development for basic data.

[0005] Waves have a significant impact on the safety and efficiency of dredging ships and are key influencing parameters in dredging projects; currently, on the one hand, the dredging industry needs to develop intelligent dredging, which requires the collection of data throughout the dredging process, including ship data and environmental data such as waves in the surrounding waters, and intelligent analysis is carried out based on this. Therefore, synchronous monitoring of waves during construction is very important.

[0006] Currently, shore-based wave image recognition can only effectively identify wave parameters in fixed sea areas. Because the observation equipment is fixed, the monitoring range and reference system are relatively clear. Since ships are in navigation during construction operations, obviously, the image acquisition method of laying instruments on the water surface is not applicable.

[0007] In the field of intelligent vehicles, for mobile image recognition technology, the on-vehicle scenarios to be recognized by the recognition technology have obvious road and building line separations (such as road markings, the facades of street buildings, etc.). Therefore, when collecting images, the scope and objects to be recognized are relatively easy to determine. However, in the open sea scenario, the waves do not have obvious and stable line separations. Due to factors such as the sea area, wave propagation, ship navigation, and weather effects, it is difficult to directly perform recognition. Therefore, if the existing mobile image recognition technology in the intelligent vehicle field is used as an application for shipborne wave recognition, due to the complexity of wave propagation in the open sea, as well as the influence of the surrounding sea environment (visibility, weather conditions) and the relative relationship between the ship and the waves, it is simply impossible to effectively collect and recognize wave images. Summary of the Invention

[0008] Under the background of the current development of artificial intelligence technology, there has been a certain amount of technical accumulation in the related technology of the network prediction model based on sea wave images in the industry. By inputting sea wave images into the network, the basic data of waves can be predicted and recognized. The purpose of the present invention is to further disclose a method for obtaining shipborne images for synchronous monitoring in waters related to marine engineering, and a shipborne monitoring system for waters related to marine engineering. By optimizing the image acquisition parameters during the navigation state through existing ships, their on-vehicle cameras, operating state parameters, etc., high-precision recognition and monitoring of waves in the working sea area can be achieved.

[0009] Technical Solution 1

[0010] A method for obtaining shipborne images for waters related to marine engineering, characterized by including:

[0011] Step 1: Algorithm operating conditions

[0012] Install a high-precision night vision binocular camera as the acquisition hardware and determine the initial installation parameters;

[0013] When the system enters the monitoring target distance M < 2000m, the system will control and enable the binocular camera to collect wave surface image data in the form of optical images;

[0014] Determine different monitoring target distances M, that is, monitoring ranges, for different sea conditions;

[0015] Step 2. Determine the optimal shooting parameters at the current moment t, and perform shooting under the optimal shooting parameters at this moment to obtain the original video data

[0016] 2.1 According to the initial installation parameters in Step 1, further optimize the shooting parameters of the optical hardware sensing device (camera) at the current moment t. The shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0017] 2.2 Based on the optimal parameter shooting direction D determined by calculation in step 2.1, adjust the center line of the high-definition camera device to coincide with the shooting direction D, and

[0018] Based on the optimal parameter maximum shooting area distance lmax determined by calculation in step 2.1, then execute shooting the sea area wave image at the current moment t, and store the original wave image in video form;

[0019] Step 3. Preprocessing

[0020] Preprocess the original sea area wave image taken in step 2 to form a standardized image;

[0021] Step 4. Use the existing neural network to identify the input standardized image, and real-time predict the recognition parameters H t , HT t , C t ;

[0022] Step 5. Iteratively update the monitoring parameters and hardware shooting parameters to ensure real-time high-precision monitoring and recognition and optimal acquisition parameters of the shipborne system.

[0023] Calculate the waiting time T for the next shooting time interval, then enter step 2 for equipment adjustment and execute shooting, and iterate in this way.

[0024] Technical solution two

[0025] A shipborne monitoring system for waters of sea-related engineering projects, including a sensing and acquisition system, a target setting module, a preprocessing system, an intelligent network recognition system, and a real-time shooting and recognition accuracy regulation system. The sensing and acquisition system includes an image acquisition and acquisition system, a ship GPS module, and a storage module; where:

[0026] The sensing and acquisition system is synchronized with the ship's navigation. Among them, the image acquisition and acquisition system is selected as a high-precision night vision binocular camera. The binocular camera is the acquisition hardware and is installed at the head of the ship. As Figure 4 shown, collect wave surface image data in the form of optical images, and its execution is controlled by the real-time shooting and recognition accuracy regulation system;

[0027] The ship GPS module outputs the current ship speed V and the current ship heading S;

[0028] The storage module stores video original data, initial installation parameters, and the data output by the ship GPS module; the initial installation parameters are the installation position height of the camera on the ship, and the position height is also called the shooting height hc;

[0029] The target setting module sets different monitoring target distances M (i.e., monitoring ranges) for different sea conditions. When the on-board system enters the monitoring target distance M < 2000m, the system adjusts the image acquisition and obtaining system to enable the high-precision night vision binocular camera.

[0030] The preprocessing system performs real-time preprocessing on the original sea area wave images stored in the image acquisition and obtaining system to form standardized images.

[0031] The intelligent network recognition system is connected to the preprocessing system. After inputting the standardized images, it predicts and outputs the wave recognition parameters at the current moment t in real time. The wave recognition parameters include H t 、HT t 、C t ;

[0032] The shooting and recognition accuracy real-time regulation system is used to iteratively update the monitoring parameters and hardware shooting parameters to ensure the real-time high-precision monitoring and recognition and the optimal acquisition parameters of the on-board system. The shooting and recognition accuracy real-time regulation system is respectively connected to the image acquisition and obtaining system, the intelligent network recognition system, and the target setting module. The shooting and recognition accuracy real-time regulation system includes a shooting parameter optimization module and a recognition accuracy real-time regulation module, which are interconnected.

[0033] The shooting and recognition accuracy real-time regulation system determines the optimal shooting parameters at the current moment t, so that the image acquisition and obtaining system performs shooting under the optimal shooting parameters at this moment to obtain and store the video raw data.

[0034] The shooting parameter optimization module further calculates and optimizes the shooting parameters of the optical hardware sensing device (camera) at the current moment t according to the initial installation parameters in the storage module. The shooting parameters include the shooting direction D and the farthest shooting area distance lmax.

[0035] The shooting parameter optimization module adjusts the center line of the high-definition camera device to coincide with the shooting direction D based on the calculated optimal parameter shooting direction D, and

[0036] based on the calculated optimal parameter farthest shooting area distance lmax;

[0037] Furthermore, the image acquisition and obtaining system executes shooting of the sea area wave images at the current moment t and stores the original wave images in video form.

[0038] The recognition accuracy real-time regulation module uses the current ship speed V and the current ship heading S provided by the ship GPS module to calculate the relative ship speed Vs affected by the waves. The relative ship speed:

[0039] The recognition accuracy real-time control module sets the target monitoring distance M according to the ship monitoring requirements, and calculates the wave height recognition accuracy parameter e at the current moment t t , e t = V s / M * 0.1 + (H - hb) / hb * 0.9;

[0040] Furthermore, it is compared with the accuracy e at the previous moment t - 1 t-1 , and adjustments are made accordingly:

[0041] If e t > e t-1 , then adjust the focal length f = f - 10 * sz * lmax / (H - hb). Since the parameter setting of the focal length f of the camera is too large, it needs to be adjusted smaller;

[0042] If e t < e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H - hb). Since the measured wave height range is too large, the focal length f needs to be increased;

[0043] The recognition accuracy real-time control module calculates the acquisition parameters and provides them to the optimal shooting parameter module. Specifically:

[0044] According to the above adjustments, calculate the parameters of the forward or reverse adjustment of the focal length f and the angle as of the camera's central axis horizontally downward;

[0045] Based on the ship's heading S t and the recognized wave direction C t at the current moment t, calculate the shooting direction D t+1 at the next moment t + 1 according to formula (8);

[0046] Calculate the next shooting time interval T for the binocular optical camera to collect, T = Min(0.25, M / Vs) (9);

[0047] The above calculation results are passed as instructions to the image acquisition and obtaining system through the optimal shooting parameter module, and the image acquisition and obtaining system performs shooting after waiting for T time;

[0048] The shipborne monitoring system is synchronized with the ship's navigation. Following time, it continuously iterates to ensure real-time high-precision monitoring and recognition of the shipborne system and optimal acquisition parameters.

[0049] Summary: According to the ship's size L and navigation parameters (direction, speed), the shooting range (monitoring target distance M) and recognition accuracy of the wave image are determined in real time to shoot the sea surface of the key area, realizing the effectiveness of the shipborne acquired image and laying a foundation for the effective recognition of wave parameters in the industry. Brief Description of the Drawings

[0050] Figure 1 Example 1 Principle of the Process of the Method of the Present Invention

[0051] Figure 2 Example 2 Composition of the System of the Present Invention

[0052] Figure 3 Schematic Diagram of Adjusting the Monitoring Direction of a Ship during Navigation

[0053] Figure 4 Schematic Diagram of the Scene of Arranging Binocular Cameras on the Top Deck of the Ship's Cab and Monitoring the Sea Surface: a Overall Side View; b Schematic Diagram of the Relationship between Shooting Parameters Specific Embodiment

[0054] The technical solution of the present invention will be further introduced below with reference to the accompanying drawings.

[0055] Example 1

[0056] The overall algorithm design of the present invention is as Figure 1 shown.

[0057] A shipborne image acquisition method for waters of marine engineering projects, characterized by comprising:

[0058] Step 1 Operating Conditions of the Algorithm

[0059] Install a high-precision night vision binocular camera as the acquisition hardware, and determine the initial installation parameters, including the installation position height of the camera on the ship, and the position height is also called the shooting height hc;

[0060] When the system enters the monitoring target distance M < 2000m, the system will control and enable the high-precision night vision binocular camera to collect wave surface image data in the form of optical images;

[0061] Determine the theoretical monitoring range of the camera according to the monitoring requirements. According to the international standard sea state grade division, the present invention determines different monitoring target distances M, that is, the monitoring range, for different sea states by means of variable focal length:

[0062] Sea state level Monitoring target distance M (L: ship's length) Level 0 - 1 3L Level 2 - 3 6L Level 4 - 5 9L Level 6 and above 12L

[0063] Step 2. Determine the optimal shooting parameters at the current moment t, and perform shooting under the optimal shooting parameters at this moment to obtain the original video data

[0064] 2.1 Further optimize the shooting parameters of the optical hardware sensing device (camera) at the current moment t according to the initial installation parameters in Step 1. The shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0065] 2.2 Based on the optimal parameter shooting direction D calculated and determined in Step 2.1, adjust the center line of the high-definition imaging device to coincide with the shooting direction D, and

[0066] The maximum shooting area distance lmax determined based on the optimal parameters calculated in Step 2.1;

[0067] Furthermore, execute shooting the sea area wave image at the current moment t, and store the original wave image in video form.

[0068] Step 3. Preprocessing

[0069] Perform preprocessing such as flipping, light transformation, and scaling on the original sea area wave image taken in Step 2, so that the target sea area wave images taken under different distances and different weather (sunny / cloudy) conditions form standardized images.

[0070] Step 4. Use the existing neural network (i.e., the prediction model) to identify the input standardized image, and predict the recognition parameters H t , HT t , C t .

[0071] Step 5. Iteratively update the monitoring parameters and the hardware shooting parameters

[0072] Iteratively adjust the recognition and monitoring accuracy of the system at the current moment t:

[0073] Based on the ship's heading S t , ship's speed V t , and the recognition parameters H t , HT t , C t at the current moment t, calculate the current recognition accuracy e t , judge the difference between the current accuracy and the previous accuracy, and according to the accuracy adjustment, calculate the parameters of the forward or reverse adjustment of the focal length f and the angle as of the camera's central axis horizontally downward;

[0074] According to the ship's heading S t , recognition wave direction C t at the current moment t, calculate the shooting direction D at the next moment t + 1 according to formula (8) t+1 ;

[0075] Calculate the recognition period (i.e., the shooting time interval T) using formula (9);

[0076] Feed the calculated adjusted camera focal length f t+1 , the angle as of the camera's central axis horizontally downward t+1 , shooting direction D t+1 and recognition period T t+1 at the next moment t + 1 back to Step 2, and accordingly adjust the collection parameters of the binocular optical camera;

[0077] After waiting for time T, the system enters Step 2 for calculation and acquisition;

[0078] The above iterations are carried out to ensure real-time, highly accurate monitoring and identification of the shipborne system and optimal acquisition parameters.

[0079] The algorithm details of the present invention are described in detail below

[0080] Step 2.1 [Innovative and key step]

[0081] Step 2.1.1 Determine the optimal parameter shooting direction D

[0082] To adjust the monitoring direction, as Figure 3 shown, that is, to adjust and determine the optimal parameter shooting direction D, and provide it to the shooting step (Step 2.2) to control the image to capture the wave direction movement.

[0083] Construct calculation formula (8), that is, the calculation formula for the shooting direction D is:

[0084]

[0085] Parameter annotations and explanations:

[0086] The course of the engineering ship S (unit: °);

[0087] The shooting direction D (unit: °);

[0088] The horizontal field of view angle acw;

[0089] To adjust and control the highest wave direction accuracy, the present invention designs formula (8) to determine the relationship between the camera central axis and the wave direction during the monitoring process and the adjustment method. When the shipborne visual monitoring camera is installed, there is an inclination. In the algorithm design, the present invention sets the angle between the wave direction and the camera central axis to 0 or 180.

[0090] The wave direction C (unit: °);

[0091] For the identification of the wave direction C, a network model is used for identification. The network model is not the technical contribution of the present invention, and existing technologies in the industry are all adopted. For example, but not limited to: the Faster R-CNN framework can be adopted. For the wave image, the method of transfer learning is used, and the automatic detection of internal waves in the ocean on the image is realized by training a large number of data images. The input parameters of this prediction model are the preprocessed sea surface images. For the specific method, please refer to the article "Automatic Detection of Internal Waves in the South China Sea from Satellite SAR Images Based on Faster R-CNN" Journal: Journal of Remote Sensing, Volume 27, Issue 4, 2023, Pages: 905-918, Paper Publication Date: 2023-04-07

[0092] 2.1.2 Determine the farthest shooting area distance (lmax)

[0093] Calculate and determine the maximum shooting area distance \(l_{max}\) and the horizontal downward angle \(\alpha_s\) of the camera central axis, which are used to provide to Step 2.2 to execute the shooting step and input to Step 5.

[0094] (1) The maximum monitoring distance \(l_{max}\) is calculated by the following formula:

[0095] \(l_{max}=h_b\times f / (s_z\times10)\) (1)

[0096] Where, the high resolution \(h_b\), the focal length \(f\), the height \(s_z\) of a single pixel of the CMOS

[0097] (2) Calculate the horizontal downward angle \(\alpha_s\) of the camera central axis:

[0098] As Figure 4 shown in the scene and geometric schematic diagram, the shooting area is a trapezoidal area, and the following relationship exists:

[0099] The vertical field of view angle \(\alpha_{ch}\), \(\alpha_{ch}=2\times\arctan(cmos_h / (2f))\) (2)

[0100] The horizontal field of view angle \(\alpha_{cw}\), \(\alpha_{cw}=2\times\arctan(cmos_w / (2f))\) (3)

[0101] The horizontal downward angle \(\alpha_s\) of the camera central axis, \(\alpha_s=\arctan(h_c / l_{max})+\alpha_{ch} / 2\) (4)

[0102] Parameter annotation and algorithm design description:

[0103] \(cmos_h\) is the height of the camera sensor CMOS, and \(cmos_w\) is the width of the camera sensor CMOS;

[0104] The maximum monitoring distance \(l_{max}\) is determined by the camera hardware and is the upper limit of the measurement distance. It is required to meet the monitoring requirements of the specified sea conditions. Otherwise, the focal length \(f\) needs to be adjusted to meet the monitoring requirements. If it is too small, large wave heights cannot be measured, and if \(f\) is too small, the accuracy cannot meet the wave height measurement accuracy.

[0105] When the measured wave height \(H\) is less than the high resolution \(h_b\) of the wave height, \(f\) needs to be adjusted and reduced, and vice versa.

[0106] Step 3: Preprocessing

[0107] This part does not constitute a component of the core algorithm design of the present invention. That is to say, this part is not the key technical contribution of the present invention; but as an application example, in order to give play to the technical solution of the present invention in the implementation application effect, the following specific preprocessing process is further given in this example:

[0108] Step 3.1, Image screening

[0109] Since the captured wave image covers irrelevant image content such as the sky and ships, which can easily have an adverse effect on the wave recognition process, the collected video images are focused on the operation sea area and processed frame by frame to form a large number of effective image datasets.

[0110] Step 3.2, Image standardization: Images captured under different distances and different weather conditions (sunny / cloudy) are formed into standardized processed images.

[0111] 3.2.1 Image flipping

[0112] The purpose of image flipping is to change its direction or perspective to meet the processing requirements in data augmentation and subsequent model recognition algorithms. The image flipping operation is one of the most commonly used techniques in the image enhancement process. By randomly flipping the training data, the diversity of the data can be increased, and the robustness and generalization ability of the model can be improved. In addition, for images with direction errors caused by reasons such as shooting or scanning angles, the image can be corrected in the correct direction through image flipping, improving the recognition accuracy.

[0113] The image flipping method can be divided into two types: horizontal flipping and vertical flipping. Its principle is to obtain the output coordinates by subtracting the width and height of the image from the input coordinates, and this process belongs to the subtraction operation of unsigned numbers. The following formula expresses the mathematical process of image flipping:

[0114] I[X in ,Y in =O[X,Y] (14)

[0115] X in =TW-1-X in (15)

[0116] Y in =TH”-1-Y in (16)

[0117] X in ∈[0,TW) (17)

[0118] Y in ∈[0,TH”) (18)

[0119] Where I represents the input, O represents the output, X in and Y in represent the horizontal and vertical coordinates of the input pixels respectively, and TW and TH” represent the width and height of the image.

[0120] 3.2.2 Light transformation

[0121] The operation scenarios of dredging vessels include the wave image process under different weather conditions. Due to the differences in shooting light, there may be significant differences in recognition in the deep convolutional network. Therefore, light change processing is performed on the wave image data, based on adjusting the pixel values of the image to change the light characteristics such as brightness, contrast, and color of the image, forming a wave training set with different brightness intervals from 0.4 to 2.0 and different contrast intervals from 0.4 to 2.0, so as to enhance the generalization ability of the established deep convolutional network.

[0122] By scaling or offsetting the brightness value of each pixel, the brightness of the entire wave image is changed. The adjustment process can be achieved by multiplying the RGB channel value of each pixel by the brightness factor or adding the brightness offset value. The following formula expresses the mathematical principle process of brightness transformation:

[0123] L out =α*L in +cl (19)

[0124] In the formula, L out represents the pixel output value of the wave image, L in represents the pixel input value, α represents the scaling factor for brightness adjustment, and cl represents the brightness adjustment constant.

[0125] Contrast adjustment is achieved by increasing or decreasing the difference degree between bright and dark pixel values, thereby changing the contrast of the wave image. High contrast indicates a large difference between bright and dark parts, and low contrast indicates a small difference between bright and dark parts. Contrast adjustment can be achieved through linear transformation or non-linear transformation methods such as logarithmic transformation or histogram equalization.

[0126] Logarithmic transformation can transform the gray levels concentrated in the low brightness area into evenly distributing from the low brightness area to the high brightness area. The transformation function is:

[0127] Lt out =cx*Ln(1+r) (20)

[0128] In the formula, r is the gray value of the input image, Lt out is the gray value of the output image, and cx is the scaling coefficient.

[0129] The histogram equalization method enhances the contrast of the image by redistributing the gray levels of the image. Its principle is to transform the gray histogram of the original image into a uniformly distributed histogram, so that the brightness range of the image is wider and the details are clearer. Usually, it is based on the cumulative distribution function (CDF) and the mapping of gray levels. The mapping method is:

[0130]

[0131] In the formula, S kRefers to the value after the current gray level is mapped by the cumulative distribution function. n is the total number of pixels in the image, and n j is the number of pixels at the current gray level, and Lh is the total number of gray levels in the image.

[0132] 3.2.3 Image Scaling and Cropping

[0133] In the wave image processing task, the primary task is to resize the image to a fixed size for input into an algorithm or model, i.e., image scaling. This is achieved through an interpolation algorithm.

[0134] 3.2.3.1 Bilinear Interpolation Scaling of Images

[0135] By performing a weighted average of the gray values of the four surrounding original pixels to determine the gray value of the new pixel, more refined interpolation can be achieved. Bilinear interpolation first determines the position of the target pixel in the original image and then calculates the weighted average of the gray values of the four pixels adjacent to the target pixel to determine the gray value of the new pixel. This interpolation method utilizes the local gray-level variations in the original image and can therefore provide smoother and more accurate results compared to nearest-neighbor interpolation. In bilinear interpolation, assuming the width of the original wave image is W, the height is TH", the width of the target image is W', and the height is H', the gray value of the target pixel can be calculated using the following formula:

[0136] X = X' * (TW - 1) / (TW' - 1) (22)

[0137] Y = Y' * (TH" - 1) / (TH' - 1) (23)

[0138] I'(X', Y') = I(X, Y) (24)

[0139] Find the four nearest pixels (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2) in the original image, where (X1, Y1) is the top-left pixel closest to (X, Y) and (X2, Y2) is the bottom-right pixel closest to (X, Y). The gray value of the target image coordinates (X', Y') is calculated by bilinear interpolation based on these four nearest pixels:

[0140] I'(X', Y') = (1 - dX) * (1 - dY) * I(X1, Y1)

[0141] + dX * (1 - dY) * I(X2, Y1)

[0142] + (1 - dX) * dY * I(X1, Y2)

[0143] + dX * dY * I(X2, Y2) (25)

[0144] Among them, I(X, Y) represents the grayscale value at the original image coordinates (X, Y), I'(X', Y') represents the grayscale value at the target image coordinates (X', Y'), dX = X - x, dY = Y - y, and x, y represent the floor values of X, Y.

[0145] 3.2.3.2 Image Cropping

[0146] Extract a specific area from the original image to generate an image with new dimensions, achieving the purpose of removing some noise interference from the original picture and retaining the original features of the image, so as to facilitate feature extraction and analysis processing in the subsequent convolution process. It can improve the accuracy and efficiency of subsequent processing.

[0147] During the image acquisition process, the acquired images often have interference elements of other unnecessary information. By cropping the image, these interference elements can be excluded, enabling the analysis to focus on the part where feature recognition is required. In addition, the cropped image can reduce the number of pixels to be processed, thereby improving the computational efficiency. Therefore, this technology is particularly important for the wave image scenario of the working sea area during the dredging process.

[0148] The mathematical principle of image cropping is based on restricting and intercepting the pixel coordinates of the image. For the input image I(x, y), the cropped wave image Q(x′, y′) can be expressed as:

[0149] Q(x′, y′) = I((x s , x s + w), (y s , y s + h)) (26)

[0150] x s + w ≤ x (27)

[0151] y s + h ≤ y (28)

[0152] Among them, x s , y s represent the starting pixel coordinate points of cropping, w, h represent the width and height of cropping, (x s , x s + w) represents the x s to x s + w rows of the x pixel coordinate, (y s , y s + h) represents the y s to y s + h rows of the y pixel coordinate. By intercepting the pixel coordinates of the input image, the effect of image cropping can be achieved, and the required area can be extracted as the cropped image.

[0153] In step 4, various existing neural networks (i.e., prediction models) are used to identify the input image, and the recognition parameters H at the current moment are predicted in real time. t 、HT t 、C t 。

[0154] Under the background of the current development of artificial intelligence technology, there has been a certain amount of technical accumulation in the industry regarding the network prediction model related to sea wave images. When a sea wave image is input into the network, the basic data of the wave can be predicted and recognized. The basic data includes three types of recognition parameters: H t 、HT t 、C t 。

[0155] To meet the requirements of safe and efficient construction of deep - sea and far - sea projects and the need for wave basic data in the development of intelligent dredging technology, examples of the achievements accumulated in this field are as follows:

[0156] "Automatic Detection of Ocean Internal Waves in South China Sea Satellite SAR Images Based on Faster R - CNN". For the recognition of wave direction C, the Faster R - CNN framework is adopted. For sea wave images, the method of transfer learning is used, and the automatic detection of ocean internal waves on the image is realized by training a large amount of data. The input parameter of this model is the pre - processed sea surface image. (Journal: Journal of Remote Sensing, Volume 27, Issue 4, 2023, Pages: 905 - 918, Paper Publication Date: 2023 - 04 - 07)

[0157] "Engineering - scale Wave Recognition Method Based on Machine Vision". An unmanned aerial vehicle is used to conduct patrol surveys on the sea area environment around the construction ship to obtain real - time image information. A deep neural network model is applied to recognize the wave characteristics in the engineering sea area, achieving the purpose of real - time prediction of the wave height H and period HT of the waves in the construction area. (On July 28, 2023, CN116503765A)

[0158] "A Method for Improving the High Precision of Predicted Waves in Construction Areas Based on Image Prediction Models" proposes to combine the survey data with the model data. An unmanned aerial vehicle is equipped with an acoustic wave - measuring device to supplement the local wave observation data during construction, solving the problem of lack of some data. In addition, a model ConvLSTM combining convolutional operation and long short - term memory network (LSTM) is proposed. The performance of this model network in capturing the spatio - temporal changes of sequences is superior to that of deep feed - forward neural networks and other advanced machine learning algorithms. It provides technical support for the high - precision regional wave prediction required by engineering operation ships.

[0159] (On February 13, 2024, CN117556932A)

[0160] Vol. 58, No. 24 / December 2021 / Laser & Optoelectronics Progress Research Paper Nearshore Wave Period Detection Based on Video Spatiotemporal Feature Learning; See

[0161] https: / / www.researching.cn / ArticlePdf / m00002 / 2021 / 58 / 24 / 2401001.pdf

[0162] "CCTV-Based Sea Condition Classification and Wave Height Identification on Sea Routes", See

[0163] https: / / www.sohu.com / a / 583618087_121051793

[0164] In the text, the "prediction model" and "model" refer to various existing neural networks described in Step 4. Images are input into such neural networks, and the neural networks identify and predict the basic data of the waves: H t , HT t , C t . This part of Step 4 is not the innovative part of the present invention.

[0165] Step 5. Iteratively update the monitoring parameters and the hardware shooting parameters

[0166]

Innovation and Key Steps

[0167] The process is as follows ([ Figure 1 shown):

[0168] Step 5.1. Using the current ship speed V and the current ship heading S, calculate the relative ship speed Vs affected by the waves. The relative ship speed:

[0169]

[0170] Step 5.2. Set the target monitoring distance M according to the ship monitoring requirements, and calculate the wave height recognition accuracy parameter e t , e t = V s / M * 0.1 + (H - hb) / hb * 0.9;

[0171] Then compare it with the accuracy e t-1 at the previous moment t - 1, and make adjustments accordingly:

[0172] If e t > e t-1 , then adjust the focal length f = f - 10 * sz * lmax / (H - hb). Since the parameter setting of the focal length f of the camera is too large, it needs to be adjusted smaller;

[0173] If e t < e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H - hb). Since the measured wave height range is too large, the focal length f needs to be increased;

[0174] Step 5.3, calculate the acquisition parameters

[0175] Adjust according to Step 5.2, and calculate the parameters of the forward or reverse adjusted focal length f and the angle as of the camera's central axis downward horizontally;

[0176] Based on the ship's heading S at the current moment t t and the identified wave direction C t , calculate the shooting direction D at the next moment t + 1 according to formula (8) t+1 ;

[0177] Calculate the next shooting time interval T collected by the binocular optical camera, T = Min(0.25, M / Vs) (9);

[0178] Step 5.4, the next shooting parameters calculated in Step 5.3 are provided to Step 2.1, and Step 2.1 is started after waiting for T time;

[0179] Perform continuous cyclic iteration.

[0180] Parameter annotation:

[0181] Monitoring parameter: That is, the recognition accuracy, specifically the wave height recognition accuracy parameter e.

[0182] The ship speed V (unit: km / h), ship heading angle S (unit: °), monitoring accuracy factor e, monitoring target distance M (unit: km), sea surface wave height H (unit: m), wave period HT (unit: s), estimated wave direction C0, wave direction C (unit: °) of the engineering ship.

[0183] Embodiment 2

[0184] Based on Figure 1 and the algorithm design principle of Embodiment 1, the shipborne monitoring system of the present invention is further introduced.

[0185] As Figure 2 shown in the system composition diagram, the present invention further discloses a shipborne monitoring system for offshore engineering waters, including a sensing and acquisition system, a target setting module, a preprocessing system, an intelligent network recognition system, and a shooting and recognition accuracy real-time regulation system. The sensing and acquisition system includes an image acquisition and acquisition system, a ship GPS module, and a storage module; wherein:

[0186] The sensing and acquisition system is synchronized with the ship's navigation. Among them, the image acquisition and acquisition system is selected as a high-precision night vision binocular camera. The binocular camera is the acquisition hardware and is installed at the head of the ship. As Figure 4 shown, it collects wave surface image data in the form of optical images, and its execution is controlled by the shooting and recognition accuracy real-time regulation system;

[0187] The ship GPS module outputs the current ship speed V and the current ship heading S;

[0188] The storage module stores the original video data, the initial installation parameters, and the data output by the ship GPS module; the initial installation parameters are the installation position height of the camera on the ship, and the position height is also called the shooting height hc;

[0189] The target setting module sets different monitoring target distances M (i.e., monitoring ranges) for different sea conditions. When the on-board system enters the monitoring target distance M < 2000m, the system adjusts the image acquisition and obtaining system to enable the high-precision night vision binocular camera;

[0190] The preprocessing system performs real-time preprocessing on the original sea area wave images stored in the image acquisition and obtaining system to form standardized images;

[0191] The intelligent network recognition system is connected to the preprocessing system; after inputting the standardized images, it predicts and outputs the wave recognition parameters at the current moment t in real time. The wave recognition parameters include H t 、HT t 、C t ;

[0192] The shooting and recognition accuracy real-time regulation system is used to iteratively update the monitoring parameters and the hardware shooting parameters to ensure the real-time high-precision monitoring and recognition and the optimal acquisition parameters of the on-board system; the shooting and recognition accuracy real-time regulation system is respectively connected to the image acquisition and obtaining system, the intelligent network recognition system, and the target setting module; the shooting and recognition accuracy real-time regulation system includes an optimal shooting parameter module and a real-time recognition accuracy regulation module, which are interconnected;

[0193] The optimal shooting parameters at the current moment t are determined by the shooting and recognition accuracy real-time regulation system, so that the image acquisition and obtaining system performs shooting under the optimal shooting parameters at this moment to obtain and store the original video data;

[0194] The optimal shooting parameter module further calculates and optimizes the shooting parameters of the optical hardware sensing device (camera) at the current moment t according to the initial installation parameters in the storage module. The shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0195] The optimal shooting parameter module adjusts the center line of the high-definition imaging device to coincide with the shooting direction D based on the calculated optimal parameter shooting direction D, and

[0196] based on the calculated optimal parameter farthest shooting area distance lmax;

[0197] Furthermore, the image acquisition and obtaining system captures the sea area wave image at the current moment t and stores the original wave image in the form of a video;

[0198] The recognition accuracy real-time regulation module uses the current ship speed V and the current ship heading S provided by the ship GPS module to calculate the relative ship speed Vs affected by the waves. The relative ship speed is:

[0199] The recognition accuracy real-time regulation module sets the target monitoring distance M according to the ship monitoring requirements and calculates the wave height recognition accuracy parameter e at the current moment t t , e t = V s / M * 0.1 + (H - hb) / hb * 0.9;

[0200] Furthermore, it is compared with the accuracy e at the previous moment t - 1, and adjustments are made accordingly: t-1 If e

[0201] > e t , then adjust the focal length f = f - 10 * sz * lmax / (H - hb). Since the parameter setting of the focal length f of the camera is too large, it needs to be adjusted smaller; t-1 If e

[0202] < e t < e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H - hb). Since the measured wave height range is too large, the focal length f needs to be increased;

[0203] The recognition accuracy real-time regulation module calculates the acquisition parameters and provides them to the shooting parameter optimization module. Specifically:

[0204] According to the above adjustments, calculate the parameters of the forward or reverse adjustment of the focal length f and the angle as of the camera's central axis horizontally downward;

[0205] Based on the ship heading S at the current moment t t , the recognized wave direction C t , calculate the shooting direction D at the next moment t + 1 according to formula (8) t+1 ;

[0206] Calculate the next shooting time interval T collected by the binocular optical camera, T = Min(0.25, M / Vs)(9);

[0207] The above calculation results are transmitted to the image acquisition and obtaining system through the shooting parameter optimization module to issue an instruction. After waiting for T time, the image acquisition and obtaining system performs shooting;

[0208] The on-board monitoring system is synchronized with the ship's navigation and continuously iterates over time to ensure real-time, highly accurate monitoring and identification as well as optimal acquisition parameters for the on-board system.

[0209] Furthermore, the target setting module: determines the theoretical monitoring range of the camera according to the monitoring requirements. According to the international standard sea state level classification, in the present invention, by means of variable focal length, different monitoring target distances M are determined for different sea states, that is, the monitoring range:

[0210] Sea state level Monitoring target distance M (L: ship's length) Level 0 - 1 3L Level 2 - 3 6L Level 4 - 5 9L Level 6 and above 12L

[0211] The shooting and recognition accuracy real-time regulation system includes an optimal shooting parameter module and a recognition accuracy real-time regulation module, which are interconnected;

[0212] The optimal shooting parameter module determines the optimal parameter shooting direction D and provides it to the image acquisition and obtaining system to control the image to capture the wave direction movement.

[0213] Specifically, the optimal shooting parameter module constructs calculation formula (8), that is, the calculation formula for the shooting direction D is:

[0214]

[0215] The course S of the engineering ship (unit: °);

[0216] The shooting direction D (unit: °);

[0217] The horizontal field of view angle acw;

[0218] In order to adjust and control the wave direction accuracy to the highest level, the present invention designs formula (8) to determine the relationship between the camera's central axis and the wave direction during the monitoring process and the adjustment method. When the camera is installed on the engineering ship, there is an inclination in visual monitoring, and when the included angle between the wave direction and the camera's central axis is 0 or 180.

[0219] The optimal shooting parameter module determines the farthest shooting area distance (lmax), calculates the farthest shooting area distance lmax and the angle as of the camera's central axis horizontally downward, and provides them to the image acquisition and obtaining system to perform shooting.

[0220] Specifically, the farthest monitoring distance lmax is calculated by the following formula:

[0221] lmax = hb * f / (sz * 10) (1)

[0222] Where, the high resolution hb, the focal length f, the height sz of a single pixel of the cmos,

[0223] The calculation of the angle as of the camera's central axis horizontally downward:

[0224] Height field of view angle ach, ach = 2 * arctan(cmosh / (2f)) (2)

[0225] Horizontal field of view angle acw, acw = 2 * arctan(cmosw / (2f)) (3)

[0226] Camera central axis horizontal downward angle as, as = arctan(hc / lmax) + ach / 2 (4)

[0227] Parameter annotation and algorithm design description:

[0228] cmosh is the height of the camera's CMOS image sensor, and cmosw is the width of the camera's CMOS image sensor.

[0229] The preprocessing system, a non-critical design module, may include the following functional modules:

[0230] 3.1 Image screening module, which focuses on the collected video images centered on the operation sea area, processes them frame by frame, and forms an effective image dataset.

[0231] 3.2, Image normalization module, which forms normalized images from images taken under different distances and different weather (sunny / cloudy) conditions.

[0232] 3.2.1 Image flipping module

[0233] The flipping operation of the image. In addition, for the direction error images caused by reasons such as shooting or scanning angles, the image can be corrected in the correct direction through image flipping, improving the recognition accuracy.

[0234] 3.2.2 Light transformation module

[0235] Perform light change processing on the wave image data, adjust based on the pixel values of the image to change the light characteristics such as brightness, contrast, and color of the image, and form a wave training set with different brightness intervals from 0.4 to 2.0 and different contrast intervals from 0.4 to 2.0, so as to enhance the generalization ability of the established deep convolutional network.

[0236] By scaling or offsetting the brightness value of each pixel, the brightness of the entire wave image is changed. The adjustment process can be achieved by multiplying the RGB channel value of each pixel by the brightness factor or adding the brightness offset value. The following formula expresses the mathematical principle process of brightness transformation:

[0237] L out = α * L in + cl (19)

[0238] In the formula, L out represents the pixel output value of the wave image, Lin represents the input value of the pixel, α represents the scaling factor for brightness adjustment, and cl represents the brightness adjustment constant.

[0239] 3.2.3 Image Scaling and Cropping Module

[0240] In the wave image processing task, the primary task is to adjust the image to a fixed size for input into an algorithm or model, i.e., image scaling. This is achieved through an interpolation algorithm.

[0241] 3.2.3.1 Image Bilinear Interpolation Scaling Module

[0242] By performing a weighted average of the gray values of the four surrounding original pixels to determine the gray value of the new pixel, more refined interpolation can be achieved. Bilinear interpolation first determines the position of the target pixel in the original image and then calculates the weighted average of the gray values of the four pixels adjacent to the target pixel to determine the gray value of the new pixel. This interpolation method utilizes the local gray value changes in the original image. Therefore, it can provide smoother and more accurate results compared to nearest-neighbor interpolation. In bilinear interpolation, assuming the width of the original wave image is W, the height is TH", the width of the target image is W', and the height is H', the gray value of the target pixel can be calculated using the following formula:

[0243] X = X' * (TW - 1) / (TW' - 1) (22)

[0244] Y = Y' * (TH" - 1) / (TH' - 1) (23)

[0245] I'(X', Y') = I(X, Y) (24)

[0246] Find the four nearest pixels (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2) in the original image, where (X1, Y1) is the top-left pixel closest to (X, Y) and (X2, Y2) is the bottom-right pixel closest to (X, Y). The gray value of the target image coordinates (X', Y') is calculated by bilinear interpolation based on these four nearest pixels:

[0247] I'(X', Y') = (1 - dX) * (1 - dY) * I(X1, Y1)

[0248] + dX * (1 - dY) * I(X2, Y1)

[0249] + (1 - dX) * dY * I(X1, Y2)

[0250] + dX * dY * I(X2, Y2) (25)

[0251] Among them, I(X, Y) represents the gray value at the original image coordinates (X, Y), I'(X', Y') represents the gray value at the target image coordinates (X', Y'), dX = X - x, dY = Y - y, and x, y represent the floor of X, Y.

[0252] 3.2.3.2 Image Cropping Module

[0253] Extract a specific area from the original image to generate an image with new dimensions, achieving the purpose of removing partial noise interference from the original picture and retaining the original features of the image, so as to facilitate feature extraction and analysis processing in the subsequent convolution process. It can improve the accuracy and efficiency of subsequent processing.

Claims

1. A ship-borne monitoring system for marine engineering waters, characterized in that: include: A sensing acquisition system, a target setting module, a preprocessing system, an intelligent network recognition system, and a real-time control system for shooting and recognition accuracy. The sensing acquisition system includes an image acquisition system, a ship GPS module, and a storage module.

2. A ship-borne monitoring system for marine engineering waters according to claim 1, characterized in that: The sensing acquisition system is synchronized with the navigation of the ship, wherein the image acquisition system is selected to be a high-precision night vision binocular camera. The binocular camera is acquisition hardware installed on the bow of the ship to collect wavefront image data in the form of optical images, and its execution is controlled by a real-time control system for shooting and recognition accuracy.

3. A ship-borne monitoring system for marine engineering waters according to claim 1, characterized in that: The ship GPS module outputs the current ship speed V and the current ship heading S.

4. A ship-borne monitoring system for marine engineering waters according to claim 1, characterized in that: The storage module stores raw video data, initial installation parameters and output data of the ship GPS module; the initial installation parameters are the height of the camera installation position on the ship, and the height is also called the shooting height hc.

5. A ship-borne monitoring system for marine engineering waters according to claim 1, characterized in that: The target setting module sets different monitoring target distances M, i.e., monitoring ranges, for different sea conditions. When the shipborne system enters a monitoring target distance M<2000m, the system adjusts the image acquisition system to enable the high-precision night vision binocular camera.

6. A ship-borne monitoring system for marine engineering waters according to claim 1, characterized in that: The preprocessing system performs real-time preprocessing on the original sea wave image stored in the image acquisition system to form a standardized image.

7. A ship-borne monitoring system for marine engineering waters according to claim 1, characterized in that: The intelligent network recognition system is connected to the preprocessing system; after inputting the standardized image, the wave recognition parameters at the current time t are predicted and output in real time, and the wave recognition parameters include H t , HT t , C t ; H t is the sea surface wave height at time t, in meters; HT t is the wave period at time t, in seconds; C t is the wave direction at time t, in units of o; The shooting and recognition accuracy real-time control system is used to iteratively update monitoring parameters and hardware shooting parameters to ensure real-time and high-precision monitoring and recognition and optimal acquisition parameters of the shipborne system; the shooting and recognition accuracy real-time control system is connected to the image acquisition system, the intelligent network recognition system, and the target setting module respectively; the shooting and recognition accuracy real-time control system includes a shooting parameter optimization module and a recognition accuracy real-time control module, which are interconnected; The optimal shooting parameters at the current time t are determined by the real-time control system for shooting and recognition accuracy, so that the image acquisition system performs shooting under the optimal shooting parameters at the time to obtain and store the original video data; The shooting parameter optimization module further calculates and optimizes the shooting parameters of the optical hardware sensor device at the current moment t according to the initial installation parameters in the storage module, wherein the shooting parameters include a shooting direction D and a distance lmax of the farthest shooting area; The shooting parameter optimization module adjusts the center line of the high-definition camera device to coincide with the shooting direction D based on the optimal parameter shooting direction D determined by calculation, and The farthest shooting area distance lmax determined based on the optimal parameters calculated; Then, the image acquisition system takes the sea wave image at the current time t and stores the original wave image in the form of video; The recognition accuracy real-time control module uses the current ship speed V and the current ship heading S provided by the ship GPS module to calculate the relative ship speed Vs under the influence of waves. The relative ship speed is: Where g is the acceleration due to gravity; The recognition accuracy real-time control module sets the target monitoring distance M according to the ship monitoring requirements and calculates the wave height recognition accuracy parameter e at the current moment. t , e t =V s / M*0.1+(H-hb) / hb*0.9; Then, the accuracy e of the last time t-1 is t-1 Compare and adjust accordingly: If e t >e t-1 , then adjust the focal length f = f-10*sz*lmax / (H-hb). Since the focal length f parameter of the camera is set too large, it needs to be reduced; If e t <e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H-hb), because the measured wave height range is too large, so the focal length f needs to be increased; The recognition accuracy real-time control module calculates the acquisition parameters and provides them to the shooting parameter optimization module, specifically: According to the above adjustment, calculate the parameters of positive or negative adjustment of focal length f and horizontal downward angle as of the camera center axis; According to the current time t, the ship's heading S t , identify the wave direction C t , calculate the shooting direction D at the next moment t+1 according to the calculation formula of shooting direction D t+1 ; Calculate the next shooting time interval T collected by the binocular optical camera, T = Min (0.25, M / Vs); The above calculation results are transmitted to the image acquisition system through the shooting parameter optimization module, and the image acquisition system executes shooting after waiting for T time; The shipboard monitoring system is synchronized with the ship's navigation, follows time, and iterates continuously to ensure real-time and highly accurate monitoring and identification, and optimal acquisition parameters.

8. A ship-borne monitoring system for marine engineering waters according to claim 7, characterized in that: The target setting module: determines the theoretical monitoring range of the camera according to the monitoring requirements, and determines different monitoring target distances M, i.e., monitoring ranges, for different sea conditions by zooming according to the international standard sea condition classification: When the sea condition level is 0 to 1, the monitoring target distance M is 3L; When the sea condition level is 2-3, the monitoring target distance M is 6L; When the sea condition level is 4-5, the monitoring target distance M is 9L; When the sea condition level is 6 or above, the monitoring target distance M is 12L; Where L is the captain of the ship.

9. A ship-borne monitoring system for marine engineering waters according to claim 7, characterized in that: The shooting parameter optimization module constructs the calculation formula (8), that is, the calculation formula of the shooting direction D is: Engineering ship heading S, unit o; Shooting direction D, unit o; Horizontal field of view acw; Wave direction C, unit o.

10. A ship-borne monitoring system for marine engineering waters according to claim 7, characterized in that: The shooting parameter optimization module determines the farthest shooting area distance lmax, determines the farthest shooting area distance lmax and the horizontal downward angle as of the camera center axis through calculation, and provides them to the image acquisition system to perform shooting; Specifically, the maximum monitoring distance lmax is calculated by the following formula: lmax=hb*f / (sz*10) (1) Among them, high resolution hb, focal length f, CMOS single pixel height sz, The horizontal downward angle of the camera axis is calculated as: Altitude field angle ach, ach = 2*arctan(cmosh / (2f))(2) Horizontal field of view acw, acw = 2*arctan(cmosw / (2f))(3) The horizontal angle of the camera axis is as, as=arctan(hc / lmax)+ach / 2(4) Parameter annotation and algorithm design description: Cmosh is the height of the camera's photosensitive device CMOS, cmosw is the width of the camera's photosensitive device CMOS, and hc is the shooting height.

Images