A shipborne image acquisition method for synchronous monitoring of marine engineering waters

Abstract

A shipborne image acquisition method for marine engineering waters, characterized by comprising: step 1. algorithm operation conditions; step 2. determining the optimal shooting parameters at the current time t, performing shooting under the optimal shooting parameters at the time to obtain original video data; preprocessing the original sea wave image shot in step 2 to form a standardized image; step 4. using an existing neural network to recognize the input standardized image, and predicting the recognition parameter H at the current time t in real time t , HT t , C t ; Step 5. Iteratively update the monitoring parameters and hardware shooting parameters to ensure real-time and highly accurate monitoring and identification and optimal acquisition parameters of the shipborne system.

Description

Technical Field

[0001] The present application relates to mobile image recognition technology, which is applied on board a ship, specifically a ship-borne image acquisition method for marine engineering waters, and a ship-borne monitoring system for marine engineering waters. Background Art

[0002] Waves are a key factor affecting the implementation of marine engineering projects. The safety of engineering vessels, construction efficiency and the operation of key equipment are all significantly affected by the waves in the surrounding waters. As my country's dredging industry continues to develop into overseas markets, the operating waters have expanded from offshore to offshore, and the wave conditions have become increasingly complex, significantly affecting the construction efficiency. The safety of engineering vessels has also been subject to great challenges.

[0003] In the process of developing a new generation of dredging ships, new technologies such as big data and artificial intelligence are being used more and more widely. As the cornerstone of the development of intelligent dredging, the synchronous monitoring of basic data of the hydrological environment of the operating sea area is an extremely critical component for the digitalization of the dredging process. However, the acquisition of wave data for the sea area currently relies mainly on specialized hydrological, meteorological and maritime management agencies, and the data obtained are also limited to large-scale regional forecasts and monitoring of individual relatively fixed stations. For engineering ships sailing in different waters for construction, the existing hydrological technology or acquisition methods have poor real-time performance, low integration, and low accuracy, and cannot achieve real-time monitoring synchronized with construction, which does not meet the needs of safe and efficient construction of deep-sea projects and the development of intelligent dredging technology for basic data.

[0004] Waves have a significant impact on the safety and efficiency of dredging vessels and are a key influencing parameter of dredging projects. At present, the dredging industry is developing intelligent dredging, which requires the collection of data on the entire dredging process, including ship data and environmental data such as waves in the surrounding waters, and intelligent analysis is carried out based on this data. Therefore, synchronous monitoring of waves during construction is very important.

[0005] At present, shore-based wave image recognition can only effectively identify wave parameters in fixed sea areas, because the observation equipment is fixed, so the monitoring range and reference system are relatively clear. Since ships are operating while sailing, the image acquisition method that relies on dropping instruments on the water surface is obviously not applicable.

[0006] Mobile image recognition technology in the field of smart cars requires that the vehicle-mounted scenes to be recognized have obvious road and building line separations (such as road markings, street building facades, etc.), so when collecting images, the scope and objects of recognition are easier to determine. However, waves in open sea scenes do not have obvious, stable and unchanging line separations. Due to factors such as sea area, wave propagation, ship navigation, and weather influences, it is difficult to directly identify them. Therefore, if the existing mobile image recognition technology in the field of smart cars is used as a ship-borne wave recognition application, due to the complexity of wave propagation in open seas, the surrounding sea environment (visibility, weather conditions), and the relative relationship between ships and waves, it is impossible to achieve effective collection and recognition of wave images. Summary of the invention

[0007] Under the background of the current development of artificial intelligence technology, the network prediction model related technology based on ocean wave images has accumulated certain technical accumulation in the industry. The ocean wave image can be input into the network to predict and identify the basic data of the wave. The purpose of the present invention is to further disclose a ship-borne image acquisition method for synchronous monitoring of marine engineering waters, and a ship-borne monitoring system for marine engineering waters, which optimizes the image acquisition parameters under navigation state through existing ships and their ship-borne cameras, operating state parameters, etc., so as to realize high-precision recognition and monitoring of waves in the operating sea area.

[0008] Technical Solution 1

[0009] A method for acquiring shipborne images in marine engineering waters, characterized by comprising:

[0010] Step 1 Algorithm operating conditions

[0011] Install a high-precision night vision binocular camera as the acquisition hardware and determine the initial installation parameters;

[0012] When the system enters the monitoring target distance M < 2000m, the system controls and enables the binocular camera to collect wavefront image data in the form of optical images;

[0013] Determine different monitoring target distances M, i.e. monitoring ranges, for different sea conditions;

[0014] Step 2. Determine the optimal shooting parameters at the current time t, and perform shooting under the optimal shooting parameters at this time to obtain the original video data

[0015] 2.1 According to the initial installation parameters in step 1, further optimize the shooting parameters of the optical hardware sensor device (camera) at the current time t, wherein the shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0016] 2.2 Based on the optimal parameter shooting direction D calculated and determined in step 2.1, adjust the center line of the high-definition camera device so that it coincides with the shooting direction D, and

[0017] Based on the optimal parameter farthest shooting area distance lmax calculated and determined in step 2.1, the sea wave image at the current time t is shot, and the original wave image in video form is stored;

[0018] Step 3. Preprocessing

[0019] Preprocessing the original sea wave image taken in step 2 to form a standardized image;

[0020] Step 4. Use the existing neural network to identify the input standardized graph and predict the recognition parameter H at the current time t in real time. t , HT t , C t ;

[0021] Step 5. Iteratively update monitoring parameters and hardware shooting parameters to ensure real-time and highly accurate monitoring and identification and optimal acquisition parameters of the shipborne system.

[0022] Calculate the next shooting time interval and wait for T time before going into step 2 to adjust the equipment and perform shooting, and repeat this process.

[0023] Technical Solution 2

[0024] A ship-borne monitoring system for marine engineering waters, including a sensor acquisition system, a target setting module, a preprocessing system, an intelligent network recognition system, and a real-time control system for shooting and recognition accuracy. The sensor acquisition system includes an image acquisition system, a ship GPS module, and a storage module; wherein:

[0025] The sensor acquisition system is synchronized with the ship's navigation, wherein the image acquisition system is selected as a high-precision night vision binocular camera, which is the acquisition hardware and is installed at the head of the ship. Figure 4 As shown, wavefront image data is collected in an optical image manner, and its execution is controlled by a real-time control system for shooting and recognition accuracy;

[0026] The ship GPS module outputs the current ship speed V and the current ship heading S;

[0027] The storage module stores raw video data, initial installation parameters and output data of the ship GPS module; the initial installation parameters are the height of the camera installation position on the ship, and the height is also called the shooting height hc;

[0028] The target setting module sets different monitoring target distances M, i.e., monitoring ranges, for different sea conditions. When the shipborne system enters a monitoring target distance M < 2000m, the system adjusts the image acquisition system to enable the high-precision night vision binocular camera;

[0029] The preprocessing system performs real-time preprocessing on the original sea wave image stored in the image acquisition system to form a standardized image;

[0030] The intelligent network recognition system is connected to the preprocessing system; after inputting the standardized image, the wave recognition parameters at the current time t are predicted and output in real time, and the wave recognition parameters include H t , HT t , C t ;

[0031] The shooting and recognition accuracy real-time control system is used to iteratively update monitoring parameters and hardware shooting parameters to ensure real-time and high-precision monitoring and recognition and optimal acquisition parameters of the shipborne system; the shooting and recognition accuracy real-time control system is connected to the image acquisition system, the intelligent network recognition system, and the target setting module respectively; the shooting and recognition accuracy real-time control system includes a shooting parameter optimization module and a recognition accuracy real-time control module, which are interconnected;

[0032] The optimal shooting parameters at the current time t are determined by the real-time control system for shooting and recognition accuracy, so that the image acquisition system performs shooting under the optimal shooting parameters at the time to obtain and store the original video data;

[0033] The shooting parameter optimization module further calculates and optimizes the shooting parameters of the optical hardware sensor device (camera) at the current moment t according to the initial installation parameters in the storage module, wherein the shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0034] The shooting parameter optimization module adjusts the center line of the high-definition camera device to coincide with the shooting direction D based on the optimal parameter shooting direction D determined by calculation, and

[0035] The farthest shooting area distance lmax determined based on the optimal parameters calculated;

[0036] Then, the image acquisition system takes the sea wave image at the current time t and stores the original wave image in the form of video;

[0037] The recognition accuracy real-time control module uses the current ship speed V and the current ship heading S provided by the ship GPS module to calculate the relative ship speed Vs under the influence of waves. The relative ship speed is:

[0038] The recognition accuracy real-time control module sets the target monitoring distance M according to the ship monitoring requirements and calculates the wave height recognition accuracy parameter e at the current moment. t , e t =V s / M*0.1+(H-hb) / hb*0.9;

[0039] Then, the accuracy e of the last time t-1 is t-1 Compare and adjust accordingly:

[0040] If e t >e t-1 , then adjust the focal length f = f-10*sz*lmax / (H-hb). Since the focal length f parameter of the camera is set too large, it needs to be reduced;

[0041] If e t <e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H-hb), because the measured wave height range is too large, so the focal length f needs to be increased;

[0042] The recognition accuracy real-time control module calculates the acquisition parameters and provides them to the shooting parameter optimization module, specifically:

[0043] According to the above adjustment, calculate the parameters of positive or negative adjustment of focal length f and horizontal downward angle as of the camera center axis;

[0044] According to the current time t, the ship's heading S t , identify the wave direction C t , calculate the shooting direction D at the next time t+1 according to formula (8) t+1 ;

[0045] Calculate the next shooting time interval T collected by the binocular optical camera, T = Min (0.25, M / Vs) (9);

[0046] The above calculation results are transmitted to the image acquisition system through the shooting parameter optimization module, and the image acquisition system executes shooting after waiting for T time;

[0047] The shipboard monitoring system is synchronized with the ship's navigation, follows time, and iterates continuously to ensure real-time and highly accurate monitoring and identification, and optimal acquisition parameters.

[0048] Summary: According to the size L of the ship and navigation parameters (direction, speed), the range of wave image shooting (monitoring target distance M) and recognition accuracy are determined in real time to capture images of the sea surface in key areas, thereby achieving the effectiveness of shipborne image acquisition and laying the foundation for effective identification of wave parameters in the industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Example 1 Principle of the process of the present invention

[0050] Figure 2 Embodiment 2 System composition of the present invention

[0051] Figure 3 Schematic diagram of monitoring direction adjustment of a ship in navigation

[0052] Figure 4 Schematic diagram of the arrangement of binocular cameras on the top deck of the ship's wheelhouse and the scene of monitoring the sea surface: a. Overall side view; b. Schematic diagram of the relationship between shooting parameters DETAILED DESCRIPTION

[0053] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0054] Example 1

[0055] The whole algorithm design of the present invention is as follows: Figure 1 shown.

[0056] A method for acquiring shipborne images in marine engineering waters, characterized by comprising:

[0057] Step 1 Algorithm operating conditions

[0058] Install a high-precision night vision binocular camera as the acquisition hardware, and determine the initial installation parameters, including the camera installation position height on the ship, which is also called the shooting height hc;

[0059] When the system enters the monitoring target distance M<2000m, the system will control and enable the high-precision night vision binocular camera to collect wavefront image data in the form of optical images;

[0060] The theoretical monitoring range of the camera is determined according to the monitoring requirements. According to the international standard sea condition classification, the present invention determines different monitoring target distances M, i.e., monitoring ranges, for different sea conditions by zooming:

[0061] Sea state level Monitoring target distance M (L: ship length) Level 0-1 3L Level 2-3 6L Level 4-5 9L Level 6 and above 12L

[0062] Step 2. Determine the optimal shooting parameters at the current time t, and perform shooting under the optimal shooting parameters at this time to obtain the original video data

[0063] 2.1 According to the initial installation parameters in step 1, further optimize the shooting parameters of the optical hardware sensor device (camera) at the current time t, wherein the shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0064] 2.2 Based on the optimal parameter shooting direction D calculated and determined in step 2.1, adjust the center line of the high-definition camera device so that it coincides with the shooting direction D, and

[0065] The optimal parameter farthest shooting area distance lmax is calculated and determined based on step 2.1;

[0066] Then, the sea wave image at the current time t is captured, and the original wave image is stored in the form of a video.

[0067] Step 3. Preprocessing

[0068] The original sea wave image taken in step 2 is preprocessed by flipping, light transformation, and scaling, so that the target sea wave images taken at different distances and under different weather conditions (sunny / cloudy) form standardized images.

[0069] Step 4. Use the existing neural network (i.e., prediction model) to identify the input standardized graph and predict the recognition parameter H at the current time t in real time. t , HT t , C t .

[0070] Step 5. Iteratively update monitoring parameters and hardware shooting parameters

[0071] Iteratively adjust the recognition and monitoring accuracy of the system at the current time t:

[0072] Based on the ship's heading S at the current time t t , Ship speed V t , and the identification parameter H at the current time t t , HT t , C t , calculate the current recognition accuracy e t , determine the difference between the current accuracy and the previous accuracy, adjust the accuracy according to the calculation, and calculate the parameters of the forward or reverse adjustment of the focal length f and the horizontal downward angle as of the camera center axis;

[0073] According to the current time t, the ship's heading S t , identify the wave direction C t , calculate the shooting direction D at the next time t+1 according to formula (8) t+1 ;

[0074] Formula (9) is used to calculate the recognition period (i.e., the shooting time interval T);

[0075] The calculated result is used to adjust the camera focal length f at the next moment t+1 t+1 、The horizontal downward angle of the camera axis as t+1 , Shooting direction D t+1 and identification period T t+1 , and feed it back to step 2 to adjust the acquisition parameters of the binocular optical camera accordingly;

[0076] After waiting for T time, the system enters step 2 for calculation and collection;

[0077] The above iterations ensure real-time and highly accurate monitoring and identification of the shipborne system and optimal acquisition parameters.

[0078] The details of the algorithm of the present invention are described in detail below

[0079] Step 2.1 [Innovation and key steps]

[0080] Step 2.1.1 Determine the optimal parameter shooting direction D

[0081] In order to adjust the monitoring direction, such as Figure 3 As shown, the optimal parameter shooting direction D is adjusted and determined, and provided to the shooting step (step 2.2) to control the image to capture the wave motion

[0082] Construct calculation formula (8), that is, the calculation formula of the shooting direction D is:

[0083]

[0084] Parameter annotation and description:

[0085] Engineering vessel heading S (unit: °);

[0086] Shooting direction D (unit: °);

[0087] Horizontal field of view acw;

[0088] In order to adjust the wave direction to achieve the highest accuracy, the present invention designs formula (8) to determine the relationship between the camera center axis and the wave direction and the adjustment method during the monitoring process. The camera is installed on an engineering vessel and visual monitoring is tilted. In the algorithm design, the present invention sets the angle between the wave direction and the camera center axis to 0 or 180.

[0089] Wave direction C (unit: °);

[0090] The wave direction C is identified by a network model. The network model is not a technical contribution of the present invention. All existing technologies in the industry are used. For example, but not limited to: the Faster R-CNN framework can be used to use the transfer learning method for wave images. The automatic detection of ocean internal waves on the image can be achieved by training a large number of data images. The input parameter of the prediction model is the preprocessed sea surface image. For specific methods, please refer to the article "Automatic Detection of South China Sea Ocean Internal Waves from Satellite SAR Images Based on Faster R-CNN" Journal: Journal of Remote Sensing 2023 Volume 27 Issue 4 Page Number: 905-918 Paper Publication Date: 2023-04-07

[0091] 2.1.2 Determine the distance of the farthest shooting area (lmax)

[0092] Calculate and determine the farthest shooting area distance lmax and the horizontal downward angle as of the camera center axis, and provide them to step 2.2 to execute the shooting step and input them into step 5.

[0093] (1) The maximum monitoring distance lmax is calculated by the following formula:

[0094] lmax=hb*f / (sz*10)(1)

[0095] Among them, high resolution hb, focal length f, CMOS single pixel height sz,

[0096] (2) Calculation of the horizontal downward angle as of the camera's central axis:

[0097] like Figure 4 In the scene and geometric diagram shown, the shooting area is a trapezoidal area, and the relationship is as follows:

[0098] Altitude field angle ach, ach = 2*arctan(cmosh / (2f))(2)

[0099] Horizontal field of view acw, acw = 2*arctan(cmosw / (2f))(3)

[0100] The horizontal angle of the camera axis is as, as=arctan(hc / lmax)+ach / 2(4)

[0101] Parameter annotation and algorithm design description:

[0102] Cmosh is the height of the camera's photosensitive device CMOS, and cmosw is the width of the camera's photosensitive device CMOS;

[0103] The maximum monitoring distance lmax is determined by the camera hardware and is the upper limit of the measurement distance. It is required to meet the monitoring requirements of the specified sea conditions. Otherwise, the focal length f needs to be adjusted to meet the monitoring requirements. If it is too small, large wave heights cannot be measured. If f is too small, the accuracy cannot meet the wave height precision measurement.

[0104] When the measured wave height H is smaller than the high resolution of the wave height hb, it is necessary to adjust and reduce f, and vice versa.

[0105] Step 3: Preprocessing

[0106] This part does not constitute an integral part of the core algorithm design of the present invention. In other words, this part is not the key technical contribution of the present invention; but as an application example, from the perspective of giving full play to the application effect of the technical solution of the present invention, this example further provides the following specific preprocessing process:

[0107] Step 3.1, Image screening

[0108] Since the captured wave images include irrelevant image content such as the sky and ships, which may easily have an adverse effect on the wave recognition process, the collected video images are focused on the operating sea area and processed frame by frame to form a large number of effective image data sets.

[0109] Step 3.2, image standardization, images taken at different distances and under different weather conditions (sunny / cloudy) are formed into standardized processed images.

[0110] 3.2.1 Image Flip

[0111] The purpose of image flipping is to change its direction or perspective to meet the processing requirements of data enhancement and subsequent model recognition algorithms. Image flipping is one of the most commonly used techniques in image enhancement. By randomly flipping the training data, the diversity of the data can be increased, and the robustness and generalization ability of the model can be improved. In addition, for images with directional errors caused by shooting or scanning angles, image flipping can correct them in the correct direction to improve recognition accuracy.

[0112] Image flipping methods can be divided into two types: horizontal flipping and vertical flipping. The principle is to obtain the output coordinates by subtracting the width and height of the image from the input coordinates. This process belongs to the subtraction operation of unsigned numbers. The following formula expresses the mathematical process of image flipping:

[0113] I[X in , Y in ]=O[X,Y] (14)

[0114] X in =TW-1-X in (15)

[0115] Y in =TH"-1-Y in (16)

[0116] X in ∈[0,TW) (17)

[0117] Y in ∈[0,TH”) (18)

[0118] Where I represents input, O represents output, and X in and Y in Represent the horizontal and vertical coordinates of the input pixel, respectively, and TW and TH" represent the width and height of the image.

[0119] 3.2.2 Light Transformation

[0120] The dredging ship operation scene includes the wave image process under different weather conditions. Due to the difference in shooting light, the recognition difference may be large in the deep convolutional network. Therefore, the wave image data is processed with light changes. Based on the adjustment of the pixel value of the image, the light characteristics such as brightness, contrast and color of the image are changed to form a wave training set with different brightness ranges of 0.4 to 2.0 and different contrast ranges of 0.4 to 2.0, so as to enhance the generalization ability of the established deep convolutional network.

[0121] By scaling or offsetting the brightness value of each pixel, the brightness of the entire wave image is changed. The adjustment process can be achieved by multiplying the RGB channel value of each pixel by the brightness factor or adding the brightness offset value. The following formula expresses the mathematical principle of brightness transformation:

[0122] L out =α*L in +cl (19)

[0123] Where, L out Represents the pixel output value of the wave image, L in Represents the input value of the pixel, α represents the scaling factor of the brightness adjustment, and cl represents the brightness adjustment constant.

[0124] Contrast adjustment is to change the contrast of the wave image by increasing or decreasing the difference between the light and dark pixel values. High contrast means a large difference between the light and dark parts, and low contrast means a small difference between the light and dark parts. Contrast adjustment can be achieved through linear transformation or nonlinear transformation methods such as logarithmic transformation or histogram equalization.

[0125] Logarithmic transformation can transform the grayscale concentrated in the low brightness area into a low brightness area that is equally divided into high brightness areas. The change function is:

[0126] Lt out =cx*Ln(1+r) (20)

[0127] Where r is the grayscale value of the input image, Lt out is the grayscale value of the output image, and cx is the scaling factor.

[0128] The histogram equalization method enhances the contrast of an image by redistributing the grayscale of the image. The principle is to transform the grayscale histogram of the original image into a uniformly distributed histogram, so that the brightness range of the image is wider and the details are clearer. It is usually based on the mapping of the cumulative distribution function (CDF) and the grayscale. The mapping method is:

[0129]

[0130] In the formula, S kRefers to the value of the current gray level after being mapped by the cumulative distribution function, n is the sum of the pixels in the image, n j is the number of pixels at the current gray level, and Lh is the total number of gray levels in the image.

[0131] 3.2.3 Image Scaling and Cropping

[0132] In the task of wave image processing, the first task is to adjust the image to a fixed size so as to be input into the algorithm or model, that is, image scaling. This is achieved through interpolation algorithms.

[0133] 3.2.3.1 Image Bilinear Interpolation Scaling

[0134] A more refined interpolation can be achieved by taking a weighted average of the grayscale values ​​of the four surrounding original pixels to determine the grayscale value of the new pixel. Bilinear interpolation first determines the grayscale value of the new pixel based on the position of the target pixel in the original image, and then takes a weighted average of the grayscale values ​​of the four pixels adjacent to the target pixel. This interpolation method takes advantage of the local grayscale changes in the original image and can therefore provide smoother and more accurate results than nearest neighbor interpolation. In bilinear interpolation, assuming that the width of the original wave image is W and the height is TH", and the width of the target image is W' and the height is H', the grayscale value of the target pixel can be calculated using the following formula:

[0135] X=X′*(TW-1) / (TW′-1)(22)

[0136] Y=Y′*(TH"-1) / (TH′-1)(23)

[0137] I′(X′,Y′)=I(X,Y)(24)

[0138] Find the four nearest pixels (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2) in the original image, where (X1, Y1) is the upper left pixel closest to (X, Y), (X2, Y2) is the lower right pixel closest to (X, Y), and the grayscale value of the target image coordinate (X', Y') is calculated by bilinear interpolation based on these four nearest pixels:

[0139] I'(X',Y')=(1-dX)*(1-dY)*I(X1,Y1)

[0140] +dX*(1-dY)*I(X2,Y1)

[0141] +(1-dX)*dY*I(X1,Y2)

[0142] +dX*dY*I(X2,Y2)(25)

[0143] Where I(X,Y) represents the grayscale value at the coordinate (X,Y) of the original image, I'(X',Y') represents the grayscale value at the coordinate (X',Y') of the target image, dX=Xx, dY=Yy, and x,y represent the rounding down of X,Y.

[0144] 3.2.3.2 Image Cropping

[0145] Extract a specific area from the original image and generate an image of a new size, so as to remove some noise interference from the original image and retain the original features of the image, so as to facilitate feature extraction and analysis in the subsequent convolution process. This can improve the accuracy and efficiency of subsequent processing.

[0146] During the image acquisition process, the collected images often contain interference elements of other non-essential information. By cropping the image, these interference elements can be excluded, so that the analysis can be focused on the part where feature recognition is required. In addition, the cropped image can reduce the number of processed pixels, thereby improving the efficiency of the calculation. Therefore, this technology is particularly important for the wave image scene in the working sea area of ​​the dredging process.

[0147] The mathematical principle of image cropping is based on limiting and intercepting the pixel coordinates of the image. For the input image I(x,y), the cropped wave image Q(x′,y′) can be expressed as:

[0148] Q(x′,y′)=I((x s ,x s +w),(y s ,y s +h))(26)

[0149] x s +w≤x(27)

[0150] y s +h≤y(28)

[0151] where x s ,y s Indicates the starting pixel coordinate point of the cropping, w, h indicates the width and height of the cropping, (x s ,x s +w) means to crop the xth pixel coordinate s to x s +w line, (y s ,y s +h) means to crop the yth pixel coordinate s to y s +h line. By intercepting the pixel coordinates of the input image, the image cropping effect can be achieved, and the required area can be extracted as the cropped image.

[0152] Step 4: Use various existing neural networks (i.e. prediction models) to identify the input image and predict the recognition parameter H at the current moment in real time. t , HT t , C t .

[0153] In the context of the current development of artificial intelligence technology, the network prediction model technology based on wave images has accumulated a certain amount of technology in the industry. The basic data of waves can be predicted and identified by inputting wave images into the network. The basic data includes three types of identification parameters: H t , HT t , C t .

[0154] In order to meet the needs of safe and efficient construction of deep-sea projects and the demand for basic wave data in the development of intelligent dredging technology, the achievements accumulated in this field are as follows:

[0155] "Automatic Detection of Ocean Internal Waves in the South China Sea Based on Faster R-CNN" uses the Faster R-CNN framework to identify wave directions. It uses the transfer learning method for wave images and trains a large amount of data to achieve automatic detection of ocean internal waves on images. The input parameter of the model is the preprocessed sea surface image. (Journal: Journal of Remote Sensing 2023 Vol. 27 No. 4 Pages: 905-918 Paper Publication Date: 2023-04-07)

[0156] "Engineering scale wave recognition method based on machine vision" uses drones to patrol the sea environment around construction ships to obtain real-time image information, and applies deep neural network models to identify the wave characteristics of the engineering sea area, so as to achieve the purpose of real-time prediction of the wave height H and period HT of the waves in the construction area. (Published on 2023-07-28CN116503765A)

[0157] "A method for improving the high accuracy of construction area prediction waves based on image prediction model" proposes to combine patrol data with model data, and use drones equipped with acoustic wave measurement equipment to supplement the local wave observation data of the construction, solving the problem of lack of some data. In addition, a model ConvLSTM that combines convolution operation with long short-term memory network (LSTM) is proposed. The performance of this model network in capturing the temporal and spatial changes of sequences is better than that of deep feedforward neural networks and other advanced machine learning algorithms. Provide technical support for high-precision regional wave prediction required by engineering operation ships.

[0158] (2024-02-13CN117556932A)

[0159] Volume 58, Issue 24 / December 2021 / Laser and Optoelectronics Progress Research Paper Nearshore wave period detection based on video spatiotemporal feature learning; "Sea condition classification and wave height identification based on CCTV on the route".

[0160] The "prediction model" and "model" in this article refer to the various existing neural networks described in step 4. Images are input to such neural networks, and the neural networks recognize and predict the basic data of waves: H t , HT t , C t Step 4 is not an innovative part of the present invention.

[0161] Step 5. Iteratively update monitoring parameters and hardware shooting parameters

[0162]

Innovation and key steps

[0163] The process is as follows ( Figure 1 shown):

[0164] Step 5.1, using the current ship speed V and the current ship heading S, calculate the relative ship speed Vs under the influence of waves, the relative ship speed:

[0165]

[0166] Step 5.2: Set the target monitoring distance M according to the ship monitoring requirements and calculate the wave height recognition accuracy parameter e at the current moment t , e t =V s / M*0.1+(H-hb) / hb*0.9;

[0167] Then, the accuracy e of the last time t-1 is t-1 Compare and adjust accordingly:

[0168] If e t >e t-1 , then adjust the focal length f = f-10*sz*lmax / (H-hb). Since the focal length f parameter of the camera is set too large, it needs to be reduced;

[0169] If e t <e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H-hb), because the measured wave height range is too large, so the focal length f needs to be increased;

[0170] Step 5.3, calculate acquisition parameters

[0171] Adjust according to step 5.2, and calculate the parameters of the forward or reverse adjustment of the focal length f and the horizontal downward angle as of the camera center axis;

[0172] According to the current time t, the ship's heading S t , identify the wave direction C t , calculate the shooting direction D at the next time t+1 according to formula (8) t+1 ;

[0173] Calculate the next shooting time interval T collected by the binocular optical camera, T = Min (0.25, M / Vs) (9);

[0174] Step 5.4: Calculate the next shooting parameters in step 5.3 and provide them to step 2.1. Start step 2.1 after waiting for T time.

[0175] Repeat the cycle continuously.

[0176] Parameter Notes:

[0177] Monitoring parameters: that is, recognition accuracy, specifically the wave height recognition accuracy parameter e.

[0178] The ship speed V (unit km / h) of the engineering ship, the ship heading angle S (unit °), the monitoring accuracy factor e, the monitoring target distance M (unit km), the sea surface wave height H (unit m), the wave period HT (unit s), the estimated wave direction C0, and the wave direction C (unit °).

[0179] Example 2

[0180] based on Figure 1 As well as the algorithm design principle of Example 1, the shipborne monitoring system of the present invention is further introduced.

[0181] like Figure 2 As shown in the system composition diagram, the present invention further discloses a ship-borne monitoring system for marine engineering waters, including a sensor acquisition system, a target setting module, a preprocessing system, an intelligent network recognition system, and a real-time control system for shooting and recognition accuracy. The sensor acquisition system includes an image acquisition system, a ship GPS module, and a storage module; wherein:

[0182] The sensor acquisition system is synchronized with the ship's navigation, wherein the image acquisition system is selected as a high-precision night vision binocular camera, which is the acquisition hardware and is installed at the head of the ship. Figure 4 As shown, wavefront image data is collected in an optical image manner, and its execution is controlled by a real-time control system for shooting and recognition accuracy;

[0183] The ship GPS module outputs the current ship speed V and the current ship heading S;

[0184] The storage module stores raw video data, initial installation parameters and output data of the ship GPS module; the initial installation parameters are the height of the camera installation position on the ship, and the height is also called the shooting height hc;

[0185] The target setting module sets different monitoring target distances M, i.e., monitoring ranges, for different sea conditions. When the shipborne system enters a monitoring target distance M < 2000m, the system adjusts the image acquisition system to enable the high-precision night vision binocular camera;

[0186] The preprocessing system performs real-time preprocessing on the original sea wave image stored in the image acquisition system to form a standardized image;

[0187] The intelligent network recognition system is connected to the preprocessing system; after inputting the standardized image, the wave recognition parameters at the current time t are predicted and output in real time, and the wave recognition parameters include H t , HT t , C t ;

[0188] The shooting and recognition accuracy real-time control system is used to iteratively update monitoring parameters and hardware shooting parameters to ensure real-time and high-precision monitoring and recognition and optimal acquisition parameters of the shipborne system; the shooting and recognition accuracy real-time control system is connected to the image acquisition system, the intelligent network recognition system, and the target setting module respectively; the shooting and recognition accuracy real-time control system includes a shooting parameter optimization module and a recognition accuracy real-time control module, which are interconnected;

[0189] The optimal shooting parameters at the current time t are determined by the real-time control system for shooting and recognition accuracy, so that the image acquisition system performs shooting under the optimal shooting parameters at the time to obtain and store the original video data;

[0190] The shooting parameter optimization module further calculates and optimizes the shooting parameters of the optical hardware sensor device (camera) at the current moment t according to the initial installation parameters in the storage module, wherein the shooting parameters include the shooting direction D and the farthest shooting area distance lmax;

[0191] The shooting parameter optimization module adjusts the center line of the high-definition camera device to coincide with the shooting direction D based on the optimal parameter shooting direction D determined by calculation, and

[0192] The farthest shooting area distance lmax determined based on the optimal parameters calculated;

[0193] Then, the image acquisition system takes the sea wave image at the current time t and stores the original wave image in the form of video;

[0194] The recognition accuracy real-time control module uses the current ship speed V and the current ship heading S provided by the ship GPS module to calculate the relative ship speed Vs under the influence of waves. The relative ship speed is:

[0195] The recognition accuracy real-time control module sets the target monitoring distance M according to the ship monitoring requirements and calculates the wave height recognition accuracy parameter e at the current moment. t , e t =V s / M*0.1+(H-hb) / hb*0.9;

[0196] Then, the accuracy e of the last time t-1 is t-1 Compare and adjust accordingly:

[0197] If e t >e t-1 , then adjust the focal length f = f-10*sz*lmax / (H-hb). Since the focal length f parameter of the camera is set too large, it needs to be reduced;

[0198] If e t <e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H-hb), because the measured wave height range is too large, so the focal length f needs to be increased;

[0199] The recognition accuracy real-time control module calculates the acquisition parameters and provides them to the shooting parameter optimization module, specifically:

[0200] According to the above adjustment, calculate the parameters of positive or negative adjustment of focal length f and horizontal downward angle as of the camera center axis;

[0201] According to the current time t, the ship's heading S t , identify the wave direction C t , calculate the shooting direction D at the next time t+1 according to formula (8) t+1 ;

[0202] Calculate the next shooting time interval T collected by the binocular optical camera, T = Min (0.25, M / Vs) (9);

[0203] The above calculation results are transmitted to the image acquisition system through the shooting parameter optimization module, and the image acquisition system executes shooting after waiting for T time;

[0204] The shipboard monitoring system is synchronized with the ship's navigation, follows time, and iterates continuously to ensure real-time and highly accurate monitoring and identification, and optimal acquisition parameters.

[0205] Furthermore, the target setting module determines the theoretical monitoring range of the camera according to the monitoring requirements. According to the international standard sea condition classification, the present invention determines different monitoring target distances M, i.e., monitoring ranges, for different sea conditions by zooming:

[0206] Sea state level Monitoring target distance M (L: ship length) Level 0-1 3L Level 2-3 6L Level 4-5 9L Level 6 and above 12L

[0207] The shooting and recognition accuracy real-time control system includes a shooting parameter optimization module and a recognition accuracy real-time control module, which are interconnected;

[0208] The shooting parameter optimization module determines the optimal parameter shooting direction D and provides it to the image acquisition system to control the image to capture the wave direction movement.

[0209] Specifically, the shooting parameter optimization module constructs the calculation formula (8), that is, the calculation formula of the shooting direction D is:

[0210]

[0211] Engineering vessel heading S (unit: °);

[0212] Shooting direction D (unit: °);

[0213] Horizontal field of view acw;

[0214] In order to adjust the wave direction to achieve the highest accuracy, the present invention designs formula (8) to determine the relationship between the camera center axis and the wave direction and the adjustment method during the monitoring process. When the camera is installed on the engineering vessel and visual monitoring is tilted, the angle between the wave direction and the camera center axis is set to 0 or 180.

[0215] The shooting parameter optimization module determines the farthest shooting area distance (lmax), determines the farthest shooting area distance lmax and the horizontal downward angle as of the camera center axis through calculation, and provides them to the image acquisition system to perform shooting.

[0216] Specifically, the maximum monitoring distance lmax is calculated by the following formula:

[0217] lmax=hb*f / (sz*10)(1)

[0218] Among them, high resolution hb, focal length f, CMOS single pixel height sz,

[0219] The horizontal downward angle of the camera axis is calculated as:

[0220] Altitude field angle ach, ach = 2*arctan(cmosh / (2f))(2)

[0221] Horizontal field of view acw, acw = 2*arctan(cmosw / (2f))(3)

[0222] The horizontal angle of the camera axis is as, as=arctan(hc / lmax)+ach / 2(4)

[0223] Parameter annotation and algorithm design description:

[0224] Cmosh is the height of the camera's photosensitive device CMOS, and cmosw is the width of the camera's photosensitive device CMOS.

[0225] The pre-processing system, a non-critical design module, may include the following functional modules:

[0226] 3.1 Image screening module focuses the collected video images on the operating sea area, processes them frame by frame, and forms an effective image data set.

[0227] 3.2, the image standardization module forms standardized images from images taken at different distances and under different weather conditions (sunny / cloudy).

[0228] 3.2.1 Image Flip Module

[0229] Image flipping operation. In addition, for images with directional errors caused by shooting or scanning angles, image flipping can correct them in the correct direction to improve recognition accuracy.

[0230] 3.2.2 Light Transformation Module

[0231] The wave image data is processed by light changes. The pixel values ​​of the image are adjusted to change the light characteristics of the image, such as brightness, contrast and color. A wave training set with different brightness ranges of 0.4 to 2.0 and different contrast ranges of 0.4 to 2.0 is formed to enhance the generalization ability of the established deep convolutional network.

[0232] By scaling or offsetting the brightness value of each pixel, the brightness of the entire wave image is changed. The adjustment process can be achieved by multiplying the RGB channel value of each pixel by the brightness factor or adding the brightness offset value. The following formula expresses the mathematical principle of brightness transformation:

[0233] L out =α*L in +cl (19)

[0234] Where, L out Represents the pixel output value of the wave image, L in Represents the input value of the pixel, α represents the scaling factor of the brightness adjustment, and cl represents the brightness adjustment constant.

[0235] 3.2.3 Image Scaling and Cropping Module

[0236] In the task of wave image processing, the first task is to adjust the image to a fixed size so as to be input into the algorithm or model, that is, image scaling. This is achieved through interpolation algorithms.

[0237] 3.2.3.1 Image Bilinear Interpolation Scaling Module

[0238] A more refined interpolation can be achieved by taking a weighted average of the grayscale values ​​of the four surrounding original pixels to determine the grayscale value of the new pixel. Bilinear interpolation first determines the grayscale value of the new pixel based on the position of the target pixel in the original image, and then takes a weighted average of the grayscale values ​​of the four pixels adjacent to the target pixel. This interpolation method takes advantage of the local grayscale changes in the original image and can therefore provide smoother and more accurate results than nearest neighbor interpolation. In bilinear interpolation, assuming that the width of the original wave image is W and the height is TH", and the width of the target image is W' and the height is H', the grayscale value of the target pixel can be calculated using the following formula:

[0239] X=X′*(TW-1) / (TW′-1)(22)

[0240] Y=Y′*(TH"-1) / (TH′-1)(23)

[0241] I′(X′,Y′)=I(X,Y)(24)

[0242] Find the four nearest pixels (X1, Y1), (X1, Y2), (X2, Y1), (X2, Y2) in the original image, where (X1, Y1) is the upper left pixel closest to (X, Y), (X2, Y2) is the lower right pixel closest to (X, Y), and the grayscale value of the target image coordinate (X', Y') is calculated by bilinear interpolation based on these four nearest pixels:

[0243] I'(X',Y')=(1-dX)*(1-dY)*I(X1,Y1)

[0244] +dX*(1-dY)*I(X2,Y1)

[0245] +(1-dX)*dY*I(X1,Y2)

[0246] +dX*dY*I(X2,Y2)(25)

[0247] Where I(X,Y) represents the grayscale value at the coordinate (X,Y) of the original image, I'(X',Y') represents the grayscale value at the coordinate (X',Y') of the target image, dX=Xx, dY=Yy, and x,y represent the rounding down of X,Y.

[0248] 3.2.3.2 Image Cropping Module

[0249] Extract a specific area from the original image and generate an image of a new size, so as to remove some noise interference from the original image and retain the original features of the image, so as to facilitate feature extraction and analysis in the subsequent convolution process. This can improve the accuracy and efficiency of subsequent processing.

Claims

1. A shipborne image acquisition method for marine engineering waters, characterized in that: include: Step 1 Algorithm operating conditions Install a high-precision night vision binocular camera as the acquisition hardware and determine the initial installation parameters; When the system enters the monitoring target distance M<2000m, the system controls and enables the binocular camera to collect wavefront image data in the optical image mode; Determine different monitoring target distances M, i.e. monitoring ranges, for different sea conditions; Step 2. Determine the optimal shooting parameters at the current time t, and perform shooting under the optimal shooting parameters at this time to obtain the original video data 2.1 According to the initial installation parameters in step 1, further optimize the shooting parameters of the optical hardware sensor device at the current time t, wherein the shooting parameters include the shooting direction D and the farthest shooting area distance lmax; 2.2 Based on the optimal parameter shooting direction D calculated and determined in step 2.1, adjust the center line of the high-definition camera device so that it coincides with the shooting direction D, and Based on the optimal parameter farthest shooting area distance lmax calculated and determined in step 2.1, the sea wave image at the current time t is shot, and the original wave image in the form of video is stored; Step 3. Preprocessing Preprocessing the original sea wave image taken in step 2 to form a standardized image; Step 4. Use the existing neural network to identify the input standardized graph and predict the recognition parameter H at the current time t in real time. t , HT t , C t ; H t is the sea surface wave height at time t, in meters; HT t is the wave period at time t, in seconds; C t is the wave direction at time t, in degrees; Step 5. Iteratively update monitoring parameters and hardware shooting parameters to ensure real-time and high-precision monitoring and identification of the shipborne system and optimal acquisition parameters; Calculate the next shooting time interval and wait for T time before going into step 2 to adjust the equipment and perform shooting, thus iterating; The step 5 comprises: Step 5.1, using the current ship speed V and the current ship heading S, calculate the relative ship speed Vs under the influence of waves, the relative ship speed: Where g is the acceleration due to gravity; Step 5.2: Set the target monitoring distance M according to the ship monitoring requirements and calculate the wave height recognition accuracy parameter e at the current moment t , e t =V s / M*0.1+(H-hb) / hb*0.9; Then, the accuracy e of the last time t-1 is t-1 Compare and adjust accordingly: If e t >e t-1 , then adjust the focal length f = f-10*sz*lmax / (H-hb). Since the focal length f parameter of the camera is set too large, it needs to be reduced; If e t <e t-1 , then adjust the focal length f = f + 10 * sz * lmax / (H-hb), because the measured wave height range is too large, so the focal length f needs to be increased; Step 5.3, Calculate acquisition parameters Adjust according to step 5.2, calculate the parameters of positive or negative adjustment of focal length f and horizontal downward angle as of the camera center axis; According to the current time t, the ship's heading S t , identify the wave direction C t , calculate the shooting direction D at the next moment t+1 according to the calculation formula of shooting direction D t+1 ; Calculate the next shooting time interval T collected by the binocular optical camera, T = Min (0.25, M / Vs) (9); Step 5.4: Calculate the next shooting parameters in step 5.3 and provide them to step 2.

1. Start step 2.1 after waiting for T time. Repeat the cycle continuously.

2. A method for acquiring shipborne images for marine engineering waters as claimed in claim 1, characterized in that: In step 1: Determine different monitoring target distances M, i.e. monitoring ranges, for different sea conditions: When the sea condition level is 0 to 1, the monitoring target distance M is 3L; When the sea condition level is 2-3, the monitoring target distance M is 6L; When the sea condition level is 4-5, the monitoring target distance M is 9L; When the sea condition level is 6 or above, the monitoring target distance M is 12L; Where L is the captain of the ship.

3. A method for acquiring shipborne images for marine engineering waters as claimed in claim 1, characterized in that: Step 2.1: Step 2.1.1 Determine the optimal parameter shooting direction D Construct calculation formula (8), that is, the calculation formula of the shooting direction D is: Engineering ship heading S, unit: °; Shooting direction D, unit: °; Horizontal field of view acw; Wave direction C, unit: °; 2.1.2 Determine the farthest shooting area distance lmax (1) The maximum monitoring distance lmax is calculated by the following formula: lmax=hb*f / (sz*10) (1) Among them, high resolution hb, focal length f, CMOS single pixel height sz, (2) Calculation of the horizontal downward angle as of the camera's central axis: Altitude field angle ach, ach = 2*arctan(cmosh / (2f)) (2) Horizontal field of view acw, acw = 2*arctan(cmosw / (2f)) (3) The horizontal angle of the camera axis is as, as=arctan(hc / lmax)+ach / 2 (4) Cmosh is the height of the camera's photosensitive device CMOS, cmosw is the width of the camera's photosensitive device CMOS, and hc is the shooting height.

Images