A visual-based full-automatic state detection and early warning method for shuttle tanker

Abstract

The application provides a kind of full-automatic state detection and early warning method of shuttle tanker based on visual sensor, utilizes monocular sensor and other three-dimensional true value acquisition sensor to carry out joint calibration, obtains monocular image detection data set, monocular segmentation data set and three-dimensional true value data set of offshore shuttle tanker scene, completes the construction of training data set;The sequence image data of monocular camera is used as input, and the depth estimation information of the target area of shuttle tanker is output, then the three-dimensional true value data under the same time sequence and the depth information of the target area extracted are used for scale recovery of monocular scene, and the absolute distance field of the target area is obtained;The absolute distance between the ship head and the camera optical center is extracted using filtering algorithm and sorting algorithm, the ranging information and alarm information are visualized output combined with early warning strategy, and the state automatic detection and early warning function of shuttle tanker is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of active safety of shuttle tankers of offshore oil and gas production platforms, and particularly relates to a full-automatic state detection and early warning method for shuttle tankers based on visual sensors. BACKGROUND

[0002] In the world, offshore oil and gas production platforms are widely used in the field of oil and gas production, and the processed crude oil is regularly transported to shuttle tankers and then transported to the shore. Offshore oil and gas production platforms rely on their own advantages, and the number is increasing, and the frequency of export operations is becoming higher and higher. At the same time, the safety of export operations is concerned by the industry and society. The risks of export operations mainly include collision, mooring cable fracture, oil leakage and fire caused by damage of oil transfer hose, explosion and the like, and the collision risk is one of the most important risks in the export process.

[0003] Most of the collision accidents are caused by excessive relative motion between offshore oil and gas development platforms and shuttle tankers. When the export operation starts, the offshore oil and gas production platform is full of water and has a large draft, and is mainly affected by the flow and wave. At this time, the ballast draft of the tanker is relatively shallow, the wind effect is obvious, and the flow effect is large. As the export operation proceeds, the draft of the offshore oil and gas production platform is continuously reduced, and the effect of wind will be greater than that of wave or flow. At the same time, the draft of the tanker is continuously increased, and the effect of wave and flow will be greater than that of wind. When there is a considerable difference in the hydrodynamic characteristics between the tanker and the offshore oil and gas production platform, it may lead to the reduction of the distance between the two ships, and increase the probability of collision between the two ships.

[0004] Therefore, the present application proposes a real-time monitoring scheme for shuttle tankers based on visual sensors, so that the shuttle tanker can be real-time warned when it enters the safety warning distance of the offshore oil and gas production platform, and effectively reduce the unnecessary resource and property loss caused by the misjudgment or insufficient experience of the observer. SUMMARY

[0005] This invention provides a fully automated status detection and early warning method for shuttle oil tankers based on visual sensors. Addressing the issue that observers need to periodically observe and estimate the distance between shuttle oil tankers and offshore oil and gas extraction platforms, the accuracy of these estimates relies heavily on the observer's experience. This leads to problems such as large estimation errors and safety hazards caused by the inability to timely predict the status of shuttle oil tankers in low visibility conditions during severe weather. By using a low-cost visual sensor, automatic detection, ranging, and early warning functions for shuttle oil tankers within a certain distance range can be achieved. The output terminal displays real-time information on the distance between the shuttle oil tanker and the offshore oil and gas extraction platform, along with early warning information. This transforms the status early warning system for shuttle oil tankers from a "manual observation" stage to an "intelligent" stage, improving the proactive safety of offshore oil extraction operations. The approach of this method is as follows: The dataset is augmented by data augmentation to increase its size. A two-dimensional target detection method is used to extract the two-dimensional detection box region containing the target, and a target segmentation method is used to remove background pixels within the two-dimensional detection box. A depth prediction model is trained, and the depth information within the segmented two-dimensional detection box region is filtered multiple times to calculate the relative depth of the shuttle oil tanker from the camera's optical center. A scale factor is calculated using point cloud data and predicted depth information at the same timestamp to recover the absolute depth between the shuttle oil tanker and the camera's optical center. The absolute depth information of the segmented target is searched to obtain the absolute depth of the shuttle oil tanker from the camera's optical center. The detection box information, absolute depth information, and warning information of the shuttle oil tanker are visualized at the output end. This method can achieve fully automatic detection and warning functions, effectively reducing the risk of collisions between offshore oil and gas exploration platforms and shuttle oil tankers during offshore operations.

[0006] The technical solution of this invention is as follows:

[0007] A fully automated condition detection and early warning method for shuttle oil tankers based on vision sensors, the specific steps of which are as follows:

[0008] Step 1: Perform joint calibration on the monocular camera and the 3D point cloud data acquisition device that can acquire true values ​​to obtain the spatial transformation relationship between the monocular camera and the 3D point cloud data acquisition device.

[0009] (1.1) Select calibration plates with different length and width dimensions, flat surfaces without white edges and non-reflective surfaces, and fix the relative position between the monocular camera and the 3D point cloud data acquisition device;

[0010] (1.2) Use a calibration board to calibrate the intrinsic parameters of the monocular camera to obtain the intrinsic parameter matrix;

[0011] (1.3) Use a calibration board to jointly calibrate the monocular camera and the 3D point cloud data acquisition device to obtain the extrinsic parameter transformation matrix between the two sensors;

[0012] Step 2: Using the monocular camera and 3D point cloud data acquisition device calibrated in Step 1, monocular images and 3D point cloud ground truth data of the shuttle oil tanker of the offshore oil and gas development platform are collected, and the monocular image data and 3D point cloud ground truth data under the same time series are labeled to complete the construction of the dataset.

[0013] (2.1) Fix the calibrated monocular camera and 3D point cloud data acquisition device on the offshore oil and gas development platform to collect and store the monocular images and 3D point cloud data of the shuttle oil tanker in a multimodal synchronous manner.

[0014] (2.2) Organize the monocular image data collected by the monocular camera, perform two-dimensional detection box annotation of the region of interest, and construct a monocular detection dataset of the shuttle oil tanker;

[0015] (2.3) Organize the monocular image data collected by the monocular camera, perform target segmentation and annotation, and construct the target segmentation dataset of the shuttle oil tanker;

[0016] (2.4) Perform three-dimensional annotation on the ground truth data of three-dimensional point clouds collected under the same time series to construct the ground truth dataset of the shuttle oil tanker;

[0017] Step 3: Train the target detector using the monocular image dataset of the shuttle oil tanker, train the target segmenter using the monocular image dataset of the shuttle oil tanker, and train the depth predictor using the monocular image dataset of the shuttle oil tanker.

[0018] (3.1) A monocular detector is trained using a monocular detection dataset from a shuttle oil tanker through deep learning.

[0019] (3.2) A monocular image segmenter is trained using a monocular segmentation dataset of shuttle oil tankers through deep learning.

[0020] (3.3) A monocular depth predictor is trained using a deep learning self-supervised method based on the monocular detection dataset of shuttle oil tankers;

[0021] Step 4: Take the sequence image data from the monocular camera as input and output the depth estimation information of the target area of ​​the shuttle oil tanker.

[0022] (4.1) The sequence image data of the monocular camera is used as input. First, it is processed by the monocular detector obtained from step (3.1) to output a two-dimensional target detection box of the region of interest of the shuttle oil tanker in the sequence image, including the corresponding location information and the category information of the oil tanker.

[0023] (4.2) Use the position information of the two-dimensional target detection box output in step (4.1) to crop the sequence image to obtain the region of interest image data;

[0024] (4.3) Input the region of interest image data obtained in step (4.2) into the trained monocular image segmenter to obtain a segmented image of the shuttle oil tanker target separated from the background information within the region of interest;

[0025] (4.4) Input the segmented image of the shuttle oil tanker target obtained in step (4.3) into the trained monocular depth predictor to obtain the depth information map of the shuttle oil tanker target area.

[0026] Step 5: Calculate the scale restoration factor of the monocular image using the 3D point cloud data obtained in Step 2 and the depth information map of the target area obtained in Step 4, perform scale restoration on the target area, and obtain the absolute distance field information of the target area.

[0027] (5.1) Obtain the depth information map of the target area of ​​the shuttle oil tanker from step (4.2) and obtain the three-dimensional point cloud data of the corresponding time series from step 2;

[0028] (5.2) Calculate the statistical characteristics of the 3D point cloud data and depth map within the 2D target detection box area: mean, median, maximum, and minimum values;

[0029] (5.3) Scale recovery is performed by calculating the scale restoration factor using the median of the depth map, and the absolute distance field information of the shuttle oil tanker from the optical center of the monocular camera is obtained.

[0030] Step 6: Filter the absolute distance field information of the shuttle oil tanker from the optical center of the monocular camera obtained in Step 5, and use a sorting algorithm to search the absolute depth information of the shuttle oil tanker to obtain the true distance between the bow position of the shuttle oil tanker and the optical center of the camera.

[0031] (6.1) Perform multi-step, multi-scale filtering on the absolute depth information of the target region;

[0032] (6.2) Sort the absolute depth information of the filtered shuttle oil tanker pixels using a sorting algorithm, determine the position of the shuttle oil tanker bow, and output its true distance from the camera optical center.

[0033] Step 7: Determine the warning level based on the distance information of the shuttle oil tanker obtained in Step 6 according to the export operation level of the offshore oil and gas exploration platform.

[0034] (7.1) Develop an early warning strategy for shuttle tankers according to the offshore oil and gas production platform's external transportation operation level table; define the warning distance DoWL1 to represent the supremum of the first-level warning; define the warning distance DoWL2 to represent the supremum of the second-level warning; define the warning distance DoWL3 to represent the supremum of the third-level warning; where the distance parameter 0 < DoWL1 < DoWL2 < DoWL3, for example, DoWL1 = 42m, DoWL2 = 52m, DoWL3 = 56m can be taken. When the distance between the shuttle tanker and the offshore oil and gas production platform is in the range (0m, DoWL1], a first-level warning is issued; when the distance is in the range (DoWL1, DoWL2], a second-level warning is issued; when the distance is in the range (DoWL2, DoWL3], a third-level warning is issued; the warning strategy needs to be synchronized according to the update of the offshore oil and gas production platform's external transportation operation level table. Once the distance between the shuttle tanker and the oil and gas production platform triggers the alarm condition, the system will immediately issue the corresponding alarm;

[0035] (7.2) According to the developed early warning strategy for shuttle tankers, the early warning level of the shuttle tanker under the current state is displayed in real time at the output end;

[0036] Step 8, visually display the two-dimensional target detection box information in Step 3, the distance information between the bow of the shuttle tanker and the camera optical center in Step 6, and the early warning information in Step 7 on the output interface;

[0037] (8.1) Display the shuttle tanker detection box generated in Step 3 and the absolute distance information of the bow of the shuttle tanker from the camera optical center calculated in Step 6 at the output end;

[0038] (8.2) Display the early warning area effect diagram in Step 7 at the output end.

[0039] The beneficial effects of the present invention: Since there is a risk of collision between the offshore oil and gas production platform and the shuttle tanker during offshore operations, currently, the monitoring of the distance between the shuttle tanker and the offshore oil and gas production platform and the early warning work are still carried out manually by observers. The present invention can achieve full-automatic real-time detection of shuttle tankers through the target detection network, and issue real-time warnings when the distance between the shuttle tanker and the offshore oil and gas production platform is less than the safe range, leaving enough time and distance for the shuttle tanker to adjust its course in time, effectively reducing the risk of collision between the two ships and improving the safety of offshore operations of the offshore oil and gas production platform.

[0040] (1) By building an experimental platform for shuttle tankers with a visual sensor + three-dimensional point cloud data collector, collecting data and constructing a monocular detection database, a monocular segmentation dataset, and a true value dataset for shuttle tankers, and realizing full-automatic perception and prediction alarm of the state of shuttle tankers through the trained target detector, target segmenter, and depth predictor;

[0041] (2) In real-time detection schemes for shuttle oil tankers based on binocular or multi-view sensors, if one of the binocular cameras malfunctions, this scheme can still complete the real-time monitoring and early warning tasks for shuttle oil tankers. Therefore, this invention has irreplaceable significance in real-time detection tasks for shuttle oil tankers based on pure vision, ensuring the proactive safety of offshore oil and gas exploration platforms. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention.

[0043] Figure 2 The diagram shows an experimental platform for a visual sensor plus other sensors. In (a), the other sensors are lidar and in (b), the other sensors are binocular cameras.

[0044] Figure 3 The specifications of the calibration board to be used for joint calibration of vision sensors and other sensors.

[0045] Figure 4 This is a rendering of the calibration plate after 3D annotation.

[0046] Figure 5 (a) is the result of the target detection output, and (b) is the result after cropping the region of interest.

[0047] Figure 6 This is the depth rendering of the segmented target.

[0048] Figure 7 This is a depth ground truth image projected onto the three-dimensional ground truth data collected from the same time series.

[0049] Figure 8 This is a flowchart of the scale recovery process.

[0050] Figure 9 This is a diagram illustrating the early warning strategy.

[0051] Figure 10 This is a schematic diagram of the output visual interface. Detailed Implementation

[0052] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Please see Figure 1This invention provides a fully automated state detection and early warning method for shuttle oil tankers based on vision sensors. It utilizes a monocular sensor and other sensors for joint calibration to acquire a monocular image dataset and a ground truth dataset of the shuttle oil tanker scene at sea, and annotates the data to construct a database. The monocular image data is input into a target detection and segmentation network and a depth estimation network to obtain the depth information of the segmented target region. Then, the scale of the monocular scene is restored using the ground truth data and the extracted target region depth information to obtain the absolute distance field of the target region. A search algorithm is then used to extract the absolute distance from the bow to the camera's optical center. Finally, an early warning strategy is used to visualize the target ranging information and alarm information, achieving fully automated state detection and early warning functions for shuttle oil tankers. The aforementioned fully automated state detection and early warning method for shuttle oil tankers based on vision sensors mainly includes the following steps:

[0054] Step 1 specifically includes the following sub-steps:

[0055] (1.1) Please refer to Figure 2 The monocular camera and other sensors, such as lidar, binocular cameras, or other sensors that can acquire three-dimensional truth values, are rigidly fixed to ensure that their relative positions in space remain unchanged during operation.

[0056] (1.2) Use calibration tools to calibrate the intrinsic parameters of the monocular camera to obtain the intrinsic parameter matrix;

[0057] (1.3) Please refer to Figure 3 We plan to use a 6x11 calibration board with a side length of 60mm to perform joint calibration of the monocular camera and the 3D point cloud data acquisition device using calibration tools, and obtain the extrinsic parameter transformation matrix between the two sensors.

[0058] Step 2 specifically includes the following sub-steps:

[0059] (2.1) Fix the calibrated monocular camera and 3D point cloud collector on the offshore oil and gas development platform to collect and store the monocular images and point cloud data of the shuttle oil tanker in a multimodal synchronous manner.

[0060] (2.2) Mark the two-dimensional detection boxes of the regions of interest in the monocular image data acquired by the monocular camera to construct the monocular detection dataset of the shuttle oil tanker;

[0061] (2.3) Perform target segmentation and annotation on the monocular image data acquired by the monocular camera to construct a target segmentation dataset for the shuttle oil tanker;

[0062] (2.4) Please refer to Figure 4 The image shows the effect of 3D annotation on the calibration board; 3D annotation is performed on the ground truth data of 3D point clouds collected under the same time series to construct the ground truth dataset of the shuttle oil tanker.

[0063] Step 3 specifically includes the following sub-steps:

[0064] (3.1) Using the monocular detection dataset of shuttle oil tankers, a monocular detector is obtained by training it through deep learning methods such as using the Yolov5 object detection network;

[0065] (3.2) Using the monocular segmentation dataset of shuttle oil tankers, a monocular segmenter is obtained by training a deep learning method such as Mask R-CNN object segmentation network;

[0066] (3.3) Using the monocular dataset of the shuttle oil tanker, a monocular depth predictor is obtained by training it with deep learning methods such as Monodepth monocular self-supervised depth estimation network.

[0067] Step 4 specifically includes the following sub-steps:

[0068] (4.1) The sequence image data of the monocular camera is used as input. First, it is processed by the target detector obtained from step (3.1) to output a two-dimensional target detection box of the region of interest of the shuttle oil tanker in the sequence image, including the corresponding location information and the category information of the oil tanker.

[0069] (4.2) Please refer to Figure 5 Using the position information (x, y, w, h) of the two-dimensional target detection box output in step (4.1), the sequence image is cropped to obtain the region of interest image data, where x and y represent the pixel coordinates of the center point of the two-dimensional detection box, and w and h represent the width and height of the two-dimensional detection box, respectively.

[0070] (4.3) Input the region of interest image data obtained in step (4.2) into the trained monocular image segmenter to obtain a segmented image of the shuttle oil tanker target separated from the background information within the region of interest;

[0071] (4.4) Please refer to Figure 6 The target segmentation image of the shuttle oil tanker obtained in step (4.3) is input into the trained monocular depth predictor to obtain the depth information map of the target area of ​​the shuttle oil tanker.

[0072] Step 5 specifically includes the following sub-steps:

[0073] (5.1) Please refer to Figure 7 From step (4.2), obtain the depth information map of the target area of ​​the shuttle oil tanker and from step 2, obtain the true value data of the three-dimensional point cloud under the corresponding time series;

[0074] (5.2) Calculate the statistical characteristics of the ground truth data and depth map of the 3D point cloud within the 2D target detection box area: mean, median, maximum, and minimum values;

[0075] (5.3) Refer to Figure 8 , and use the median value of the depth map to calculate the scale recovery factor for scale recovery, so as to obtain the absolute distance field information of the shuttle tanker from the optical center of the monocular camera;

[0076] Step 6 specifically includes the following sub-steps:

[0077] (6.1) The depth information data generated by using the weights of the depth prediction model contains various complex noises. According to the longitudinal length information of the shuttle tanker as a prior, set the outlier data threshold α and the outlier data threshold β to filter the depth information within the detection frame area;

[0078] (6.2) Use the quicksort algorithm to traverse and search all pixel points of the filtered shuttle tanker, sort its absolute distance field information, determine the position of the bow of the shuttle tanker, and output the average value of the absolute distances of the m pixel points closest to the optical center of the monocular camera at the bow. The time complexity of the algorithm is O(N).

[0079] Step 7 specifically includes the following sub-steps:

[0080] (7.1) Refer to Figure 9 . Develop a warning strategy for the shuttle tanker according to the offshore oil and gas production platform's external transportation operation level table; define the warning distance DoWL1 to represent the supremum of the first-level warning; define the warning distance DoWL2 to represent the supremum of the second-level warning; define the warning distance DoWL3 to represent the supremum of the third-level warning; where the distance parameter 0 < DoWL1 < DoWL2 < DoWL3, for example, DoWL1 = 42m, DoWL2 = 52m, DoWL3 = 56m can be taken. When the distance between the shuttle tanker and the offshore oil and gas production platform is in the range of (0m, DoWL1], a first-level warning is issued; when the distance is in the range of (DoWL1, DoWL2], a second-level warning is issued; when the distance is in the range of (DoWL2, DoWL3], a third-level warning is issued; the warning strategy needs to be synchronized according to the update of the offshore oil and gas production platform's external transportation operation level table. Once the distance between the shuttle tanker and the oil and gas production platform triggers the alarm condition, the system will immediately issue the corresponding alarm;

[0081] (7.2) According to the formulated warning strategy for the shuttle tanker, the warning level of the shuttle tanker under the current state is displayed in real time at the output end to implement the warning function.

[0082] Step 8 specifically includes the following sub-steps:

[0083] (8.1) Refer to Figure 10 . Display the detection frame of the shuttle tanker generated in Step 4 and the absolute distance information of the position of the bow of the shuttle tanker from the optical center of the camera calculated in Step 6 at the output end;

[0084] (8.2) Display the effect diagram of the warning area in step 7 at the output terminal.

Images