A shuttle tanker real-time state monitoring method based on binocular camera

Abstract

A kind of shuttle tanker real-time state monitoring method based on binocular camera, binocular camera acquisition image is obtained, and parallax map is calculated, high-quality point cloud is obtained after processing. Construct data set to train deep learning network, input left eye image to obtain the bow position of target shuttle tanker. Based on point cloud data, the real-time distance and angle between offshore oil and gas exploitation platform and shuttle tanker are calculated, and through drawing and network transmission, left eye image, optimal bounding box of shuttle tanker, early warning information, real-time distance and angle, camera state, environmental brightness are output, so that the crew can master the running state of shuttle tanker in real time. The invention realizes real-time acquisition of shuttle tanker during external transport operation by using binocular camera, calculates the real-time distance and angle between offshore oil and gas exploitation platform and shuttle tanker, gives the corresponding early warning information, and uploads the data to the client in real time, so that the operator can obtain the state of shuttle tanker in real time, and make corresponding adjustment, to ensure the safety of external transport operation.

Claims

1. A method for real-time status monitoring of a shuttle oil tanker based on a binocular camera, characterized in that, The steps are as follows: Step 1: Obtain the raw data collected by the binocular camera, and obtain a dense disparity map after image preprocessing; (1.1) Configure the binocular camera and the local PC network, and transmit the raw data of the binocular camera to the PC. (1.2) Parse the raw data to obtain the original left eye image, the original right eye image, camera calibration data, ambient brightness and camera status acquired by the binocular camera; among which, the camera calibration data includes intrinsic and extrinsic parameters, distortion coefficients and baseline; (1.3) Based on the original images of the left and right eyes and the camera calibration data, after distortion correction, stereo correction and image preprocessing, the calibration images of the left and right cameras are obtained. (1.4) Use the left camera calibration image and the right camera calibration image obtained in step (1.3) to perform stereo matching to obtain the original disparity map; use a filtering algorithm to clean the original disparity map to obtain a dense disparity map; Step 2: Calculate the dense disparity map obtained in Step 1, and filter it to obtain high-quality 3D point cloud data; (2.1) Based on the dense disparity map, the principle of triangle ranging is used, and the original point cloud data is obtained according to the known intrinsic parameters of the binocular camera and the baseline; (2.2) Set the dimension and threshold parameters, specify the points within the parameter range to pass through, and filter out the points outside the parameter range to obtain high-quality point cloud data; Step 3: Collect data on shuttle oil tanker export operations, construct a shuttle oil tanker dataset, expand the shuttle oil tanker dataset using data augmentation methods for different operating conditions, and complete accurate labeling; (3.1) Collect data on the operation of shuttle oil tankers, clean the collected data, remove duplicate and low-quality images, and construct a shuttle oil tanker dataset; expand the shuttle oil tanker dataset using data augmentation methods to address the floating and swaying of shuttle oil tankers and the variable weather conditions at sea. (3.2) Use image annotation software to accurately annotate the constructed dataset and obtain the corresponding labels; Step 4: Construct an object detection network, train it, optimize its model parameters, and deploy it. Input the left camera calibration image obtained in step (1.3) to perform object detection and obtain the observation box of the target ship. (4.1) Build a deep learning object detection network, input the augmented shuttle oil tanker dataset and the corresponding label file for training, and obtain the model training weights; (4.2) Using a deep learning inference framework, optimize and deploy the model parameters of the deep learning object detection network built in step (4.1) to obtain an engineered object detection network; (4.3) Input the left camera calibration image obtained in step (1.3) into the engineered target detection network for prediction to obtain the observation box of the target ship; Step 5: Use a target tracking algorithm to track the target ship, and merge and update the results with the target detection results to obtain the optimal bounding box of the target ship; (5.1) Establish the kinematic formula of the target ship, use the kinematic equation of motion of the Kalman filter algorithm to predict the position of the target ship, and obtain the prediction box of the target ship; (5.2) Construct an association matrix based on the IOU between the observation box and the prediction box of the target ship as a metric, and match the observation box and the prediction box to obtain the best matching pair; (5.3) Establish the target ship position update formula, and use the update equation of the Kalman filter algorithm to fuse and update the best-matched observation box and prediction box to obtain the optimal bounding box of the target ship. Step 6: Extract features from the shuttle oil tanker within the optimal bounding box area, and calculate the relative distance and angle between the shuttle oil tanker and the offshore oil and gas exploration platform by combining the high-quality point cloud data obtained in step (2.2). (6.1) Extract features from the shuttle oil tanker within the bounding box area to obtain the significant texture features of the bow edge of the target shuttle oil tanker; (6.2) Combine the high-quality point cloud data obtained in step (2.2) to calculate the three-dimensional coordinate values ​​of significant texture features, and use the correlation filtering algorithm to remove outliers to obtain effective point cloud data; (6.3) Combining the relative position parameters of the binocular camera and the offshore oil and gas extraction platform, the effective point cloud data obtained in step (6.2) is converted into coordinates, and the relative distance and angle between the shuttle oil tanker and the oil and gas extraction platform are calculated using the following formula. ； ； Among them, X i For the effective point cloud data, the X-axis coordinates and Z-axis coordinates are... i Z represents the Z-axis coordinates of the valid point cloud data, Z0 is the distance between the camera and the stern of the offshore oil and gas exploration platform, n is the number of valid point cloud data, and L is the distance between the offshore oil and gas exploration platform and the shuttle tanker. The relative angle between the oil and gas exploration platform and the shuttle tanker; Step 7: Develop an early warning strategy for shuttle oil tankers based on the offshore oil and gas exploration platform export operation level table, judge the distance and angle obtained in Step 6, provide early warning information of different levels, and draw the information. (7.1) Define the safety range and level of distance and angle; when the distance between the shuttle oil tanker and the oil and gas extraction platform is (0, DoWL1] or the angle is (0, α], a level 1 warning is issued; when the distance is (DoWL1, DoWL2] or the angle is (α, β], a level 2 warning is issued; when the distance is (DoWL2, DoWL3] or the angle is (β, γ], a level 3 warning is issued. The unit of distance is m and the unit of angle is °. The distance parameters DoWL1, DoWL2, DoWL3 and the angle parameters α, β, γ are formulated and updated according to the offshore oil and gas extraction platform export operation level table. Once the calculation result enters the corresponding warning range, the system issues an alarm of the corresponding level. (7.2) Draw the optimal bounding box of the shuttle oil tanker, the warning scale line, the real-time distance and angle between the shuttle oil tanker and the offshore oil and gas exploration platform, the warning level, the ambient brightness and the camera status on the left camera calibration image to obtain the composite information of the left image; Step 8: Store the composite information of the left image and, based on the principle of network communication, send the composite information data of the left image to the client to display the status of the shuttle oil tanker in real time. (8.1) Convert the composite information of the left figure drawn in step (7.2) into binary data stream format and write it into the database in real time; (8.2) Compress and convert the format of the composite information of the left figure drawn in step (7.2); (8.3) Based on the principle of network communication, a client and a server are built. The server sends the composite information of the left figure to the client through the sending interface; (8.4) The client receives data, decodes and converts the format, and displays the composite information of the shuttle oil tanker in real time.

2. The method for real-time status monitoring of shuttle oil tankers based on binocular cameras according to claim 1, characterized in that, The aforementioned filtering algorithm selects between weighted median filtering or hole filling.

3. The method for real-time status monitoring of shuttle oil tankers based on binocular cameras according to claim 1, characterized in that, The deep learning object detection network selected is either YOLOv5 or YOLOX.

4. The method for real-time status monitoring of shuttle oil tankers based on binocular cameras according to claim 1, characterized in that, The deep learning inference framework chosen is TensorRT.

5. The method for real-time status monitoring of shuttle oil tankers based on binocular cameras according to claim 1, characterized in that, The database chosen is MySQL.

Images