A method for monitoring and warning the status of a marine shuttle tanker based on binocular vision

Abstract

The present invention provides a method for monitoring and early warning of the state of an offshore shuttle tanker based on binocular vision. The internal and external parameters of the binocular camera are calibrated using a checkerboard calibration board, and at the same time, the lidar and the left camera are jointly calibrated to obtain the rigid transformation matrix between coordinate systems; the point cloud-image data collected on site is used to construct a depth estimation dataset and a three-dimensional detection dataset for the external transportation operation scenario of the oil and gas production platform - shuttle tanker; a deep learning algorithm is trained to estimate the three-dimensional pose information and depth of the shuttle tanker by inputting binocular images; the safety ranges and levels of relative distance and relative angle are delimited, and corresponding-level early warnings are issued according to the real-time monitoring results to remind the operator to make corresponding adjustment measures in time, effectively preventing the collision between the shuttle tanker and the production platform during the external transportation operation.

Description

Technical Field

[0001] The present invention relates to the technical field of marine ship engineering safety, and specifically, it is a method for monitoring and warning the state of an offshore shuttle tanker based on binocular vision. Background Technique

[0002] An offshore oil and gas production platform is a platform for offshore and deep-sea oil exploitation, which plays an irreplaceable role in the process of offshore oil and gas exploitation. It can be used for oil production, oil-gas separation of the collected crude oil, treatment of oily sewage, and preliminary treatment and storage. The platform needs to transport the stored crude oil or preliminary extracts to a land factory through a shuttle tanker for further fine processing and treatment. When the oil production platform conducts an oil transportation operation on the shuttle tanker, low-pressure crude oil is transported to the shuttle tanker through an external transfer hose.

[0003] However, due to the need to complete the external transfer operation in a marine working condition, it is restricted by factors such as harsh and rapidly changing weather conditions and complex sea conditions. At the same time, each external transfer operation lasts for 24 hours. The personnel responsible for monitoring and command are extremely prone to fatigue at night, and the timeliness of manual command is poor, and there are deviations in relaying information. Once the relative distance between the oil and gas production platform and the shuttle tanker is too small or the relative deflection angle is too large, and due to the large volume and weight of the shuttle tanker, it is very likely that a serious collision event will occur due to untimely rescue. Both of them store a large amount of flammable and explosive items, and serious consequences beyond imagination will occur after a collision, posing a huge threat to the safety of people's lives and property. Therefore, it is extremely important to monitor the three-dimensional pose of the shuttle tanker in real time.

[0004] Patent (Publication No.: CN 113536544 A), a method for detecting ship probability conflicts based on a minimum safety distance model, proposes a method to obtain information such as the longitude and latitude of the target ship and the own ship by using the Automatic Identification System (AIS), and calculate the relative angle and relative distance between the two through geometric relationships. However, the accuracy of the positioning means of this system is not very applicable to the complex and small-spacing specific physical scenarios such as oil and gas production platforms - shuttle tankers. The length dimension of the shuttle tanker itself is larger than the intermediate safety distance, and the positioning error of the bow and stern will be amplified into the estimation of the distance and angle, and it cannot meet the monitoring safety requirements of the external transfer operation working condition.

[0005] Compared with monocular cameras, binocular cameras can achieve more accurate depth calculation and estimation through stereo matching of left and right views, and impose tighter constraints on them. In the application of deep learning in the field of computer vision, the use of binocular vision to solve the three-dimensional detection of targets has also made great progress. This method uses laser radar and binocular cameras to collect field data of the working environment of oil and gas production platforms-shuttle tankers. After data cleaning, screening, labeling and other steps, it is processed into a data set required for training deep learning algorithms. The above data set is used to train deep learning algorithms for image depth estimation and target three-dimensional detection. Based on the algorithm, the input binocular image can accurately estimate the position state information of the shuttle tanker. According to the detected state information and the demarcation of different levels of safety ranges, once the shuttle tanker is detected to enter a certain safety warning range, the system will issue a corresponding level of warning prompt to the ship's driving operator. Summary of the invention

[0006] The present invention proposes a method for monitoring and warning the status of offshore shuttle tankers based on binocular vision. It can detect the distance, angle and other position status information of the shuttle tanker in real time through a binocular camera and issue a warning for the complex external transmission operation of the oil and gas production platform and the shuttle tanker.

[0007] To achieve the above objectives, the main idea of this method is as follows: calibrate the internal and external parameters of the binocular camera, jointly calibrate the lidar and binocular camera, build a three-dimensional detection and depth estimation data set for the binocular camera, train a deep learning algorithm and use the algorithm to estimate the three-dimensional pose information and depth of the shuttle tanker, and give necessary early warning to the shuttle tanker in real time according to the estimated status results.

[0008] The specific steps of the technical solution of the present invention are as follows:

[0009] A method for monitoring and warning the state of a marine shuttle tanker based on binocular vision, the steps are as follows:

[0010] Step 1: calibrate the internal and external parameters of the binocular camera, and jointly calibrate the lidar and binocular camera to obtain the transformation matrix between the lidar coordinate system, the camera coordinate system, and the image coordinate system;

[0011] (1.1) Make a checkerboard calibration plate, use a laser radar and a binocular camera to jointly collect data on the checkerboard calibration plate, and obtain point cloud data and image data of the checkerboard calibration plate;

[0012] (1.2) Manually filter out all point clouds except the checkerboard calibration plate in the point cloud data to obtain point cloud data containing only the checkerboard calibration plate point cloud;

[0013] (1.3) Use the binocular image data containing the checkerboard calibration board to calibrate the internal and external parameter matrices of the binocular camera, and obtain the internal parameter matrix of the camera, the distortion coefficients, and the external parameter rotation matrix and translation matrix for the rigid transformation between the left and right camera coordinate systems;

[0014] (1.4) Perform joint calibration of the lidar and the binocular camera using the image data obtained in step (1.1) and the point cloud data after filtering in step (1.2), and obtain the rotation matrix and translation matrix for the rigid transformation between the lidar coordinate system and the camera coordinate system;

[0015] Step 2: Transform the point cloud data collected by the lidar into the corresponding depth map data through the internal parameter matrix of the left camera in step (1.3) and the transformation matrix from the lidar coordinate system to the camera coordinate system obtained in step (1.4), and construct a dataset for binocular depth estimation;

[0016] (2.1) Use the lidar and the binocular camera to jointly collect data for the actual shuttle tanker's external transfer operation work scene, and perform time soft synchronization on the point cloud data and image data of each frame;

[0017] (2.2) Project the point cloud data in step (2.1) onto the camera coordinate system through the transformation matrix from the lidar coordinate system to the camera coordinate system obtained in step (1.4);

[0018] (2.3) Use the internal parameter matrix of the camera obtained in step (1.3) for the point cloud data in the camera coordinate system obtained in step (2.2) to obtain the point cloud data corresponding to the field of view of the binocular camera; Convert this point cloud data from the camera coordinate system to the image coordinate system, generate the two-dimensional projection of the lidar point cloud data in the image, and obtain the depth map;

[0019] (2.4) Perform depth completion on the depth map obtained in step (2.3) to obtain a dense and continuous depth map, which is used to construct a depth estimation dataset;

[0020] Step 3: Perform three-dimensional annotation on the shuttle tanker using the point cloud data obtained in step (2.1), and represent the annotation information in the camera coordinate system in the same way as in step (2.2). The annotation information and the binocular images jointly construct a dataset for binocular three-dimensional detection;

[0021] (3.1) Perform data cleaning and screening on the image data collected by the binocular camera in step (2.1) to remove low-quality and blurred binocular image data;

[0022] (3.2) Annotate the point cloud data collected by the laser radar in step (2.1) with a three-dimensional box of the shuttle tanker to obtain the category C of the shuttle tanker. The three-dimensional box annotation information includes the coordinates X of the center point of the three-dimensional box. O , Y O and Z O , the length, width and height of the three-dimensional box, and the orientation R' of the three-dimensional box y ;

[0023] (3.3) Using the 3D frame annotation information obtained in step (3.2), calculate the 3D coordinate values of the 8 corner points of the 3D frame. The X coordinate calculation formula is as follows:

[0024]

[0025] The Y coordinate calculation formula is as follows:

[0026]

[0027] The Z coordinate calculation formula is as follows:

[0028]

[0029] (3.4) The three-dimensional coordinates of the eight corner points obtained in step (3.3) are transformed into the camera coordinate system by the same method as step (2.2);

[0030] (3.5) The binocular camera internal parameter matrix, distortion coefficient and external parameter rotation matrix and translation matrix obtained in step (1.3), the rotation matrix and translation matrix from the laser radar coordinate system to the camera coordinate system obtained in step (1.4), and the three-dimensional frame annotation information of the shuttle tanker in the camera coordinate system obtained in the series of operations of steps (3.2)-(3.4) constitute a binocular three-dimensional detection data set;

[0031] (3.6) Divide the binocular 3D detection data set in step (3.5) into training set, validation set and test set in proportion;

[0032] Step 4, training a deep learning algorithm based on the depth estimation data set of step (2.4) and the binocular 3D detection data set of step (3.5) and using the deep learning algorithm to predict the depth and 3D pose of the shuttle tanker;

[0033] (4.1) performing feature extraction on the binocular image in the binocular 3D detection data set in step (3.5) to obtain feature information of the shuttle tanker in the image;

[0034] (4.2) constructing a plane scanning volume based on the features obtained in step (4.1), calculating the matching loss based on the plane scanning volume, matching the horizontal distance of pixel i in the left and right images of the binocular image, the horizontal distance is the disparity disp, and constructing the calculation loss based on the disparity;

[0035] (4.3) According to the disparity disp in step (4.2), convert it into depth depth data and calculate the expectation of all depth candidates. Use the depth estimation data obtained in step (2.4) as supervision information to regress the depth value corresponding to each pixel of the image. The conversion formula between disparity and depth is:

[0036]

[0037] Where f x and baseline are the lateral focal length and baseline distance of the camera, respectively, both obtained through step (1.4);

[0038] (4.4) The camera internal parameter matrix obtained by step (1.4) includes the lateral focal length f x , vertical focal length f y and the actual position of the principal point c u 、c v Four parameters, through inverse three-dimensional projection, transform the last feature map of the plane scan volume in step (4.2) from the camera coordinate system (u, v, d) to the world coordinate system (x, y, z):

[0039]

[0040] (4.5) Take the N anchor points closest to the true value of the shuttle tanker 3D box annotation obtained by the series of operations in steps (3.2) to (3.4) as positive samples, define the sum of the distances between the anchor box corresponding to the aforementioned anchor points and the 8 corner points of the true 3D box as the loss function, and realize the regression of the shuttle tanker 3D box;

[0041] Step 5: calibrate the installation position of the binocular camera, convert the detection results of step (4.5), and issue a warning in real time based on the conversion results;

[0042] (5.1) Calibrate the relative position coordinates of the binocular camera and the oil and gas production platform, transform the three-dimensional frame of the shuttle tanker in step (4.5) from the camera coordinate system to the platform coordinate system, and obtain the relative position and posture between the shuttle tanker and the production platform;

[0043] (5.2) according to step (5.1), the straight-line distance between the front end of the shuttle tanker and the oil and gas production platform and the angle between the front end of the shuttle tanker and the production platform are calculated;

[0044] (5.3) Define the safety range and level of distance and angle. Once the monitoring results show that the shuttle tanker has entered the corresponding danger warning state, the system will issue an alarm of the corresponding emergency level.

[0045] Beneficial results of the present invention: The high-risk external transmission operating environment of the oil and gas production platform and the shuttle tanker, the close distance between the two, the excessive volume and mass, and the delay in attitude position control are all very likely to cause the two to have an immeasurable collision event. The present invention uses a binocular camera as a monitoring sensor to calculate the relative posture information of the shuttle tanker in real time, and quickly warns when it enters a dangerous state, so that the operator can accurately adjust the position of the shuttle tanker in time according to the posture and warning information, effectively avoiding the occurrence of serious collision accidents between the production platform and the shuttle tanker.

[0046] (1) Using LiDAR and binocular cameras to jointly collect real data, a binocular depth estimation dataset and a 3D detection dataset for the offshore oil and gas production platform-shuttle tanker outflow operation scenario were produced, providing data support for the real-time detection of shuttle tankers.

[0047] (2) A deep learning algorithm is used to obtain the mapping relationship between the stereo image and the depth and three-dimensional posture of the shuttle tanker. The position and status of the shuttle tanker are calculated in real time through the stereo image data transmitted by the binocular camera. This can help the operator accurately monitor the posture status of the shuttle tanker so that reasonable adjustment measures can be made in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural flow chart of the present invention.

[0049] Figure 2 Schematic diagram of a chessboard.

[0050] Figure 3 Schematic diagram of joint data collection by lidar and camera.

[0051] Figure 4 Schematic diagram of collecting radar point cloud data of the calibration plate.

[0052] Figure 5 Schematic diagram of collecting image data of the calibration plate.

[0053] Figure 6 Schematic diagram of point cloud data after filtering.

[0054] Figure 7 Schematic diagram of projecting point cloud data into image data.

[0055] Figure 8 A schematic diagram of a labeled three-dimensional box.

[0056] Figure 9A top view diagram of the orientation of the labeled 3D box.

[0057] Figure 10 A three-dimensional schematic diagram of a shuttle tanker. DETAILED DESCRIPTION

[0058] In order to explain the steps of the present invention in more detail. The specific implementation process of the present invention is introduced by way of drawings and cases. The cases described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative labor conditions belong to the protection scope of the present invention.

[0059] See also Figure 1 The present invention provides a method for monitoring and early warning of offshore shuttle oil tankers based on binocular vision. By using laser radar and binocular camera to jointly collect real external transmission operation conditions to produce depth estimation and three-dimensional detection data sets, a deep learning algorithm is used to perform depth and three-dimensional detection on the input stereo image in real time, and the position information of the shuttle oil tanker is accurately estimated. In addition, a shuttle oil tanker entering the danger range is warned in real time, which can effectively avoid the occurrence of major collision accidents.

[0060] The method for monitoring and early warning of offshore shuttle oil tankers based on binocular vision mainly includes the following steps:

[0061] Step 1 specifically includes the following sub-steps:

[0062] (1.1) Make a checkerboard calibration plate. The checkerboard is composed of small black and white squares. The number of small squares in the horizontal and vertical directions is set differently to construct a black and white checkerboard calibration plate with unequal overall length and width. For example, see Figure 2 In this example, the side length of the small square in the middle is set to 7cm. The vertical direction consists of 9 small squares and the horizontal direction consists of 10 small squares. Please refer to Figure 3 , the laser radar and the binocular camera are fixed together to collect the radar point cloud data and image data on the calibration plate, and the calibration plate position is continuously changed for multiple collections to obtain multiple sets of point cloud-image joint data. For examples, please refer to Figure 4 and Figure 5 For example, in the example, it is necessary to ensure that the chessboard is completely within the camera's field of view;

[0063] (1.2) Point cloud filtering: When calibrating, manually filter the point cloud data in each point cloud-image data pair, remove all point clouds except the calibration plate, and only keep the point cloud where the calibration plate is located. For example, please refer to Figure 6 ;

[0064] (1.3) Calibration of the internal and external parameters of the binocular camera: Use the binocular images containing the calibration board for calibration. Find and match the feature corner points of the calibration board in the left and right images to obtain the internal and external parameters of the left and right cameras, including the internal parameter matrix, distortion coefficients, and external parameter matrix, that is, the rotation matrix and translation matrix of the right camera coordinate system relative to the left camera coordinate system;

[0065] (1.4) Perform the joint calibration of the lidar and the binocular camera on the image data obtained in step (1.1) and the point cloud data after filtering in step (1.2) to obtain the rotation matrix and translation matrix of the rigid transformation between the lidar coordinate system and the camera coordinate system. After completing the calibration in the example, the lidar point cloud can be projected into the image through the above rigid transformation matrix. For details, please refer to Figure 7 ;

[0066] Step 2, which specifically includes the following sub-steps:

[0067] (2.1) To ensure the validity of the calibration data in step (1.4), the fixed connection position of the lidar and the camera is required to be the same as that during the joint acquisition in step (1.1). Conduct joint acquisition on the actual working scene of the shuttle tanker's external transfer operation, perform soft synchronization of the time stamps for each frame of the point cloud and the picture, and capture the synchronized data to obtain the point cloud-image data pair;

[0068] (2.2) Project the point cloud data in step (2.1) into the camera coordinate system through the transformation matrix from the lidar coordinate system to the camera coordinate system obtained in step (1.4);

[0069] (2.3) Pass the point cloud data in the camera coordinate system obtained in step (2.2) through the camera internal parameter matrix obtained in step (1.3) to obtain the point cloud data corresponding to the field of view of the binocular camera; Convert this point cloud data from the camera coordinate system to the image coordinate system to generate the two-dimensional projection of the lidar point cloud data in the image and obtain the depth map;

[0070] (2.4) Perform a completion operation on the depth map projected from the radar point cloud obtained in step (2.3) to make the sparse radar point cloud into a dense and continuous depth map, and use this depth map as the training label for binocular depth estimation.

[0071] Step 3, which specifically includes the following sub-steps:

[0072] (3.1) For the binocular image data collected by the binocular camera in step (2.1), remove the low-quality image data with motion blur caused by camera jitter and excessive movement amplitude of the FPSO, and at the same time remove the radar point cloud data with the same time stamp as this image, and only retain the image data and point cloud data that are clearly visible for the shuttle tanker;

[0073] (3.2) Refer to Figure 8 and Figure 9 . Using the point cloud annotation tool, annotate the 3D bounding box of the shuttle tanker for the radar point cloud data to obtain the category C of the shuttle tanker, the center point coordinates X O 、Y O and Z O of the 3D bounding box, the length length, width width and height height of the 3D bounding box, the orientation R' y and other label information;

[0074] (3.3) Refer to Figure 8 . Based on the 3D bounding box characterization information obtained in step (3.2), calculate the 3D coordinate values of the 8 corner points of the 3D bounding box. The X coordinate calculation formulas for corner points 1, 2, 5, and 6 are as follows (corner points 4, 3, 8, and 7 are the same):

[0075]

[0076] The Y coordinate calculation formulas are as follows (corner points 4, 3, 8, and 7 are the same):

[0077]

[0078] The Z coordinates of the four corner points on the upper and lower surfaces of the 3D bounding box are the same. Therefore, the Z coordinate calculation formulas for the upper and lower surfaces are as follows:

[0079]

[0080] (3.4) Transform the 3D coordinates of the 8 corner points of the top and bottom surfaces obtained in step (3.3) to the camera coordinate system by the method in step (2.2);

[0081] (3.5) The binocular camera internal parameter matrix, distortion coefficients, and external parameter rotation matrix and translation matrix obtained in step (1.3), the rotation matrix and translation matrix from the lidar coordinate system to the camera coordinate system obtained in step (1.4), and the 3D bounding box annotation information of the shuttle tanker in the camera coordinate system obtained from the series of operations in steps (3.2)-(3.4) constitute the dataset for binocular 3D detection;

[0082] (3.6) Shuffle the dataset in any order and divide it according to the ratio of training set: validation set: test set. For example, the ratio in the example is 7:1:2. The training set and validation set are used for training the algorithm, and the test set is used for verifying the accuracy of the algorithm;

[0083] Step 4 specifically includes the following sub-steps:

[0084] (4.1) Set the initial parameters of the deep learning model and the training termination conditions, extract features from the binocular images in the dataset of binocular three-dimensional detection in step (3.5) to obtain the feature information of the shuttle tanker in the images;

[0085] (4.2) Construct a planar sweep volume from the features obtained in step (4.1), calculate the matching loss based on the planar sweep volume, calculate the horizontal distance of the matching pixel i in the left and right images of the binocular image respectively, this horizontal distance is the disparity disp, and construct a calculation loss based on the disparity;

[0086] (4.3) According to the disparity disp in step (4.2), convert it to depth data and calculate the expectation of all depth candidates, use the depth estimation data obtained in step (2.4) as the supervision information to regress the depth value corresponding to each pixel point in the image; the conversion formula between disparity and depth is:

[0087]

[0088] where f x and baseline are the horizontal focal length and baseline distance of the camera respectively, both obtained through step (1.4);

[0089] (4.4) The camera internal parameter matrix obtained in step (1.4) includes the horizontal focal length f x , the vertical focal length f y and the actual position of the principal point c u , c v four parameters. Through inverse three-dimensional projection, map the last feature of the planar sweep volume in step (4.2) from the camera coordinate system (u, v, d) to the world coordinate system (x, y, z):

[0090]

[0091] (4.5) Take the N anchor points with the shortest distance to the true value of the three-dimensional box annotation of the shuttle tanker obtained from the series of operations in steps (3.2)-(3.4) as positive samples, define the sum of the distances between the anchor boxes corresponding to the aforementioned anchor points and the 8 corner points of the true three-dimensional box as the loss function to realize the regression of the three-dimensional box of the shuttle tanker. Please refer to Figure 10 , to obtain the position information of the three-dimensional box surrounding the shuttle tanker. At this time, the position information is represented in the camera coordinate system;

[0092] Step 5 specifically includes the following sub-steps:

[0093] (5.1) Because the binocular camera is not installed at the front end of the oil and gas production platform, it is necessary to calibrate the relative position coordinates of the binocular camera and the oil and gas production platform, and convert the detection results in step (4.5) from the camera coordinate system to the platform coordinate system to obtain the relative position and posture of the shuttle tanker and the production platform;

[0094] (5.2) according to step (5.1), the straight-line distance between the front end of the shuttle tanker and the oil and gas production platform and the angle between the front end of the shuttle tanker and the production platform are calculated;

[0095] (5.3) Define the safety range and level of distance and angle. Once the monitoring results show that the shuttle tanker has entered the corresponding danger warning state, the system will issue an alarm of the corresponding emergency level.

Claims

1. A method for monitoring and warning the status of a seagoing shuttle tanker based on binocular vision, characterized in that, The steps are as follows: Step 1: Calibrate the internal and external parameters of the binocular camera, and jointly calibrate the lidar and the binocular camera to obtain the transformation matrices between the lidar coordinate system, the camera coordinate system, and the image coordinate system; (1.1) Make a checkerboard calibration board, and use the lidar and the binocular camera to jointly collect the checkerboard calibration board to obtain the point cloud data and image data of the checkerboard calibration board; (1.2) Manually filter all the point clouds in the point cloud data except the checkerboard calibration board to obtain the point cloud data containing only the point clouds of the checkerboard calibration board; (1.3) Use the binocular image data containing the checkerboard calibration board to calibrate the internal and external parameter matrices of the binocular camera to obtain the internal camera parameter matrix, the distortion coefficients, and the external parameter rotation matrix and translation matrix of the rigid transformation between the left and right camera coordinate systems; (1.4) Perform joint calibration of the lidar and the binocular camera on the image data obtained in step (1.1) and the point cloud data after filtering in step (1.2) to obtain the rotation matrix and translation matrix of the rigid transformation between the lidar coordinate system and the camera coordinate system; Step 2: Transform the point cloud data collected by the lidar into the corresponding depth map data through the left camera internal parameter matrix in step (1.3) and the transformation matrix from the lidar coordinate system to the camera coordinate system obtained in step (1.4), and construct a dataset for binocular depth estimation; (2.1) Use the lidar and the binocular camera to jointly collect the shuttle tanker during the actual external transfer operation work scene, and perform time soft synchronization on the point cloud data and image data of each frame; (2.2) Project the point cloud data in step (2.1) into the camera coordinate system through the transformation matrix from the lidar coordinate system to the camera coordinate system obtained in step (1.4); (2.3) Pass the point cloud data in the camera coordinate system obtained in step (2.2) through the camera internal parameter matrix obtained in step (1.3) to obtain the point cloud data corresponding to the binocular camera's field of view; convert this point cloud data from the camera coordinate system to the image coordinate system to generate a two-dimensional projection of the lidar point cloud data in the image and obtain a depth map; (2.4) Perform depth completion on the depth map obtained in step (2.3) to obtain a dense and continuous depth map, which is used to construct a depth estimation dataset; Step 3: Perform three-dimensional annotation on the shuttle tanker with the point cloud data obtained in step (2.1), and represent the annotation information in the camera coordinate system in the same way as in step (2.2). The annotation information and the binocular images jointly construct a dataset for binocular three-dimensional detection; (3.1) Clean and screen the image data collected by the binocular camera in step (2.1) to remove low-quality and blurred binocular image data; (3.2) Label the point cloud data collected by the lidar in step (2.1) with the 3D bounding box of the shuttle tanker to obtain the category C of the shuttle tanker. The 3D bounding box annotation information includes the coordinates X of the center point of the 3D bounding box O , Y O and Z O , the length length, width width and height height of the 3D bounding box, and the orientation R' of the 3D bounding box y ; (3.3) Calculate the three-dimensional coordinate values of the eight corner points of the three-dimensional box through the three-dimensional box annotation information obtained in step (3.2). The X coordinate calculation formula is as follows: The Y coordinate calculation formula is as follows: The Z coordinate calculation formula is as follows: (3.4) Transform the three-dimensional coordinates of the eight corner points obtained in step (3.3) into the camera coordinate system by the same method as in step (2.2); (3.5) The binocular camera internal parameter matrix, distortion coefficient and external parameter rotation matrix and translation matrix obtained in step (1.3), the rotation matrix and translation matrix from the laser radar coordinate system to the camera coordinate system obtained in step (1.4), and the three-dimensional frame annotation information of the shuttle tanker in the camera coordinate system obtained in the series of operations of steps (3.2)-(3.4) constitute a binocular three-dimensional detection data set; (3.6) Divide the binocular 3D detection data set in step (3.5) into training set, validation set and test set in proportion; Step 4, training a deep learning algorithm based on the depth estimation data set of step (2.4) and the binocular 3D detection data set of step (3.5) and using the deep learning algorithm to predict the depth and 3D pose of the shuttle tanker; (4.1) performing feature extraction on the binocular image in the binocular 3D detection data set in step (3.5) to obtain feature information of the shuttle tanker in the image; (4.2) constructing a plane scanning volume based on the features obtained in step (4.1), calculating the matching loss based on the plane scanning volume, matching the horizontal distance of pixel i in the left and right images of the binocular image, the horizontal distance is the disparity disp, and constructing the calculation loss based on the disparity; (4.3) According to the disparity disp in step (4.2), convert it into depth depth data and calculate the expectation of all depth candidates. Use the depth estimation data obtained in step (2.4) as supervision information to regress the depth value corresponding to each pixel of the image. The conversion formula between disparity and depth is: where f x and baseline are the horizontal focal length and baseline distance of the camera respectively, both of which are obtained through step (1.4); (4.4) The camera internal parameter matrix obtained in step (1.4) includes the horizontal focal length f x , the vertical focal length f y and the actual position c of the principal point u , c v These four parameters are used to convert the last feature map of the planar scanned volume in step (4.2) from the camera coordinate system (u, v, d) to the world coordinate system (x, y, z) through inverse three-dimensional projection: (4.5) Take the N anchor points closest to the true value of the shuttle tanker 3D box annotation obtained by the series of operations of steps (3.2)-(3.4) as positive samples, define the sum of the distances between the anchor box corresponding to the aforementioned anchor points and the 8 corner points of the true 3D box as the loss function, and realize the regression of the shuttle tanker 3D box; Step 5: calibrate the installation position of the binocular camera, convert the detection results of step (4.5), and issue a warning in real time based on the conversion results; (5.1) Calibrate the relative position coordinates of the binocular camera and the oil and gas production platform, transform the three-dimensional frame of the shuttle tanker in step (4.5) from the camera coordinate system to the platform coordinate system, and obtain the relative position and posture between the shuttle tanker and the production platform; (5.2) according to step (5.1), the straight-line distance between the front end of the shuttle tanker and the oil and gas production platform and the angle between the front end of the shuttle tanker and the production platform are calculated; (5.3) Define the safety range and level of distance and angle. Once the monitoring results show that the shuttle tanker has entered the corresponding danger warning state, the system will issue an alarm of the corresponding emergency level.

Images