Monocular camera-based marine ship distance and azimuth detection method

Abstract

The invention discloses a seaborne ship distance and azimuth detection method based on a monocular camera, and relates to the technical field of computer image processing. Comprising the steps that data content is acquired, and a Yolov5s detection model is established; the Yolov5s detection model is trained; calibrating and correcting the monocular camera, and capturing a real-time video stream of the marine ship; performing real-time ship target identification on the captured video stream by the trained Yolov5s detection model, judging whether a ship is detected in the video stream or not, if so, taking a bottom midpoint of a detection frame as a target point, entering the next step, and if not, returning to the previous step; establishing a pixel coordinate system, an image coordinate system and a camera coordinate system, associating a conversion relation among the pixel coordinate system, the image coordinate system and the camera coordinate system, and converting the position of the ship target point in the pixel coordinate system into the position of the ship target point in the camera coordinate system; and calculating the distance and azimuth angle of the detected ship target point, and outputting the real-time position data of the ship and the calculated ship distance and azimuth information. According to the invention, efficient detection and accurate positioning of the ship are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method for detecting the distance and orientation of a marine ship based on a monocular camera. Background Art

[0002] With the rapid development of the global shipping industry and ocean development, the number and size of ships at sea are increasing, and the problems of ship collisions and navigation errors are becoming increasingly prominent. Most of the existing marine ship detection and positioning technologies rely on equipment such as radar, laser rangefinders or multi-lens cameras. Although these systems can achieve the detection and positioning of ship targets to a certain extent, they are complex, costly, and difficult to deploy, and cannot meet the real-time and accuracy requirements of ship navigation.

[0003] In recent years, with the rapid development of deep learning and computer vision technology, target detection methods based on image recognition have been widely used and studied, especially the YOLO series of models, which have attracted attention for their efficient real-time target detection performance. However, when the traditional YOLO model is applied to marine ship detection, it is difficult to accurately measure the distance and orientation of the target due to the lack of calculation methods for the target recognition accuracy of the monocular camera and the distance and orientation, which leads to its limitations in practical applications.

[0004] Therefore, it is an urgent problem for technicians in this field to propose a method for detecting the distance and orientation of marine ships based on a monocular camera to solve the difficulties existing in the prior art. Summary of the invention

[0005] In view of this, the present invention provides a method for detecting the distance and orientation of marine ships based on a monocular camera, which can achieve efficient, low-cost and real-time detection of the distance and orientation of marine ships.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for detecting distance and orientation of a marine ship based on a monocular camera, comprising:

[0008] S101, obtain data content and establish Yolov5s detection model;

[0009] S102, training the Yolov5s detection model;

[0010] S103, calibrating and correcting the monocular camera to capture real-time video stream of ships at sea;

[0011] S104, the trained Yolov5s detection model performs real-time ship target recognition on the video stream captured by the monocular camera, and determines whether there is a ship detected in the video stream. If so, the bottom midpoint of the detection frame is taken as the target point, and the process proceeds to S105; if not, the process returns to S103;

[0012] S105, establishing a pixel coordinate system, an image coordinate system, and a camera coordinate system, and associating conversion relationships between them, and converting the position of the ship target point in the pixel coordinate system into the position in the camera coordinate system;

[0013] S106, calculating the distance and azimuth of the detected ship target point, and outputting the real-time position data of the ship and the calculated ship distance and azimuth information.

[0014] In the above method, optionally, in S101, the Yolov5s detection model established includes: an input end, a Backbone reference network, a Neck network and a Head output layer;

[0015] Input: Image preprocessing is performed during the model training phase, including mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling.

[0016] Backbone benchmark network: including Focus structure and CBS module;

[0017] Neck network: Add CBAM attention mechanism, introduce BottleneckCSP module to replace C3 module, use FPN+PAN feature pyramid structure to extract feature maps of different scales, and perform multi-scale fusion processing on the features extracted by the backbone network;

[0018] Head output layer: includes the loss function GIOU_Loss of the output layer during training and IOU_hms of the prediction box screening.

[0019] In the above method, optionally, in S102, the Yolov5s detection model is trained, specifically:

[0020] The model is trained using ship datasets from multiple different maritime scenarios, and the trained model is converted into a format suitable for TensorRT.

[0021] In the above method, optionally, in S103, the monocular camera is calibrated and corrected:

[0022] The monocular camera is calibrated using a chessboard calibration plate to obtain the camera's intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient, and the captured image is geometrically corrected.

[0023] In the above method, optionally, in S104, for the ship identified in the video stream image, the bottom midpoint of the target frame is selected as the target point, and the target point is corrected by offsetting the reference point upward by d pixels to obtain the coordinates of the upper left corner and the lower right corner of the target frame. The coordinates of the target point are (x p ,y p ), the coordinates of the upper left corner and the lower right corner are (x L ,y L ) and (x R ,y R ), so the target point coordinates can be expressed as:

[0024]

[0025] In the above method, optionally, in S105, a pixel coordinate system, an image coordinate system, and a camera coordinate system are established, and a conversion relationship is associated therebetween, so that the ship target point is converted from the pixel coordinate system to the image coordinate system, and then to the camera coordinate system; (u, v) represents the pixel coordinate; (x, y) represents the image coordinate;

[0026] (X C , Y C , Z C ) represents the camera coordinates;

[0027] From pixel coordinate system to image coordinate system:

[0028]

[0029] From image coordinate system to camera coordinate system:

[0030]

[0031] In the above method, optionally, in S106, the distance and azimuth of the detected ship target point are calculated, specifically:

[0032] In the process of monocular ranging, the target point Q corresponds to point Q' on the image, and the projection of point Q' on the y-axis is point P'. C The angle between the optical axis and Z is a. C The angle between the optical axis and PP' is b, and the angle between the straight line OP and the x1 axis of the ground is c;

[0033] Determine the height h of the camera center position from the horizontal plane;

[0034] Measure the angle ∠a between the camera optical axis and the horizontal line;

[0035] The internal parameter matrix can be used to calculate Z C Angle ∠b between the optical axis and PP':

[0036]

[0037] The hull's Z can be calculated by ∠a, ∠b and camera height h C Values ​​on the optical axis:

[0038] ∠c=∠a+∠b

[0039]

[0040] Z C =OP*cos(∠b);

[0041] Through the internal parameter matrix parameters and Z C X can be calculated C and Y C Values, (u1, v1) are the coordinates of the target point on the image, and (u0, v0) are the coordinates of the camera's principal point in the image:

[0042]

[0043] The distance and orientation between the camera and the target ship are calculated by the value of the target point in the camera coordinate system.

[0044]

[0045] It can be seen from the above technical solutions that compared with the prior art, the present invention provides a method for detecting the distance and azimuth of a marine ship based on a monocular camera, which has the following beneficial effects: 1) Compared with the YOLOv5s model, as well as equipment such as radar, laser rangefinder or multi-camera, the method for calculating the distance and azimuth of a marine ship based on monocular camera perception has the advantages of simple equipment, low cost and low deployment difficulty; 2) By improving the YOLOv5s model structure, the model can learn more features, while increasing the network depth, reducing the number of parameters and calculation complexity, and significantly improving the feature representation capability, greatly improving the ship recognition rate, while improving the recognition accuracy and the stability of the recognition frame; 3) Due to the reasonable design of the model structure, the ship detection accuracy is improved, thereby enhancing the accuracy of target calculation, which can provide effective support for marine navigation and obstacle avoidance, and improve the safety and reliability of marine operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1A flow chart of a method for detecting distance and orientation of a marine ship based on a monocular camera provided by the present invention;

[0048] Figure 2 A model structure diagram of the yolov5s detection model provided by the present invention;

[0049] Figure 3 A structural diagram of the BottleneckCSP module in the yolov5s detection model provided by the present invention;

[0050] Figure 4 A structural diagram of the FPN+PAN feature pyramid structure in the yolov5s detection model provided by the present invention;

[0051] Figure 5 A schematic diagram of the calculation and search for target points provided by the present invention;

[0052] Figure 6 The principle diagram of converting the coordinates in the pixel coordinate system provided by the present invention into the point coordinates in the camera coordinate system;

[0053] Figure 7 This is a schematic diagram of the distance and orientation calculation principle of the monocular camera provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] Reference Figure 1 As shown, a method for detecting distance and orientation of a marine ship based on a monocular camera comprises:

[0057] S101, obtain data content and establish Yolov5s detection model;

[0058] S102, training the Yolov5s detection model;

[0059] S103, calibrating and correcting the monocular camera to capture real-time video stream of ships at sea;

[0060] S104, the trained Yolov5s detection model performs real-time ship target recognition on the video stream captured by the monocular camera, and determines whether there is a ship detected in the video stream. If so, the bottom midpoint of the detection frame is taken as the target point, and the process proceeds to S105; if not, the process returns to S103;

[0061] S105, establishing a pixel coordinate system, an image coordinate system, and a camera coordinate system, and associating conversion relationships between them, and converting the position of the ship target point in the pixel coordinate system into the position in the camera coordinate system;

[0062] S106, calculating the distance and azimuth of the detected ship target point, and outputting the real-time position data of the ship and the calculated ship distance and azimuth information.

[0063] Further, see Figure 2 As shown, in S101, the Yolov5s detection model established includes:

[0064] Input: Image preprocessing is performed during the model training phase, including mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling.

[0065] Backbone benchmark network: including Focus structure and CBS module;

[0066] Neck network: Add CBAM attention mechanism, a simple and effective attention module for feedforward convolutional neural networks. CBAM is a lightweight universal module that can be seamlessly integrated into any CNN architecture with negligible overhead and can be trained end-to-end with the base CNN.

[0067] Reference Figure 3As shown in the figure, the BottleneckCSP module is introduced to replace the C3 module. The BottleneckCSP module is introduced to improve the extraction efficiency of deep image features. Compared with C3, BottleneckCSP has stronger feature expression ability and larger receptive field, especially in small target detection. The module consists of two parts: Bottleneck and CSP. Bottleneck adopts the classic residual structure and realizes output through the combination of 1×1 and 3×3 convolution layers. CSP realizes cross-stage connection by dividing the features into two parts and convolving them separately. This design enhances the feature transferability and diversity, promotes information flow, and improves detection performance. At the same time, BottleneckCSP reduces the number of parameters and keeps the input and output channels consistent through a dual-branch structure, effectively reducing the computational complexity while improving the network depth, and enhancing the feature representation capability.

[0068] Reference Figure 4 As shown in the figure, the FPN+PAN feature pyramid structure is used to extract feature maps of different scales, and the features extracted by the backbone network are subjected to multi-scale fusion processing; the FPN+PAN structure is used to extract feature maps of different scales, and the target detection capability is enhanced through multi-scale fusion; the original feature pyramid contains three layers: the P3 layer is used to detect small targets, and the feature map size is 80×80; the P4 layer is used to detect medium targets, and the feature map size is 40×40; the P5 layer is used to detect large targets, and the feature map size is 20×20; in order to improve the detection performance of small targets, the new network structure adds the P2 layer to the feature fusion, and uses the position information of the shallow feature map to enhance the detection capability of small targets; the feature maps at lower levels have higher resolution and can capture more detailed information, while the deep feature maps contain stronger semantic information; by fusing the deep and shallow feature maps, the detection performance of small targets can be improved;

[0069] The Concat operation is used to fuse feature maps, changing only the number of channels without changing the size. The specific operation formula is:

[0070] Concat(X1,X2)=(B,C1,H,W)+(B,C2,H,W)=(B,C1+C2,H,W);

[0071] Among them, X1 and X2 are two feature maps. To adapt to the increased large-scale fusion, the original anchor frame is adjusted and a new layer of Detect is added;

[0072] The initial anchor boxes of YOLOv5s are set to [10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119],

[0073] [116,90,156,198,373,326], combined with large-scale feature maps, the anchor box is adjusted to four layers:

[0074] [4,5,8,10,22,18], [10,13,16,30,33,23], [30,61,62,45,59,119],

[0075] [116,90,156,198,373,326]; Using these small-sized anchor boxes can more accurately capture small features and improve the detection and recognition capabilities of small objects;

[0076] Head output layer: includes the loss function GIOU_Loss of the output layer during training and IOU_hms of the prediction box screening.

[0077] Furthermore, in S102, the Yolov5s detection model is trained, specifically:

[0078] By collecting sufficient data sets and training based on the improved structural model of YOLOv5s, the trained model is converted into a format suitable for TensorRT to accelerate the reasoning process and improve real-time detection performance.

[0079] Furthermore, in S103, the monocular camera is calibrated and corrected:

[0080] The monocular camera is calibrated using a chessboard calibration plate to obtain the camera's intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient. The intrinsic parameter matrix is ​​used to represent the camera's focal length, principal point offset and other parameters to perform geometric correction on the captured image. Based on these calibration parameters, the captured image is geometrically corrected to ensure the accuracy and precision of the image.

[0081] Further, see Figure 5 As shown in S104, for the ship identified in the video stream image, the bottom midpoint of the target frame is selected as the target point, and the target point is corrected by offsetting. The reference point is offset upward by d pixels to obtain the coordinates of the upper left corner and the lower right corner of the target frame. The coordinates of the target point are (x p ,y p ), the coordinates of the upper left corner and the lower right corner are (x L ,y L ) and (x R ,y R ), so the target point coordinates can be expressed as:

[0082]

[0083] Further, see Figure 6As shown, in S105, a pixel coordinate system, an image coordinate system, and a camera coordinate system are established, and the conversion relationship between them is associated. The intrinsic parameter matrix and the extrinsic parameter matrix of the camera are used to convert the ship target point from the pixel coordinate system to the image coordinate system, and then to the camera coordinate system; (u, v) represents the pixel coordinate; (x, y) represents the image coordinate; (x C , Y C , Z C ) represents the camera coordinates;

[0084] From pixel coordinate system to image coordinate system:

[0085]

[0086] From image coordinate system to camera coordinate system:

[0087]

[0088] Further, see Figure 7 As shown, in S106, the distance and azimuth of the detected ship target are calculated, and the accurate distance and azimuth information between the ship target and the camera is obtained by taking the average value after multiple calculations, which is specifically:

[0089] In the process of monocular ranging, the target point Q corresponds to point Q' on the image, and the projection of point Q' on the y-axis is point P'. C The angle between the optical axis and Z is a. C The angle between the optical axis and PP' is b, and the angle between the straight line OP and the x1 axis of the ground is c;

[0090] Determine the height h of the camera center position from the horizontal plane;

[0091] Measure the angle ∠a between the camera optical axis and the horizontal line;

[0092] The internal parameter matrix can be used to calculate Z C Angle ∠b between the optical axis and PP':

[0093]

[0094] The hull's Z can be calculated by ∠a, ∠b and camera height h C Values ​​on the optical axis:

[0095] ∠c=∠a+∠b

[0096]

[0097] Z C =OP*cos(∠b);

[0098] Through the internal parameter matrix parameters and Z C X can be calculatedC and Y C Values, (u1, v1) are the coordinates of the target point on the image, and (u0, v0) are the coordinates of the camera's principal point in the image:

[0099]

[0100] The distance and direction between the camera and the target ship can be calculated by the value of the target point in the camera coordinate system.

[0101]

[0102] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0103] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting the distance and orientation of a ship at sea based on a monocular camera, characterized in that: include: S101, obtain data content and establish Yolov5s detection model; S102, training the Yolov5s detection model; S103, calibrating and correcting the monocular camera to capture real-time video stream of ships at sea; S104, the trained Yolov5s detection model performs real-time ship target recognition on the video stream captured by the monocular camera, and determines whether there is a ship detected in the video stream. If so, the bottom midpoint of the detection frame is taken as the target point, and the process proceeds to S105; if not, the process returns to S103; S105, establishing a pixel coordinate system, an image coordinate system, and a camera coordinate system, and associating conversion relationships between them, and converting the position of the ship target point in the pixel coordinate system into the position in the camera coordinate system; S106, calculating the distance and azimuth of the detected ship target point, and outputting the real-time position data of the ship and the calculated ship distance and azimuth information.

2. According to claim 1, a method for detecting distance and orientation of a marine ship based on a monocular camera is characterized in that: In S101, the Yolov5s detection model established includes: Input input end, Backbone benchmark network, Neck network and Head output layer; Input: Image preprocessing is performed during the model training phase, including mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling. Backbone benchmark network: including Focus structure and CBS module; Neck network: Add CBAM attention mechanism, introduce BottleneckCSP module to replace C3 module, use FPN+PAN feature pyramid structure to extract feature maps of different scales, and perform multi-scale fusion processing on the features extracted by the backbone network; Head output layer: includes the loss function GIOU_Loss of the output layer during training and IOU_hms of the prediction box screening.

3. The method for detecting distance and orientation of a ship at sea based on a monocular camera according to claim 1, characterized in that: In S102, the Yolov5s detection model is trained, specifically: The model is trained using ship datasets in multiple different maritime scenarios, and the trained model is converted into a format suitable for TensorRT.

4. The method for detecting distance and orientation of a ship at sea based on a monocular camera according to claim 1, characterized in that: In S103, the monocular camera is calibrated and corrected: The monocular camera is calibrated using a chessboard calibration plate to obtain the camera's intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient, and the captured image is geometrically corrected.

5. The method for detecting distance and orientation of a marine ship based on a monocular camera according to claim 1, characterized in that: In S104, for the ship identified in the video stream image, the midpoint of the bottom of the target frame is selected as the target point, and the target point is corrected by offsetting. The reference point is offset upward by d pixels to obtain the coordinates of the upper left corner and the lower right corner of the target frame. The coordinates of the target point are (x p ,y p ), the coordinates of the upper left corner and the lower right corner are (x L ,y L ) and (x R ,y R ), the target point coordinates are expressed as:

6. The method for detecting distance and orientation of a ship at sea based on a monocular camera according to claim 1, characterized in that: In S105, a pixel coordinate system, an image coordinate system, and a camera coordinate system are established, and the conversion relationship between them is associated, and the ship target point is converted from the pixel coordinate system to the image coordinate system, and then to the camera coordinate system; (u, v) represents the pixel coordinate; (x, y) represents the image coordinate; (x C , Y C , Z C ) represents the camera coordinates; From pixel coordinate system to image coordinate system: From image coordinate system to camera coordinate system:

7. The method for detecting distance and orientation of a ship at sea based on a monocular camera according to claim 1, characterized in that: In S106, the distance and azimuth of the detected ship target point are calculated, specifically: In the process of monocular ranging, the target point Q corresponds to point Q' on the image, and the projection of point Q' on the y-axis is point P'. C The angle between the optical axis and Z is a. C The angle between the optical axis and PP' is b, and the angle between the straight line OP and the x1 axis of the ground is c; Determine the height h of the camera center position from the horizontal plane; Measure the angle ∠a between the camera optical axis and the horizontal line; Calculate Z through the internal parameter matrix C Angle ∠b between the optical axis and PP': Calculate the ship's Z position by ∠a, ∠b and camera height h C Values ​​on the optical axis: ∠c=∠a+∠b Z C =OP*cos(∠b); Through the internal parameter matrix parameters and Z C Calculate X C and Y C Values, (u1, v1) are the coordinates of the target point on the image, and (u0, v0) are the coordinates of the camera's principal point in the image: The distance and orientation between the camera and the target ship are calculated by the value of the target point in the camera coordinate system.

Images