Ship water gauge tracking method, device, storage medium and electronic equipment

Abstract

The present invention relates to the field of target tracking technology, and in particular to a ship draft gauge tracking method, device, storage medium and electronic device. The method comprises: S100, detecting a real-time image acquired by a camera device based on a pre-trained detection model; S200, if it is detected that the real-time image includes a draft gauge line, outputting the minimum rectangular area including the draft gauge line in the real-time image to the central area of ​​a display device; S300, if it is not detected that the real-time image includes a draft gauge line, adjusting the orientation of the camera device until the real-time image includes a draft gauge line; the present invention can adjust the orientation of the camera device based on whether there is a minimum rectangular area including the draft gauge line in the real-time image and the position of the minimum rectangular area in the real-time image, thereby improving the accuracy of ship draft gauge tracking.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a ship draft tracking method, device, storage medium and electronic equipment. Background Art

[0002] As dock loading continues to evolve towards unmanned and intelligent operations, observing the ship's draft gauge during the loading process is crucial for both loading and measurement. This has led to the emergence of numerous ship draft gauge observation devices. Most ship draft gauge observation devices involve installing cameras at the dock or attaching pan-tilt cameras to the ship draft gauge observation equipment. Manual adjustments are made via a computer's control interface to track and observe the ship's draft gauge.

[0003] However, during the loading process, the ship's water level line will constantly change in the image of the observation camera due to changes in waves, tides, cables, draft, etc., and the camera device needs to be frequently adjusted manually to track the ship's water level line, which is very labor-intensive.

[0004] There is an urgent need in this field for a solution to enable a camera pan / tilt to automatically track the water gauge line of a ship. Summary of the Invention

[0005] The present invention provides a ship water gauge tracking method, device, storage medium and electronic equipment, which solve the technical problem that some technical solutions require frequent manual adjustment of a camera device to track the ship's water gauge line.

[0006] In a first aspect, the present invention provides a method for tracking a ship's water gauge, comprising:

[0007] S100, detecting the real-time image acquired by the camera device based on a pre-trained detection model;

[0008] S200, if it is detected that the real-time image includes a water gauge line, outputting the smallest rectangular area including the water gauge line in the real-time image to the central area of ​​the display device;

[0009] S300: If the water gauge line is not detected in the real-time image, adjust the direction of the camera device until the water gauge line is detected in the real-time image.

[0010] In some embodiments, in S100, the detection model is trained by the following steps:

[0011] Get a preset number of original images including water level lines;

[0012] Mark the water level line in the original image to obtain a marked image;

[0013] Grayscale the labeled images and divide them into training and test sets;

[0014] Train the Yolo model based on the training set;

[0015] The trained Yolo model is tested for recognition accuracy based on the test set, and the Yolo model with recognition accuracy greater than the threshold is regarded as the trained detection model.

[0016] In some embodiments, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0017] Calculate the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image;

[0018] When the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image is greater than a threshold, the angle of the camera device is adjusted.

[0019] In some embodiments, adjusting the angle of the camera device includes:

[0020] When the center coordinates of the minimum rectangular area are above the center coordinates of the real-time image, the pitch angle of the camera device is adjusted upwards;

[0021] When the center coordinates of the minimum rectangular area are below the center coordinates of the real-time image, the pitch angle of the camera device is adjusted downward;

[0022] When the center coordinates of the minimum rectangular area are to the left of the center coordinates of the real-time image, the rotation angle of the camera device is adjusted to the left;

[0023] When the center point coordinates of the minimum rectangular area are to the right of the center point coordinates of the real-time image, the rotation angle of the camera device is adjusted to the right.

[0024] In some embodiments, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0025] Intercept the smallest rectangular area including the water gauge line in the real-time image;

[0026] The center point of the smallest rectangular area including the water gauge line in the intercepted real-time picture is aligned with the center of the display device, and the smallest rectangular area is enlarged.

[0027] In some embodiments, in S300, the method further includes:

[0028] Reducing the focal length of the camera device and detecting whether the real-time image after the focal length is reduced includes the water gauge line;

[0029] If the real-time picture after the focal length is reduced includes the water gauge line, re-performing the step of reducing the focal length of the camera device and detecting whether the real-time picture after the focal length is reduced includes the water gauge line, until the real-time picture after the focal length is reduced does not include the water gauge line;

[0030] If the real-time image after reducing the focal length does not include the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected at the increased focal length.

[0031] In some embodiments, detecting whether the real-time image after reducing the focus includes a water gauge line includes:

[0032] Detect whether there are any two vertically arranged adjacent water gauge marks in the real-time image;

[0033] If there are any two vertically arranged adjacent water gauge marks in the real-time image, it is determined that the real-time image includes a water gauge line;

[0034] If there are no two vertically adjacent water gauge marks in the real-time image, it is determined that the real-time image does not include a water gauge line;

[0035] Among them, the water level marks include the numbers 2, 4, 6, 8 and the letter M.

[0036] In a second aspect, the present invention provides a ship water gauge tracking device, comprising:

[0037] A detection module, comprising a pre-trained detection model, for detecting real-time images acquired by the camera device based on the pre-trained detection model;

[0038] an output module, configured to output a minimum rectangular area including the water gauge line in the real-time image to a central area of ​​the display device if it is detected that the real-time image includes the water gauge line;

[0039] The adjustment module is used to adjust the direction of the camera device if the water gauge line is not detected in the real-time image, until the water gauge line is detected in the real-time image.

[0040] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the methods in the first aspect when the computer program is executed by a processor.

[0041] In a fourth aspect, the present invention provides an electronic device comprising a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method of any one of the first aspects is implemented.

[0042] The present invention provides a ship draft gauge tracking method, device, storage medium and electronic device, which detect a real-time image acquired by a camera device based on a pre-trained detection model; if a draft gauge line is detected in the real-time image, the minimum rectangular area including the draft gauge line in the real-time image is output to the central area of ​​a display device; if no draft gauge line is detected in the real-time image, the orientation of the camera device is adjusted until a draft gauge line is detected in the real-time image; the present invention can adjust the orientation of the camera device based on whether a minimum rectangular area including the draft gauge line exists in the real-time image and the position of the minimum rectangular area in the real-time image, thereby improving the accuracy of ship draft gauge tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Hereinafter, the present invention will be described in more detail based on embodiments and with reference to the accompanying drawings:

[0044] Figure 1 Schematic diagram of a ship water gauge tracking method according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a ship water gauge tracking principle according to an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a ship water gauge line according to an embodiment of the present invention;

[0047] Figure 4 The figure is a schematic diagram of a ship water gauge tracking device according to an embodiment of the present invention.

[0048] In the drawings, like components are given like reference numerals, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solutions of the present invention, and to fully understand and implement the process of how the present invention applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. The embodiments of the present invention and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] As dock loading continues to evolve towards unmanned and intelligent operations, observing the ship's draft gauge during the loading process is crucial for both loading and measurement. This has led to the emergence of numerous ship draft gauge observation devices. However, most ship draft gauge observation devices rely on cameras installed at the dock or PTZ cameras installed on the ship draft gauge observation equipment. Manual adjustments are made via a computer control interface to track and observe the ship's draft gauge.

[0053] The principle and function of ship draft weighing: Draft weighing (checking weight by draft) involves measuring the ship's draft to calculate the ship's displacement and determine the onboard stores. The weight of bulk solid cargo is then calculated based on accurate ship charts. Draft weighing is generally applicable to low-value, difficult-to-weigh bulk commodities such as coal, salt, bulk fertilizer, ore, pig iron, scrap steel, petroleum coke, sulfur, and phosphate.

[0054] During the loading process, the ship's water gauge scale line will constantly change in the image of the observation camera device due to changes in waves, tides, cables, draft, etc. The camera device needs to be frequently adjusted to track the ship's water gauge, which is very labor-intensive.

[0055] It can be seen that there is an urgent need in this field for a solution to enable the camera pan / tilt to automatically track the ship's water level.

[0056] Example 1

[0057] Figure 1 This is a flow chart of a ship water gauge tracking method according to an embodiment of the present invention. Figure 2FIG. 1 is a schematic diagram of the ship water gauge tracking principle according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, this embodiment provides a ship water gauge tracking method, including:

[0058] S100, detecting the real-time image acquired by the camera device based on a pre-trained detection model;

[0059] S200, if it is detected that the real-time image includes a water gauge line, outputting the smallest rectangular area including the water gauge line in the real-time image to the central area of ​​the display device;

[0060] S300: If the water gauge line is not detected in the real-time image, adjust the direction of the camera device until the water gauge line is detected in the real-time image.

[0061] The ship draft gauge tracking method provided in this embodiment detects the real-time image acquired by the camera device based on a pre-trained detection model; if the real-time image is detected to include a draft gauge line, the minimum rectangular area including the draft gauge line in the real-time image is output to the central area of ​​the display device; if the real-time image is not detected to include a draft gauge line, the orientation of the camera device is adjusted until the real-time image is detected to include a draft gauge line; the orientation of the camera device can be adjusted based on whether there is a minimum rectangular area including the draft gauge line in the real-time image and the position of the minimum rectangular area in the real-time image, thereby improving the accuracy of ship draft gauge tracking.

[0062] Example 2

[0063] Based on the above embodiment, in S100, the detection model is trained through the following steps:

[0064] Get a preset number of original images including water level lines;

[0065] Mark the water level line in the original image to obtain a marked image;

[0066] Grayscale the labeled images and divide them into training and test sets;

[0067] Train the Yolo model based on the training set;

[0068] The trained Yolo model is tested for recognition accuracy based on the test set, and the Yolo model with recognition accuracy greater than the threshold is regarded as the trained detection model.

[0069] In this embodiment, a camera is used to take pictures of the ship's water level gauge to obtain, for example, ten thousand original pictures.

[0070] The original image is annotated using annotation software to obtain an annotated image, that is, the position of the ship draft gauge is selected on the original image with a minimum rectangular frame using the annotation software to obtain an annotated image; wherein the annotation software is, for example, labelme software. After annotating, for example, 10,000 original images using labelme software, annotated images in the PASCAL VOC dataset format can be obtained, and the sample size thereof is, for example, 10,000; wherein the PASCAL VOC dataset provides a complete set of standardized and excellent datasets for image recognition and classification, and therefore the annotated images of the ship draft gauge are produced according to this standard; wherein the annotated images are divided into a training set and a test set, wherein the ratio of the training set to the test set is, for example, 8:2.

[0071] The annotated image is grayscaled and the grayscale value is calculated using the expression Gray = (Red * 0.299 + Green * 0.587 + Blue * 0.114). The grayscale value obtained based on this expression can clearly distinguish the color of the ship's draft gauge from the color of the hull and the water surface, improving the recognition efficiency of the ship's draft gauge. It is understandable that grayscale conversion can also be performed before generating the training and test sets.

[0072] The training set is fed into a water gauge line recognition model built based on the Yolo network architecture for parameter training, which takes two hours. After training, the accuracy of the current Yolo model is tested on the test set. When the accuracy reaches a preset value, the current Yolo model is selected as the final detection model. The preset accuracy value is [0.9, 1). The Yolo model is a segmentation algorithm. It can be used for "object detection," "object instance segmentation," and "object key point detection." It performs particularly well in situations where the objects are small, numerous, contain a lot of impurities, or overlap.

[0073] This embodiment obtains a preset number of original images including water gauge lines; marks the water gauge lines in the original images to obtain marked images; grayscales the marked images and divides them into a training set and a test set; trains a Yolo model based on the training set; performs a recognition accuracy test on the trained Yolo model based on the test set, and uses the Yolo model with a recognition accuracy greater than a threshold as a trained detection model; and can obtain a detection model with higher accuracy.

[0074] Example 3

[0075] Based on the above embodiment, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0076] Calculate the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image;

[0077] When the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image is greater than a threshold, the angle of the camera device is adjusted.

[0078] In some implementations of this embodiment, adjusting the angle of the camera device includes:

[0079] When the center coordinates of the minimum rectangular area are above the center coordinates of the real-time image, the pitch angle of the camera device is adjusted upwards;

[0080] When the center coordinates of the minimum rectangular area are below the center coordinates of the real-time image, the pitch angle of the camera device is adjusted downward;

[0081] When the center coordinates of the minimum rectangular area are to the left of the center coordinates of the real-time image, the rotation angle of the camera device is adjusted to the left;

[0082] When the center point coordinates of the minimum rectangular area are to the right of the center point coordinates of the real-time image, the rotation angle of the camera device is adjusted to the right.

[0083] In this embodiment, the detection model trained in the above embodiment is used to detect the real-time image captured by the ship's water gauge camera device. If the water gauge line is detected, the detection model will output the minimum rectangular area of ​​the water gauge line in the real-time image, that is, the detection model will output the vertex coordinates of the minimum rectangle surrounding the water gauge line; if the water gauge line is not detected, the detection model will output 0;

[0084] If the water gauge line is not detected, the camera device pan / tilt is controlled by the ONVIF universal camera device protocol to adjust the camera device orientation until the minimum rectangular area of ​​the water gauge line is detected. The pitch and rotation angles of the camera device have a step value of [0.1, 1] degrees, for example, 0.5 degrees.

[0085] If the water gauge line is detected, the position of the water gauge line is calculated from the minimum rectangular area of ​​the water gauge line in the real-time image, that is, the coordinates of the four vertices of the rectangle surrounding the water gauge line, that is, the coordinates of the center point of the rectangle surrounding the water gauge line are calculated.

[0086] When the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image exceeds, for example, 200 pixels, the camera orientation is fine-tuned. Specifically, when the center of the minimum rectangular area is below the center of the real-time image, the camera pitch angle is adjusted downward; when the center of the minimum rectangular area is above the center of the real-time image, the camera pitch angle is adjusted upward; when the center of the minimum rectangular area is to the left of the center of the real-time image, the camera rotation angle is adjusted leftward; when the center of the minimum rectangular area is to the right of the center of the real-time image, the camera rotation angle is adjusted rightward. This continues until the center coordinates of the minimum rectangular area are within 200 pixels of the center of the real-time image. At this point, the step value of the camera pitch and rotation angles is [0.1, 1] degrees, for example, a value of 0.1 degree can be taken.

[0087] In this embodiment, by adjusting the angle of the camera device by calculating the difference in the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image, the minimum rectangular area including the water gauge line in the real-time image can be output to the central area of ​​the display device.

[0088] Example 4

[0089] Based on the above embodiment, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0090] Intercept the smallest rectangular area including the water gauge line in the real-time image;

[0091] The center point of the smallest rectangular area including the water gauge line in the intercepted real-time picture is aligned with the center of the display device, and the smallest rectangular area is enlarged.

[0092] It is understandable that in the real-time image captured by the camera device, the smallest rectangular area including the water gauge line usually only occupies a portion of the area, rather than the entire area. Therefore, the water gauge line can be displayed at the center of the display area by intercepting and displaying only the smallest rectangular area of ​​the water gauge line.

[0093] In this embodiment, the tracking of the water gauge line is completed by intercepting and displaying only the minimum rectangular area of ​​the water gauge line to move the water gauge line to the center of the display area, thereby locking the minimum rectangular area to the center area of ​​the display area of ​​the display device without rotating the camera device, thereby completing the tracking of the ship's water gauge.

[0094] Example 5

[0095] Based on the above embodiment, in S300, the following steps are further included:

[0096] Reducing the focal length of the camera device and detecting whether the real-time image after the focal length is reduced includes the water gauge line;

[0097] If the real-time picture after the focal length is reduced includes the water gauge line, re-performing the step of reducing the focal length of the camera device and detecting whether the real-time picture after the focal length is reduced includes the water gauge line, until the real-time picture after the focal length is reduced does not include the water gauge line;

[0098] If the real-time image after reducing the focal length does not include the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected at the increased focal length.

[0099] In some implementations of this embodiment, detecting whether the real-time image after reducing the focal length includes a water gauge line includes:

[0100] Detect whether there are any two vertically arranged adjacent water gauge marks in the real-time image;

[0101] If there are any two vertically arranged adjacent water gauge marks in the real-time image, it is determined that the real-time image includes a water gauge line;

[0102] If there are no two vertically adjacent water gauge marks in the real-time image, it is determined that the real-time image does not include a water gauge line;

[0103] Among them, the water level marks include the numbers 2, 4, 6, 8 and the letter M.

[0104] Figure 3 Schematic diagram of a ship water gauge line according to an embodiment of the present invention. Figure 3 As shown, the ship's draft line includes multiple vertically arranged draft marks. The draft marks include the numbers 2, 4, 6, 8, and the letter M. Therefore, when at least two vertically arranged and adjacent draft marks are identified, the ship's draft line can be considered to have been identified. Among them, there are five types of vertically arranged adjacent draft marks: the first type is vertically arranged 2 and 4, the second type is vertically arranged 4 and 6, the third type is vertically arranged 6 and 8, the fourth type is vertically arranged 8 and M, and the fifth type is vertically arranged M and 2.

[0105] In this embodiment, the focal length of the camera is reduced, and the real-time image after the reduction is detected to see whether it contains the water gauge line. If the real-time image after the reduction is found to contain the water gauge line, the steps of reducing the focal length of the camera and detecting whether the real-time image after the reduction is found to contain the water gauge line are repeated until the real-time image after the reduction is found to contain the water gauge line. If the real-time image after the reduction is found to contain the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected at the increased focal length. Thus, the method of this embodiment obtains the minimum focal length required to identify the water gauge line. This embodiment identifies the water gauge line at the minimum focal length, i.e., the maximum field of view, thereby reducing the probability of the water gauge line drifting out of the camera's field of view and the number of times the camera needs to re-track the water gauge line. Furthermore, if the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image is fine-tuned to a distance of less than, for example, 200 pixels, as in the above-mentioned embodiment, the probability of the water gauge line drifting out of the camera's field of view can be further reduced, reducing the number of times the camera needs to re-track the water gauge line.

[0106] Example 6

[0107] Figure 4 FIG. 1 is a schematic diagram of a ship water gauge tracking device according to an embodiment of the present invention. Figure 4 As shown, based on the above embodiment, this embodiment provides a ship water gauge tracking device, including:

[0108] A detection module, comprising a pre-trained detection model, for detecting real-time images acquired by the camera device based on the pre-trained detection model;

[0109] an output module, configured to output a minimum rectangular area including the water gauge line in the real-time image to a central area of ​​the display device if it is detected that the real-time image includes the water gauge line;

[0110] The adjustment module is used to adjust the direction of the camera device if the water gauge line is not detected in the real-time image, until the water gauge line is detected in the real-time image.

[0111] In this embodiment, the detection module includes a pre-trained detection model, which is used to detect the real-time picture obtained by the camera device based on the pre-trained detection model; the output module is used to output the minimum rectangular area including the water gauge line in the real-time picture to the central area of ​​the display device if the water gauge line is detected in the real-time picture; the adjustment module is used to adjust the orientation of the camera device until the water gauge line is detected in the real-time picture if the water gauge line is not detected in the real-time picture.

[0112] In some embodiments, the detection model is trained by the following steps:

[0113] Get a preset number of original images including water level lines;

[0114] Mark the water level line in the original image to obtain a marked image;

[0115] Grayscale the labeled images and divide them into training and test sets;

[0116] Train the Yolo model based on the training set;

[0117] The trained Yolo model is tested for recognition accuracy based on the test set, and the Yolo model with recognition accuracy greater than the threshold is regarded as the trained detection model.

[0118] In some embodiments, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0119] Calculate the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image;

[0120] When the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image is greater than a threshold, the angle of the camera device is adjusted.

[0121] In some embodiments, adjusting the angle of the camera device includes:

[0122] When the center coordinates of the minimum rectangular area are above the center coordinates of the real-time image, the pitch angle of the camera device is adjusted upwards;

[0123] When the center coordinates of the minimum rectangular area are below the center coordinates of the real-time image, the pitch angle of the camera device is adjusted downward;

[0124] When the center coordinates of the minimum rectangular area are to the left of the center coordinates of the real-time image, the rotation angle of the camera device is adjusted to the left;

[0125] When the center point coordinates of the minimum rectangular area are to the right of the center point coordinates of the real-time image, the rotation angle of the camera device is adjusted to the right.

[0126] In some embodiments, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0127] Intercept the smallest rectangular area including the water gauge line in the real-time image;

[0128] The center point of the smallest rectangular area including the water gauge line in the intercepted real-time picture is aligned with the center of the display device, and the smallest rectangular area is enlarged.

[0129] In some embodiments, further comprising:

[0130] Reducing the focal length of the camera device and detecting whether the real-time image after the focal length is reduced includes the water gauge line;

[0131] If the real-time picture after the focal length is reduced includes the water gauge line, re-performing the step of reducing the focal length of the camera device and detecting whether the real-time picture after the focal length is reduced includes the water gauge line, until the real-time picture after the focal length is reduced does not include the water gauge line;

[0132] If the real-time image after reducing the focal length does not include the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected at the increased focal length.

[0133] In some embodiments, detecting whether the real-time image after reducing the focus includes a water gauge line includes:

[0134] Detect whether there are any two vertically arranged adjacent water gauge marks in the real-time image;

[0135] If there are any two vertically arranged adjacent water gauge marks in the real-time image, it is determined that the real-time image includes a water gauge line;

[0136] If there are no two vertically adjacent water gauge marks in the real-time image, it is determined that the real-time image does not include a water gauge line;

[0137] Among them, the water level marks include the numbers 2, 4, 6, 8 and the letter M.

[0138] It should be understood that the device of this embodiment has all the beneficial effects of the method embodiment.

[0139] Example 7

[0140] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method of the above embodiment is implemented.

[0141] The above-mentioned storage media can be flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc.

[0142] The method implemented in this embodiment includes:

[0143] S100, detecting the real-time image acquired by the camera device based on a pre-trained detection model;

[0144] S200, if it is detected that the real-time image includes a water gauge line, outputting the smallest rectangular area including the water gauge line in the real-time image to the central area of ​​the display device;

[0145] S300: If the water gauge line is not detected in the real-time image, adjust the direction of the camera device until the water gauge line is detected in the real-time image.

[0146] In some embodiments, in S100, the detection model is trained by the following steps:

[0147] Get a preset number of original images including water level lines;

[0148] Mark the water level line in the original image to obtain a marked image;

[0149] Grayscale the labeled images and divide them into training and test sets;

[0150] Train the Yolo model based on the training set;

[0151] The trained Yolo model is tested for recognition accuracy based on the test set, and the Yolo model with recognition accuracy greater than the threshold is regarded as the trained detection model.

[0152] In some embodiments, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0153] Calculate the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image;

[0154] When the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image is greater than a threshold, the angle of the camera device is adjusted.

[0155] In some embodiments, adjusting the angle of the camera device includes:

[0156] When the center coordinates of the minimum rectangular area are above the center coordinates of the real-time image, the pitch angle of the camera device is adjusted upwards;

[0157] When the center coordinates of the minimum rectangular area are below the center coordinates of the real-time image, the pitch angle of the camera device is adjusted downward;

[0158] When the center coordinates of the minimum rectangular area are to the left of the center coordinates of the real-time image, the rotation angle of the camera device is adjusted to the left;

[0159] When the center point coordinates of the minimum rectangular area are to the right of the center point coordinates of the real-time image, the rotation angle of the camera device is adjusted to the right.

[0160] In some embodiments, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0161] Intercept the smallest rectangular area including the water gauge line in the real-time image;

[0162] The center point of the smallest rectangular area including the water gauge line in the intercepted real-time picture is aligned with the center of the display device, and the smallest rectangular area is enlarged.

[0163] In some implementations, in S300, the method further includes:

[0164] Reducing the focal length of the camera device and detecting whether the real-time image after the focal length is reduced includes the water gauge line;

[0165] If the real-time picture after the focal length is reduced includes the water gauge line, re-performing the step of reducing the focal length of the camera device and detecting whether the real-time picture after the focal length is reduced includes the water gauge line, until the real-time picture after the focal length is reduced does not include the water gauge line;

[0166] If the real-time image after reducing the focal length does not include the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected at the increased focal length.

[0167] In some embodiments, detecting whether the real-time image after reducing the focus includes a water gauge line includes:

[0168] Detect whether there are any two vertically arranged adjacent water gauge marks in the real-time image;

[0169] If there are any two vertically arranged adjacent water gauge marks in the real-time image, it is determined that the real-time image includes a water gauge line;

[0170] If there are no two vertically adjacent water gauge marks in the real-time image, it is determined that the real-time image does not include a water gauge line;

[0171] Among them, the water level marks include the numbers 2, 4, 6, 8 and the letter M.

[0172] Example 8

[0173] This embodiment provides an electronic device, including a processor and a memory. The memory stores a computer program, and the method of the above embodiment is implemented when the processor executes the computer program.

[0174] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0175] The memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0176] The method implemented in this embodiment includes:

[0177] S100, detecting the real-time image acquired by the camera device based on a pre-trained detection model;

[0178] S200, if it is detected that the real-time image includes a water gauge line, outputting the smallest rectangular area including the water gauge line in the real-time image to the central area of ​​the display device;

[0179] S300: If the water gauge line is not detected in the real-time image, adjust the direction of the camera device until the water gauge line is detected in the real-time image.

[0180] In some embodiments, in S100, the detection model is trained by the following steps:

[0181] Get a preset number of original images including water level lines;

[0182] Mark the water level line in the original image to obtain a marked image;

[0183] Grayscale the labeled images and divide them into training and test sets;

[0184] Train the Yolo model based on the training set;

[0185] The trained Yolo model is tested for recognition accuracy based on the test set, and the Yolo model with recognition accuracy greater than the threshold is regarded as the trained detection model.

[0186] In some embodiments, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0187] Calculate the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image;

[0188] When the distance between the center coordinates of the minimum rectangular area and the center coordinates of the real-time image is greater than a threshold, the angle of the camera device is adjusted.

[0189] In some embodiments, adjusting the angle of the camera device includes:

[0190] When the center coordinates of the minimum rectangular area are above the center coordinates of the real-time image, the pitch angle of the camera device is adjusted upwards;

[0191] When the center coordinates of the minimum rectangular area are below the center coordinates of the real-time image, the pitch angle of the camera device is adjusted downward;

[0192] When the center coordinates of the minimum rectangular area are to the left of the center coordinates of the real-time image, the rotation angle of the camera device is adjusted to the left;

[0193] When the center point coordinates of the minimum rectangular area are to the right of the center point coordinates of the real-time image, the rotation angle of the camera device is adjusted to the right.

[0194] In some embodiments, in S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes:

[0195] Intercept the smallest rectangular area including the water gauge line in the real-time image;

[0196] The center point of the smallest rectangular area including the water gauge line in the intercepted real-time picture is aligned with the center of the display device, and the smallest rectangular area is enlarged.

[0197] In some implementations, in S300, the method further includes:

[0198] Reducing the focal length of the camera device and detecting whether the real-time image after the focal length is reduced includes the water gauge line;

[0199] If the real-time picture after the focal length is reduced includes the water gauge line, re-performing the step of reducing the focal length of the camera device and detecting whether the real-time picture after the focal length is reduced includes the water gauge line, until the real-time picture after the focal length is reduced does not include the water gauge line;

[0200] If the real-time image after reducing the focal length does not include the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected at the increased focal length.

[0201] In some embodiments, detecting whether the real-time image after reducing the focus includes a water gauge line includes:

[0202] Detect whether there are any two vertically arranged adjacent water gauge marks in the real-time image;

[0203] If there are any two vertically arranged adjacent water gauge marks in the real-time image, it is determined that the real-time image includes a water gauge line;

[0204] If there are no two vertically adjacent water gauge marks in the real-time image, it is determined that the real-time image does not include a water gauge line;

[0205] Among them, the water level marks include the numbers 2, 4, 6, 8 and the letter M.

[0206] Embodiment 9

[0207] Based on the above embodiment, an application example is provided here, which includes steps (1) to (12).

[0208] (1) Use a camera to take pictures of the ship's water level gauge to obtain, for example, 10,000 original pictures.

[0209] (2) Annotating the original image using annotation software to obtain an annotated image, that is, using the annotation software to frame the position of the ship's draft gauge on the original image with a minimum rectangular frame to obtain an annotated image; wherein the annotation software is, for example, labelme software, and after annotating, for example, 10,000 original images using labelme software, annotated images in the PASCAL VOC dataset format can be obtained, and its sample size is, for example, 10,000; wherein the PASCAL VOC dataset provides a complete set of standardized and excellent datasets for image recognition and classification, so the annotated images of the ship's draft gauge are produced according to this standard; wherein the annotated images are divided into a training set and a test set, wherein the ratio of the training set to the test set is 8:2.

[0210] (3) The above-mentioned annotated image is gray-scaled, and the grayscale value is calculated using the expression Gray = (Red*0.299+Green*0.587+Blue*0.114). The grayscale value obtained based on this expression can clearly distinguish the color of the ship's draft gauge from the color of the hull and the water surface, thereby improving the recognition efficiency of the ship's draft gauge.

[0211] (4) The training set is input into the water gauge line recognition model built based on the Yolo network structure for parameter training. The training time is two hours. After the training is completed, the accuracy of the current Yolo model is tested on the test set. When the accuracy reaches the preset value, the current Yolo model is determined as the final detection model. Among them, the accuracy preset value is [0.9, 1). The Yolo model is a body segmentation algorithm. It can be used for "target detection", "target instance segmentation", and "target key point detection". It performs particularly well in cases where the target size is small, the number is large, there are many impurities, and the targets overlap.

[0212] (5) Use the detection model to detect the real-time image captured by the ship's water gauge camera. If the water gauge line is detected, the detection model will output the rectangular area of ​​the water gauge line in the real-time image, that is, the detection model will output the vertex coordinates of the minimum rectangle surrounding the water gauge line; if the water gauge line is not detected, the detection model will output 0;

[0213] (6) If the water gauge line is not detected, the camera device pan / tilt is controlled by the ONVIF universal camera device protocol to adjust the camera device orientation until the rectangular area of ​​the water gauge line is detected. The step value of the pitch angle and rotation angle of the camera device is [0.1, 1] degrees, for example, a value of 0.5 degrees can be taken.

[0214] (7) The position of the water-scale line is calculated from the rectangular area of ​​the water-scale line in the real-time image, that is, the coordinates of the four vertices of the rectangle surrounding the water-scale line. That is, the coordinates of the center point of the rectangle surrounding the water-scale line are calculated. When the distance between the coordinates of the center point of the rectangle and the coordinates of the center point of the image exceeds 200 pixels, the direction of the camera device is fine-tuned. Specifically, when the center point of the scale line is below the center point of the image, the pitch angle of the camera device is adjusted downward; specifically, when the center point of the scale line is above the center point of the image, the pitch angle of the camera device is adjusted upward; specifically, when the center point of the scale line is to the left of the center point of the image, the rotation angle of the camera device is adjusted to the left; specifically, when the center point of the scale line is to the right of the center point of the image, the rotation angle of the camera device is adjusted to the right. This is done until the coordinates of the center point of the scale line are within 200 pixels around the center point of the image. At this time, the step value of the pitch and rotation angle of the camera device is [0.1, 1] degrees, for example, a value of 0.1 degree can be taken.

[0215] (8) In some cases, when the rectangular area of ​​the water gauge line is detected, the water gauge line can be tracked by intercepting and displaying only the rectangular area of ​​the water gauge line to move the water gauge line to the center of the display area;

[0216] (9) In some cases, if the water gauge line is not detected, the focal length of the camera device is reduced to expand the field of view; each time the focal length is reduced, a real-time image is acquired again, the real-time image is grayscale processed / binarized, and whether there are at least two water gauge marks within the preset pixels centripetal to the edge of the real-time image, wherein the water gauge marks include numbers 2, 4, 6, 8 and the letter M; when at least two water gauge marks are detected at the edge of the real-time image, the direction of the edge is determined to be the direction in which the camera device needs to be deflected; in some cases, after the water gauge line is detected by (7), the focal length of the camera is reduced to expand the field of view; each time the focal length is reduced, a real-time image is acquired again, the real-time image is grayscale processed / binarized, and whether there are at least two water gauge marks within the preset pixels centripetal to the edge of the real-time image, wherein the water gauge marks include numbers 2, 4, 6, 8 and the letter M; when at least two water gauge marks are detected at the edge of the real-time image, the direction of the edge is determined to be the direction in which the camera needs to be deflected;

[0217] (10) In some cases, if the water gauge line is detected, the focal length of the camera device is reduced to expand the field of view of the camera device, thereby reducing the probability that the ship's water gauge line will drift out of the real-time image;

[0218] (11) Based on (9), the water gauge mark is identified based on the rectangular area of ​​the water gauge line; when any two adjacent water gauge marks arranged vertically can be identified, the focal length of the camera device is reduced; the process is repeated until any two adjacent water gauge marks arranged vertically are difficult to identify, and the focal length of the camera device is increased, and the focal length is used as the focal length for obtaining real-time images; wherein, the adjacent water gauge marks arranged vertically include five types: the first type is 2 and 4 arranged vertically, the second type is 4 and 6 arranged vertically, the third type is 6 and 8 arranged vertically, the fourth type is 8 and M arranged vertically, and the fifth type is M and 2 arranged vertically;

[0219] (12) Based on (10), before identifying the water gauge mark based on the rectangular area of ​​the water gauge line, the rectangular area of ​​the water gauge line is binarized and denoised, and the five vertically arranged adjacent water gauge marks are compared with the rectangular area of ​​the water gauge line respectively to increase the probability of identifying any two vertically arranged adjacent water gauge marks.

[0220] To address the shortcomings of existing ship draft gauge observation equipment, this application example proposes a deep learning-based ship draft gauge line tracking method. This method includes: an original image acquisition step, in which the original ship draft gauge image is acquired through the device's camera; an image preprocessing step, in which the original image is scaled and grayscaled to reduce or even offset the influence of the ship's color on subsequent identification; a ship draft gauge line detection step, in which the preprocessed image is input into a convolutional neural network-based object detection model for waterline position detection; a draft gauge line position calculation step, in which the center point of the detected draft gauge line rectangle is compared with the center point of the image; if the difference in the x and y directions exceeds a certain threshold, the pan / tilt position is determined to need adjustment; and a pan / tilt adjustment step, in which the pan / tilt is adjusted based on the difference obtained in the previous step, and the previous and current steps are repeated until the difference is within the threshold range.

[0221] This application example realizes that the pan-tilt camera device in the ship's water gauge observation equipment automatically tracks the ship's water gauge scale line, reducing the labor of manual adjustment.

[0222] In some cases, the image in the original image acquisition step is not required to contain waterlines.

[0223] In some cases, the convolutional neural network recognition model is constructed and trained based on the Yolo network structure.

[0224] In some cases, for models built based on the Yolo network structure, the training images are changed from ordinary RGB to grayscale images with equal three channels for training. The grayscale value is obtained according to the expression Gray = (Red*0.299+Green*0.587+Blue*0.114). The grayscale value obtained based on this expression can clearly distinguish the color of the ship's draft gauge from the color of the hull, thereby improving the recognition efficiency of the ship's draft gauge.

[0225] In some cases, other combinations of images that affect waterline recognition results include images taken during both day and night.

[0226] On the other hand, an embodiment of the present invention also provides a device for tracking and observing the water gauge line of a ship, including: an original image acquisition module, which is used to obtain the original image of the ship's water gauge through a collection vehicle with a pan-tilt camera device; an image feature recognition module, which is used to input the original image into a water gauge line recognition model constructed based on a convolutional neural network module to obtain the position of the water gauge line rectangular frame in the image; a pan-tilt automatic adjustment module, which is used to perform pan-tilt self-adjustment based on the coordinate difference between the water gauge line rectangular frame position obtained by the detection model and the center position of the image.

[0227] Compared with the existing technology, the technical solution of this application example brings the following technical effects:

[0228] 1. The ship draft detection and tracking proposed in this technical solution is based on a deep learning model trained on a large number of samples. The sample environment is a port terminal, so it can adapt well to actual scenarios.

[0229] 2. This technical solution uses the ONVIF universal control protocol to control the PTZ camera device to adjust the pitch and rotation angle, replacing the control SDK of various PTZ camera devices. It is highly universal for most PTZ camera devices.

[0230] 3. A large number of samples (10,000 samples of ship draft scale lines) are used in the model construction of this technical solution to ensure the accuracy of the model. The neural network feature fusion ensures the detection rate of the model. After a large number of tests, the detection rate of ship draft scale lines reaches more than 98%.

[0231] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0232] It should be noted that, in the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element limited by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0233] Although the embodiments disclosed herein are as described above, the above contents are merely embodiments for facilitating understanding of the present invention and are not intended to limit the present invention. Any person skilled in the art may make any modifications and variations in the form and details of the embodiments without departing from the spirit and scope of the present invention. However, the scope of patent protection of the present invention shall still be subject to the scope defined by the appended claims.

Claims

1. A ship water gauge tracking method, characterized in that: include: S100, detecting the real-time image acquired by the camera device based on a pre-trained detection model; S200, if it is detected that the real-time image includes a water gauge line, outputting the smallest rectangular area including the water gauge line in the real-time image to a central area of ​​a display device; S300, if the real-time image does not contain a water gauge line, adjusting the direction of the camera device until the real-time image contains a water gauge line; Reducing the focal length of the camera device and detecting whether the real-time image after the focal length is reduced includes the water gauge line; If the real-time picture after the focal length is reduced includes the water gauge line, re-performing the step of reducing the focal length of the camera device and detecting whether the real-time picture after the focal length is reduced includes the water gauge line, until the real-time picture after the focal length is reduced does not include the water gauge line; If the real-time image after reducing the focal length does not include the water gauge line, the focal length is increased, and the real-time image is acquired and the water gauge line is detected with the maximum field of view corresponding to the increased focal length.

2. The ship water gauge tracking method according to claim 1, characterized in that: In S100, the detection model is trained through the following steps: Get a preset number of original images including water level lines; Annotating the watermark line in the original image to obtain an annotated image; Grayscale the labeled images and divide them into training and test sets; Train the Yolo model based on the training set; The trained Yolo model is tested for recognition accuracy based on the test set, and the Yolo model with recognition accuracy greater than the threshold is regarded as the trained detection model.

3. The ship water gauge tracking method according to claim 1, characterized in that: In S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes: Calculate the center point coordinates of the minimum rectangular area based on the minimum rectangular area of ​​the water gauge line in the real-time image; When the distance between the center point coordinates of the minimum rectangular area and the center point coordinates of the real-time picture is greater than a threshold, the angle of the camera device is adjusted.

4. The ship water gauge tracking method according to claim 3, characterized in that: The adjusting the angle of the camera device includes: When the center coordinates of the minimum rectangular area are above the center coordinates of the real-time image, the pitch angle of the camera device is adjusted upwards; When the center coordinates of the minimum rectangular area are below the center coordinates of the real-time image, the pitch angle of the camera device is adjusted downward; When the center coordinates of the minimum rectangular area are to the left of the center coordinates of the real-time image, the rotation angle of the camera device is adjusted to the left; When the center point coordinates of the minimum rectangular area are to the right of the center point coordinates of the real-time image, the rotation angle of the camera device is adjusted to the right.

5. The ship water gauge tracking method according to claim 1, characterized in that: In S200, outputting the minimum rectangular area including the water gauge line in the real-time image to the central area of ​​the display device includes: intercepting a minimum rectangular area including a water gauge line in the real-time image; The center point of the smallest rectangular area including the water gauge line in the intercepted real-time picture is aligned with the center of the display device, and the smallest rectangular area is enlarged.

6. The ship water gauge tracking method according to claim 1, characterized in that: The detecting whether the real-time image after the focal length is reduced includes a water gauge line includes: Detecting whether there are any two vertically arranged adjacent water gauge marks in the real-time image; If any two vertically arranged adjacent water gauge marks exist in the real-time image, it is determined that the real-time image includes a water gauge line; If there are no two vertically adjacent water gauge marks in the real-time image, it is determined that the real-time image does not include a water gauge line; Wherein, the water level mark includes numbers 2, 4, 6, 8 and the letter M.

7. A ship draft tracking device based on the ship draft tracking method according to any one of claims 1 to 6, characterized in that: include: A detection module, comprising a pre-trained detection model, for detecting real-time images acquired by the camera device based on the pre-trained detection model; an output module, configured to output a minimum rectangular area including the water gauge line in the real-time image to a central area of ​​a display device if it is detected that the real-time image includes the water gauge line; The adjustment module is used to adjust the direction of the camera device if the water gauge line is not detected in the real-time picture, until the water gauge line is detected in the real-time picture.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the ship draft tracking method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a processor and a memory, characterized in that: The memory stores a computer program, and the processor implements the ship water gauge tracking method according to any one of claims 1 to 6 when executing the computer program.

Images