Other ship tracking and monitoring system and method

Abstract

The present invention relates to tracking and monitoring of other ships around a ship. A system for tracking and monitoring other ships comprises: a sensor information collection unit; a selection information collection unit; a tracking camera unit for tracking a selected other ship and collecting an enlarged image thereof; and a control unit, wherein the control unit controls the tracking camera unit by considering the size and position of the selected other ship shown in the image information.

Description

Line tracking monitoring system and line tracking monitoring method

[0001] The present invention relates to a ship tracking monitoring system and a ship tracking monitoring method applied to a ship to monitor the surroundings of the ship.

[0002] According to the Convention on the International Regulations for Preventing Collisions at Sea (COLREGs), an international law for preventing collisions at sea, all ships must maintain a lookout at all times using all appropriate means to be aware of their surroundings and to fully assess the risk of collision. All appropriate means include all available means, such as crew visual, auditory, or radar.

[0003] Due to this duty of watchkeeping, crew members can become increasingly fatigued and lose concentration as voyages lengthen. Furthermore, failure to recognize potential hazards in the sea, even if they are difficult to identify with the naked eye, can lead to navigational accidents that can result in significant damage.

[0004] Prior art in this field includes a ship's omnidirectional monitoring system using a PTZ camera and a ship's omnidirectional monitoring method using the same, which are described in Korean Patent No. 10-1545276.

[0005] The purpose of the present invention is to provide an overall monitoring of the surroundings of a ship and an enlarged and tracking monitoring image of a specific target using a wide-angle camera and a narrow-angle enlarged camera.

[0006] In addition, the present invention aims to automatically control a tracking camera that provides an enlarged image of a line that requires observation.

[0007] The purpose of the present invention is not limited thereto, and other purposes not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] In order to achieve the above-mentioned purpose, the present invention provides the following line tracking monitoring system.

[0009] A tracking monitoring system for a ship according to one embodiment of the present invention comprises a sensor information collection unit for collecting wide-angle images or radar information about the sea around a ship, a selection information collection unit for receiving selection information about a selected ship based on the wide-angle images or radar information, a tracking camera unit for tracking the selected ship and collecting an enlarged image, and a control unit including a tracking camera control value generation module for generating a control value for controlling the tracking camera unit, wherein the tracking camera control value generation module generates a control value for controlling the tracking camera unit in consideration of the size and position of the selected ship appearing in the wide-angle images or radar information.

[0010] A method for monitoring tracking of a ship according to one embodiment of the present invention includes a sensor information collection step for collecting wide-angle images or radar information about the sea around a ship, a selection information collection step for receiving selection information about a selected ship based on the wide-angle images or radar information, a tracking camera control value generation step for generating a control value for controlling a tracking camera in consideration of the size and position of the selected ship appearing in the image information, and an enlarged image collection step for collecting an enlarged image of the selected ship based on the control value from the tracking camera.

[0011] According to one embodiment of the present invention, it is possible to perform tracking monitoring of a target to be tracked as well as overall monitoring of the surroundings of a ship using a wide-angle camera and an enlarged image collection camera.

[0012] According to one embodiment of the present invention, control of a tracking camera for an object to be tracked can be automatically performed.

[0013] According to one embodiment of the present invention, an enlarged image of a vessel selected by a crew member, as well as a newly detected or tracked vessel, can be automatically provided when information on that vessel changes. This allows crew members on lookout duty to easily identify objects at sea that are difficult to identify with the naked eye. This can improve crew fatigue and concentration, and prevent navigational accidents.

[0014] According to one embodiment of the present invention, by simultaneously displaying a wide-angle image, an enlarged image, information on own ship, and information on other ships selected as observation targets, the user can immediately and intuitively check the necessary information, and the problem of difficulty in finding other ships at once when using binoculars and difficulty in tracking and observing the movement of other ships and own ship can be solved.

[0015] FIG. 1 is a drawing for explaining a line tracking monitoring system according to one embodiment of the present invention.

[0016] Fig. 2 shows a wide-angle image including a line, Fig. 3 shows an enlarged image of a line selected from Fig. 2, and Fig. 4 shows an image with the center offset adjusted in the enlarged image of Fig. 3.

[0017] Figure 5 illustrates an image displayed on a display unit according to one embodiment of the present invention.

[0018] FIG. 6 is a drawing for explaining the operation of a control unit according to one embodiment of the present invention.

[0019] FIG. 7 is a flowchart for explaining a line tracking monitoring method according to one embodiment of the present invention.

[0020] A system according to one embodiment of the present invention for solving the above technical problem includes a sensor information collection unit that generates a wide-angle image of the sea around the ship; a tracking camera unit that tracks another ship included in the wide-angle image and collects an enlarged image; and a control unit that is configured to control the tracking camera unit by considering the position of the other ship included in the wide-angle image.

[0021] In the above system, the sensor information collection unit may further include a wide-angle image collection module that generates the wide-angle image by aligning images acquired using a plurality of cameras; a radar information collection module that collects radar information on the sea around the own ship; an other ship information collection module that collects other ship information including at least one of the size, position, speed, and direction of the other ship; and an own ship information collection module that collects own ship information including at least one of the attitude, position, speed, and direction of the own ship.

[0022] In the above system, the control unit may be set to control the tracking camera unit based on the own ship information and the other ship information, correct the coordinate state value of the tracking camera unit based on the attitude information of the own ship, and control the tracking camera unit based on the corrected coordinate state value of the tracking camera unit, the position information of the other ship, and the relative position information of the own ship and the other ship.

[0023] In the above system, a selection information collection unit may further be included that receives selection information on a selected line among a plurality of lines included in the wide-angle image.

[0024] In the above system, if the line is not located in a preset central region in the enlarged image, the control unit can calculate an offset between the center of the enlarged image and the center of the bounding box of the line, and control the tracking camera unit based on the offset so that the line is located in the preset central region of the enlarged image.

[0025] In the above system, the tracking camera unit performs a continuous scanning operation on the sea around the own ship, and when another ship is detected during the continuous scanning operation, the tracking camera control value at the time the other ship is detected, the own ship information, and the offset at which the center of the other ship is located in the enlarged image are taken into consideration to extract the position coordinates of the other ship, and the control unit may be set to update the list of previously detected ships based on the extracted position coordinates of the other ship.

[0026] The above system may further include a display unit that simultaneously provides the wide-angle image and an enlarged image of the selected line.

[0027] According to another embodiment of the present invention for solving the above technical problem, a method includes a sensor information collection step of generating a wide-angle image of the sea around a ship; a selection information collection step of receiving selection information on a selected other ship based on the wide-angle image; a tracking camera control value generation step of generating a control value for controlling a tracking camera in consideration of the position of the selected other ship appearing in the wide-angle image; and an enlarged image collection step of collecting an enlarged image of the selected other ship from the tracking camera based on the control value.

[0028] In the above method, the step of generating the tracking camera control value may include a step of correcting the coordinate state value of the tracking camera unit based on the attitude information of the own ship; and a step of generating the control value based on the corrected coordinate state value of the tracking camera unit, the position information of the other ship, and the relative position information of the own ship and the other ship.

[0029] In the above method, the step of generating the tracking camera control value may include the step of calculating an offset between the center of the enlarged image and the center of the bounding box of the line when the line is not located in a preset central region of the enlarged image; and the step of generating a control value based on the offset so that the line is located in a preset central region of the enlarged image.

[0030] Hereinafter, with reference to the attached drawings, preferred embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention. However, when describing preferred embodiments of the present invention in detail, if a detailed description of a related known function or configuration is judged to unnecessarily obscure the gist of the present invention, such detailed description will be omitted. In addition, the same reference numerals are used throughout the drawings for parts that perform similar functions and actions.

[0031] Additionally, throughout the specification, when a part is said to be "connected" to another part, this includes not only cases where it is "directly connected" but also cases where it is "indirectly connected" with other elements in between. Furthermore, "including" a component does not exclude other components unless specifically stated to the contrary, but rather means that it may include other components.

[0032] The present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0033] FIG. 1 is a drawing for explaining a line tracking monitoring system according to one embodiment of the present invention.

[0034] Referring to FIG. 1, a tracking monitoring system (10) according to one embodiment of the present invention may include a sensor information collection unit (100), a display unit (200), a selection information collection unit (300), a tracking camera unit (400), and a control unit (500).

[0035] The sensor information collection unit (100) can acquire at least one piece of sensor information. The sensor information collection unit (100) can process and / or align at least one piece of sensor information and transmit it to the display unit (200) and the control unit (500). In addition, the sensor information collection unit (100) can include at least one of a wide-angle image collection module (110), a radar information collection module (120), a ship information collection module (130), and a ship information collection module (140).

[0036] The wide-angle image collection module (110) can collect wide-angle images of the vessel's surroundings using at least one camera fixed to the vessel. At least one camera can be mounted on the vessel's bow, stern, or central portion to acquire images of the vessel's surroundings. For example, the camera can be installed on the vessel's compass deck or foremast in a forward direction. The camera's installation location can be changed as long as it does not obstruct the forward field of view.

[0037] According to an embodiment, the wide-angle image collection module (110) may include at least three cameras that capture images with an angle of greater than 80 degrees and less than 120 degrees. Accordingly, the wide-angle image collection module (110) may generate a panoramic image with an angle of greater than 180 degrees and less than 225 degrees. In another embodiment, the wide-angle image collection module (110) may include at least one camera that captures images with an angle of greater than 45 degrees and less than 270 degrees, but the present invention is not limited thereto.

[0038] The wide-angle image collection module (110) can generate a wide-angle image by aligning images acquired by multiple cameras. For example, the wide-angle image collection module (110) can generate a panoramic image or panoramic image by aligning images acquired by multiple cameras.

[0039] The wide-angle image collection module (110) can preprocess the aligned wide-angle image to correct the image that is distorted in a fixed coordinate axis. Accordingly, the wide-angle image collection module (110) can generate a wide-angle image based on a fixed coordinate axis (e.g., X-axis). For example, the wide-angle image collection module (110) can generate a corrected wide-angle image in which the horizontal line is parallel to the fixed coordinate axis (e.g., X-axis).

[0040] For example, the wide-angle image collection module (110) can receive information analyzing motions such as the attitude and speed of the vessel from the self-charity information collection module (140) and / or the control unit (500), and can perform preprocessing to correct the aligned wide-angle image without shaking based on the received information. The information analyzing motions such as the attitude and speed of the vessel can include information on the roll, yaw, and pitch of the vessel.

[0041] The wide-angle image collection module (110) can detect objects in the corrected wide-angle image and estimate their positions based on the magnetic field. For example, the wide-angle image collection module (110) can detect a line in the corrected wide-angle image and convert and store the world coordinate values ​​of the line based on the detected line information. According to an embodiment, the wide-angle image collection module (110) can generate a bounding box based on the detected line information. The bounding box can be generated for each detected object and stored in the wide-angle image collection module (110).

[0042] The wide-angle image collection module (110) can store the corrected wide-angle image. The wide-angle image collection module (110) can also store the detected line information and the converted world coordinate values ​​of the line together with the corrected wide-angle image. According to an embodiment, the wide-angle image collection module (110) can also store the bounding box of each line together with the corrected wide-angle image. The wide-angle image collection module (110) can transmit the generated wide-angle image to the display unit (200) and / or the control unit (500).

[0043] The radar information collection module (120) can collect radar signals from the sea surrounding the vessel using radar. The radar information collection module (120) can collect radar signals within a certain range centered around the vessel. The radar information collection module (120) can collect radar signals using radar, generate radar images, and transmit them to the display unit (200).

[0044] The other ship information collection module (130) can collect other ship information including at least one of the position, speed, and direction of other ships within a certain distance from the own ship. The other ship information may further include identification information of other ships within a certain distance from the own ship, information on the type, length, and width of the ship, static information on the position of the antenna, dynamic information that changes according to the sailing state of the ship such as the bow heading, sailing state, turning rate, and heel angle, sailing information on the route plan, destination, and expected time of arrival, etc. The other ship information collection module (130) may include an Automatic Identification System (AIS). AIS is a navigational device that can transmit information about the own ship to other ships within the radio wave reach using wireless communication and receive information from other ships and information from a land-based control system.

[0045] The ship information collection module (140) can collect ship information including at least one of the ship's attitude, location, speed, and direction using at least one sensor equipped on the ship. The ship information collection module (140) can include at least some of an IMU (Inertial Measurement Unit), an AIS (Automatic Identification System), GPS, Speedlog, and a gyro sensor. The IMU is a device that collects information related to the ship's speed, attitude change, etc., and can include at least one of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, and can measure the inclination angle of the ship by measuring inertia. The AIS can be a system that collects information on the ship's position, speed, and direction, etc. The ship information collection module (140) can be omitted depending on the embodiment.

[0046] At least one of the wide-angle image, radar signal, own ship information, and other ship information collected by the sensor information collection unit (100) can be transmitted to the control unit (500). At least one of the wide-angle image, radar signal, own ship information, and other ship information collected by the sensor information collection unit (100) can be used to track the other ship and generate a tracking camera control value. For example, the control unit (500) can calculate the relative speed and direction of the own ship and the other ship using at least one of the collected wide-angle image, radar signal, own ship information, and other ship information, and can generate a control value for tracking the corresponding other ship.

[0047] The sensor information collection unit (100) can transmit at least one piece of sensor information to the display unit (200). At least one of the wide-angle image, radar signal, own ship information, and other ship information collected by the sensor information collection unit (100) can be transmitted to the display unit (200). Accordingly, the user can check at least one piece of sensor information acquired by the sensor information collection unit (100).

[0048] The display unit (200) can receive at least one of a wide-angle image, a radar signal, own ship information, and other ship information from the sensor information collection unit (100). The display unit (200) can display at least one of the wide-angle image, the radar signal, own ship information, and other ship information received from the sensor information collection unit (100). The wide-angle image, the radar signal, and the enlarged image may be displayed on the same display device, or may be displayed on different display devices. If they are displayed on different devices, the display unit (200) may be composed of a wide-angle display module and a tracking display module.

[0049] The display unit (200) can display a wide-angle image or radar signal collected by the image information collection unit (100) or an enlarged image collected by the tracking camera unit (400). The display unit (200) will be described in more detail with reference to FIG. 5.

[0050] The selection information collection unit (300) can receive selection information. The selection information can be input by a user. The selection information can include information about at least one other line selected based on information provided by the sensor information collection unit (100). The selection information can include information about at least one other line selected as an observation target among multiple other lines.

[0051] For example, a user can check the wide-angle image information displayed through the display unit (200), select at least one target line to be observed from among multiple target lines included in the wide-angle image information, and input the selected target line information through the selection information collection unit (300). Accordingly, the selection information collection unit (300) can receive the selection information.

[0052] The selection information collection unit (300) may include various input means, such as a display capable of touch input, a keyboard, or a mouse. Depending on the embodiment, the selection information collection unit (300) may be omitted or may be included in the display unit (200). The selection information collection unit (300) may transmit the collected selection information to the control unit (500).

[0053] The control unit (500) can receive information collected by the sensor information collection unit (100) from the sensor information collection unit (100). The control unit (500) can generate a tracking camera control value based on at least one of a wide-angle image, a radar signal, own ship information, and other ship information received from the sensor information collection unit (100). The control unit (500) can generate a tracking camera control value so that the enlarged image collection module (410) continuously scans the wide-angle image. The control unit (500) can generate a tracking camera control value so that the enlarged image collection module (410) automatically collects an enlarged image when another ship is detected while continuously scanning the wide-angle image. According to an embodiment, the control unit (500) can receive selection information from the selection information collection unit (300) and generate a tracking camera control value based on further consideration of the selection information. The control unit (500) can track the selected other ship based on the selection information and generate a control value of the tracking camera unit. The control unit (500) may include a line tracking module (510) and a tracking camera control value generation module (520).

[0054] The ship tracking module (510) can track other ships using at least one of the wide-angle image, radar signal, own ship information, and other ship information received from the sensor information collection unit (100). The ship tracking module (510) can track other ships around the ship by fusing at least a portion of the wide-angle image, radar signal, own ship information, and other ship information received from the sensor information collection unit (100). For example, the ship tracking module (510) can track other ships by mapping ship-specific tags of radar information to the wide-angle image.

[0055] For example, the line tracking module (510) can use artificial intelligence to select lines from video information and track their movements. The line tracking module (510) can track lines using a trained artificial intelligence model. The artificial intelligence model can be trained through supervised learning using images containing lines as learning data, and can recognize and track lines using a camera image recognition algorithm.

[0056] According to an embodiment, the ship tracking module (510) may track at least one ship detected in a wide-angle image received from the sensor information collection unit (100). At this time, the ship tracking module (510) may further utilize at least one of the radar signal, own ship information, and ship information received from the sensor information collection unit (100). The ship tracking module (510) may transmit the tracked ship information to the tracking camera control value generation module (520).

[0057] According to an embodiment, the line tracking module (510) may receive selection information from the selection information collection unit (300). The selection information may be information on selecting at least one of a plurality of lines. The line tracking module (510) may select and track the selected line and transmit the tracked line information to the tracking camera control value generation module (520).

[0058] According to an embodiment, the line tracking module (510) can receive an enlarged image of the line, position information, and size information of the line from the tracking camera unit (400). The line tracking module (510) can improve line tracking performance by using the enlarged image, position information, and size information of the line received from the tracking camera unit (400).

[0059] The tracking camera control value generation module (520) can generate a tracking camera control value based on detected line information received from the sensor information collection unit (100), selection information received from the selection information collection unit (300), and / or tracked line information received from the line tracking module (510).

[0060] The tracking camera control value may be a value for controlling the tracking camera unit (400) to collect enlarged image information. The enlarged image information may include image information captured by enlarging a detected, selected, or tracked line. The tracking camera control value generation module (520) may generate a tracking camera control value for controlling the tracking camera unit (400) by considering at least one of the size, position, and coordinates of the center point of the line appearing in the image information.

[0061] The tracking camera control value may include instruction information for at least one of the horizontal rotation angle, the vertical rotation angle, and the magnification ratio of the magnified image collection module (410) included in the tracking camera unit (400). The tracking camera control value generation module (520) may use own ship information and / or other ship information received from the sensor information collection unit (100) to calculate the tracking camera control value. For example, the tracking camera control value generation module (520) may calculate at least one of the relative speed, position, and direction of the own ship and the other ship by considering own ship information including at least one of the speed, attitude, position, and heading direction of the own ship and other ship information including at least one of the speed, position, and heading direction of the other ship, and may determine the tracking camera control value according to the calculation result.

[0062] According to an embodiment, the tracking camera control value generation module (520) can automatically control the continuous scanning operation of the enlarged image collection module (410) even when there is no other line selected as an observation target, and when a other line is detected while performing the continuous scanning operation, can control the transmission of information and / or an enlarged image of the detected other line to the other line tracking module (510). Accordingly, the other line tracking module (510) and the tracking camera control value generation module (520) can obtain other line information missed in the wide-angle image, and can improve object tracking performance by using object information detected by the tracking camera unit (400).

[0063] According to an embodiment, the tracking camera control value generation module (520) may generate a control value that controls the enlarged image collection module (410) to capture multiple designated regions of interest at regular intervals to collect information on other lines that may be missed in wide-angle images. Accordingly, the enlarged image collection module (410) can intensively track and capture enlarged images of multiple designated regions of interest.

[0064] The tracking camera control value generation module (520) can receive the magnified image collected by the tracking camera unit (400) after tracking has started and provide feedback to determine whether the tracking camera unit (400) is properly tracking the selected line. If the image of the selected line is not properly visible in the magnified image or the selected line is not observed in the magnified image, feedback can be provided to perform fine adjustments to the horizontal rotation angle, vertical rotation angle, and magnification ratio of the tracking camera unit (400).

[0065] The tracking camera control value generation module (520) can generate a feedback control value to position the selected line at the center of the enlarged image acquired through the tracking camera unit (400) through feedback control when the selected line is not located at the center of the enlarged image. Feedback control will be described in more detail later with reference to FIGS. 3 and 4.

[0066] The collision risk detection module (530) can generate a warning message for a ship with a high risk of collision with the own ship among the ships tracked by the other ship tracking module (510). The collision risk detection module (530) can detect the risk of collision between the own ship and the other ship and generate a warning message. The collision risk detection module (530) can transmit the warning message to the display unit (200), and the display unit (200) can display the warning message in a pop-up form. Depending on the embodiment, the collision risk detection module (530) may be omitted.

[0067] The tracking camera unit (400) can collect an enlarged image of the other line based on the tracking camera control value received from the control unit (500). The tracking camera unit (400) can transmit the collected enlarged image to the control unit (500). The tracking camera unit (400) can include an enlarged image collection module (410) and a tracking camera control module (420).

[0068] The tracking camera control module (420) can control the operation of the enlarged image collection module (410). The tracking camera control module (420) can receive a tracking camera control value from the control unit (500) and control the mechanical operation of the enlarged image collection module (410) based on the tracking camera control value.

[0069] In some embodiments, the tracking camera control module (420) may independently track the target line. In this case, the tracking camera control module (420) may independently control the magnification image collection module (410) to track the target line and collect magnification images without instructions from the control unit (500).

[0070] For example, the tracking camera control module (420) can control the enlarged image collection module (410) to detect and track other ships by automatically performing continuous scanning or taking pictures of the location of interest at regular intervals even when there is no other ship selected as the observation target. The tracking camera control module (420) can calculate size and / or location information of a ship detected in the enlarged image based on the operation control value of the enlarged image collection module (410), and can transmit the calculated location information of the ship to the control unit (500) to transmit other ship information missed in the wide-angle image.

[0071] The tracking camera control module (420) can detect vessels by continuously scanning the maritime situation by zooming in or by photographing a designated location of interest at regular intervals. The vessels detected by the tracking camera control module (420) can include vessels not detected by the wide-angle image collection module (110), such as small wooden vessels, vessels without AIS installed, and low-height vessels. When a vessel is detected, the tracking camera control module (420) can transmit the tracking camera control value at the time the vessel is detected to the control unit (500).

[0072] At this time, the control unit (500) can extract the world coordinate values ​​of the other ship detected by the tracking camera unit (400) using the tracking camera control values, own ship information, and tracking camera correction values ​​received from the tracking camera control module (420). The control unit (500) can use the extracted world coordinate values ​​to check whether the other ship matches the previously detected ship information, and if there is no matching information, can update new ship information in the list of ships to be tracked. Accordingly, the user can check the new ship information through the display unit (200) or select it through the selection information collection unit (300).

[0073] The target tracking module (510) of the control unit (500) can improve object tracking performance by utilizing object information detected by the tracking camera unit. To collect information on targets that may be missed in wide-angle images, the tracking camera can intensively track and capture an enlarged image of the relevant area.

[0074] In another embodiment, the tracking camera control module (420) may be omitted, in which case the operations of the aforementioned tracking camera control module (420) may be performed by the control unit (500). The enlarged image collection module (410) may collect an enlarged image of a selected target based on the control of the tracking camera control module (420). The enlarged image collection module (410) may include a camera having a pan-tilt-zoom (PTZ) function. The enlarged image collection module (410) may include a camera having a left-right rotation (Pan), up-down rotation (Tilt), and zoom function, and tracking a selected target. The enlarged image collection module (410) may transmit the collected enlarged image of the selected target to the control unit (500).

[0075] Figures 2 to 5 are drawings for explaining the operation of the control unit. Figure 2 shows a wide-angle image including a line, Figure 3 shows an enlarged image of a line selected from Figure 2, and Figure 4 shows an image with the center offset adjusted in the enlarged image of Figure 3. The following description will be given with reference to Figure 1.

[0076] Referring to FIG. 2, the control unit (500) can receive at least some of the detected line information received from the sensor information collection unit (100), the selection information received from the selection information collection unit (300), and the tracked line information received from the line tracking module (510).

[0077] The control unit (500) can receive selection information (S). The selection information (S) may include information about the target line of observation. For example, the selection information (S) may include information input through the selection information collection unit (300). Accordingly, the selection information (S) may include information about the target line selected by the user.

[0078] According to an embodiment, the selection information (S) may include a bounding box formed around the target line. The bounding box may or may not be displayed to the user via the display unit (200). The bounding box will be described in more detail with reference to FIG. 6.

[0079] The operation of the control unit (500) of FIG. 2 can be performed by the line tracking module (510) and the tracking camera control value generation module (520). For example, the line tracking module (510) receives selection information (S), transmits the selection information (S) and the tracking information of the selected line to the tracking camera control value generation module (520), and the tracking camera control value generation module (520) can generate the tracking camera control value based on the received selection information (S) and the tracking information of the selected line.

[0080] Referring to FIG. 3, the control unit (500) can receive an enlarged image of the target line from the tracking camera unit (400). The target line tracking module (510) can receive an enlarged image of the target line determined as an observation target according to selection information (S) from the enlarged image collection module (410).

[0081] The tracking camera control value generation module (520) can receive the enlarged image collected by the tracking camera unit (400) and provide feedback to determine whether the tracking camera unit (400) is properly tracking the selected line. If the image of the selected line is not properly visible in the enlarged image or the selected line is not observed in the enlarged image, feedback can be provided to perform fine adjustments to the horizontal rotation angle, vertical rotation angle, and enlargement ratio of the tracking camera unit (400).

[0082] The tracking camera control value generation module (520) can generate a feedback control value to position the selected line at the center of the enlarged image obtained through the tracking camera unit (400) through feedback control when the selected line is not located at the center of the enlarged image.

[0083] For example, in an ideal case, the target line of observation can be located at the center of the enlarged image. However, due to reasons such as an error in the control value of the tracking camera control module (420) or the control unit (500), the center (P1) of the bounding box of the target line of observation may not be located at the center (P2) of the enlarged image.

[0084] The tracking camera control value generation module (520) can estimate the offset between the center (P1) of the bounding box of the target line and the center (P2) of the enlarged image, and correct the control value of the tracking camera based on the estimated offset. The tracking camera control value generation module (520) can transmit the corrected tracking camera control value to the tracking camera control module (420). Referring to FIG. 4, it can be confirmed that the center offset is adjusted in the enlarged image of FIG. 3 so that the center (P1) of the bounding box of the target line and the center (P2) of the enlarged image coincide.

[0085] In another embodiment, the tracking camera control module (420) of the tracking camera unit (400) may correct the control value of the tracking camera. In this case, the tracking camera control module (420) may estimate the offset between the center (P1) of the bounding box of the other line and the center (P2) of the magnified image, correct the control value of the tracking camera based on the estimated offset, and control the magnified image collection module (410) based on the corrected control value of the tracking camera.

[0086] Figure 5 illustrates an image displayed on a display unit according to one embodiment of the present invention.

[0087] Referring to FIG. 5, the display unit (200) can display at least one of a wide-angle image (W), an enlarged image (N), a radar signal, self-charitable information (I1), and other ship information (I2) received from the sensor information collection unit (100).

[0088] According to an embodiment, the display unit (200) can additionally display an enlarged image (N) collected by the tracking camera unit (400). The display unit (200) can display a wide-angle image (W) and an enlarged image (N) simultaneously. The display unit (200) can display the wide-angle image (W) and the enlarged image (N) in a PIP (Picture in Picture Extension) format, and when a line is selected in the wide-angle image (W), an enlarged image (N) for the selected line can be displayed in a pop-up format. However, the present invention is not limited thereto, and the wide-angle image (W) and the enlarged image (N) can be displayed in various ways on a single display device.

[0089] At this time, the display unit (200) can display the line shown in the enlarged image (N) as a marker (M) in the wide-angle image (W). Accordingly, the user can intuitively determine the location of the enlarged line in the wide-angle image (W).

[0090] The display unit (200) can display charity information (I1) together with a wide-angle image (W). The charity information (I1) can include information on at least one of the charity's position, speed, direction, and attitude. The charity information (I1) can be information collected from the charity information collection module (140).

[0091] The display unit (200) can display the enlarged image (N) together with the information on other ships (I2) included in the enlarged image. The information on other ships (I2) can include information on at least one of the position, speed, and direction of the other ships. The information on other ships (I2) can be information collected from the information collection module (130). The information on other ships (I2) can further include identification information of other ships within a certain distance from the own ship, information on the type, length, width, etc. of the ship, static information on the position of the antenna, etc., dynamic information that changes depending on the sailing state of the ship, such as the bow heading, sailing state, turning rate, and angle of inclination, and sailing information on the route plan, destination, and expected time of arrival.

[0092] By simultaneously displaying a wide-angle image (W), an enlarged image (N), own ship information (I1), and information on other ships (I2) selected as observation targets on the display unit (200), the user can immediately and intuitively check the necessary information, and can solve the problem of difficulty in finding other ships at once when using binoculars and difficulty in tracking and observing the movement of other ships and own ships.

[0093] Additionally, the display unit (200) can automatically provide an enlarged image of a vessel selected by a crew member, as well as a newly detected or tracked vessel, even when information on that vessel changes. This allows crew members on lookout duty to easily identify objects at sea that are difficult to identify with the naked eye. Consequently, crew members' fatigue and concentration levels can be improved, and navigational accidents can be prevented.

[0094] FIG. 6 is a drawing for explaining the operation of a control unit according to one embodiment of the present invention.

[0095] Referring to Fig. 6, the wide-angle image collected by the sensor information collection unit (100) and provided to the user by the display unit (200) may include at least one line. In Fig. 6, at least one line is depicted in various shapes such as a hexagon and an asterisk.

[0096] The line tracking module (510) of the control unit (500) can form bounding boxes (B1 to B5) for each of the plurality of detected lines, and can track each of the plurality of lines based on the bounding boxes (B1 to B5). According to an embodiment, the wide-angle image collection module (110) can form bounding boxes (B1 to B5) for each of the plurality of detected lines, and the line tracking module (510) can also track the lines based on the formed bounding boxes (B1 to B5).

[0097] The line tracking module (510) can transmit information of bounding boxes (B1 to B5) to the tracking camera control module (520). Depending on the embodiment, the line tracking module (510) may transmit only bounding box information of a line selected as an observation target among the bounding boxes (B1 to B5) to the tracking camera control module (520). Whether a line is selected as an observation target can be determined based on selection information (S of FIG. 2). The selection information (S of FIG. 2) can include selection information input through the selection information collection unit (300). The tracking camera control module (520) can generate a tracking camera control value using the center point position and size of the bounding boxes (B1 to B5) of each line.

[0098] For example, when a user selects one of a plurality of lines appearing in the image information through the selection information collection unit (300), the line tracking module (510) can transmit bounding box (B3) information corresponding to the selected line among a plurality of bounding boxes (B1 to B5) to the tracking camera control module (510). The tracking camera control value generation module (520) can generate a tracking camera control value by considering the center point position and size of the bounding box (B3) of the selected line. The tracking camera unit (400) can obtain an enlarged image based on the tracking camera control value.

[0099] Thereafter, the tracking camera control value generation module (520) can perform feedback control to align the center lines when the position of the center point of the bounding box and the position of the center point of the enlarged image of the tracking camera do not match.

[0100] Figure 7 is a flowchart illustrating a method for monitoring line tracking according to one embodiment of the present invention. The description of the aforementioned line tracking monitoring system can be applied to the method for monitoring line tracking, and any redundant description will be omitted.

[0101] Referring to Figure 7, in the image information collection step (S710), image information about the sea surrounding the vessel can be collected. The image information may include wide-angle images or radar signals. The image information may be collected using a wide-angle camera or radar.

[0102] In the ship tracking step (S720), the ship included in the image information can be tracked based on artificial intelligence or deep learning. The ship tracking module (510) can track the ship using a trained artificial intelligence model. The artificial intelligence model can be trained through supervised learning using a maritime image including the ship as learning data, and can recognize and track the ship using a camera image recognition algorithm. The ship tracking module (510) can track other ships or buoys, etc. around the ship, and obtain the relative position values ​​of the ship and the ship.

[0103] In the selection information collection step (S730), selection information about a selected line may be received based on image information. The user may select one of multiple lines appearing in the image information. The selection information may include information including the location and size of the selected line within the image information. The selection information collection step (S730) may include a step of collecting information obtained when new line information is detected or tracked line information changes based on line information detected in a wide-angle image received from the wide-angle image collection module (110) or line information tracked from the line tracking module (510).

[0104] In the tracking camera control value generation step (S740), the tracking camera control value generation module (620) can control the tracking camera by considering the size and position of the selected target line appearing in the image information.

[0105] In the tracking camera control value generation step (S740), the tracking camera can be controlled by considering not only selection information on the position and size within the image information of the selected target ship, but also the target ship information including at least one of the attitude, position, speed, and direction of the target ship, and / or the target ship information including at least one of the position, speed, and direction of the selected target ship.

[0106] In one embodiment, the tracking camera control value generation module (620) can generate the tracking camera control value based on the relative position of the line tracked by the line tracking module (510).

[0107] In one embodiment, the tracking camera control value generation module (620) may generate tracking camera control values ​​based on estimated values ​​of the vessel's speed, attitude, etc. acquired by the IMU sensor of the vessel information collection module and the relative position of the vessel tracked by the vessel tracking module (510). In the enlarged image collection step (S750), an enlarged image of the selected vessel may be collected from the tracking camera. The tracking camera may collect the enlarged image while tracking the selected vessel.

[0108] In the feedback control step (S760), the tracking camera can be fine-tuned to ensure that the selected target line appears at the center of the enlarged image obtained from the tracking camera. Since the enlarged image can vary significantly depending on the motion and movement of the target, the offset between the center of the enlarged image and the center of the target can be estimated so that the selected target line can be located at the center of the enlarged image, and the roll, pitch, and yaw control values ​​of the tracking camera can be precisely controlled.

[0109] In the display step (S770), the display unit (200) can simultaneously provide a wide-angle image and an enlarged image of a selected line by clicking or double-clicking on the wide-angle image.

[0110] In describing the present invention, the '~ part' can be implemented in various ways, for example, by a processor, program instructions executed by a processor, a software module, microcode, a computer program product, a logic circuit, an application-specific integrated circuit, firmware, etc.

[0111] The contents of the method disclosed in the embodiments of the present application can be directly implemented in a hardware processor, or can be implemented and performed by a combination of hardware and software modules within the processor. The software modules can be stored in conventional storage media such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage media is located in memory, and the processor reads the information stored in the memory and combines it with the hardware to complete the contents of the above-described method. To avoid duplication, a detailed description is omitted herein.

[0112] The present invention is not limited to the embodiments described above and the attached drawings. The scope of the invention is defined by the appended claims, and it will be apparent to those skilled in the art that various substitutions, modifications, and alterations can be made without departing from the technical spirit of the invention as defined in the claims.

Claims

1. Sensor information collection unit that generates wide-angle images of the sea around the charity ship; A tracking camera unit that tracks the target line included in the above wide-angle image and collects an enlarged image; A line tracking monitoring system, comprising a control unit configured to control the tracking camera unit by considering the position of the line included in the wide-angle image.

2. In paragraph 1, The above sensor information collection unit, A wide-angle image collection module that generates the wide-angle image by aligning images acquired using multiple cameras; A radar information collection module for collecting radar information about the sea surrounding the above charity; A line information collection module that collects line information including at least one of the size, position, speed, and direction of the line; and A ship tracking monitoring system further comprising a charity information collection module that collects charity information including at least one of the attitude, position, speed, and direction of the charity.

3. In paragraph 2, A ship tracking monitoring system, wherein the control unit is set to control the tracking camera unit based on the own ship information and the other ship information, correct the coordinate status value of the tracking camera unit based on the own ship's attitude information, and control the tracking camera unit based on the corrected coordinate status value of the tracking camera unit, the other ship's position information, and the relative position information of the own ship and the other ship.

4. In paragraph 1, A line tracking monitoring system further comprising a selection information collection unit that receives selection information on a selected line among a plurality of lines included in the wide-angle image.

5. In paragraph 1, A line tracking monitoring system, wherein the control unit calculates an offset between the center of the enlarged image and the center of the bounding box of the line when the line is not located in a preset central area in the enlarged image, and controls the tracking camera unit based on the offset so that the line is located in the preset central area of ​​the enlarged image.

6. In paragraph 1, The above tracking camera unit performs a continuous scanning operation on the sea around the own ship, and when another ship is detected during the continuous scanning operation, the tracking camera control value at the time the other ship is detected, the own ship information, and the offset at which the center of the other ship is located in the enlarged image are taken into consideration to extract the position coordinates of the other ship. A ship tracking monitoring system, wherein the control unit is set to update a list of detected ships based on the extracted ship location coordinates.

7. In paragraph 1, The above line tracking monitoring system is, A line tracking monitoring system further comprising a display unit that simultaneously provides the wide-angle image and an enlarged image of the selected line.

8. Sensor information collection step to create a wide-angle image of the sea around the ship; A selection information collection step for receiving selection information on a selected line based on the above wide-angle image; A tracking camera control value generation step for generating a control value for controlling a tracking camera by considering the position of the selected line appearing in the wide-angle image; and A method for monitoring line tracking, comprising a zoom-in image collection step of collecting a zoom-in image of the selected line from the tracking camera based on the control value.

9. In paragraph 8, The above tracking camera control value generation step is: A step of correcting the coordinate status value of the tracking camera unit based on the detailed information of the charity; and A method for monitoring line tracking, comprising a step of generating a control value based on the coordinate status value of the above-mentioned corrected tracking camera unit, the position information of the other line, and the relative position information of the own line and the other line.

10. In paragraph 8, The above tracking camera control value generation step is: If the line is not located in the preset central area in the enlarged image, a step of calculating an offset between the center of the enlarged image and the center of the bounding box of the line; and A method for monitoring line tracking, comprising the step of generating a control value so that the line is located in a preset central area of ​​the enlarged image based on the offset.

Images