Target monitoring device, ship maneuvering system, target monitoring method, and program

Abstract

To provide a target monitoring device which allows image recognition suitable for a scene.SOLUTION: A target monitoring device comprises: an image acquisition unit which acquires an image including a target on the sea picked up by a camera installed in a ship; a scene determination unit which determines whether the image is picked up in the daytime or in the nighttime; an image recognition unit for the daytime which detects the target included in the image by using a learned model for the daytime when it is determined that the image is picked up in the daytime; and an image recognition unit for the nighttime which detects the target included in the image by using the learned model for the nighttime when it is determined that the image is picked up in the nighttime.SELECTED DRAWING: Figure 4

Description

Technical Field

[0004] ,

[0006] , , ، , , ,

[0005] , , ,

[0003] , , , , , ,

[0001] The present invention relates to an object monitoring device, a ship control system, an object monitoring method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique in which a camera device for sequentially acquiring image data of the surrounding waters of a construction ship is installed, the image data of the surrounding waters is sequentially input into an arithmetic device, and the arithmetic device stores in advance a prediction model that has been machine-learned using, as teacher data, image data of a plurality of types of ships including other construction ships that are construction ships other than the construction ship in question and image data other than ships, and uses the arithmetic device to sequentially determine whether or not a ship of a preset monitoring target type exists in the image data of the surrounding waters based on the prediction model and the image data of the surrounding waters.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when constructing an AI image recognition system for monitoring marine objects over 24 hours, it is costly and difficult to improve accuracy to create one learned model that can recognize various images captured in various scenes such as daytime or nighttime.

[0005] The present invention has been made in view of the above problems, and its main object is to provide an object monitoring device, a ship control system, an object monitoring method, and a program capable of image recognition suitable for the scene.

Means for Solving the Problems

[0006] To solve the above problems, a target monitoring device according to one aspect of the present invention includes: an image acquisition unit that acquires an image including a target at sea captured by a camera installed on a ship; a scene determination unit that determines whether the image was captured during the day or at night; a daytime image recognition unit that, when it is determined that the image was captured during the day, detects the target included in the image using a pre-trained model for daytime use; and a nighttime image recognition unit that, when it is determined that the image was captured at night, detects the target included in the image using a pre-trained model for nighttime use. This enables image recognition appropriate to the scene.

[0007] In the above embodiment, the scene determination unit may use a pre-trained model for scene determination to determine whether the image was taken during the day or at night. This makes it possible to improve the accuracy of scene determination by using a pre-trained model for scene determination.

[0008] In the above embodiment, the nighttime image recognition unit may detect target candidates that have been detected by the nighttime trained model and have a brightness above a predetermined level as the target. While the nighttime trained model alone may not be able to detect an independent single light as a target, by combining it with the rule that the light has a brightness above a predetermined level, it becomes possible to improve the accuracy of target detection even in images taken at night.

[0009] In the above embodiment, the daytime image recognition unit may use the daytime trained model to detect the object contained in the image and to determine the type of object. This makes it possible not only to detect the object but also to determine the type of object.

[0010] In the above embodiment, the night-time image recognition unit may use the night-time trained model to detect the targets included in the image and determine whether or not they are targets. Since it is difficult to determine the type of target in images taken at night, determining whether or not something is a target, rather than determining the type of target, makes it possible to use a trained model suitable for images taken at night.

[0011] In the above embodiment, the scene determination unit may further include a sunrise / sunset image recognition unit that determines whether the image was taken during sunrise or sunset, and if it is determined that the image was taken during sunrise or sunset, detects the object contained in the image using a pre-trained model for sunrise / sunset. During sunrise or sunset, the reflection from the sea surface is strong, so the accuracy of object detection may not be sufficient with pre-trained models for daytime or nighttime. However, by using a pre-trained model for sunrise / sunset, it is possible to improve the accuracy of object detection even for images taken during sunrise or sunset.

[0012] In the above embodiment, the sunrise / sunset image recognition unit may use the sunrise / sunset trained model to detect the object included in the image and determine the type of object. This makes it possible not only to detect objects but also to determine the type of object.

[0013] In the above embodiment, the scene determination unit may further include a preprocessing unit that determines whether the image was captured in backlight, and if it is determined that the image was captured in backlight, performs gamma correction or contrast adjustment on the image before it is input to the trained model. This makes it possible to improve the accuracy of target detection even with images captured in backlight.

[0014] In the above embodiment, the scene determination unit may further include a pre-processing unit that determines whether the image was captured in fog, and if it is determined that the image was captured in fog, performs image sharpening processing on the image before it is input to the trained model. This makes it possible to improve the accuracy of target detection even for images captured in fog.

[0015] In the above embodiment, the daytime trained model may be a trained model generated by machine learning using training images, which include images taken during the daytime, as input data, and the image position and type of targets included in the training images as training data. This makes it possible to use a trained model suitable for images taken during the daytime.

[0016] In the above embodiment, the nighttime trained model may be a trained model generated by machine learning using training images, which include images taken at night, as input data, and the in-image positions of targets included in the training images as training data. This makes it possible to use a trained model suitable for images taken at night.

[0017] In the above embodiment, the pre-trained model for sunrise and sunset may be a pre-trained model generated by machine learning, using training images including images taken during sunrise or sunset as input data, and the image position and type of targets included in the training images as training data. This makes it possible to use a pre-trained model suitable for images taken during sunrise or sunset.

[0018] Furthermore, another aspect of the present invention provides a ship handling system comprising: a target monitoring device; a ship handling decision unit that makes a ship handling decision based on a target detected by the target monitoring device; and a ship handling control unit that performs ship handling control based on the ship handling decision. This enables ship handling decisions and ship handling control based on targets detected by image recognition appropriate to the scene.

[0019] In addition, the object monitoring method according to another aspect of the present invention acquires an image including an object on the sea captured by a camera installed on a ship, determines whether the image was captured during the day or at night, and when it is determined that the image was captured during the day, uses a pre-trained model for daytime to detect the object included in the image, and when it is determined that the image was captured at night, uses a pre-trained model for nighttime to detect the object included in the image. According to this, image recognition suitable for the scene becomes possible.

[0020] In addition, the program according to another aspect of the present invention causes a computer to execute acquiring an image including an object on the sea captured by a camera installed on a ship, determining whether the image was captured during the day or at night, when it is determined that the image was captured during the day, using a pre-trained model for daytime to detect the object included in the image, and when it is determined that the image was captured at night, using a pre-trained model for nighttime to detect the object included in the image. According to this, image recognition suitable for the scene becomes possible.

Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing a configuration example of an object monitoring system. [Figure 2] It is a diagram showing a configuration example of an object monitoring device. [Figure 3] It is a diagram showing a content example of an object management DB. [Figure 4] It is a diagram showing a configuration example of an image processing unit. [Figure 5] It is a diagram showing a recognition example by a pre-trained model for scene determination. [Figure 6] It is a diagram showing a recognition example by a pre-trained model for daytime. [Figure 7] It is a diagram showing a recognition example by a pre-trained model for nighttime. [Figure 8] It is a diagram showing a recognition example by a pre-trained model for sunrise / sunset. [Figure 9] It is a diagram showing a procedure example of an object monitoring method. [Figure 10] This figure shows an example of the pre-processing procedure. [Figure 11] This figure shows an example of the procedure for image recognition processing. [Modes for carrying out the invention]

[0022] Embodiments of the present invention will be described below with reference to the drawings.

[0023] Figure 1 is a block diagram showing an example configuration of the target monitoring system 100. The target monitoring system 100 is a system installed on a ship. In the following description, the ship on which the target monitoring system 100 is installed will be referred to as "our ship," and other ships will be referred to as "other ships."

[0024] The target monitoring system 100 comprises a target monitoring device 1, a display unit 2, a radar 3, an AIS 4, a camera 5, a GNSS receiver 6, a gyrocompass 7, an ECDIS 8, a wireless communication unit 9, and a ship handling control unit. These devices are connected to a network N, such as a LAN, and are capable of network communication with each other.

[0025] Target monitoring device 1 is a computer including a CPU, RAM, ROM, non-volatile memory, and input / output interfaces. The CPU of target monitoring device 1 performs information processing according to a program loaded from ROM or non-volatile memory into RAM.

[0026] The program may be supplied via an information storage medium such as an optical disc or memory card, or via a communication network such as the Internet or LAN.

[0027] Display unit 2 displays display images generated by target monitoring device 1. Display unit 2 also displays radar images, camera images, or electronic charts.

[0028] The display unit 2 is, for example, a display device with a touch sensor, a so-called touch panel. The touch sensor detects the position indicated on the screen by the user's finger or the like. However, the position may also be input by a trackball or the like.

[0029] Radar 3 emits radio waves around the vessel and receives the reflected waves, generating echo data based on the received signals. Radar 3 also identifies targets from the echo data and generates TT data (Target Tracking Data) representing the target's position and speed.

[0030] The Automatic Identification System (AIS) 4 receives AIS data from other vessels or shore-based control systems in the vicinity of the vessel. While AIS is not the only option, a VHF Data Exchange System (VDES) may also be used. AIS data includes the identification code, name, position, course, speed, vessel type, length, and destination of other vessels.

[0031] Camera 5 is a digital camera that captures images of the outside from the ship and generates image data. Camera 5 is installed, for example, on the ship's bridge, facing the bow. Camera 5 may also be a camera with pan-tilt and optical zoom functions, a so-called PTZ camera.

[0032] The GNSS receiver 6 detects the ship's position based on radio waves received from the GNSS (Global Navigation Satellite System). The gyrocompass 7 detects the ship's heading. A GPS compass may be used instead of a gyrocompass.

[0033] The ECDIS (Electronic Chart Display and Information System) 8 obtains the ship's position from the GNSS receiver 6 and displays the ship's position on the electronic chart. The ECDIS 8 also displays the ship's planned route on the electronic chart. A GNSS plotter may be used instead of the ECDIS.

[0034] The radio communication unit 9 includes various radio equipment for enabling communication with other ships or land-based air traffic control, such as radio equipment in the ultra-high frequency, very high frequency, medium frequency, or short frequency bands.

[0035] The ship steering control unit 10 is a control device for realizing automatic ship steering and controls the ship's steering gear. The ship steering control unit 10 may also control the ship's engine.

[0036] In this embodiment, the target monitoring device 1 is an independent device, but it is not limited to this and may be integrated with other devices such as ECDIS8. In other words, the functions of the target monitoring device 1 may be implemented by other devices.

[0037] In this embodiment, the target monitoring device 1 is mounted on the vessel itself and used to monitor targets such as other vessels in the vicinity of the vessel; however, its use is not limited to this. For example, the target monitoring device 1 may be installed at a land-based control center and used to monitor vessels in a controlled area.

[0038] Figure 2 is a block diagram showing an example configuration of the target monitoring device 1. The control unit 20 of the target monitoring device 1 includes an image acquisition unit 11, an image processing unit 12, a display control unit 13, and a ship maneuvering decision unit 14. These functions are realized by the control unit 20 executing information processing according to a program. The ship maneuvering decision unit 14 may be located outside the target monitoring device 1.

[0039] The control unit 20 of the target monitoring device 1 further includes a target management DB (database) 19. The target management DB 19 is stored in the memory of the target monitoring device 1.

[0040] The image acquisition unit 11 acquires images including other vessels and other targets at sea captured by a camera 5 installed on its own vessel. The image acquisition unit 11 sequentially acquires time-series images from the camera 5 and provides them sequentially to the image processing unit 12. The time-series images are, for example, still images (frames) included in video data.

[0041] The image processing unit 12 performs predetermined image processing, such as image recognition, on the image acquired by the image acquisition unit 11, generates target data for the recognized targets from the image, and registers it in the target management DB 19. Details of the image processing unit 12 will be described later.

[0042] The target management DB 19 is a database that manages target data generated by the image processing unit 12. The target management DB 19 may integrate not only target data generated by the image processing unit 12, but also other target data such as TT data generated by the radar 3 or AIS data received by AIS 4.

[0043] The display control unit 13 generates a display image containing objects representing targets based on target data registered in the target management DB 19 and outputs it to the display unit 2. The display image is, for example, a radar image, an electronic chart, or an image combining the two, and the objects representing targets are placed in positions within the image that correspond to the actual locations of the targets.

[0044] The ship handling decision unit 14 makes a ship handling decision based on the target data registered in the target management DB 19, and if it determines that it is necessary to avoid a target, it instructs the ship handling control unit 10 to perform avoidance maneuvers. Specifically, the ship handling control unit 10 calculates an avoidance route to avoid the target using an avoidance maneuvering algorithm, and controls the steering gear or engines, etc., so that the ship follows the avoidance route.

[0045] Figure 3 shows an example of the contents of the target management DB19. The target management DB19 includes fields such as "Target ID," "Type," "Location in Image," "Actual Location," "Velocity," and "Course." The target management DB19 may also include other information, such as the size of the target and the time elapsed since detection.

[0046] "Type" represents the type of target identified from the image captured by camera 5. Target types include, for example, vessel types such as tankers, pleasure boats, and fishing boats. Target types may also include, for example, offshore structures such as buoys.

[0047] "Image Position" represents the location of the target within the image. "Actual Position" represents the position of the target in real space, calculated based on the target's image position. The actual position is calculated by first calculating the target's relative position to the ship from the image position, and then converting it to the target's absolute position using the ship's position. Note that "Position" may be calculated by integrating the relative position of the target detected by radar 3 or the actual position of the target received by AIS 4, or alternatively. "Speed" and "Course" represent the target's speed and course, calculated based on the temporal change of the target's actual position.

[0048] The target management DB19 may also register target data for targets recognized from images captured by cameras 5, as well as target data for targets recognized from images captured by separately installed PZT cameras, fixed cameras, 360-degree cameras, or infrared cameras.

[0049] Figure 4 shows an example of the configuration of the image processing unit 12. The image processing unit 12 includes a scene determination unit 21, a pre-processing unit 22, a daytime image recognition unit 23, a nighttime image recognition unit 24, and a sunrise / sunset image recognition unit 25. These functional units are realized by the control unit 20 executing information processing according to a program.

[0050] The image processing unit 12 further includes a determination model holding unit 31, a daytime model holding unit 33, a nighttime model holding unit 34, and a sunrise / sunset model holding unit 35. These storage units are provided in the memory of the target monitoring device 1.

[0051] The scene determination unit 21 uses a trained scene determination model held in the determination model holding unit 31 to determine whether the image acquired by the image acquisition unit 11 was taken during the daytime, at night, or during sunrise or sunset. The scene determination unit 21 further determines whether the image was taken in backlight or in fog.

[0052] As shown in Figure 5, when an image P captured by the camera 5 mounted on the ship is input to the trained model DM for scene determination, the trained model DM outputs a determination result representing the scene in which the image P was captured.

[0053] A pre-trained model (DM) for scene classification is, for example, an image classification model such as a convolutional neural network (CNN). The DM is a pre-trained model generated by machine learning, using training images as input data and the classes associated with those training images as training data.

[0054] The training images include images of the sea taken during the day, images of the sea taken at night, images of the sea taken during sunrise or sunset (hereinafter also referred to as "sunrise / sunset times"), images of the sea taken against the light, and images of the sea taken in fog. The training images may also include images of the sea generated by a Generative Adversarial Network (GAN) or 3-Dimensional Computer Graphics (3DCG).

[0055] The classes associated with training images include "daytime," "nighttime," "sunrise / sunset," "backlit," and "foggy."

[0056] The output layer of the pre-trained model DM for scene determination contains elements corresponding to classes. The elements corresponding to "daytime," "nighttime," and "sunrise / sunset" are set, for example, using a softmax function so that the sum of their probabilities equals 1. The scene determination unit 21 applies the class with the highest probability among "daytime," "nighttime," and "sunrise / sunset."

[0057] In other words, the scene determination unit 21 determines that image P was taken during the daytime when the probability of "daytime" is highest, determines that image P was taken at night when the probability of "nighttime" is highest, and determines that image P was taken at sunrise or sunset when the probability of "sunrise or sunset" is highest.

[0058] Furthermore, elements corresponding to "backlight" and "fog" are set to output an accuracy of 0 to 1, for example, using a sigmoid function. The scene determination unit 21 determines that image P was captured with backlight if the accuracy of "backlight" is above a threshold, and determines that image P was captured in fog if the accuracy of "fog" is above a threshold.

[0059] The scene determination unit 21 may also determine whether image P was captured during the daytime, at night, or at sunrise or sunset, based on the time of image P's capture and the current position of the ship, according to the sunrise and sunset times calculated from these times.

[0060] The sunrise period is a predetermined duration that includes the time of sunrise, and the sunset period is a predetermined duration that includes the time of sunset. Daytime is the period from sunrise to sunset, excluding the sunrise and sunset periods. Nighttime is the period from sunset to sunrise, excluding the sunrise and sunset periods.

[0061] Furthermore, the scene determination unit 21 may determine whether the image P was captured during the daytime, at night, or at sunrise or sunset, based on the ambient brightness detected by the illuminance sensor installed on the vessel.

[0062] If the scene determination unit 21 determines that image P was captured in backlight, the preprocessing unit 22 performs gamma correction or contrast adjustment on image P to process it into an image suitable for input to a later-stage trained model for daytime, nighttime, or sunrise / sunset.

[0063] Furthermore, if the scene determination unit 21 determines that image P was captured in fog, the preprocessing unit 22 performs image enhancement processing such as defog processing on image P to process it into an image suitable for input to the subsequent trained models for daytime, nighttime, or sunrise / sunset.

[0064] When the scene determination unit 21 determines that the image P was captured during the daytime, the daytime image recognition unit 23 uses the daytime trained model stored in the daytime model holding unit 33 to detect targets contained in the image P.

[0065] Furthermore, the daytime image recognition unit 23 determines the type of target detected from the image P. The type of target may be, for example, a type of vessel such as a tanker, pleasure boat, or fishing boat. The type of target may also be, for example, a buoy or other object installed at sea.

[0066] The pre-trained model for daytime use is, for example, an object detection model such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once), which outputs a bounding box surrounding the object in the image. However, the pre-trained model for daytime use may also be a segmentation model such as Semantic Segmentation or Instance Segmentation.

[0067] A pre-trained model for daytime use is a model generated by machine learning, using training images, including images of the sea taken during the daytime, as input data, and the image location and type of targets included in the training images as training data. The training images may include daytime images of the sea generated by a Generative Adversarial Network (GAN) or 3DCG.

[0068] The in-image location of an object is identified by the coordinates of a rectangular area containing the object within image P. The in-image location of the object is associated with a class representing the type of object (e.g., "tanker," "pleasure board," "fishing vessel," "buoy") and an estimated accuracy.

[0069] Figure 6 shows an example of recognition of a daytime image DP using a pre-trained model for daytime use. As shown in the figure, targets SH, such as other ships, included in the daytime image DP are enclosed by a rectangular boundary box BB. A label CF indicating the type of target and the accuracy of the estimation is attached to the boundary box BB.

[0070] When the scene determination unit 21 determines that image P was captured at night, the night-time image recognition unit 24 uses a trained night-time model stored in the night-time model storage unit 34 to detect targets contained in image P.

[0071] Furthermore, the night-time image recognition unit 24 determines whether an image P is a target or not, rather than determining the type of target detected from the image P. That is, the night-time image recognition unit 24 determines that an image is a target if the accuracy of the estimation output from the night-time trained model is above a threshold.

[0072] The pre-trained model for nighttime use may be an object detection model such as SSD or YOLO, similar to the pre-trained model for daytime use described above, or it may be a region segmentation model such as Semantic Segmentation or Instance Segmentation.

[0073] The nighttime pre-trained model is a model generated by machine learning using training images, including images of the sea taken at night, as input data, and the in-image locations of targets included in the training images as training data. The nighttime pre-trained model also learns the arrangement patterns of lights as parameters. The training images may include nighttime images of the sea generated by a Generative Adversarial Network (GAN) or 3DCG. A class representing the target is associated with the in-image location of the target.

[0074] Figure 7 shows an example of recognizing an image NP taken at night using a pre-trained model for nighttime use. As shown in the figure, in an image NP taken at night, almost only the light L emitted by targets such as other ships is visible. When a pre-trained model for nighttime use is applied to such an image NP, the light L of the targets is enclosed by a rectangular bounding box BB, and a label CF is added to the bounding box BB indicating that it is a target and the accuracy of the estimation.

[0075] Furthermore, the nighttime image recognition unit 24 detects target candidates as targets that have been detected by a nighttime trained model and have a brightness above a predetermined level. In other words, simply applying a nighttime trained model may not be able to detect an independent single light as a target, but by combining this with the rule that the light must have a brightness above a predetermined level, it becomes possible to improve the accuracy of target detection even for images NP captured at night.

[0076] When the scene determination unit 21 determines that the image was captured at sunrise or sunset, the sunrise / sunset image recognition unit 25 uses the sunrise / sunset model stored in the sunrise / sunset model holding unit 35 to detect objects contained in the image P. In addition, the sunrise / sunset image recognition unit 25, similar to the daytime image recognition unit 23, determines the type of object detected from the image P.

[0077] The pre-trained model for sunrise and sunset may be an object detection model such as SSD or YOLO, similar to the pre-trained models for daytime and nighttime described above, or it may be a region segmentation model such as Semantic Segmentation or Instance Segmentation.

[0078] The pre-trained model for sunrise and sunset is a model generated by machine learning, using training images, which include images of the sea taken at sunrise and sunset, as input data, and the image location and type of targets included in the training images as training data. The training images may also include images of the sea at sunrise and sunset generated by a Generative Adversarial Network (GAN) or 3DCG.

[0079] Figure 8 shows an example of image recognition SP taken during sunrise and sunset using a pre-trained model for sunrise and sunset. As shown in the figure, targets SH, such as other ships, included in the image SP taken during sunrise and sunset are enclosed by a rectangular boundary box BB. A label CF indicating the type of target and the accuracy of the estimation is attached to the boundary box BB.

[0080] Because image SPs taken at sunrise and sunset contain reflections from the sea surface, the accuracy of target detection may be insufficient with pre-trained models for daytime or nighttime use. However, by preparing a separate pre-trained model for sunrise and sunset, it is possible to improve the accuracy of target detection even for image SPs taken at sunrise and sunset.

[0081] Figure 9 shows an example of the procedure for the target monitoring method implemented in the target monitoring system 100. Figure 10 shows an example of the procedure for the preprocessing routine. Figure 11 shows an example of the procedure for the image recognition processing routine. The control unit 20 of the target monitoring device 1 executes the information processing shown in the figure according to the program.

[0082] First, the control unit 20 acquires the image P generated by the camera 5 (S11, processing as the image acquisition unit 11).

[0083] Next, the control unit 20 uses a trained model for scene determination to determine whether the acquired image P was taken during the daytime, at night, or during sunrise or sunset, and furthermore, whether the image was taken against the light or in fog (S12, processing by the scene determination unit 21).

[0084] Next, the control unit 20 executes a preprocessing routine (S13, processing as the preprocessing unit 22).

[0085] As shown in Figure 10, in the preprocessing routine, if the control unit 20 determines that image P was captured in backlight (S21: YES), it performs gamma correction or contrast adjustment on image P (S22).

[0086] Furthermore, if the control unit 20 determines that image P was captured in fog (S23:YES), it performs image enhancement processing such as defog processing on image P (S24).

[0087] With the above steps complete, the preprocessing routine is finished, and the process returns to the main routine shown in Figure 9.

[0088] Next, the control unit 20 executes the image recognition processing routine (S14).

[0089] As shown in Figure 11, in the image recognition processing routine, if the control unit 20 determines that image P was captured during the daytime (S31: YES), it uses a pre-trained model for daytime use to detect targets in image P and determine the type of target (S32, processing as daytime image recognition unit 23).

[0090] Furthermore, if the control unit 20 determines that image P was captured at night (S33: YES), it uses a pre-trained model for nighttime use to detect target candidates included in image P, and extracts target candidates with a brightness above a predetermined level as targets (S34, S35, processing as the nighttime image recognition unit 24).

[0091] Furthermore, if the control unit 20 determines that image P was captured at sunrise or sunset (S36: YES), it uses a pre-trained model for sunrise and sunset to detect objects contained in image P and determine the type of object (S37, processing as the sunrise / sunset image recognition unit 25).

[0092] With the above steps complete, the image recognition processing routine is finished, and the main routine shown in Figure 9 is also completed. Subsequently, the control unit 20 generates target data for the targets detected from image P and registers it in the target management DB 19.

[0093] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are of course possible for those skilled in the art. [Explanation of Symbols]

[0094] 1 Target monitoring device, 2 Display unit, 3 Radar, 4 AIS, 5 Camera, 6 GNSS receiver, 7 Gyrocompass, 8 ECDIS, 9 Wireless communication unit, 10 Ship handling control unit, 11 Image acquisition unit, 12 Image processing unit, 13 Display control unit, 14 Ship handling decision unit, 19 Target management DB, 20 Control unit, 21 Scene determination unit, 22 Pre-processing unit, 23 Daytime image recognition unit, 24 Nighttime image recognition unit, 25 Daytime entry / exit image recognition unit, 31 Determination model holding unit, 33 Daytime model holding unit, 34 Nighttime model holding unit, 35 Daytime entry / exit model holding unit, 100 Target monitoring system

Claims

1. An image acquisition unit that acquires images including objects at sea captured by a camera installed on a ship, A scene determination unit that determines whether the aforementioned image was taken during the daytime, at night, or during sunrise or sunset, If it is determined that the image was taken during the daytime, a daytime image recognition unit detects the object contained in the image using a pre-trained model for daytime use. If it is determined that the image was taken at night, a night-time image recognition unit detects the object contained in the image using a pre-trained model for nighttime use. If it is determined that the aforementioned image was taken during sunrise or sunset, a sunrise / sunset image recognition unit detects the object contained in the image using a pre-trained model for sunrise / sunset, A target monitoring device equipped with the following features.

2. The scene determination unit uses a pre-trained model for scene determination to determine whether the image was taken during the day or at night. The target monitoring device according to claim 1.

3. The nighttime image recognition unit detects target candidates that have been detected by the nighttime trained model and have a brightness level above a predetermined level as the target. The target monitoring device according to claim 1 or 2.

4. The daytime image recognition unit uses the daytime trained model to detect the object contained in the image and to determine the type of object. A target monitoring device according to any one of claims 1 to 3.

5. The nighttime image recognition unit uses the nighttime trained model to detect the target contained in the image and to determine whether or not it is a target. A target monitoring device according to any one of claims 1 to 4.

6. The sunrise / sunset image recognition unit uses the sunrise / sunset trained model to detect the object contained in the image and to determine the type of object. A target monitoring device according to any one of claims 1 to 5.

7. The scene determination unit further determines whether the image was captured in backlight, The system further includes a preprocessing unit that performs gamma correction or contrast adjustment on the image before it is input to the trained model, if it is determined that the image was captured in backlight. A target monitoring device according to any one of claims 1 to 6.

8. The scene determination unit further determines whether the image was taken in fog. The system further includes a preprocessing unit that performs image sharpening on the image before it is input to the trained model if it is determined that the image was taken in fog. A target monitoring device according to any one of claims 1 to 7.

9. The aforementioned daytime trained model is a trained model generated by machine learning, using training images, which include images taken during the daytime, as input data, and the image position and type of targets included in the training images as training data. A target monitoring device according to any one of claims 1 to 8.

10. The aforementioned nighttime trained model is a trained model generated by machine learning, using training images, which include images taken at night, as input data, and the in-image positions of targets included in the training images as training data. A target monitoring device according to any one of claims 1 to 9.

11. The aforementioned pre-trained model for sunrise and sunset is a pre-trained model generated by machine learning, using training images, which include images taken during sunrise or sunset, as input data, and the image positions and types of targets included in the training images as training data. A target monitoring device according to any one of claims 1 to 10.

12. A target monitoring device as described in any one of claims 1 to 11, A ship maneuvering decision unit that makes a ship maneuvering decision based on targets detected by the aforementioned target monitoring device, A ship handling control unit that controls the operation of the ship based on the ship handling judgment, A ship handling system equipped with the following features.

13. By computer, Images including maritime targets are acquired by cameras installed on ships. Determine whether the aforementioned image was taken during the daytime, at night, or during sunrise or sunset. If it is determined that the image was taken during the daytime, the target contained in the image is detected using a pre-trained model for daytime use. If it is determined that the image was taken at night, the target contained in the image is detected using a pre-trained model for nighttime use. If it is determined that the image was taken during sunrise or sunset, the target contained in the image is detected using a pre-trained model for sunrise and sunset. Target monitoring method.

14. To acquire images including objects at sea captured by cameras installed on ships, Determine whether the aforementioned image was taken during the daytime, at night, or during sunrise or sunset. If it is determined that the image was taken during the daytime, the target contained in the image is detected using a pre-trained model for daytime use. If it is determined that the image was taken at night, the target contained in the image is detected using a pre-trained model for nighttime use, and If it is determined that the image was taken during sunrise or sunset, the object contained in the image is detected using a pre-trained model for sunrise and sunset. A program that causes a computer to execute something.

Images