Target monitoring device, target monitoring method, and program

Abstract

[Problem] To provide a target monitoring device which facilitates identification of unknown targets. [Solution] A target monitoring device comprises: an image acquisition unit which sequentially acquires images including marine views captured by a camera performing a panning operation; an image recognition unit which detects targets included in the images; a target identification unit which determines whether a detected target is the same as a target registered in a database in which target data of targets detected by a target detection unit which is different from the camera is registered; and a camera control unit which stops the panning operation of the camera in a state in which the detected target is included in the angle of view if the detected target is not a target registered in the database.

Description

Technical Field

[0001] The present invention relates to an object monitoring device, an object monitoring method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique for associating and combining information obtained by a device or equipment such as a radar with information obtained from camera images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, a radar may cause false detections due to the influence of sea surface reflections or false echo images. In addition, an AIS cannot detect ships that do not carry or use an AIS, or objects other than ships.

[0005] The present invention has been made in view of the above problems, and its main object is to provide an object monitoring device, an object monitoring method, and a program that can easily focus on unrecognized objects.

Means for Solving the Problems

[0006] To solve the above problems, a target monitoring device according to one aspect of the present invention comprises: an image acquisition unit that sequentially acquires images including a seascape captured by a camera in panning motion; an image recognition unit that detects targets included in the images; a target identification unit that determines whether the detected target is the same as a target registered in a database where target data of targets detected by a target detection unit different from the camera is registered; and a camera control unit that, if the detected target is not a target registered in the database, stops the panning motion of the camera while including the detected target in the field of view. This makes it easy to focus on targets that have not yet been detected.

[0007] In the above embodiment, the camera control unit may continue the panning operation of the camera if the detected target is the same as a target registered in the database. This makes it possible to continue monitoring by panning.

[0008] In the above embodiment, the image recognition unit may generate target data of the detected target from the image acquired while the panning motion is stopped and register it in the database. This makes it possible to generate target data from the image acquired while the panning motion is stopped.

[0009] In the above embodiment, the camera control unit may cause the camera to zoom in and capture an image of the detected target while the panning motion is stopped. This makes it possible to generate target data from the zoomed-in image.

[0010] In the above embodiment, the camera control unit may cause the camera to resume the panning operation after a predetermined time has elapsed since the panning operation stopped. This makes it possible to resume the panning operation after a predetermined time has elapsed.

[0011] In the above embodiment, the database may register target data of targets detected by at least one of a camera, radar, and AIS (Automatic Identification System) other than the camera. This makes it possible to suppress the failure to detect targets by at least one of the camera, radar, and AIS.

[0012] Furthermore, another aspect of the present invention involves acquiring an image including a seascape captured by a camera during panning, detecting targets included in the image, determining whether the detected target is identical to a target registered in a database where target data of targets detected by a different target detection unit than the camera is registered, and, if the detected target is not a target registered in the database, stopping the panning operation of the camera while including the detected target in the field of view. This makes it easier to focus on targets that have not yet been detected.

[0013] Furthermore, a program in another aspect of the present invention causes a computer to sequentially acquire images including a seascape captured by a camera during panning, detect targets included in the images, determine whether the detected targets are identical to targets registered in a database where target data of targets detected by a target detection unit different from the camera is registered, and, if the detected targets are not targets registered in the database, stop the panning operation of the camera while including the detected target in the field of view. This makes it easier to focus on targets that have not yet been detected. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram showing an example configuration of a target monitoring system. [Figure 2] This is a diagram showing an example of the configuration of a target monitoring device. [Figure 3] This diagram shows an example of an integrated management database. [Figure 4] This figure shows an example of an image captured by a camera. [Figure 5] This is a diagram showing an example of the panning operation of a camera. [Figure 6] This is a diagram showing an example of the panning operation of a camera. [Figure 7] This is a diagram showing an example of the panning operation of a camera. [Figure 8] This is a diagram showing an example of the panning operation of a camera. [Figure 9] This is a diagram showing an example of the procedure of the target monitoring method.

Embodiments of the Invention

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0016] FIG. 1 is a block diagram showing a configuration example of a target monitoring system 100. The target monitoring system 100 is a system mounted on a ship. In the following description, the ship on which the target monitoring system 100 is mounted is referred to as the "own ship", and other ships are referred to as "other ships".

[0017] The target monitoring system 100 includes a target monitoring device 1, a display unit 2, a radar 3, an AIS 4, a camera 5, a GNSS receiver 6, a gyrocompass 7, an ECDIS 8, a wireless communication unit 9, and a ship control unit 10. These devices are connected to a network N such as a LAN, for example, and can communicate with each other via the network.

[0018] The target monitoring device 1 is a computer including a CPU, a RAM, a ROM, a non-volatile memory, an input / output interface, and the like. The CPU of the target monitoring device 1 executes information processing according to a program loaded from the ROM or the non-volatile memory into the RAM.

[0019] The program may be supplied via an information storage medium such as an optical disk or a memory card, or may be supplied via a communication network such as the Internet or a LAN.

[0020] Display unit 2 displays display images generated by target monitoring device 1. Display unit 2 also displays radar images, camera images, or electronic charts.

[0021] The display unit 2 is, for example, a display device with a touch sensor, a so-called touch panel. The touch sensor detects the position indicated on the screen by the user's finger or the like. However, the position may also be input by a trackball or the like.

[0022] Radar 3 emits radio waves around the vessel and receives the reflected waves, generating echo data based on the received signals. Radar 3 also identifies targets from the echo data and generates TT (Target Tracking) data representing the target's position and speed. The TT data may also be generated by the target monitoring device 1.

[0023] The Automatic Identification System (AIS) 4 receives AIS data from other vessels or shore-based control systems in the vicinity of the vessel. While AIS is not the only option, a VHF Data Exchange System (VDES) may also be used. AIS data includes the identification code, name, position, course, speed, vessel type, length, and destination of other vessels.

[0024] Camera 5 is a digital camera that captures images of the outside from the ship and generates image data. Camera 5 is installed, for example, on the ship's bridge, facing the bow. Camera 5 is a camera with pan, tilt, and zoom functions, a so-called PTZ camera.

[0025] Furthermore, camera 5 may include an image recognition unit that estimates the position and type of targets, such as other vessels, included in the captured image using an object detection model. The image recognition unit is not limited to camera 5, but may also be implemented in other devices such as the target monitoring device 1.

[0026] The GNSS receiver 6 detects the ship's position based on radio waves received from the GNSS (Global Navigation Satellite System). The gyrocompass 7 detects the ship's heading. A GPS compass may be used instead of a gyrocompass.

[0027] The ECDIS (Electronic Chart Display and Information System) 8 obtains the ship's position from the GNSS receiver 6 and displays the ship's position on the electronic chart. The ECDIS 8 also displays the ship's planned route on the electronic chart. A GNSS plotter may be used instead of the ECDIS.

[0028] The radio communication unit 9 includes various radio equipment for enabling communication with other ships or land-based air traffic control, such as radio equipment for the ultra-high frequency, very high frequency, medium frequency, and short frequency bands.

[0029] The ship steering control unit 10 is a control device for realizing automatic ship steering and controls the ship's steering gear. The ship steering control unit 10 may also control the ship's engine.

[0030] In this embodiment, the target monitoring device 1 is an independent device, but it is not limited to this and may be integrated with other devices such as ECDIS8. In other words, the functional parts of the target monitoring device 1 may be implemented by other devices.

[0031] In this embodiment, the target monitoring device 1 is mounted on the vessel itself and used to monitor targets such as other vessels in the vicinity of the vessel; however, its use is not limited to this. For example, the target monitoring device 1 may be installed at a land-based control center and used to monitor vessels in a controlled area.

[0032] Figure 2 shows an example of the configuration of the target monitoring device 1. The control unit 20 of the target monitoring device 1 includes a data acquisition unit 11, a data acquisition unit 12, an image acquisition unit 13, an image recognition unit 14, a data integration unit 15, a display control unit 16, a ship operation decision unit 17, a target identification unit 18, and a camera control unit 19. These functional units are realized by the control unit 20 executing information processing according to a program.

[0033] The control unit 20 of the target monitoring device 1 further includes a radar management database 21, an AIS management database 22, a camera management database 23, and an integrated management database 24. These storage units are located in the memory of the control unit 20.

[0034] The data acquisition unit 11 sequentially acquires the TT data generated by the radar 3 as target data and registers it in the radar management DB 21.

[0035] The target data registered in the radar management DB21 includes the position, speed, and course of targets such as other vessels detected by radar 3. The target data registered in the radar management DB21 may further include the target's track, elapsed time since detection, size of the echo image, and signal strength of the reflected wave.

[0036] The data acquisition unit 12 acquires the AIS data received by AIS4 as target data and registers it in the AIS management DB 22.

[0037] The target data registered in the AIS management DB22 includes the position, speed, and course of other vessels detected by AIS4. The target data registered in the AIS management DB22 may further include the type of other vessel, its name, hull length, hull width, and destination.

[0038] The image acquisition unit 13 acquires images including targets such as other vessels captured by the camera 5. The image acquisition unit 13 sequentially acquires time-series images from the camera 5 and provides them sequentially to the image recognition unit 14. The time-series images are, for example, still images (frames) included in video data.

[0039] The image recognition unit 14 performs image recognition on the image acquired by the image acquisition unit 13, generates target data for the recognized targets from the image, and registers it in the camera management DB 23. Details of the image recognition unit 14 will be described later.

[0040] The target data registered in the camera management DB23 includes the position, speed, and course of targets such as other vessels, calculated by the image recognition unit 14. The target data registered in the camera management DB23 may further include the size of the target, the type of target, and the time elapsed since detection.

[0041] The position of a target detected by radar 3 and the position of a target recognized from images captured by camera 5 are relative positions to the ship. Therefore, they are converted into absolute positions, including heading information, using the ship's position and heading detected by GNSS receiver 6. Note that the heading may be obtained from a gyro sensor or the like instead of the GNSS receiver.

[0042] The targets detected by radar 3 and those recognized from images captured by camera 5 are mainly ships, but may also include other objects such as buoys.

[0043] The target data registered in the camera management DB23 may include not only target data of targets recognized from images captured by camera 5, but also target data of targets recognized from images captured by a PZT camera of the same type as camera 5 but installed separately, a fixed-point camera of a different type than camera 5, a 360-degree camera, or an infrared camera.

[0044] The data integration unit 15 registers the target data registered in the radar management DB 21, the AIS management DB 22, and the camera management DB 23 into the integrated management DB 24, which manages these databases across the board.

[0045] As shown in Figure 3, the target data registered in the integrated management DB24 includes the position, speed, and course of targets such as other vessels. The source indicates the origin of the target data, i.e., whether the target was detected by radar 3, AIS 4, or camera 5.

[0046] The data integration unit 15 integrates target data if the position of a target registered in one of the radar management DB 21, AIS management DB 22, and camera management DB 23 is the same as or approximately the same as the position of a target registered in the other one. However, the accuracy of target position calculation by cameras is often low, so in such cases, the integration condition may be set to include not only the target position, but also at least one of the target's speed, heading (bearing), and size, instead of the target position.

[0047] The display control unit 16 generates a display image containing objects representing targets based on target data registered in the integrated management DB 24 and outputs it to the display unit 2. The display image is, for example, a radar image, an electronic chart, or an image combining them, and the objects representing targets are placed in positions within the image that correspond to the actual locations of the targets.

[0048] The ship handling decision unit 17 makes a ship handling decision based on the target data registered in the integrated management DB 24, and if it determines that it is necessary to avoid a target, it instructs the ship handling control unit 10 to perform avoidance maneuvers. Specifically, the ship handling control unit 10 calculates an avoidance route to avoid the target using an avoidance maneuvering algorithm, and controls the steering gear and engines, etc., so that the ship follows the avoidance route.

[0049] The image acquisition unit 13 and the image recognition unit 14 will be described again. In this embodiment, the target is monitored while the camera 5 is panning. The image acquisition unit 13 sequentially acquires images, including the seascape, captured by the camera 5 during the panning operation.

[0050] The image recognition unit 14 detects targets contained in the image acquired by the data acquisition unit 13. Specifically, the image recognition unit 14 uses a pre-trained model generated by machine learning to calculate the region of the target contained in the image, the type of target, and the confidence level of the estimation. The type of target is, for example, a type of vessel such as a tanker or a fishing boat. However, the image recognition unit 14 may also recognize the region and type of target contained in the image using a rule-based method.

[0051] The trained model is, for example, an object detection model such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once), which detects the bounding box surrounding an object in an image as the region of the object. However, the trained model may also be a region segmentation model such as Semantic Segmentation or Instance Segmentation.

[0052] As shown in Figure 4, the other vessel SH included in the image P captured by camera 5 is surrounded by a rectangular boundary box BB. A label CF indicating the type of target and the confidence level of the estimation is attached to the boundary box BB.

[0053] The target identification unit 18 determines whether the target detected by the image recognition unit 14 is the same as a target registered in the integrated management DB 24. Whether the targets are the same or not is determined by whether the positions of the targets are the same or approximate. Note that the accuracy of calculating the target position by the camera is often low, in which case the condition may include not only the target's position, but also, instead of the target's position, at least one of the target's speed, heading (azimuth), and size.

[0054] In this embodiment, whether or not the targets are the same was determined by referring to the integrated management DB24 in which target data detected by radar 3, AIS 4, or camera 5 is registered, but the database to be referred to is not limited to these.

[0055] For example, the system may determine whether the targets are the same by referring to the radar management DB21 or AIS management DB22, which contains target data detected by a different target detection unit, such as the radar 3 or AIS 4, from the camera 5.

[0056] The camera control unit 19 controls the panning, tilting, or zooming of the camera 5. In this embodiment, the camera control unit 19 causes the camera 5 to repeatedly perform a panning motion when monitoring a target.

[0057] The camera control unit 19 continues the panning operation of the camera 5 if the target detected by the image recognition unit 14 is the same as a target registered in the integrated management DB 24, and stops the panning operation of the camera 5 with the detected target included in the field of view if the detected target is not a target registered in the integrated management DB 24.

[0058] The image recognition unit 14 performs image recognition on images acquired while the camera 5's panning motion is stopped, generates target data for detected targets, and registers it in the camera management DB 23. The registered target data is also registered in the integrated management DB 24. This makes it possible to register target data while the camera 5's panning motion is stopped, preventing missed detections.

[0059] The camera control unit 19 causes the camera 5 to zoom in on a target and take an image while the camera 5's panning motion is stopped. The image recognition unit 14 performs image recognition on the zoomed-in image, generates target data for the detected target, and registers it in the camera management DB 23. This makes it possible to generate more accurate target data.

[0060] The camera control unit 19 instructs the camera 5 to resume panning after a predetermined time (for example, a few seconds or 10 seconds) has elapsed since the camera 5 stopped panning.

[0061] Figures 5-8 illustrate the panning motion of camera 5. SA represents the field of view of camera 5. RS represents the start angle of the panning motion, and RE represents the end angle of the panning motion. The field of view SA of camera 5 moves from the start angle RS to the end angle RE.

[0062] Furthermore, SC represents a target recognized from the image of camera 5 (hereinafter referred to as image-recognized target SC). SN represents a target registered in the integrated management DB24 (hereinafter referred to as DB-registered target SN).

[0063] Figure 5 shows a situation where the image recognition target SC is not within the field of view SA of camera 5. In this case, the camera control unit 19 updates the orientation of camera 5, i.e., continues the panning operation.

[0064] Figure 6 shows a situation where an image-recognized target SC exists within the field of view SA of camera 5, and an identical DB-registered target SN also exists. This indicates a situation where no detection failures (oversights) have occurred. At this time, the camera control unit 19 updates the orientation of camera 5.

[0065] Figure 7 shows a situation where an image-recognized target SC exists within the field of view SA of camera 5, but no identical DB-registered target SN exists. This indicates a detection failure. At this time, the camera control unit 19 temporarily suspends the orientation update of camera 5, i.e., temporarily suspends the panning operation. As a result, camera 5 remains in a state where the image-recognized target SC is included in the field of view SA (target-locked state).

[0066] The image recognition unit 14 performs image recognition on the image captured in the target-lock state, generates target data for the detected target, and registers it in the camera management DB 23. The registered target data is also registered in the integrated management DB 24. As a result, the situation is the same as in Figure 6 above, where an image-recognized target SC exists within the field of view SA of the camera 5, and an identical DB-registered target SN also exists.

[0067] Figure 8 shows a situation where the image recognition target SC is not present within the field of view SA of camera 5, but the DB-registered target SN is present. This also represents a situation where no detection failures have occurred. At this time, the camera control unit 19 updates the orientation of camera 5.

[0068] Figure 9 shows an example of the procedure for target monitoring implemented in the target monitoring system 100. The figure mainly illustrates the process of monitoring a target while panning the camera 5. The target monitoring device 1 executes the process shown in the figure according to the program.

[0069] When the target monitoring device 1 acquires an image captured by the camera 5 during panning, it performs image recognition processing (S11, processing as the image recognition unit 14).

[0070] If no target is detected by the image recognition process (S12: NO), the target monitoring device 1 continues the panning operation of the camera 5 (S13, the situation shown in Figure 5 above).

[0071] On the other hand, if a target is detected by image recognition processing (S12: YES), the target monitoring device 1 calculates the position of the detected target (S14). The position of the target is calculated based on the position of the target in the image, the orientation of the camera 5, and the position of the ship.

[0072] Next, the target monitoring device 1 determines whether the detected target is the same as a target registered in the integrated management DB 24 (S15, processing as the target identification unit 18). Whether the targets are the same is determined by whether the locations of the targets are the same or similar.

[0073] If the target detected by the image recognition process is the same as the target registered in the integrated management DB24 (S15: YES), the target monitoring device 1 continues the panning operation of the camera 5 (S13, processing by the camera control unit 19, the situation shown in Figure 6 above).

[0074] On the other hand, if the target detected by the image recognition process is not the same as the target registered in the integrated management DB24 (S15: NO), the target monitoring device 1 stops the panning operation of the camera 5 (S16, processing by the camera control unit 19, the situation shown in Figure 7 above).

[0075] Next, the target monitoring device 1 performs image recognition on the images acquired while the camera 5's panning motion is stopped, generates target data for the detected targets, and registers it in the camera management DB 23 (S17-S19, processing as the image recognition unit 14).

[0076] If a predetermined time has elapsed after the camera 5 has stopped panning (S20: YES), the target monitoring device 1 instructs the camera 5 to resume panning (S13, processing by the camera control unit 19, the situation shown in Figure 6 above).

[0077] The target monitoring device 1 repeats the above processes S11-S20 while the camera 5 is panning.

[0078] According to the embodiment described above, it becomes possible to focus on targets detected during the panning operation of camera 5 that are not yet known in the integrated management DB 24, and furthermore, it becomes possible to generate target data from images acquired when the panning operation is stopped and register it in the integrated management DB 24.

[0079] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are of course possible for those skilled in the art. [Explanation of symbols]

[0080] 1 Target monitoring device, 2 Display unit, 3 Radar, 4 AIS, 5 Camera, 6 GNSS receiver, 7 Gyrocompass, 8 ECDIS, 9 Wireless communication unit, 10 Ship handling control unit, 20 Control unit, 11,12 Data acquisition unit, 13 Image acquisition unit, 14 Image recognition unit, 15 Data integration unit, 16 Display control unit, 17 Ship handling decision unit, 18 Target identification unit, 19 Camera control unit, 21 Radar management DB, 22 AIS management DB, 23 Camera management DB, 24 Integrated management DB, 100 Target monitoring system

Claims

1. An image acquisition unit that sequentially acquires images including the seascape captured by a camera in panning motion, An image recognition unit for detecting targets included in the aforementioned image, A target identification unit determines whether the detected target is the same as a target registered in a database where target data of targets detected by a target detection unit different from the camera is registered. If the detected target is not a target registered in the database, the camera control unit stops the panning operation of the camera while the detected target is included in the field of view. A target monitoring device equipped with the following features.

2. The camera control unit continues the panning motion of the camera if the detected target is the same as a target registered in the database. The target monitoring device according to claim 1.

3. The image recognition unit generates target data of the detected target from the image acquired while the panning motion is stopped, and registers it in the database. The target monitoring device according to claim 1 or 2.

4. The camera control unit, while the panning motion is stopped, causes the camera to zoom in on the detected target and take an image. A target monitoring device according to any one of claims 1 to 3.

5. The camera control unit, after a predetermined time has elapsed since the panning motion stopped, instructs the camera to resume the panning motion. A target monitoring device according to any one of claims 1 to 4.

6. The aforementioned database contains target data of targets detected by at least one of a camera different from the aforementioned camera, a radar, and an AIS (Automatic Identification System). A target monitoring device according to any one of claims 1 to 5.

7. Images including the seascape captured by the camera during panning are acquired. The target contained in the aforementioned image is detected, The system determines whether the detected target is the same as a target registered in a database where target data of targets detected by a target detection unit different from the camera is registered. If the detected target is not a target registered in the database, the camera's panning motion is stopped while the detected target is included in the field of view. Target monitoring method.

8. To sequentially acquire images including the seascape captured by a camera during panning, To detect targets included in the aforementioned image, The process involves determining whether the detected target is identical to a target registered in a database where target data of targets detected by a target detection unit different from the camera is registered, and, if the detected target is not a target registered in the database, stopping the panning motion of the camera while the detected target is included in the field of view. A program that causes a computer to execute something.

Images