Ship monitoring system, distance measurement method, distance measurement program, display control method, and display control program

Abstract

To provide a vessel monitoring system, a distance measuring method, a distance measuring program, a display control method and a display control program, capable of increasing the distance measurement accuracy of a vessel recognized from an image.SOLUTION: A camera 1 photographs an image. Marine measurement equipment 2 detects a vessel from around a photographing position by the camera 1 and measures the direction and distance of a detected vessel. An information processor 3 measures, based on a photographed image by the camera, a first direction indicating a direction from the photographing position for a vessel A recognized from the photographed image. determines, of vessels B and C detected by the marine measurement equipment 2, the vessel B whose direction measurement matches the first direction, as a ship corresponding to the vessel A, and determines the distance measurement result for the vessel A based on the distance of the vessel B measured by the marine measurement equipment 2.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a ship monitoring system, a distance measuring method, a distance measuring program, a display control method, and a display control program. [Background technology]

[0002] A common method for detecting ships at sea and measuring the distance to the ships from a measuring point on the ship or on land is to use marine measuring equipment such as radar and AIS (Automatic Identification System) receivers. In addition, with the development of image recognition technology, it is now possible to recognize ships from photographed images. Furthermore, it is also possible to measure the distance to the ship based on the photographed image, for example by using triangulation.

[0003] The following technologies have been proposed for recognizing and measuring distances to ships. For example, a ship navigation support device has been proposed that uses the results of ship detection from photographed images and the results of ship detection from radar to display information about the ship. In addition, a false image removal device has been proposed that converts the photographed images into positioning signals of targets, outputs the signals, and removes false images from the radar by combining the obtained positioning signals with those from the radar. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 208422 [Patent Document 2] Japanese Patent Application Publication No. 2-61581 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, when using radar, there is a possibility that waves may be mistaken for small ships, but such misrecognition is unlikely to occur when ships are recognized from photographed images. On the other hand, the accuracy of measuring the distance to a ship based on photographed images is lower than when using marine measuring equipment such as radar. Therefore, in order to improve the accuracy of measuring the distance to a ship recognized from photographed images, it is possible to combine it with distance measurements using marine measuring equipment. In order to perform such a combination, it is necessary to accurately match the ship recognized from the photographed images with the ship detected by the marine measuring equipment, but there is a problem that accurate matching is difficult due to the difference in distance measurement accuracy as described above.

[0006] In one aspect, the present invention aims to provide a ship monitoring system, a ranging method, a ranging program, a display control method, and a display control program that can improve the ranging accuracy of ships recognized from images. [Means for solving the problem]

[0007] One proposal provides a ship monitoring system having a camera that captures images, a ship measuring device that detects ships from around the position captured by the camera and measures the direction and distance of the detected ship, and an information processing device as follows: The information processing device measures a first direction indicating the direction from the shooting position of a first ship recognized from the captured image based on the image captured by the camera, determines a second ship among the ships detected by the ship measuring device whose measured direction value matches the first direction to be a ship corresponding to the first ship, and determines a distance measurement result for the first ship based on the distance of the second ship measured by the ship measuring device.

[0008] Also, one proposal provides a distance measuring method in which a computer executes the same processing as the above-mentioned information processing device. Furthermore, in one proposal, a distance measuring program is provided that causes a computer to execute the same processing as the above-mentioned information processing device.

[0009] In addition, one proposal provides a display control method in which, when a computer displays on a map the position of a ship recognized from an image captured by a camera, if the distance from the camera's shooting position to the ship is obtained by both measurement based on the captured image and measurement by a ship measuring device, a first position of the ship is identified based on the distance measurement result by the ship measuring device and the first position is displayed on the map, and if the distance from the shooting position to the ship is obtained by measurement based on the captured image but not by measurement by the ship measuring device, a second position of the ship is identified based on the distance measurement result based on the captured image and the second position is displayed on the map in a display manner different from the first position.

[0010] In addition, in one proposal, a display control program is provided that causes a computer to execute a process similar to the above-mentioned display control method. Effect of the Invention

[0011] In one aspect, the accuracy of measuring the distance of a ship recognized from an image can be improved. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration and processing of a ship monitoring system according to a first embodiment; [Diagram 2] FIG. 11 is a diagram illustrating an example of the configuration of a vessel monitoring system according to a second embodiment. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a monitoring processing device. [Figure 4] FIG. 2 is a diagram illustrating an example of a configuration of processing functions included in a monitoring processing device. [Diagram 5] 11 is a diagram for explaining the recognition and distance measurement process of a ship from a camera image. FIG. [Figure 6] FIG. 13 is a diagram for explaining a matching method for a recognized ship. [Figure 7] FIG. 1 is a diagram for explaining region division in a camera image. [Figure 8]13 is a flowchart showing an example of a recognition and ranging process performed by a monitoring processing device. [Figure 9] FIG. 11 is a diagram illustrating a first example of an image display process. [Figure 10] FIG. 11 is a diagram illustrating a second example of the image display process. [Figure 11] FIG. 11 is a diagram illustrating a third example of the image display process. [Figure 12] 13 is a flowchart illustrating an example of a recognition and ranging process performed by a monitoring processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First Embodiment 1 is a diagram showing an example of the configuration and processing of a ship monitoring system according to a first embodiment. The ship monitoring system shown in FIG.

[0014] The camera 1 captures an image in order to recognize the ship from around the capturing position. The captured image (captured image) is input to the information processing device 3. The vessel measuring device 2 detects ships from around the position photographed by the camera 1, and measures the direction and distance of the detected ship. The measured distance refers to the distance from the position photographed by the camera 1 to the ship. The vessel measuring device 2 is, for example, a measuring device that uses radar. Alternatively, the vessel measuring device 2 may be a measuring device that uses the results of receiving AIS information.

[0015] The information processing device 3 uses the images captured by the camera 1 and the measurement results obtained by the marine measuring instrument 2 to execute the following processes. The information processing device 3 recognizes a ship from the image captured by the camera 1. When a ship is recognized, the information processing device 3 measures the distance from the image capture position to the ship and the direction of the ship along the horizontal plane based on the captured image (step S1). In the example of Fig. 1, it is assumed that ship A is recognized, and the distance to ship A is measured as 2 km and the direction is determined as 30 degrees.

[0016] The information processing device 3 also acquires from the ship measuring device 2 the measurement results of distance and direction for the ships detected by the ship measuring device 2 (step S2). In the example of Fig. 1, it is assumed that ships B and C are detected by the ship measuring device 2. For ship B, it is assumed that the distance is measured to be 2.5 km and the direction is measured to be 30 degrees. For ship C, it is assumed that the distance is measured to be 1.1 km and the direction is measured to be 60 degrees.

[0017] When ship A is recognized from the captured image in step S1, the information processing device 3 determines which of ships B and C detected by the ship measuring instrument 2 in step S2 has a direction measurement value that matches the direction measurement value for ship A (step S3).Then, the information processing device 3 determines the distance measurement result for ship A based on the determination result in step S3 (step S4).

[0018] In the example of FIG. 1, in step S3, it is determined that the direction measurement value for ship B, of ships B and C, matches the direction measurement value for ship A. In this case, ship B detected by the ship measuring device 2 is presumed to correspond to ship A recognized from the captured image (is the same ship as ship A). ​​In this case, in step S4, the information processing device 3 determines the distance measurement result for ship A based on the distance measurement result for ship B measured by the ship measuring device 2. For example, the distance measurement value for ship B (2.5 km) is directly determined as the distance measurement value for ship A. Alternatively, an intermediate value between the distance measurement value for ship B (2.5 km) and the distance measurement value for ship A (2 km) may be determined as the distance measurement value for ship A.

[0019] In this way, a matching process is performed based on the direction measurement value to search for a ship corresponding to the ship recognized from the photographed image from among the ships detected by the ship measuring device 2. Unlike the distance measurement accuracy, the direction measurement accuracy is as high in measurements based on photographed images as in measurements by the ship measuring device 2. Therefore, it is possible to accurately find a ship corresponding to the ship recognized from the photographed image from among the ships detected by the ship measuring device 2. Then, when a corresponding ship is found, the distance measurement result for the ship recognized from the photographed image is determined based on the distance measurement result by the ship measuring device 2, thereby improving the distance measurement accuracy. Therefore, it is possible to improve the distance measurement accuracy for a ship recognized from a photographed image.

[0020] Although not shown, in step S3, if the direction measurement value for either ship B or C does not match the direction measurement value for ship A, it is determined that a ship corresponding to ship A (the same ship) has not been detected by the ship measuring instrument 2. In this case, in step S4, the information processing device 3 determines the distance measurement result for ship A based on the distance measurement result for ship A measured based on the captured image.

[0021] This makes it possible to recognize ships that are difficult to detect by the marine measuring instrument 2, such as ships made of FRP (Fiber Reinforced Plastics), from the captured image and output the distance measurement results.

[0022] Second Embodiment Fig. 2 is a diagram showing an example of the configuration of a ship monitoring system according to a second embodiment. The ship monitoring system shown in Fig. 2 has a configuration in which a camera 101, a radar device 102, an AIS device 103, and a display device 104 are connected to a monitoring processing device 100. The monitoring processing device 100, the camera 101, the radar device 102, the AIS device 103, and the display device 104 are mounted on a ship. Hereinafter, a ship on which the ship monitoring system is mounted may be referred to as "own ship", and other ships may be referred to as "other ships".

[0023] The camera 101 captures images of the surroundings of the ship and transmits the captured images (camera images) to the monitoring processing device 100. In this embodiment, the camera images are used to recognize other ships present around the ship and to measure the distance from the ship to the other ships.

[0024] The radar device 102 includes an antenna for transmitting radar waves (e.g., pulsed microwaves) and receiving reflected waves, a rotation mechanism for rotating the antenna, and a processing device for processing signals received by the antenna. The radar device 102 recognizes other ships present around the ship and measures the distance from the ship by emitting radar waves in the horizontal direction while rotating the antenna 360 degrees. The radar device 102 transmits radar information based on the results of receiving the reflected waves to the monitoring processing device 100. The radar information includes information on the direction and distance of other ships.

[0025] The AIS device 103 is a device for receiving AIS information from other ships, and includes an antenna for receiving the AIS information, a processing device for processing the signal received by the antenna, etc. The AIS information includes the identification code, ship name, and position information (for example, absolute position information such as latitude and longitude) of the other ship. The AIS device 103 transmits the received AIS information to the monitoring processing device 100.

[0026] The monitoring processing device 100 is a computer device that executes processing related to monitoring other ships. The monitoring processing device 100 determines the recognition result of the other ship and the distance to the recognized other ship based on information obtained from the camera 101, the radar device 102, and the AIS device 103. The monitoring processing device 100 also generates an image showing the recognition result of the other ship and the distance to the other ship, and displays it on the display device 104. For example, the monitoring processing device 100 displays the image area of ​​the recognized other ship and information showing the distance to the other ship on the camera image. Alternatively, the monitoring processing device 100 displays information showing the position of the recognized other ship on a map (nautical chart).

[0027] The display device 104 displays an image according to an instruction from the monitoring processing device 100. As the display device 104, for example, a liquid crystal display or an organic EL (ElectroLuminescence) display is used.

[0028] Such a vessel monitoring system is installed, for example, on a work ship for construction work at sea. In this case, workers can use the distance measurement results and display contents of the vessel monitoring system to improve the safety of the construction work at sea.

[0029] As another example, the ship monitoring system may be installed on land as a marine monitoring facility. Fig. 3 is a diagram showing an example of the hardware configuration of a monitoring processing device. The monitoring processing device 100 is realized, for example, as a computer as shown in Fig. 3. The monitoring processing device 100 shown in Fig. 3 has a processor 111, a RAM (Random Access Memory) 112, a HDD (Hard Disk Drive) 113, a graphic interface (I / F) 114, an input interface (I / F) 115, a reading device 116, a communication interface (I / F) 117, and a network interface (I / F) 118.

[0030] The processor 111 performs overall control of the entire monitoring processing device 100. The processor 111 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 111 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

[0031] The RAM 112 is used as a main storage device of the monitoring processing device 100. The RAM 112 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 111. The RAM 112 also stores various data necessary for processing by the processor 111.

[0032] The HDD 113 is used as an auxiliary storage device for the monitoring processing device 100. The OS program, application programs, and various data are stored in the HDD 113. Note that other types of non-volatile storage devices such as SSDs (Solid State Drives) can also be used as the auxiliary storage device.

[0033] The above-mentioned display device 104 is connected to the graphic interface 114. The graphic interface 114 causes the display device 104 to display an image in accordance with an instruction from the processor 111.

[0034] An input device 115a is connected to the input interface 115. The input interface 115 transmits a signal output from the input device 115a to the processor 111. The input device 115a may be a keyboard or a pointing device. The pointing device may be a mouse, a touch panel, a tablet, a touch pad, a trackball, or the like.

[0035] Portable recording medium 116a is detachably attached to reader 116. Reader 116 reads data recorded on portable recording medium 116a and transmits it to processor 111. Portable recording medium 116a may be an optical disk, a magneto-optical disk, a semiconductor memory, or the like.

[0036] The communication interface 117 is an interface device that transmits and receives data to and from other devices such as the camera 101, the radar device 102, and the AIS device 103. The network interface 118 transmits and receives data to and from other devices via a network 118a.

[0037] The processing functions of the monitoring processing device 100 can be realized by the above-mentioned hardware configuration. Conventionally, radar has been widely used as a means of recognizing the presence of other ships and measuring the distance. The advantage of measuring distance using radar is that it is highly accurate, especially when the distance is relatively long. On the other hand, there are problems with using radar, such as the possibility of misidentifying waves as small ships, or the possibility of not being able to recognize small FRP ships.

[0038] On the other hand, the progress of image recognition technology from camera images has been remarkable, and the technology to recognize ships from camera images is now at a practical level. There is also technology to measure the distance to a ship recognized from a camera image, for example by using trigonometry. When it comes to recognizing ships from camera images, the possibility of misidentifying waves as ships is low, and ships made of FRP can be recognized without any particular problems, so it can be said that the recognition accuracy is higher than when radar is used. However, when it comes to measuring the distance to a ship using camera images, the accuracy is inferior to when radar is used. In particular, when using camera images, there is a problem that the distance measurement accuracy decreases the further the target object is.

[0039] Therefore, it is conceivable to perform more accurate ship recognition and ranging by combining both the recognition and ranging results using radar and the recognition and ranging results using camera images. However, in order to do this, there is a problem in how to match a ship recognized using radar with the same ship recognized using camera images. For example, a method can be considered in which the position of the ship recognized from both ranging results is plotted on a two-dimensional coordinate along a horizontal plane to match the same ships. However, as mentioned above, there is a difference in accuracy between the ranging results of both, so it is difficult to accurately match the same ships on two-dimensional coordinates.

[0040] Therefore, in this embodiment, the distance to the other ship is measured by taking advantage of the fact that the direction of the other ship can be measured with high accuracy, comparable to that when radar information is used, even when a camera image is used. The monitoring processing device 100 matches the same ship recognized using radar and camera images, using the direction of the other ship based on radar information and the direction of the other ship based on camera images. If matching is successful, the monitoring processing device 100 measures the distance to the other ship mainly using the distance measurement results using radar. This enables the monitoring processing device 100 to recognize other ships with high accuracy and measure the distance to the recognized other ships with high accuracy.

[0041] Furthermore, when the monitoring and processing device 100 determines, through matching using the above-mentioned method, that a ship recognized using a camera image has not been recognized using radar, it adopts the distance measurement result based on the camera image. However, when the monitoring and processing device 100 determines that the relevant ship is present among the ships included in the AIS information, it corrects the distance measurement result based on the camera image using the position information of the ship included in the AIS information.

[0042] 4 is a diagram showing an example of the configuration of the processing functions of the monitoring processing device. The monitoring processing device 100 includes an image recognition information storage unit 121, a radar information storage unit 122, an AIS information storage unit 123, a tracking information storage unit 124, a map information storage unit 125, an image recognition processing unit 131, a radar information acquisition unit 132, an AIS information acquisition unit 133, a distance measurement processing unit 134, and a display processing unit 135.

[0043] The image recognition information storage unit 121, the radar information storage unit 122, the AIS information storage unit 123, the tracking information storage unit 124, and the map information storage unit 125 are realized as storage areas of a storage device provided in the monitoring processing device 100, such as the RAM 112 and the HDD 113. Furthermore, each process of the image recognition processing unit 131, the radar information acquisition unit 132, the AIS information acquisition unit 133, the distance measurement processing unit 134, and the display processing unit 135 is realized by the processor 111 executing a predetermined program.

[0044] The image recognition information storage unit 121 stores image recognition information indicating the results of image recognition from camera images by the image recognition processing unit 131. The image recognition information includes information on the direction and distance of other ships recognized from the camera images, area information indicating the area in which the recognized other ships appear on the camera images, information indicating the type of the recognized other ships, and the like.

[0045] The radar information storage unit 122 stores radar information acquired from the radar device 102 by the radar information acquisition unit 132. The radar information includes information on the direction and distance of recognized other ships.

[0046] The AIS information storage unit 123 stores the AIS information acquired from the AIS device 103 by the AIS information acquisition unit 133. The AIS information includes the identification codes, ship names, and position information of other ships.

[0047] The tracking information storage unit 124 stores tracking information indicating the change in position of the same ship among other ships whose distances have been measured by the distance measurement processing unit 134. The map information storage unit 125 stores information on maps (nautical charts).

[0048] The image recognition processing unit 131 receives camera images captured by the camera 101 and performs image recognition processing on the camera images to recognize other ships captured in the camera images and measure the distance to the other ships. Furthermore, in this embodiment, as described below, the image recognition processing unit 131 is also capable of determining the type of other ships. The image recognition processing unit 131 performs this type of image recognition processing at regular time intervals (for example, at the image capture period of the camera 101) and stores image recognition information indicating the image recognition results in the image recognition information storage unit 121. The image recognition information includes, for the other ships recognized from the camera images, information on the direction based on the capture direction of the camera 101, information on the distance from the ship itself, area information on the camera image, and information indicating the ship type.

[0049] The radar information acquisition unit 132 receives the results of recognition and distance measurement of other ships by the radar device 102 at regular time intervals, and stores these results as radar information in the radar information storage unit 122. The radar information includes information indicating the direction of the recognized other ships based on a predetermined reference direction and the distance from the ship. Furthermore, if the radar device 102 has a function for receiving GPS (Global Positioning System) signals, the radar information may also include the absolute coordinates of the recognized other ships.

[0050] The AIS information acquisition unit 133 acquires AIS information about other ships received by the AIS device 103, and stores it in the AIS information storage unit 123. As described above, the AIS information includes the identification code, ship name, and location information of the other ship.

[0051] The ranging processing unit 134 outputs the ranging results for other ships recognized from camera images, together with the recognition results of other ships, to the display processing unit 135 based on the information stored in the image recognition information storage unit 121, the radar information storage unit 122, and the AIS information storage unit 123. The ranging processing unit 134 also accumulates the most recent recognition and ranging results in the tracking information storage unit 124. At the same time, the ranging processing unit 134 generates tracking information indicating the change in the position of the same ship, based on the most recent recognition and ranging results and the information accumulated in the tracking information storage unit 124, and records it in the tracking information storage unit 124 as tracking information.

[0052] The display processing unit 135 generates display information for displaying the recognition results and distance measurement results of other ships, and displays the information on the display device 104. For example, the display processing unit 135 displays the image area of ​​the recognized other ship and information indicating the distance to the other ship on the camera image. Alternatively, the display processing unit 135 displays information indicating the position of the recognized other ship on a map (nautical chart).

[0053] FIG. 5 is a diagram for explaining the recognition and distance measurement process of a ship from a camera image. For example, a method for recognizing a ship from a camera image may use a large number of images of ships as teacher images to train the recognition process, while a method for measuring the distance from a camera image to an object may use triangulation based on the coordinate information of the object on the camera image.

[0054] In contrast, in this embodiment, by way of example, a large number of training images are used to train the robot not only to recognize ships, but also to measure the distance to the recognized ships. As a more specific example, in this embodiment, the robot learns the process of recognizing ships, the process of determining the type of the recognized ships, and the process of measuring the direction of the ships and the distance to the ships.

[0055] For this purpose, learning is performed in advance using training data, as shown in the upper part of Fig. 5. The training data includes many pairs of training images as input data and output data including multiple types of label information. The output data includes area information indicating the image area of ​​the ship in the training image by coordinates, ship type indicating the type of ship, the direction of the ship relative to the horizontal direction based on the shooting direction, and the distance to the ship.

[0056] By such learning, an image recognition processing unit 131 is generated that infers area information, ship type, direction, and distance from the input camera image, as shown in the lower part of Fig. 5. For example, the learning results are calculated as setting values ​​of a learning model for inference, and are used during processing by the image recognition processing unit 131. As a result, the image recognition processing unit 131 operates as a classifier that identifies each piece of label information described above.

[0057] The learning process using the teacher data may be executed on the monitoring processing device 100, or on another information processing device. Incidentally, the distance measurement process of the image recognition processing unit 131 based on the above-mentioned learning results also has lower distance measurement accuracy, especially at relatively long distances, compared to distance measurement using radar. However, the direction of a recognized ship can be measured with the measurement process by the image recognition processing unit 131 with the same high accuracy as when radar is used. Therefore, the distance measurement processing unit 134 uses the direction of other ships measured by the image recognition processing unit 131 and the direction of other ships measured by the radar device 102 to search for the same ship from among the ships recognized by each.

[0058] FIG. 6 is a diagram for explaining a matching method for a recognized ship. 6, the shooting range of the camera 101 is a range of angle of view according to the shooting parameters of the camera 101. Hereinafter, this angle of view is referred to as angle θ. Therefore, from the camera image captured by the camera 101, the direction of other ships can be measured within a range of angle θ centered on the shooting direction D of the camera 101 (a range of 1 / θ from the center to the left and right, respectively).

[0059] On the other hand, the radar measurement screen 103a shown in the lower part of Fig. 6 shows the position of the ship on an image centered on the position of the ship itself, based on the recognized direction and distance of the ship. As can be seen from this radar measurement screen 103a, in measurements by the radar device 102, the direction of other ships is measured within a range of 360 degrees centered on the position of the ship itself. By converting such direction measurement results by the radar device 102 into angles indicating a range of angle θ centered on the shooting direction D of the camera 101, it is possible to match the scales of the direction measurement results based on the camera image and the direction measurement results by the radar device 102.

[0060] The ranging processor 134 acquires the direction measurement values ​​included in the radar information from the radar device 102 in a state converted into a direction scale based on the camera image. The ranging processor 134 then recognizes a ship recognized from the camera image and a ship recognized by the radar device 102 whose direction measurement values ​​match as the same ship. By matching the same ship using the direction measurement results in this way, it becomes possible to accurately determine that they are the same ship by comparing with matching on two-dimensional coordinates based on the distance and direction measurement results.

[0061] When the distance measurement processing unit 134 succeeds in matching the same ship in this way, it adopts the distance measurement value for that ship taken by the radar device 102 as the distance measurement value to be output. As a result, of the objects recognized as ships by the radar device 102, only those that are also recognized as ships from the camera image are ultimately recognized as ships. This reduces the possibility of misidentifying waves as ships. Then, with the possibility of misrecognition reduced in this way, it becomes possible to accurately measure the distance to the ship by adopting the distance measurement value taken by the radar device 102.

[0062] Therefore, by effectively utilizing the advantages of both high recognition accuracy based on camera images and high distance measurement accuracy using the radar device 102, it is possible to improve both the problems of low distance measurement accuracy based on camera images and the occurrence of erroneous recognition using the radar device 102. In addition, by utilizing marine equipment and cameras that are generally installed on ships, it is possible to accurately recognize and measure the distance to the ship as described above without incurring new manufacturing costs or equipment purchasing costs.

[0063] In this embodiment, as an example, matching between a ship recognized from a camera image and a ship recognized by the radar device 102 is performed as follows. As shown in the upper part of Fig. 6, the angle of view (angle θ) of the camera 101 is evenly divided into a predetermined number of ranges (six in Fig. 6). Hereinafter, each of the divided areas will be referred to as an "angle range." Meanwhile, within the range of direction measurement by the radar device 102, the range of angle θ centered on the shooting direction D of the camera 101 as shown in the lower part of Fig. 6 is also evenly divided into the same number of angle ranges (six in Fig. 6).

[0064] The distance measurement processing unit 134 determines in which angle range the direction of a ship recognized from either the camera image or the radar device 102 falls. When a ship recognized from a camera image falls within a certain angle range, the distance measurement processing unit 134 determines that ships recognized by the radar device 102 that fall within the same angle range are the same ship. This matching method makes it possible to accurately determine whether or not the ships are the same, even if there is a certain degree of error between the direction measurement result based on the camera image and the direction measurement result using the radar device 102.

[0065] In addition, as shown in the following Figure 7, the angle range to which the direction of the ship recognized from the camera image belongs can also be determined based on the position (coordinates) of the ship in the camera image, without directly using the direction measurement results based on the camera image.

[0066] 7 is a diagram for explaining region division in a camera image. The direction of a ship recognized from a camera image is uniquely determined from the position (coordinates) of the ship captured on the camera image. Therefore, the angle range to which the direction of the ship belongs can be associated with the divided regions obtained by dividing the camera image.

[0067] As shown by the thick dashed lines in the camera image 101a in Fig. 7, the dividing lines dividing the regions corresponding to the angle ranges are represented by radial lines centered at a predetermined position on the vertical center line located at the horizontal center of the camera image 101a. The center position of the radial lines on the vertical center line is determined by the installation height of the camera 101, and is usually located below the bottom line of the camera image 101a as in the example of Fig. 7.

[0068] Furthermore, if the shooting direction of the camera 101 is horizontal, a horizon line H appears in the center of the camera image 101a. Since the upper limit of the center position of the ship captured in the camera image 101a is the height of the horizon line H, each divided area surrounded by radial lines (thick dashed lines in FIG. 7) and the horizon H in the camera image 101a is associated with a separate angle range. Therefore, when a ship is recognized from the camera image 101a, the distance measurement processing unit 134 can determine the angle range to which the direction of the ship belongs, depending on which divided area in the camera image 101a the center position of the ship is included in.

[0069] When multiple ships are recognized from the same divided area on the camera image 101a, the distance measurement processing unit 134 matches only the ship closest to it (the ship with the shortest distance) with the ship recognized by the radar device 102. This is because, in the same divided area, a part of a ship located at the back may overlap with a ship located at the front, which may result in an incorrect recognition of the ship located at the back. Usually, the ship located at the front is considered to be the most dangerous, such as having a high probability of colliding with the ship, so in many cases, there is no problem if the distance to the ship located at the front can be measured.

[0070] FIG. 8 is an example of a flowchart showing a processing procedure of recognition and ranging by the monitoring processing device. [Step S11] When a ship is recognized from the most recently captured camera image, the distance measurement processing unit 134 obtains image recognition information based on the camera image from the image recognition information storage unit 121, and executes the processes from step S12 onwards. If multiple ships are recognized from the camera image, the processes from steps S12 to S18 are executed for each recognized ship. Then, in step S19, an image is displayed for each recognized ship.

[0071] [Step S12] The distance measurement processor 134 extracts area information of the ship in the camera image from the acquired image recognition information, and identifies the coordinates of the center position of the ship area. The distance measurement processor 134 determines which divided area in the camera image contains the identified center position. This determines which angle range the direction of the ship is contained in.

[0072] [Step S13] The ranging processor 134 obtains the radar information most recently output from the radar device 102 from the radar information storage unit 122, and refers to the direction measurement results for each recognized ship from the radar information. The ranging processor 134 searches for a ship from among the recognized ships whose measured direction falls within an angle range that matches the angle range determined in step S12.

[0073] [Step S14] The distance measurement processing unit 134 determines whether a corresponding ship has been found by the search in step S13. If a corresponding ship has been found, the process proceeds to step S15. If a corresponding ship has not been found, the process proceeds to step S16.

[0074] [Step S15] The ranging processing unit 134 obtains the distance measurement result for the relevant ship from the radar information, and outputs the obtained distance as the ranging result for this ship to the display processing unit 135. Note that if the radar information includes the measurement result of the absolute coordinates of the ship instead of the ranging result, the distance is calculated based on the absolute coordinates and the current position of the ship, and this is used as the ranging result.

[0075] [Step S16] The ranging processing unit 134 acquires AIS information received within a certain time from the current time from the AIS information storage unit 123. The ranging processing unit 134 determines whether the ship recognized in step S11 is among the ships indicated by the acquired AIS information. For example, among the ships indicated by the AIS information, a ship that is the same ship type as the ship recognized in step S11 and has a position (absolute coordinate) error less than a certain level is determined to be the same as the ship recognized in step S11. If a corresponding ship is found, processing proceeds to step S17; if a corresponding ship is not found, processing proceeds to step S18.

[0076] [Step S17] The ranging processing unit 134 acquires position information from the AIS information for the relevant ship, and outputs the distance between that position information and the position information of the own ship to the display processing unit 135 as the ranging result for this ship (other ship).

[0077] [Step S18] The distance measurement processing unit 134 acquires a distance measurement result from image recognition information based on the camera image of the relevant ship, and outputs the acquired distance to the display processing unit 135 as the distance measurement result of this ship.

[0078] By processing any one of the above steps S15, S17, and S18, the final distance measurement result for the relevant ship is output. Here, in steps S17 and S18, the distance measurement result for the ship that could not be recognized by the radar device 102 is output. Therefore, it becomes more likely that the distance measurement result for the FRP ship that could not be recognized by the radar device 102 will also be output. Therefore, it is possible to reduce the possibility of missing the FRP ship compared to the case where the ship is recognized using only the radar device 102. Furthermore, particularly in step S17, it becomes possible to output an accurate distance measurement result for the FRP ship.

[0079] In steps S15, S17, and S18, the distance measurement results of the ship, area information in the camera image, and ship type are actually output to the display processing unit 135. The distance measurement results, direction, and ship type are stored in the tracking information storage unit 124. Then, based on these distance measurement results, direction, and ship type, and the past distance measurement results, direction, and ship type stored in the tracking information storage unit 124, tracking information indicating the change in position of the same ship is generated and recorded in the tracking information storage unit 124. If tracking information for the ship in question has already been recorded, position information based on the latest distance measurement results is added to the tracking information.

[0080] In addition, recognition type information is added to the information output to the display processing unit 135 and the generated tracking information. In step S15, recognition type information is added indicating that the information is of a ship recognized from both the camera image and the radar device 102. In step S17, recognition type information is added indicating that the information is of a ship recognized from both the camera image and the AIS information. In step S18, recognition type information is added indicating that the information is of a ship recognized only from the camera image.

[0081] [Step S19] The display processing unit 135 executes image display processing based on the distance measurement result. An example of the image display processing will be described below with reference to FIGS. Fig. 9 is a diagram showing a first example of image display processing. Fig. 9 shows a case where the image area of ​​a recognized ship, the distance to the ship, and the type of ship are displayed on a camera image. In a camera image 140 shown in Fig. 9, three ships are recognized as an example, and image areas 141-143 for each ship are shown by rectangles. Furthermore, ship information 141a-143a indicating the distance and ship type is displayed near each of the image areas 141-143.

[0082] Here, it is assumed that the distance measurement results of the ships included in image areas 141 and 142 are determined based on radar information in step S14 of Fig. 8. On the other hand, it is assumed that the distance measurement results of the ships included in image area 143 are determined based on camera images in step S18 of Fig. 8. In other words, the ships included in image areas 141 and 142 are ships for which matching between the recognition results based on the camera images and the recognition results using radar was successful. On the other hand, the ships included in image area 143 are ships for which such matching failed (ships recognized only from camera images). Such differences between each ship are determined based on the recognition type information added to the distance measurement results for each ship.

[0083] In this case, the display processing unit 135 displays the image areas 141, 142 and the image area 143 in different display modes on the camera image 140. In the example of Fig. 9, the former is displayed as a solid-line rectangle, and the latter is displayed as a dashed-line rectangle. Furthermore, in the example of Fig. 9, the ship information 141a, 142a corresponding to the image areas 141, 142, and the ship information 143a corresponding to the image area 143 are displayed in different display modes (different colors in Fig. 9).

[0084] By performing such processing, it becomes possible to notify the ship monitoring operator of the ship recognition results and distance measurement results, and to make the monitoring operator aware of differences in the distance measurement accuracy of those distance measurement results.

[0085] 8, the image area and ship information may be displayed in a display mode different from that of the ships whose distance measurement results are determined in steps S15 and S18. However, since the distance measurement result in step S17 is as high as the distance measurement result in step S15, the ships whose distance measurement results are determined in steps S15 and S17 may be displayed in the same display mode.

[0086] Next, a case where the position of the ship calculated based on the distance measurement result is displayed on a map (nautical chart) will be described with reference to Figs. Fig. 10 is a diagram showing a second example of the image display process. In the example of Fig. 10, a position image 151 showing the position of the ship itself and position images 152 and 153 showing the positions of other ships recognized from the captured image are displayed on a map 150.

[0087] The absolute coordinates indicating the position of the ship are determined, for example, from a GPS signal received by a GPS receiver equipped on the ship. The absolute coordinates indicating the position of the other ship are determined based on at least the distance measurement results output from the distance measurement processing unit 134. More specifically, the absolute coordinates indicating the position of the other ship are calculated, for example, based on the distance measurement results output from the distance measurement processing unit 134, the direction measurement results, and the ship's position.

[0088] Here, it is assumed that the distance measurement result of the ship corresponding to the position image 152 is determined based on radar information in step S14 of Fig. 8. Meanwhile, it is assumed that the distance measurement result of the ship corresponding to the position image 153 is determined based on a camera image in step S18 of Fig. 8. In this case, the display processing unit 135 displays the information indicating the former position and the information indicating the latter position on the map 150 in different display modes. In particular, in this embodiment, a display mode is used that allows visual understanding that the latter has a lower position identification accuracy.

[0089] In the example of Fig. 10, for the former, the position is indicated only by a position image 152 which is an elliptical dot centered on the position of the other ship. On the other hand, for the latter, which has a lower identification accuracy, the position of the ship is indicated by a circle 153a centered on the ship's position in addition to a position image 153 which is an elliptical dot centered on the position of the other ship. This visually indicates that the latter is a position identified based on a distance measurement result with lower accuracy, and enables the monitoring operator to intuitively understand such differences in accuracy.

[0090] Note that, since a distance measurement error may occur for the ship corresponding to the position image 152, an error may also occur in the position based on the distance measurement result. For this reason, a circle centered on the position of the ship may also be displayed around the position image 152. In this case, the circle displayed around the position image 152 and the circle 153a displayed around the position image 153 are displayed with different sizes (diameters). For example, since the position indicated by the position image 152 is calculated with higher accuracy, the circle displayed around the position image 152 is displayed with a smaller size than the circle 153a displayed around the position image 153.

[0091] 10, the thick curves connecting the position images 152 and 153 indicate the trajectory of the corresponding other ship up to the present. Such curves indicating the trajectory are generated based on the tracking information stored in the tracking information storage unit 124.

[0092] Fig. 11 is a diagram showing a third example of the image display process, in which the same components as those in Fig. 10 are denoted by the same reference numerals. The map 160 shown in Fig. 11 displays the predicted route of the other ship from the present time in addition to the display contents of the map 150 in Fig. 10. Such a predicted route is generated based on the tracking information of the corresponding ship stored in the tracking information storage unit 124.

[0093] Since errors occur in the predicted route, the range of predicted positions of other ships on the predicted route at each fixed time is represented by a circle. In the example of Fig. 11, the range of positions of other ships on the predicted route corresponding to position image 152 is represented by circles 152a and 152b. Moreover, the range of positions of other ships on the predicted route corresponding to position image 153 is represented by circles 153b and 153c.

[0094] Here, the range of predicted positions at the same time is displayed in a manner that allows the difference in the identification accuracy of the current position (i.e., the accuracy of the distance measurement used to identify it) to be visually understood. Specifically, the predicted position of another ship using the distance measurement results using radar is displayed as a circle with a larger diameter than the predicted position of another ship using the distance measurement results based on camera images. For example, circle 153b is displayed as a circle larger than circle 152a, and circle 153c is displayed as a circle larger than circle 152b. This allows the operator to intuitively understand the difference in the identification accuracy of the predicted positions.

[0095] Third embodiment The processing procedure of the monitoring processing device 100 according to the second embodiment can be modified as follows.

[0096] As mentioned above, the accuracy of distance measurement of a ship based on camera images decreases as the distance from the ship increases. Conversely, the accuracy of distance measurement of a ship using radar is low in areas where the distance is relatively close. As an example, when the resolution of the camera 101 is 7680 x 4320 pixels, the accuracy of distance measurement based on camera images is clearly low in areas where the distance from the ship exceeds 6 nautical miles, whereas the accuracy of distance measurement of a ship using radar is sufficiently high. Conversely, in areas where the distance from the ship is 6 nautical miles or less, the accuracy of distance measurement based on camera images is high enough for practical use.

[0097] Therefore, the monitoring processing device 100 according to the third embodiment uses the distance measurement results using the radar only when there is a high possibility that the distance to the ship is more than 6 nautical miles. This allows the distance measurement results using the radar to be used only when necessary, making it possible to reduce the processing load for determining the distance measurement results.

[0098] Fig. 12 is an example of a flowchart showing the processing procedure of recognition and ranging by the monitoring processing device according to the third embodiment. In Fig. 12, the same steps as those in Fig. 8 are denoted by the same step numbers.

[0099] In FIG. 12, the determination process of step S21 is executed after step S11, and if the determination result is Yes, the process of step S12 is executed, and if the determination result is No, the process of step S22 is executed.

[0100] [Step S21] The distance measurement processing unit 134 acquires a distance measurement result based on the camera image from the image recognition information acquired in step S11, and determines whether the acquired distance exceeds a predetermined threshold (here, 6 nautical miles). If the distance exceeds the threshold, the process proceeds to step S12. If the distance is equal to or less than the threshold, the process proceeds to step S22.

[0101] [Step S22] The distance measurement processing unit 134 outputs the distance measurement result obtained from the image recognition information in step S21 to the display processing unit 135 as the distance measurement result for the corresponding ship. In other words, if the result is No in step S21, the matching process for the same ship using the direction measurement results is not performed, and the distance measurement result is determined using only the measurement results based on the camera image. This simplifies the processing procedure and reduces the processing load. When the processing of step S22 is completed, the process proceeds to step S19.

[0102] Here, the accuracy of the distance measurement result in step S22 can be considered to be as high as the accuracy of the distance measurement result in step S15. For this reason, in the image display process in step S19, for example, an image related to a ship corresponding to the distance measurement result determined in step S15 and an image related to a ship corresponding to the distance measurement result determined in step S22 may be displayed in the same display mode.

[0103] The processing functions of the devices (e.g., the information processing device 3, the monitoring processing device 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical disks, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical disks include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0104] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and then sold. The program can also be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0105] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from a portable recording medium and execute processing according to the program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0106] 1 Camera 2. Marine Measuring Equipment 3. Information processing equipment S1~S4 steps

Claims

1. A camera for taking images; A vessel measuring device that detects vessels around the photographing position of the camera using the results of receiving radar or AIS (Automatic Identification System) information and measures the direction and distance of the detected vessels; measuring a first direction indicating a direction from the photographing position to a first ship recognized from the photographed image based on the photographed image by the camera; If a second ship is present among the ships detected by the ship measuring device and the measured direction thereof coincides with the first direction, the second ship is determined to be a ship corresponding to the first ship, a distance measurement result for the first ship is determined based on the distance to the second ship measured by the ship measuring device, a first position of the first ship is identified based on the distance measurement result, and the first position is displayed on a map as information indicating an area centered on the first position, an information processing device which, if there is no ship detected by the ship measuring instrument whose directional measurement value coincides with the first direction, determines the distance measurement result for the first ship based on the distance to the first ship measured from the captured image, identifies a second position of the first ship based on the distance measurement result, and displays the second position on the map as information indicating an area centered on the second position and larger than the area indicating the first position; and A ship monitoring system having:

2. When the distance of the first ship measured from the captured image exceeds a predetermined distance, the information processing device determines whether or not the second ship, whose directional measurement value coincides with the first direction, is present among the ships detected by the ship measuring device, and when the second ship is present, determines the distance measurement result for the first ship based on the distance of the second ship measured by the ship measuring device. The vessel monitoring system according to claim 1.

3. the predetermined distance is six nautical miles; The vessel monitoring system according to claim 2.

4. The computer Based on an image captured by a camera, a first direction indicating a direction from a photographing position of the camera to a first ship recognized from the photographed image is measured; If a second ship is present among the ships detected around the photographing position by a ship measuring device using the reception results of radar or AIS information, the second ship is determined to be a ship corresponding to the first ship, a ranging result for the first ship is determined based on the distance to the second ship measured by the ship measuring device, a first position of the first ship is identified based on the ranging result, and the first position is displayed on a map as information indicating an area centered on the first position, if there is no ship whose direction measurement value by the ship measuring device coincides with the first direction among the ships detected around the photographing position by the ship measuring device, a distance measurement result for the first ship is determined based on the distance to the first ship measured from the photographed image, a second position of the first ship is identified based on the distance measurement result, and the second position is displayed on the map as information indicating an area centered on the second position and larger than the area indicating the first position. Distance measurement method.

5. On the computer, Based on an image captured by a camera, a first direction indicating a direction from a photographing position of the camera to a first ship recognized from the photographed image is measured; If a second ship is present among the ships detected around the photographing position by a ship measuring device using the reception results of radar or AIS information, the second ship is determined to be a ship corresponding to the first ship, a ranging result for the first ship is determined based on the distance to the second ship measured by the ship measuring device, a first position of the first ship is identified based on the ranging result, and the first position is displayed on a map as information indicating an area centered on the first position, if there is no ship whose direction measurement value by the ship measuring device coincides with the first direction among the ships detected around the photographing position by the ship measuring device, a distance measurement result for the first ship is determined based on the distance to the first ship measured from the photographed image, a second position of the first ship is identified based on the distance measurement result, and the second position is displayed on the map as information indicating an area centered on the second position and larger than the area indicating the first position. A ranging program that performs the processing.

6. The computer When displaying the position of a ship on a map based on images captured by a camera, When the distance from the photographing position by the camera to the ship is obtained by both measurement based on the photographed image and measurement by a ship measuring device using the reception results of radar or AIS information, a first position of the ship is identified based on the distance measurement results by the ship measuring device, and the first position is displayed on the map as information indicating an area centered on the first position, when the distance from the photographing position to the ship has been obtained by measurement based on the photographed image but not by measurement using the ship measuring instrument, a second position of the ship is identified based on the distance measurement result based on the photographed image, and the second position is displayed on the map as information indicating an area centered on the second position and larger than the area indicating the first position. Display control method.

7. On the computer, When displaying the position of a ship on a map based on images captured by a camera, When the distance from the photographing position by the camera to the ship is obtained by both measurement based on the photographed image and measurement by a ship measuring device using the reception results of radar or AIS information, a first position of the ship is identified based on the distance measurement results by the ship measuring device, and the first position is displayed on the map as information indicating an area centered on the first position, when the distance from the photographing position to the ship has been obtained by measurement based on the photographed image but not by measurement using the ship measuring instrument, a second position of the ship is identified based on the distance measurement result based on the photographed image, and the second position is displayed on the map as information indicating an area centered on the second position and larger than the area indicating the first position. A display control program that executes the processing.

Images