Navigation support equipment, vessels, navigation support methods, and navigation support programs

Abstract

Using images acquired by a visible light camera, it becomes possible to detect the position of the horizon even at night based on the identification results of other ships' lights. [Solution] The navigation support device for assisting the navigation of a ship includes a light detection unit. The light detection unit detects lights based on color information contained in an image acquired by a visible light camera.

Description

Technical Field

[0001] The present invention relates to a navigation support device, a ship, a navigation support method, and a navigation support program.

Background Art

[0002] Conventionally, a method has been proposed for detecting the position of a horizon using the luminance values of each pixel included in a camera image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in an image taken during the day, the empty area in the image is brighter (has a higher luminance) than the sea area. Therefore, the position of the horizon can be detected based on the luminance change at the boundary between the empty area and the sea area. On the other hand, in an image taken at night, the entire image is dark, and there is little luminance change at the boundary between the empty area and the sea area in the image. Therefore, it is difficult to detect the position of the horizon based on the luminance change. That is, the conventional method of detecting the position of the horizon based on the luminance change is a method limited to daytime and is difficult to apply at night. Therefore, it is desired to realize a method capable of detecting the position of the horizon even at night. Moreover, for example, an infrared camera is more expensive than a visible light camera. Therefore, it is desired to realize a method capable of identifying the lights of other ships in order to detect the position of the horizon without using such an expensive camera, and to use the identification result to solve the difficulty of detecting the horizon at night.

[0005] The present invention was made to solve the above-mentioned problems, and its main purpose is to provide a navigation support device, a vessel, a navigation support method, and a navigation support program that can identify the lights of other vessels even at night using images acquired by a visible light camera, and can detect the position of the horizon based on the identification result. [Means for solving the problem]

[0006] One aspect of the present invention relates to a navigation support device that assists the navigation of a ship, and includes a light detection unit that detects lights based on color information contained in an image acquired by a visible light camera.

[0007] A vessel according to another aspect of the present invention is equipped with the above-described navigation support device.

[0008] A navigation assistance method relating to yet another aspect of the present invention is a navigation assistance method for assisting the navigation of a vessel, which includes detecting lights based on color information contained in an image acquired by a visible light camera.

[0009] A navigation support program according to yet another aspect of the present invention is a program for causing a computer to execute the above-described navigation support method. [Effects of the Invention]

[0010] With the above configuration, the position of the horizon can be detected even at night based on the identification results of other ships' lights, using images acquired by a visible light camera. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the schematic configuration of a ship equipped with a navigation support device according to one embodiment of the present invention. [Figure 2] This flowchart shows the processing flow based on the navigation support method. [Figure 3A] This is an explanatory diagram showing an example of an image acquired by a visible light camera during the daytime and displayed on the display unit. [Figure 3B] It is an explanatory diagram showing an example of a histogram created by the day / night determination unit. [Figure 4] It is a flowchart showing the flow of the light detection process. [Figure 5A] It is a flowchart showing the flow of processing on the IMU side for the horizontal line detection process at night. [Figure 5B] It is an explanatory diagram schematically showing an image acquired by the visible light camera positioned horizontally. [Figure 5C] It is an explanatory diagram schematically showing an image acquired by the visible light camera tilted downward. [Figure 6A] It is a flowchart showing the flow of processing on the navigation support device side for the horizontal line detection process at night. [Figure 6B] It is an explanatory diagram schematically showing a camera image in which a plurality of lights exist. [Figure 6C] It is an explanatory diagram showing a regression line (horizontal line) obtained by the least squares method. [Figure 7] It is an explanatory diagram showing the above regression line and the line estimated by the IMU as the horizontal line together. [Figure 8A] It is a flowchart showing the flow of the ship detection process. [Figure 8B] It is an explanatory diagram schematically showing an example of a plurality of partial images cut out from an image. [Figure 8C] It is an explanatory diagram schematically showing the area where the ship detected from the partial image by the ship detection unit is located. [Figure 9] It is an explanatory diagram showing an example of position information of the detected ship on the image. [Figure 10] It is an explanatory diagram showing an example of the display screen of the display unit on which the horizontal line and the distance etc. are displayed. [Figure 11] It is an explanatory diagram schematically showing the relationship between the camera position and the height of a person as an object. [Figure 12] It is an explanatory diagram schematically showing the positional relationship between the upper limit position and the lower limit position of the horizontal line preset in the image and the detected horizontal line. [Figure 13A]Explanatory diagrams showing the front camera image and the side camera image in the normal state where the above ship has no sway and in the state where the bow side is tilted downward, respectively. [Figure 13B] Explanatory diagram schematically showing the distance from a person to the horizon when a person is standing on the earth. [Figure 14A] Explanatory diagram schematically showing the image before correcting the detected horizon. [Figure 14B] Explanatory diagram schematically showing the image after correcting the detected horizon.

Mode for Carrying Out the Invention

[0012] The embodiments of the present invention will be described as follows based on the drawings.

[0013] 〔1. Outline of the Ship〕 FIG. 1 is a block diagram showing a schematic configuration of a ship 100 according to the present embodiment. The ship 100 includes a navigation support device 1. The navigation support device 1 is provided to support the navigation of the ship 100. In particular, the navigation support device 1 detects the horizon based on the camera image regardless of day or night (constantly for 24 hours), and detects other ships sailing off the shore by watching the vicinity of the detected horizon. Thereby, the operator can predict the risk of collision with other ships and determine the course of the own ship (ship 100) by looking at the positions of other ships.

[0014] Such a navigation support device 1 may be provided integrally with the ship 100, or may be provided separately from the ship 100. In the latter case, the navigation support device 1 may be composed of a portable communication terminal and configured to communicate with the ship 100 by wire or wirelessly. As the above communication terminal, a notebook personal computer, a tablet, etc. can be used. The details of the navigation support device 1 will be described later.

[0015] In this embodiment, the type of vessel 100 is not particularly limited. The vessel 100 may be, for example, a cargo ship, a fishing boat, a sightseeing boat, a passenger ship, etc. Also, the vessel 100 may be an ocean-going vessel or a domestic vessel. Here, an ocean-going vessel refers to a vessel that operates on foreign routes. A domestic vessel refers to a vessel that operates only on domestic routes. Whether it is an ocean-going vessel or a domestic vessel, the navigation support device 1 of this embodiment can be applied to a vessel 100 that is navigating offshore away from the coast.

[0016] [2. Regarding the configuration of a ship other than navigation support systems] The vessel 100 is equipped with a sensor unit 2. The sensor unit 2 includes a visible light camera 22, a 3D-LiDAR (3D-Light Detection and Ranging) 23, a Radar (Radio Detecting and Ranging) 24, a GNSS (Global Navigation Satellite System) device 25, an IMU (Inertial Measurement Unit) 26, an AIS (Automatic Identification System) 27, a wind direction and speed meter 28, and a rain gauge 29.

[0017] The visible light camera 22 is configured, for example, as a multi-camera array. The multi-camera array is configured by arranging nine cameras with narrow field of view (e.g., field of view of 15° to 20°) (e.g., telephoto cameras) in the circumferential direction. This makes it possible to acquire images in a range of 90° in the left and right directions (a total of 180° in the circumferential direction) centered on the front. Note that the number of cameras constituting the multi-camera array is not limited to the nine mentioned above and can be set as appropriate. Note that the captured images may be still images obtained by shooting at a predetermined period, or they may be videos obtained by continuous shooting.

[0018] The 3D-LiDAR23 detects the presence or absence of surrounding objects (including obstacles) by irradiating them with pulsed light and using the reflected light. If an object is present in the vicinity, the 3D-LiDAR23 detects the orientation and distance of the object based on the direction of the pulsed light when the reflected light is received and the time until the light is received. The 3D-LiDAR23 is composed of a 3D LiDAR that performs angular scanning in the yaw direction (left-right azimuth angle direction) and the pitch direction (front-back tilt angle direction). Therefore, if an object is present in the vicinity, the 3D-LiDAR23 outputs 3D point cloud data representing the object.

[0019] Radar24 uses radio waves with wavelengths longer than visible light to detect surrounding objects and measure the distance to them. Specifically, Radar24 measures the distance to an object based on the time it takes to receive the reflected radio waves after irradiating it with radio waves.

[0020] The 3D-LiDAR23 mentioned above uses electromagnetic waves with much shorter wavelengths than the radio waves used by Radar24. For example, ultraviolet light, visible light, and near-infrared light are used. Therefore, 3D-LiDAR23 has a narrower object detection range than Radar24, but it can detect surrounding objects with high resolution. Conversely, Radar24 can detect surrounding objects coarsely over a wide area.

[0021] The GNSS device 25 receives GNSS radio waves from satellites and obtains information on the current position of the ship 100 by performing known positioning calculations. GNSS positioning can be performed using standalone positioning, but by further utilizing RTK (Real-Time Kinematic) positioning, the position information of the ship 100 can be obtained with high accuracy.

[0022] The IMU26 is an inertial measurement device equipped with a 3-axis gyro sensor and a 3-directional accelerometer. By detecting three-dimensional angular velocity and acceleration, the IMU26 can detect the attitude information of the vessel 100. This attitude information includes the position information of the vessel 100 in the yaw direction, pitch direction, and roll direction (left and right tilt angle direction).

[0023] In this embodiment, when the nine cameras constituting the visible light camera 22 are divided into three sets of three, one IMU 26 is provided for each set. In other words, a total of three IMUs 26 are provided. Each IMU 26 can detect the attitude of the camera as well as the attitude of the ship 100. Therefore, the IMU 26 constitutes an attitude detection unit for detecting the attitude of the visible light camera 22.

[0024] AIS27 is an Automatic Identification System for Ships. More specifically, AIS27 automatically transmits and receives information about the ship 100, including its identification code, type, position, course, speed, navigation status, and other safety-related information, using VHF (Very High Frequency) radio waves. This allows for the exchange of information between ship stations and between ship stations and shore-based navigation aids. An anemometer 28 is installed on board the ship 100 to measure wind direction and speed. A rain gauge 29 is installed on board the ship 100 to measure rainfall.

[0025] The vessel 100 is further equipped with a database 3. Database 3 stores various types of information. These types of information include route information, nautical chart information, weather information, etc. Database 3 is composed of a server computer equipped with storage devices such as hard disks, optical disks, and non-volatile memory, but it may also be composed of a cloud server that exists virtually on the internet.

[0026] The vessel 100 further comprises an operating unit 4 and an actuator 5. The operating unit 4 includes, for example, an operating lever and an operating button. The operator can operate the actuator 5 by operating the operating unit 4.

[0027] Actuator 5 comprises a right-side actuator 5R and a left-side actuator 5L. The right-side actuator 5R is located on the right stern of the vessel 100. The left-side actuator 5L is located on the left stern of the vessel 100.

[0028] The right-side actuator 5R and the left-side actuator 5L constitute a propulsion system in which an engine installed inside the ship is directly connected to a drive unit installed outside the ship. Such a propulsion system is, for example, a stern drive (inboard / outboard engine). The right-side actuator 5R and the left-side actuator 5L are also provided with a rotation mechanism that moves the drive unit to change the direction of propulsion. In this embodiment, two actuators 5 are provided on the left and right sides of the ship 100, but the number of actuators 5 is not particularly limited and may be one or three or more. Furthermore, the actuator 5 may be configured to have a rudder behind the propeller driven by the engine.

[0029] [3. Configuration of the Navigation Support System] The navigation support device 1 comprises an acquisition unit 11, a display unit 12, a storage unit 13, and a control unit 14.

[0030] The acquisition unit 11 includes a communication unit 11a and an input unit 11b. The communication unit 11a is an interface for communicating with the outside. The communication may be wired or wireless. Therefore, the communication unit 11a may include a connector to which a communication cable is connected in case of wired communication. The communication unit 11a may also include an antenna, a transmitting / receiving unit, a modulation circuit, a demodulation circuit, etc., in case of wireless communication.

[0031] In this embodiment, the communication unit 11a is connected to the sensor unit 2 and the database 3 in a communicative manner. As a result, information detected by the sensor unit 2 and information recorded in the database 3 can be input to the navigation support device 1 via the communication unit 11a.

[0032] The input unit 11b accepts user-specified input (hereinafter simply referred to as "input"). Such an input unit 11b may consist of, for example, a touch panel, a mouse, or a keyboard. If the input unit 11b consists of a touch panel, it may be positioned in front of the display unit 12 and integrated with the display unit 12. When the navigation support device 1 is installed on a ship 100, the operator of the ship may be the user described above.

[0033] The display unit 12 is a display (monitor) that displays various types of information, and is composed of, for example, a liquid crystal display device. In particular, in this embodiment, the display unit 12 displays images acquired by the visible light camera 22 described above.

[0034] The storage unit 13 is a memory that stores various types of information and is composed of RAM (Random Access Memory), ROM (Read Only Memory), hard disk, optical disk, non-volatile memory, etc. The storage unit 13 stores the operation program of the control unit 14, data obtained by processing in the control unit 14, and data acquired via the acquisition unit 11 (for example, data of images captured by the visible light camera 22).

[0035] The control unit 14 is configured to include at least one of a central processing unit called a CPU (Central Processing Unit) and a high-speed processing unit called a GPU (Graphics Processing Unit). The control unit 14 operates according to the operation program stored in the memory unit 13. The control unit 14 comprises a main control unit 141, a day / night determination unit 142, a light detection unit 143, a horizon detection unit 144, a partial image extraction unit 145, a ship detection unit 146, a distance estimation unit 147, a judgment unit 148, and a calculation unit 149. Note that any of the main control unit 141, day / night determination unit 142, light detection unit 143, horizon detection unit 144, partial image extraction unit 145, ship detection unit 146, distance estimation unit 147, judgment unit 148, and calculation unit 149 may be configured as a separate CPU or GPU from the control unit 14 (they may be provided outside the control unit 14).

[0036] The main control unit 141 controls the operation of each part of the navigation support device 1. For example, the main control unit 141 controls the display of information on the display unit 12. The main control unit 141 also controls the actuators 5 (right actuator 5R, left actuator 5L) based on the operation of the operation unit 4. Functions such as the day / night determination unit 142 will be explained together in the operation description below.

[0037] [4. Regarding navigation support methods] Next, the navigation support method of this embodiment will be described. The navigation support method of this embodiment is a method for supporting the navigation of a vessel 100 and is executed by the navigation support device 1 described above. Figure 2 is a flowchart showing the processing flow according to the navigation support method of this embodiment. In this embodiment, by performing processing according to the flowchart in Figure 2, the horizon is detected day and night, and other vessels are detected based on the detected horizon.

[0038] (4-1. Image acquisition, day / night determination processing) When the acquisition unit 11 of the navigation support device 1 acquires image data from the visible light camera 22 (S1), the day / night determination unit 142 determines whether it is day or night based on the color information contained in the image (S2). Here, the color information refers to the image data of each pixel contained in the color image (original image) acquired by the visible light camera 22, specifically the red (R), green (G), and blue (B) image data. More specifically, it refers to the image data after gamma correction (brightness correction) has been applied to the RGB image data.

[0039] Figure 3A shows an example of an image acquired by a visible light camera 22 during the daytime, capturing the area in front of the vessel 100, and displayed on the display unit 12. The image includes a bow image 100a, which is an image of the bow portion of the vessel, an air area RA, and a sea area RS. The boundary between the air area RA and the sea area RS is indicated by the horizon line HL. The day / night determination unit 142 obtains the brightness value of each pixel from the color information of each pixel constituting the image in Figure 3A, and creates a histogram showing the relationship between the brightness value and the frequency (number of pixels). Figure 3B is an example of a histogram created by the day / night determination unit 142 based on the color information.

[0040] Here, the luminance value Y on the horizontal axis is calculated based on the following equation (A). Note that RGB in equation (A) is, for example, 8-bit data ranging from 0 (dark) to 255 (bright). Also, the values ​​of coefficients K1 to K3 in equation (A) are examples only and are not limited to these values. Y = K1·R + K2·G + K3·B ... (A) However, K1=0.299, K2=0.587, K3=0.114

[0041] The day / night determination unit 142 compares the total number of pixels with a brightness value Y equal to or greater than the threshold t (total number on the high-brightness side) with the total number of pixels with a brightness value Y less than the threshold t (total number on the low-brightness side). The threshold t can be set arbitrarily; for example, it could be 128, which is an intermediate value between 0 and 255. The day / night determination unit 142 determines that it was daytime when the image was acquired (when it was taken) if the total number on the high-brightness side is equal to or greater than the total number on the low-brightness side, and determines that it was nighttime when the image was acquired if the total number on the high-brightness side is less than the total number on the low-brightness side. In the equation showing the determination process in Figure 3B, the left side represents the total number on the low-brightness side, and the right side represents the total number on the high-brightness side.

[0042] (4-2. Daytime horizon detection processing) In the day / night determination process at S2 in Figure 2, if it is determined that the image was captured during the daytime, the horizon detection unit 144 performs daytime horizon detection processing (S3). For example, by configuring the horizon detection unit 144 with a horizon detector capable of machine learning using deep learning, the horizon can be detected from daytime images. A technique for performing horizon detection using deep learning is disclosed, for example, in document A, “Vision-Based Maritime Object Detection Covering Far and Tiny Obstacles”, Ryota Yoneyama, Yuichiro Dake, IFAC-PapersOnLine, Volume 55, Issue 31, 2022, Pages 210-215. In other words, the known technique disclosed in document A can be used for daytime horizon detection. Note that during the day, the brightness changes rapidly at the boundary between the sky region RA and the sea region RS. Therefore, during the day, the horizon detection unit 144 may determine the horizon by detecting the region where the brightness changes rapidly based on the brightness value of each pixel.

[0043] (4-3. Light detection process) In the day / night determination process in S2, if it is determined that the image was captured at night, the light detection unit 143 performs light detection processing (S4). Figure 4 is a flowchart showing the flow of the light detection process. The light detection unit 143 detects lights based on the color information contained in the image acquired by the visible light camera 22. In the light detection process, unlike the day / night determination process, the RGB image data of the original image, which has not undergone brightness correction by gamma correction, is used as the color information. The details of the light detection process will be explained below based on Figure 4.

[0044] When the acquisition unit 11 acquires image data of the original image from the visible light camera 22 (S41), the light detection unit 143 extracts hue information from the image data (S42). In other words, the light detection unit 143 extracts each RGB image data from the image data of the original image. Generally, there are three types of lights that ships illuminate at night: port lights (red), starboard lights (green), and mast lights (white). By acquiring each RGB image data from the original image as described above (especially by distinguishing and acquiring R and G image data), it becomes possible to reliably detect the lights of a ship.

[0045] Next, the light detection unit 143 performs noise reduction processing on the hue information (RGB image data) acquired in S42 (S43). For example, in the RGB image data, image data with extremely high numerical values ​​may be noise. By performing processing to remove image data above a predetermined value as noise, the light detection unit 143 can perform light detection with high accuracy.

[0046] Next, the light detection unit 143 converts each RGB image data, from which noise has been removed in S43, into grayscale (S44). This grayscale conversion can be performed, for example, by calculating the brightness value Y of each pixel using equation (A), which was used in the day / night determination process in S2 of Figure 2.

[0047] Next, the light detection unit 143 binarizes the image data (luminance data) of each pixel, which was converted to grayscale in S44, using a threshold (S45). For example, the light detection unit 143 sets image data above the threshold to "1" and image data below the threshold to "0".

[0048] Subsequently, the light detection unit 143 performs known shrinkage and dilation processing on the image having binarized image data (S46). This further reduces the noise in the image. Shrinkage processing is a process that converts the image data of the pixel of interest to black data "0" if there is at least one black pixel (image data "0") among the surrounding pixels (e.g., 8 pixels) of the pixel of interest. On the other hand, dilation processing is a process that converts the image data of the pixel of interest to white data "1" if there is at least one white pixel (image data "1") among the surrounding pixels of the pixel of interest.

[0049] Next, the light detection unit 143 performs a known contour extraction process on the image that has undergone shrinkage and expansion processing in S46 (S47). Through the contour extraction process, the boundary lines representing the shape of the object (in this case, the region showing the light) contained in the image are extracted as the contour of the object. Finally, the light detection unit 143 calculates and obtains the centroid of the object from the coordinates of each pixel that constitutes the contour of the object extracted in S47 (S48). The obtained centroid position (position coordinates) will point to the position of the light. In this way, the light contained in the image is detected.

[0050] (4-4. Nighttime horizon detection processing) After the light detection process in S4 of Figure 2 is performed, the horizon detection unit 144 performs the nighttime horizon detection process (S5). Figure 5A is a flowchart showing the nighttime horizon detection process and the flow of processing on the IMU 26 side (see Figure 1). When the IMU 26 acquires attitude information from the visible light camera 22 (S511), it estimates the position of the horizon based on the attitude information (S512).

[0051] For example, Figure 5B schematically shows a side view of the visible light camera 22 and the image CA (also referred to as the camera image) acquired by the visible light camera 22 when the visible light camera 22 is positioned horizontally. As shown in Figure 5B, when the visible light camera 22 is positioned horizontally, the horizontal line HL is located at the vertical center position h0 (passing through the vanishing point VP) in the camera image.

[0052] Figure 5C schematically shows a side view of the visible light camera 22 and a camera image when the visible light camera 22 tilts downward due to the rocking of the ship 100. The amount of upward displacement ΔH (corresponding to the pixel size × number of pixels) from the center position h0 of the horizontal line HL in the camera image when the visible light camera 22 is tilted downward by a predetermined angle is known in advance. Therefore, the IMU 26 can estimate the position of the horizontal line HL in the camera image from the acquired camera image (from the attitude of the visible light camera 22).

[0053] In S512, the position information of the horizon HL estimated by IMU26 is output from IMU26 to the navigation support device 1 (S513). The above process is repeated until monitoring (data acquisition) is completed (S514).

[0054] Figure 6A is a flowchart showing the processing flow on the navigation support device 1 side for nighttime horizon detection processing. The horizon detection unit 144 determines whether there are two or more lights detected in the light detection processing at S4 in Figure 2 within a single camera image (S521). If there are two or more lights at S521, the horizon detection unit 144 solves a minimization problem from the coordinates (coordinates of the centroid) of the multiple lights to estimate a straight line L0 that can approximate the horizon HL (S522). For example, as shown in Figure 6B, if there are multiple lights LP in the camera image, the horizon detection unit 144 uses the least squares method to find a regression line (=straight line L0) that passes through the vicinity of each light LP, as shown in Figure 6C. The horizon detection unit 144 then uses the obtained straight line L0 as information for the horizon HL (S523).

[0055] On the other hand, in S521, if there are fewer than two lights, the line L0 cannot be determined by the least squares method (because the line L0 cannot be identified), so the horizontal line detection unit 144 uses the position information output by the IMU 26 in the processing shown in Figure 5A (S513) as the information for the horizontal line HL (S524).

[0056] As shown in Figure 7, if there is a large difference between the line L0 (see solid line) obtained in S522 and the line L1 (see dashed line) corresponding to the position information output by the IMU 26, the horizontal line detection unit 144 may not use the obtained line L0 and restart the horizontal line detection process from the beginning. Note that a large difference between the two lines L0 and L1 refers to a case where, when one line L0 is expressed as a linear function equation y = a1x + b1 and the other line L1 is expressed as a linear function equation y = a2x + b2, the difference between the slopes a1 and a2 is greater than or equal to a predetermined value, or the difference between the intercepts b1 and b2 is greater than or equal to a predetermined value.

[0057] (4-5. Ship detection process) When the horizon is detected in S3 or S5 in Figure 2, the partial image extraction unit 145 and the ship detection unit 146 perform ship detection processing (S6). In the ship detection processing, other ships located near the horizon are detected. This will be explained in more detail below.

[0058] Figure 8A is a flowchart showing the flow of the ship detection process. First, the partial image extraction unit 145 extracts multiple partial images CR from the image CA containing the lights LP, as shown in Figure 8B (S61). Here, a partial image CR refers to an image region in one image CA that includes the lights LP and overlaps with the horizontal line HL (straight line L0 or L1). In other words, the partial image extraction unit 145 extracts a partial image CR from the image CA containing the lights that includes the lights and overlaps with the horizontal line HL. Note that two adjacent partial images CR on the image CA are extracted so that their edges overlap. By extracting the image region that overlaps with the horizontal line HL, a partial image CR focusing only on the horizontal line portion is obtained.

[0059] Next, the ship detection unit 146 takes each partial image CR extracted by the partial image extraction unit 145 as input and detects ships SH from the input image (S62). In other words, when the extracted partial image CR is input, the ship detection unit 146 detects other ships included in the partial image CR. Such detection of other ships can be performed by configuring the ship detection unit 146 with a ship detector that has been pre-machine-trained using deep learning. In Figure 8C, the region where ships SH detected from the partial image CR by the ship detection unit 146 are located is schematically shown by a rectangular dashed line.

[0060] Here, detecting a ship SH (another ship) refers to obtaining the position information of the ship SH on the image CA. The above position information is, for example, as shown in Figure 9, the position information (coordinates (x1, y1)) of the upper left point and the position information (coordinates (x2, y2)) of the bounding box (Rectangle BB) surrounding the ship SH, with respect to an arbitrary point O1 on the image CA (for example, the upper left point of the image CA). Since the position (coordinates) of each sub-image CR on the image CA is known, when a ship SH is detected in any sub-image CR, the position of the ship SH with respect to point O1 on the image CA is uniquely determined. The above position information of the ship SH may also be the position coordinates of the center O2 of the rectangle BB with respect to the reference point O1, and the numerical values ​​of the vertical width H and horizontal width W of the rectangle BB.

[0061] Subsequently, the ship detection unit 146 transmits the detection result of the ship SH to the display unit 12, which then displays it (S63). Figure 10 shows an example of the display screen of the display unit 12. Figure 10 shows an example in which the display unit 12 displays together the light LP detected in S4 of Figure 2, the ship SH detected in S62 of Figure 8A, and the horizon line HL detected in S5 of Figure 2. In addition, to clearly distinguish between the ship SH and the light LP, the display unit 12 displays the first frame F1 surrounding the ship SH in, for example, green to highlight the ship SH, and displays the second frame F2 surrounding the light LP in, for example, red to highlight the light LP.

[0062] Since light detection processing is not performed during the daytime, if ship detection processing is to be performed during the daytime, the following should be done. In other words, the partial image extraction unit 145 should extract multiple partial images CR that overlap with the horizontal line HL (straight line L0 or L1) from the image CA. Then, the ship detection unit 146 should take each partial image CR extracted by the partial image extraction unit 145 as input and detect ships from the input image.

[0063] (4-6. Distance Estimation Process) Next, as shown in Figure 10, the distance estimation unit 147 estimates the distance D1 between the light LP and the vessel S0 (ship 100) based on the relative position of the light LP and the horizon HL (S7). At the same time, the distance estimation unit 147 estimates the distance D2 between the other vessel (ship SH) and the vessel S0 (ship 100) based on the relative position of the other vessel (ship SH) and the horizon HL. As for the specific estimation method of distances D1 and D2, for example, the technology described in the known document B "Hiroshi Ando, ​​Hiroyuki Fujiyoshi, Self-camera calibration and 3D position estimation based on human detection results, Transactions of the Institute of Electrical Engineers of Japan (Industrial Applications Division), 2011, Vol. 131, No. 4, pp. 482-489, Publication date 2011 / 04 / 01, Online ISSN 1348-8163" can be used.

[0064] A brief explanation of the distance estimation method using the technique described in reference B is as follows. Figure 11 schematically shows the relationship between the camera position and the height of the object, which is a person. The coordinates of the normalized image, in which the vertical and horizontal dimensions of the image are normalized by the vertical size, are given as (u,v). On the other hand, the world coordinate system is given as (x,y,z), where y is the height, z is the depth, and x is the direction perpendicular to the yz plane. Next, the tilt angle of the camera (angle of inclination from the horizontal plane) is θ, the focal length is f, the coordinates of the camera center are (uc,vc), and the height of the camera is yc. In the world coordinate system, the camera position is defined as zc=0, xc=0 as the reference, and the ground plane is defined as y=0. In addition, the horizontal line v0 is defined as the vanishing line of the ground in the image coordinate system.

[0065] The tilt angle θ of the camera is expressed by the following equation (1).

[0066]

number

[0067] The transformation from the world coordinate system to the image coordinate system is given by equation (2) below.

[0068]

number

[0069] Solving equation (2) for the height y of the object yields the following equation (3).

[0070]

number

[0071] Here, let vt be the upper base position of the person in the image, and vb be the lower base position. When the person in the image touches the ground at the lower base position vb, y=0. Therefore, the depth z of the object can be found by equation (4).

[0072]

number

[0073] By applying the above-mentioned object to the LP light or SH ship of this embodiment, the distances D1 and D2 described above can be estimated using equation (4). This makes it possible to present the estimated distances D1 and D2 to the operator.

[0074] When the distance estimation unit 147 estimates distances D1 and D2, the display unit 12 displays distances D1 and D2 on the screen, as shown in Figure 10. In Figure 10, the estimated distance D2 between the other vessel and the vessel S0 is displayed in green, for example, as "3.5km" and "2km". Also, the estimated distance D1 between the light LP and the vessel S0 is displayed in red, for example, as "5km", "4.5km", and "3.5km". In this way, when distances D1 and D2 are displayed on the display unit 12, the operator can immediately grasp the distance between the light LP, the other vessel, and the vessel S0.

[0075] Furthermore, if image CA is an image taken offshore, the light LP is highly likely to be a ship SH even if it is not clearly detected as a ship SH. On the other hand, if image CA is an image taken of a coastal area, the light LP could be a ship SH anchored along the coast, or it could be a streetlamp or lighthouse on land. In any case, the operator can look at the distances D1 and D2 displayed on the display unit 12 to judge the risk of collision with an object ahead and ensure the safety of their ship's navigation by changing their course as necessary.

[0076] The processes S1 to S7 in Figure 2 described above are carried out until the 24-hour monitoring is completed (S8). Therefore, the processes S1 to S7 are carried out repeatedly (continuously) while the vessel 100 is underway.

[0077] As described above, the light detection unit 143 detects lights LP from the image CA acquired by the visible light camera 22 (see S4 in Figure 2 and Figure 4). Furthermore, the navigation support method of this embodiment includes the detection of lights LP (by the light detection unit 143) based on the color information contained in the image CA acquired by the visible light camera 22 (S4 in Figure 2).

[0078] In this embodiment, the light detection unit 143 can identify and detect the lights LP of other vessels from the image CA acquired by the visible light camera 22, even at night. As a result, even at night, the horizon detection unit 144 can detect the horizon based on the identification result (detection result) of the detected lights LP (see S5 in Figure 2, Figure 5A, and Figure 6A). Therefore, at night, it is possible to detect ships SH (other vessels) from images near the horizon (see S6 in Figure 2 and Figure 8A). Furthermore, there is no need to use expensive cameras such as infrared cameras, and it is possible to detect ships SH at night with an inexpensive configuration using the visible light camera 22.

[0079] In order to facilitate the operator's recognition of the position of the lights LP by looking at the image CA displayed on the display unit 12, the following configuration is desirable. That is, as shown in Figure 10, it is desirable for the display unit 12 to highlight the lights LP in addition to the image CA. In particular, in order to easily realize the highlighting of the lights, it is desirable for the display unit 12 to display a frame (second frame F2) surrounding the lights LP, as shown in Figure 10.

[0080] Because it is dark at night, it is not possible to detect the horizon line HL from camera images using the same method as during the day. In order to enable the detection of the horizon line HL regardless of whether it is day or night, it is necessary to change the method of detecting the horizon line HL depending on whether it is day or night. To do this, it is necessary to determine whether it is day or night using camera images. In this respect, a configuration in which the navigation support device 1 is equipped with a day / night determination unit 142, as in this embodiment, is desirable.

[0081] In this embodiment, when there are multiple lights LP included in the image CA, the horizon detection unit 144 detects the horizon HL based on the distribution of the lights LP (see S522 in Figure 6A). The distribution of lights LP can be detected from the image CA acquired by the visible light camera 22. Therefore, there is no need to use special and expensive cameras such as infrared cameras to detect the horizon HL at night. In this respect, a configuration in which the navigation support device 1 is equipped with the horizon detection unit 144 is desirable.

[0082] The horizon line detection unit 144 detects the horizon line HL based on the attitude information of the visible light camera 22 input from an external source (e.g., IMU 26) when there is only one light source LP in the image CA (see S524 in Figures 5A and 6A). When there is only one light source in the image CA, it is not possible to detect the horizon line HL using the least squares method. In this case, the method of detecting the horizon line HL based on the attitude information of the visible light camera 22 becomes effective.

[0083] The display unit 12 displays the horizon line HL detected by the horizon line detection unit 144, superimposed on the image CA (see Figure 10). In this case, the operator can recognize the position of the horizon line HL on the image CA displayed on the display unit 12, even at night. In this respect, the display method of superimposing the horizon line HL on the image CA is desirable.

[0084] In order to reliably detect other ships by the ship detection unit 146, it is desirable to perform ship detection using the image information (image data of each pixel) contained in the original image CA as is. In this regard, as in this embodiment, it is desirable for the partial image extraction unit 145 to extract a portion (the area overlapping with the horizon line HL) from the original image CA as a partial image CR, and for the ship detection unit 146 to perform ship detection processing using the extracted partial image CR as input (see S61 and S62 in Figure 8A).

[0085] For example, when an original image is reduced in size, pixels are removed from the original image, resulting in a decrease in the amount of information contained in the original image. In contrast, with a partial image CR, pixels are not removed from the original image, and the information contained in the original image is retained as is. Therefore, by inputting an image with a large amount of information (partial image CR) to the ship detection unit 146, it becomes possible for the ship detection unit 146 to reliably (accurately) perform ship detection processing.

[0086] [5. Regarding the decision-making unit of the navigation support system] As shown in Figure 1, the navigation support device 1 may include a determination unit 148. As shown in Figure 12, the determination unit 148 determines whether the horizontal line HL detected by the horizontal line detection unit 144 is located between a preset upper limit position HL-t and a lower limit position HL-b in the image CA.

[0087] The vessel 100 has a limit angle at which it can withstand waves at sea. When the vessel 100 is rocked by waves and its bow tilts upward, the position of the horizontal line HL detected when the tilt of the vessel 100 reaches the upper limit of the limit angle is defined as the lower limit position HL-b. Also, when the vessel 100 is rocked by waves and its bow tilts downward, the position of the horizontal line HL detected when the tilt of the vessel 100 reaches the lower limit of the limit angle is defined as the upper limit position HL-t.

[0088] In image CA, it is inappropriate for the detected horizontal line HL to be located in an area above the upper limit position HL-t. This is because it would mean that the ship 100 is tilted beyond the limit angle. Similarly, in image CA, it is inappropriate for the detected horizontal line HL to be located in an area below the lower limit position HL-b.

[0089] The determination unit 148 determines whether the horizontal line HL is located between the preset upper limit position HL-t and lower limit position HL-b, thereby determining whether the horizontal line HL detected by the horizontal line detection unit 144 is appropriate, that is, whether the horizontal line HL has been detected correctly. If the horizontal line HL has not been detected correctly, the detection unit 144 can be restarted or take other appropriate action.

[0090] [6. Regarding the calculation unit of the navigation support system] As shown in Figure 1, the navigation support device 1 may include a calculation unit 149. The calculation unit 149 calculates the roll, pitch, and heave of the vessel (ship 100) based on the position of the horizontal line HL in the image CA.

[0091] Figure 13A shows the front camera image CA-f and side camera image CA-s for the vessel 100 in a normal state with no rolling and in a state where the bow is tilted downward. The front camera image CA-f is an image taken by the camera facing forward, one of the nine cameras that make up the visible light camera 22. The side camera image CA-s is an image taken by the camera facing starboard (or port) side, one of the nine cameras that make up the visible light camera 22.

[0092] Roll is represented by the inclination of the horizontal line HL, which is superimposed on the front camera image CA-f, with respect to the horizontal direction. Pitch is represented by the inclination (θp) of the horizontal line HL, which is superimposed on the side camera image CA-s, with respect to the horizontal direction. In other words, the calculation unit 149 can determine roll and pitch, respectively, from the inclination of the horizontal line HL, which is superimposed on the front camera image CA-f and the side camera image CA-s.

[0093] On the other hand, the heave (vertical displacement) can be calculated by the calculation unit 149 based on the technique described in the aforementioned known document B. The method for calculating the heave will be described below.

[0094] In equation (2) above, the heave corresponds to the camera's installation height yc. In other words, the heave can be calculated by determining yc.

[0095] The height y is expressed by equation (3) above. Expressing equation (3) in terms of yc, we obtain the following equation (5).

[0096]

number

[0097] Assuming the ship is horizontal (θ=0), we obtain the following equation (6).

[0098]

number

[0099] Furthermore, when the image coordinate v in equation (6) is the horizontal line, y=0. Therefore, if the distance z to the horizontal line can be determined, the heave can be found using the following equation (7).

[0100]

number

[0101] The distance z to the horizon can be calculated using the height of the viewpoint and the radius of the Earth. Figure 13B schematically shows the distance z (=distance AB) from a person to the horizon when a person of height h is standing on an Earth of radius R. Expressing the geometric relationship in Figure 13B with the variables used here yields the following equation (8).

[0102]

number

[0103] Here, yc 2 Since it is sufficiently small compared to the Earth's radius R, it can be ignored. Transforming equation (8) for yc, we obtain the following equation (9).

[0104]

number

[0105] Substituting equation (9) into equation (7) and solving for z, we obtain the following equation (10).

[0106]

number

[0107] By substituting equation (10) back into equation (7), we can find yc.

[0108] The heave when ship 100 is in motion is determined as follows. In equation (5), the value of θ is the pitch angle detected by IMU26. The image coordinate v is the value of the horizontal line HL (y=0) detected by the horizontal line detection process.

[0109] As described above, the calculation unit 149 calculates the ship's roll, pitch, and heave, and by displaying the calculated roll, etc., on the display unit 12, for example, the operator can understand the ship's attitude.

[0110] [7. Additional information] The horizon detection unit 144 may detect the horizon line HL based on the lights LP detected by the light detection unit 143 and the chart information stored in the database 3 (see Figure 1). The positions of land and lighthouses can be determined from the chart information. Therefore, the horizon detection unit 144 can, for example, remove the lights of land and lighthouses as noise from the lights LP detected by the light detection unit 143 and detect the horizon line HL from the remaining lights LP. Thus, the detection accuracy of the horizon line HL is improved.

[0111] Furthermore, the horizon detection unit 144 may detect the horizon line HL based on the lights LP detected by the light detection unit 143 and the position information of other vessels received periodically. The position information of other vessels is periodically received by the vessel as AIS information. Based on the received position information of other vessels, the horizon detection unit 144 can determine whether the lights LP included in the image CA are the lights of other vessels, in other words, whether they are not lights of land or lighthouses that would cause noise. If the lights LP are the lights of other vessels, the horizon line HL can be detected with high accuracy based on the lights LP.

[0112] The horizontal line detection unit 144 may adjust the horizontal line HL in the image CA downward by a predetermined amount. Here, Figure 14A schematically shows the image CA before the detected horizontal line HL is adjusted. Figure 14B schematically shows the image CA after the detected horizontal line HL has been adjusted.

[0113] Generally, at night, lights on ships are positioned above the waterline. Therefore, the horizon line HL detected based on the light LP in the image CA is likely to be located above the actual horizon. As shown in Figure 14B, by adjusting the horizon line HL downwards by a predetermined offset value t0 (a set amount) in the image CA, the position of the detected horizon line HL can be brought closer to the position of the actual horizon. In other words, to obtain the horizon line HL with greater accuracy, it is desirable to adjust the detected horizon line HL downwards by a predetermined amount as described above.

[0114] [8. Program] The navigation support device 1 described in this embodiment can be configured, for example, as a computer (PC) on which a predetermined program (application software) is installed. By reading and executing the above program, the computer (for example, the control unit 14) can operate each part of the navigation support device 1 to execute each of the processes (each step) described above. Such a program can be obtained, for example, by downloading it from an external source via a network and stored in the storage unit 13. Alternatively, the program may be recorded on a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) or a portable non-volatile memory, and the computer can read the program from this recording medium and store it in the storage unit 13. In other words, the program in this embodiment is a program that causes the computer to execute the navigation support method of this embodiment. The recording medium in this embodiment is a computer-readable, non-transient recording medium on which the above program is recorded.

[0115] [9. Addendum] The navigation support device, vessel, navigation support method, and navigation support program described in this embodiment can also be expressed as follows.

[0116] The navigation support equipment in Appendix (1) A navigation support system that assists the navigation of a ship, It includes a light detection unit that detects lights based on color information contained in images acquired by a visible light camera.

[0117] The navigation support system in Appendix (2) is the same as the navigation support system described in Appendix (1), The system further includes a display unit for displaying the aforementioned image, In addition to the image, the display unit emphasizes and displays the lights.

[0118] The navigation support system in Appendix (3) is the same as the navigation support system described in Appendix (2), The display unit displays a frame surrounding the light.

[0119] The navigation support system in Appendix (4) is the navigation support system described in any of Appendix (1) to (3), The system further includes a day / night determination unit that determines day or night based on the aforementioned color information.

[0120] The navigation support system in Appendix (5) is a navigation support system described in any of Appendix (2) to (4), If there are multiple lights in the image, the system further includes a horizontal line detection unit that detects a horizontal line based on the distribution of the lights.

[0121] The navigation support system in Appendix (6) is the navigation support system described in Appendix (5), The horizon detection unit detects the horizon based on the attitude information of the visible light camera input from an external source when there is only one light source included in the image.

[0122] The navigation support system in Appendix (7) is the navigation support system described in Appendix (5) or (6), The display unit displays the horizontal line detected by the horizontal line detection unit, superimposed on the image.

[0123] The navigation support system in Appendix (8) is the navigation support system described in any of Appendix (5) to (7), A partial image extraction unit extracts a portion of the image from the aforementioned image that includes the light and overlaps with the horizontal line, The system further includes a ship detection unit that detects other ships included in the extracted partial image when the partial image is input.

[0124] The navigation support system in Appendix (9) is a navigation support system described in any of Appendix (5) to (8), The system further includes a determination unit that determines whether the horizontal line detected by the horizontal line detection unit is located between a preset upper limit position and a preset lower limit position in the aforementioned image.

[0125] The navigation support device in Appendix (10) is the navigation support device described in any of Appendix (5) to (9), The system further includes a calculation unit that calculates the roll, pitch, and heave of the vessel based on the position of the horizontal line in the aforementioned image.

[0126] The navigation support device in Appendix (11) is a navigation support device described in any of Appendix (5) to (10), The horizon detection unit detects the horizon based on the lights and nautical chart information.

[0127] The navigation support device in Appendix (12) is a navigation support device described in any of Appendix (5) to (11), The horizon detection unit detects the horizon based on the lights and position information of other ships that is received periodically.

[0128] The navigation support device in Appendix (13) is a navigation support device described in any of Appendix (5) to (12), The horizontal line detection unit adjusts the horizontal line in the image downward by a predetermined amount.

[0129] The navigation support device in Appendix (14) is a navigation support device described in any of Appendix (5) to (13), The system further includes a distance estimation unit that estimates the distance between the light and the ship based on the relative position of the light and the horizon.

[0130] The navigation support system in Appendix (15) is the navigation support system described in Appendix (14), The display unit displays the distance.

[0131] The vessels mentioned in Appendix (16) are The vehicle shall be equipped with a navigation support device as described in any of the appendices (1) to (15).

[0132] The navigation support methods in Appendix (17) are: A navigation support method that assists the navigation of a vessel, This includes detecting lights based on color information contained in an image acquired by a visible light camera.

[0133] The navigation support program in Appendix (18) is a navigation support program that causes the computer to execute the navigation support method described in Appendix (17).

[0134] Although embodiments of the present invention have been described above, the scope of the present invention is not limited thereto, and it can be expanded or modified without departing from the spirit of the invention. [Industrial applicability]

[0135] This invention can be used, for example, in a system that monitors the surroundings of a ship 24 hours a day. [Explanation of Symbols]

[0136] 1 Navigation aids 12 Display section 22 Visible light camera 100 Ship (own ship) 142 Day / Night Determination Section 143 Light detection unit 144 Horizontal line detection unit 145 Partial image extraction section 146 Ship detection unit 147 Distance Estimation Unit 148 Judgment Department 149 Calculation Department CA Image CR partial image F2, 2nd slot HL horizontal line HL-t Upper limit position HL-b lower limit position D2 distance LP light S0 own ship SH ship (other ship) t0 Offset value (predetermined value)

Claims

1. A navigation support system that assists the navigation of a ship, A navigation support system comprising a light detection unit that detects lights based on color information contained in images acquired by a visible light camera.

2. The system further includes a display unit for displaying the aforementioned image, The navigation support device according to claim 1, wherein the display unit displays the lights in addition to the image, with emphasis.

3. The navigation support device according to claim 2, wherein the display unit displays a frame surrounding the light.

4. The navigation support device according to claim 1, further comprising a day / night determination unit that determines day or night based on the aforementioned color information.

5. The navigation support device according to claim 2, further comprising a horizon detection unit that detects the horizon based on the distribution of lights when there are multiple lights included in the image.

6. The navigation support device according to claim 5, wherein the horizon detection unit detects the horizon based on attitude information of the visible light camera input from an external source when there is only one light in the image.

7. The navigation support device according to claim 5, wherein the display unit displays the horizon detected by the horizon detection unit superimposed on the image.

8. A partial image extraction unit extracts a portion of the image from the aforementioned image that includes the light and overlaps with the horizontal line, The navigation support device according to claim 5, further comprising: a ship detection unit that detects other ships included in the partial image when the extracted partial image is input.

9. The navigation support device according to claim 5, further comprising a determination unit that determines whether or not the horizon detected by the horizon detection unit is located between a preset upper limit position and a preset lower limit position in the aforementioned image.

10. The navigation support device according to claim 5, further comprising a calculation unit that calculates the roll, pitch, and heave of the vessel based on the position of the horizon in the aforementioned image.

11. The navigation support device according to claim 5, wherein the horizon detection unit detects the horizon based on the lights and nautical chart information.

12. The navigation support device according to claim 5, wherein the horizon detection unit detects the horizon based on the lights and position information of other vessels received periodically.

13. The navigation support device according to claim 5, wherein the horizon detection unit corrects the horizon in the image by a predetermined amount downward.

14. The navigation support device according to claim 5, further comprising a distance estimation unit that estimates the distance between the light and the ship based on the relative position of the light and the horizon.

15. The navigation support device according to claim 14, wherein the display unit displays the distance.

16. A ship equipped with a navigation support device according to any one of claims 1 to 15.

17. A navigation support method that assists the navigation of a vessel, A navigation assistance method that includes detecting lights based on color information contained in an image acquired by a visible light camera.

18. A navigation support program for causing a computer to execute the navigation support method described in claim 17.

Images