Monitoring image generation device, monitoring image generation method, and monitoring image generation program

Abstract

To provide a monitoring image generation device that has a low risk of making a ship or a structure at sea less visible when a topographic image is superimposed on a landscape image and the superimposed image is displayed.SOLUTION: A monitoring image generation device 1 comprises: an information acquisition unit 21 that acquires topographic information for indicating a topography of a sea bottom in a traveling direction in which a ship 110 is traveling; an image information generation unit 23 that generates monitoring image information for showing a monitoring image superposed on a landscape image formed by imaging a superposition image from one point, the superposition image including a topographic image which is an image of a topography seen from one point on the ship 110 between a first point on the topography at sea corresponding to the topographic information and a second point that is farther than the first point from the ship 110; and an output unit 25 that outputs the monitoring image information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a surveillance image generating device, a surveillance image generating method, and a surveillance image generating program. [Background technology]

[0002] Various technologies for supporting the movement of ships, also referred to as navigation systems, have been known for some time. For example, Patent Document 1 describes a technology in which a composite image generated based on display information including the ship's planned route, structures located on the sea such as buoys, and other ships, as well as the position and attitude of the ship, is superimposed on an image captured from the ship and displayed in three dimensions on a display. The technology described in Patent Document 1 generates a composite image to be superimposed on an image captured from the ship according to the attitude of the ship, thereby enabling the acquisition of natural and highly realistic augmented reality images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-184614 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the technology described in Patent Document 1 makes it possible to visually recognize structures and other ships located on the sea, it does not display the seabed topography that is not visible to the operator operating the navigation system. In the technology described in Patent Document 1, when an image showing the seabed topography is superimposed on a landscape image, the topographical image showing the seabed topography is superimposed over the entire surface of the sea, and the displayed screen is covered with the seabed topography, which may make it difficult for the operator to visually recognize ships and structures on the sea.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a surveillance image generating device that reduces the risk of ships and structures on the sea becoming difficult to see when a terrain image is displayed superimposed on a landscape image. [Means for solving the problem]

[0006] The surveillance image generating device of the embodiment has an information acquisition unit that acquires topography information indicating the topography of the seabed in the direction of travel of the ship, an image information generation unit that generates surveillance image information indicating a surveillance image in which a superimposed image including a topography image, which is an image of the topography seen from a point on the ship of the topography between a first point on the sea topography corresponding to the topography information and a second point remoter than the first point as viewed from the ship, is superimposed on a landscape image captured from a single point, and an output unit that outputs the surveillance image information.

[0007] In the monitoring image generating device according to the embodiment, the information acquisition unit further acquires current position information indicating the current position of the ship and direction of travel information indicating the direction of travel, and it is preferable that the first point is a point that is a first distance away from the current position in the direction of travel, and the second point is a point that is a predetermined display length away from the first point in the direction of travel.

[0008] It is preferable that the monitoring image generating device according to the embodiment further includes a collision determination unit that determines whether or not there is a risk of the ship coming into contact with the seabed based on the ship's draft and the elevation of the terrain, and outputs an alarm signal when it determines that there is a risk of the ship coming into contact with the seabed.

[0009] In the monitoring image generating device according to the embodiment, it is preferable that the collision determination unit determines whether or not there is a risk of the ship coming into contact with the seabed based on the tide level at the current location in addition to the ship's draft and altitude.

[0010] The surveillance image generating device according to the embodiment preferably further includes an imaging unit that captures a landscape image and outputs landscape image information indicating the landscape image, and the image information generating unit preferably generates the surveillance image information by superimposing a superimposed image on a landscape image corresponding to the landscape image information.

[0011] In the surveillance image generating device according to the embodiment, it is preferable that the information acquisition unit further acquires tide level information indicating the tide level at the current location, and the image information generation unit corrects the position at which the superimposed image is superimposed on the landscape image according to the tide level.

[0012] In the surveillance image generating device according to the embodiment, it is preferable that the image information generating unit acquires structure information indicating structures included in the landscape image, and generates a superimposed image that indicates the superimposed image so as to include the structures.

[0013] In the surveillance image generating device according to the embodiment, it is preferable that the image information generating section generates a superimposed image that shows the superimposed image so that it can be viewed at an elevation angle according to the inclination of the surveillance image generating device.

[0014] The surveillance image generation method of the embodiment includes acquiring topography information indicating the topography of the seabed in the direction of travel of a ship, generating surveillance image information indicating a surveillance image in which a superimposed image including a topography image, which is an image of the topography seen from a point on the ship of the topography between a first point on the sea topography corresponding to the topography information and a second point remoter than the first point as viewed from the ship, is superimposed on a landscape image captured from the single point, and outputting the surveillance image information.

[0015] The surveillance image generation program of the embodiment acquires topography information indicating the topography of the seabed in the direction of travel of a ship, generates surveillance image information indicating a surveillance image in which a superimposed image including a topography image, which is an image of the topography seen from a point on the ship of the topography between a first point on the sea topography corresponding to the topography information and a second point remoter than the first point as viewed from the ship, is superimposed on a landscape image captured from the single point, and causes a computer to execute a process of outputting the surveillance image information. [Effects of the Invention]

[0016] The surveillance image generating device according to the present invention can reduce the risk that ships and structures on the sea will become difficult to see when a terrain image is displayed superimposed on a landscape image. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a perspective view of a ship equipped with a monitoring image generation system according to an embodiment. [Figure 2] 2 is a diagram showing a monitoring image generating device mounted on the ship shown in FIG. 1. FIG. [Figure 3] FIG. 3 is a functional block diagram of the monitoring image generating device shown in FIG. 2. [Figure 4] 3 is a flowchart showing an information setting process executed by the monitoring image generating device shown in FIG. 2. [Figure 5] 3A is a flowchart showing the monitoring image generation process executed by the monitoring image generation device shown in FIG. 2, and FIG. 3B is a flowchart showing more detailed processing of the process of S205 shown in FIG. 3A. [Figure 6] (a) is a diagram (part 1) for explaining the processing of S301 shown in Figure 5(b), (b) is a diagram (part 2) for explaining the processing of S301 shown in Figure 5(b), and (c) is a diagram (part 3) for explaining the processing of S301 shown in Figure 5(b). [Figure 7] 6 is a flowchart showing the detailed processing of step S206 shown in FIG. 5(a). [Figure 8] (a) is a diagram for explaining the processing of S401 shown in Figure 7, (b) is a diagram for explaining the processing of S402 shown in Figure 7, (c) is a diagram for explaining the processing of S403 shown in Figure 7, (d) is a diagram for explaining the processing of S404 shown in Figure 7, (e) is a diagram for explaining the processing of S405 shown in Figure 7, and (f) is a diagram for explaining the processing of S406 shown in Figure 7. [Figure 9] 6 is a flowchart showing the detailed processing of step S207 shown in FIG. 5(a). [Figure 10] 4 is a diagram showing an example of a monitoring image displayed on the display unit shown in FIG. 3. FIG. [Figure 11] FIG. 2 is a functional block diagram of the vessel monitoring device. [Figure 12] 12 is a flowchart showing an image display process executed by the vessel monitoring device shown in FIG. 11. [Figure 13]12 is a diagram showing an example of a display image displayed on the monitoring display unit shown in FIG. 11. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0018] The following describes a surveillance image generating device, a surveillance image generating method, and a surveillance image generating program according to the present invention with reference to the drawings. However, please note that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions set forth in the claims and their equivalents.

[0019] (Configuration and Functions of a Surveillance Image Generation System Having a Surveillance Image Generation Device According to an Embodiment) FIG. 1 is a perspective view of a ship equipped with a surveillance image generation system according to an embodiment, FIG. 2 is a diagram showing a surveillance image generation device mounted on the ship shown in FIG. 1, and FIG. 3 is a functional block diagram of the surveillance image generation device shown in FIG. 2.

[0020] The surveillance image generation system 100 has a GNSS receiver 101, a satellite communication receiver 102, and a surveillance image generation device 1, and is mounted on a ship 110. Based on signals received from the GNSS receiver 101 to the satellite communication receiver 102, the surveillance image generation system 100 displays a landscape image and a superimposed image that is superimposed on the landscape image on a screen that is visible to an operator 112 on board the ship 110.

[0021] The GNSS receiver 101 receives signals from a satellite positioning system (Global Navigation Satellite System (GNSS)) such as a Global Positioning System (GPS). In response to receiving signals from the GNSS, the GNSS receiver 101 outputs a current position signal indicating the current position of the ship 110 and a heading signal indicating the direction in which the ship 110 is traveling to the surveillance image generation device 1. The satellite communication receiver 102 receives signals from a communication satellite that can be connected to the Internet network via a gateway earth station. The satellite communication receiver 102 receives a tide level signal indicating the tide level at the current position of the ship 110, which is extracted from information on tide levels provided by the Marine Information Department of the Japan Coast Guard via the Internet network, and outputs the received tide level signal to the surveillance image generation device 1. The satellite communications receiver 102 also receives, via the Internet network, AIS signals containing the vessel's identification code, type, position, course, speed, navigation status, and other safety-related information of an AIS vessel that is located near the vessel 110 and is equipped with an AIS signal transceiver, and outputs the received AIS signals to the surveillance image generation device 1. The satellite communications receiver 102 also receives, via the Internet network, non-AIS signals indicating the vessel's position, course, speed, navigation status, and other safety-related information of a non-AIS vessel that is located near the vessel 110 and is not equipped with an AIS signal transceiver, and outputs the received non-AIS signals to the surveillance image generation device 1. A server that outputs the AIS signals and non-AIS signals is connected to the GNSS receiver 101 via the Internet network.

[0022] The monitoring image generating device 1 is disposed inside a cabin 111 of a ship 110 so as to be visible to an operator 112 and so as to enable the built-in imaging unit 16 to capture images of the traveling direction of the ship 110 through a porthole 113. The monitoring image generating device 1 has a communication unit 11, a memory unit 12, an operation unit 13, a display unit 14, an audio output unit 15, the imaging unit 16, a tilt detection unit 17, and a processing unit 20. The communication unit 11, the tilt detection unit 17, and the processing unit 20 are connected via a bus 18.

[0023] The communication unit 11 is configured to enable the monitoring image generation device 1 to communicate with the GNSS receiver 101 to the satellite communication receiver 102, and includes a communication interface circuit. The communication interface circuit is, for example, a communication interface circuit for a wired LAN, a wireless LAN, LTE (Long Term Evolution), or the like. The communication unit 11 supplies signals input from the GNSS receiver 101 to the satellite communication receiver 102 to the processing unit 20, and outputs signals supplied from the processing unit 20 to the GNSS receiver 101 to the satellite communication receiver 102.

[0024] The storage unit 12 is configured to store data and programs and includes, for example, a semiconductor memory. The storage unit 12 stores an operating system program, a driver program, an application program, data, and the like used in processing by the processing unit 20. For example, the storage unit 12 stores, as an application program, an information setting program that causes the processing unit 20 to execute an information setting process that sets various information used when executing an image generation process that generates an image including a topographical image showing the underwater topography. The storage unit 12 also stores, as an application program, an image generation program that causes the processing unit 20 to execute the image generation process. The information setting program and the image generation program are installed into the storage unit 12 from a computer-readable, non-transitory, portable storage medium such as a compact disc (CD)-read-only memory (ROM) or a digital versatile disc (DVD)-ROM. The storage unit 12 also stores various data and information used when executing the information setting process and the image generation process. For example, the storage unit 12 stores elevation data, also referred to as topographical information, that indicates the elevation of the underwater topography as elevation tiles 12a.

[0025] The elevation tile 12a has a 256 x 256 matrix structure, with the coordinates of the seafloor located in each matrix element. The elevation tile defines elevation data at various intervals depending on the zoom level. The structure and function of the elevation tile 12a are well known, so a detailed description will be omitted here.

[0026] The memory unit 12 also stores structure information indicating structures included in the scenery in the direction of travel of the ship 110, including land-based structures such as lighthouses and port facilities, and offshore structures such as buoys and offshore wind power generation facilities. The structure information includes the name of the structure, shape data indicating the shape of the structure, and the latitude and longitude of the structure. The memory unit 12 also stores route information indicating areas that are preferably visible when maneuvering the ship 110, including route lines indicating routes along which the ship 110 can travel, and no-entry areas into which the ship 110 is prohibited from entering. The route information includes the names of areas including the route lines and no-entry areas, shape data indicating the shape of the areas, and the latitude and longitude of the areas. The memory unit 12 also stores alarm threshold information indicating an alarm threshold used to determine whether the bottom of the ship 110 will come into contact with the seabed.

[0027] The operation unit 13 is configured to receive user operations on the surveillance image generation device 1, and includes, for example, a touch panel. The operation unit 13 generates a signal in response to the operation of the operator 112, and supplies the generated signal to the processing unit 20.

[0028] The display unit 14 is configured to display an image, and includes a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 14 generates and displays an image based on a signal supplied from the processing unit 20.

[0029] The audio output unit 15 is configured to output audio and includes a speaker. The audio output unit 15 outputs audio based on a signal supplied from the processing unit 20.

[0030] The imaging unit 16 is configured to capture images of the surroundings of the monitoring image generation device 1 and includes a camera. The camera includes an imaging optical system for forming an image on a light-receiving surface, a photoelectric conversion element, and an image generation circuit for generating an image based on the output of the photoelectric conversion element. The photoelectric conversion elements included in the camera are CMOS (Complementary Metal Oxide Semiconductor) and CCD (Charge Coupled Device) sensors, etc., that are arranged two-dimensionally on the light-receiving surface and output an electrical signal according to the amount of incident light. The imaging unit 16 captures and acquires a landscape image showing the scenery in the direction of travel of the ship 110, and supplies a landscape image signal showing the acquired landscape image to the processing unit 20.

[0031] The tilt detection unit 17 is equipped with, for example, a three-axis acceleration sensor or a gyro sensor, and detects the tilt of the monitoring image generation device based on the indication value of the three-axis acceleration sensor or gyro sensor, and supplies a tilt signal indicating the detected tilt to the processing unit 20.

[0032] The processing unit 20 is configured to comprehensively control the operation of the surveillance image generation device 1, and includes one or more processors and their peripheral circuits. The processing unit 20 includes, for example, a CPU (Central Processing Unit). The processing unit 20 may also include a DSP (Digital Signal Processor), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. The processing unit 20 executes processing based on a program stored in the storage unit 12. The processing unit 20 also controls the operation of each component of the surveillance image generation device 1 so that processing is executed appropriately.

[0033] The processing unit 20 has, as functional blocks, an information acquisition unit 21, a condition setting unit 22, an image information generation unit 23, a collision determination unit 24, and an output unit 25. These functional blocks are functional modules realized based on a program executed by the processing unit 20. These functional blocks may be firmware implemented in the monitoring image generation device 1.

[0034] Fig. 4 is a flowchart showing the information setting process executed by the monitoring image generation device 1. The information setting process shown in Fig. 4 is realized by the processing unit 20 working in cooperation with each component of the monitoring image generation device 1 based on a program stored in the storage unit 12. The information setting process shown in Fig. 4 is executed every time the monitoring image generation device 1 is deployed on the ship 110.

[0035] First, the information acquisition unit 21 acquires first distance information indicating a first distance in response to the operation of the operation unit 13 by the operator 112 (S101), and stores the acquired first distance information in the storage unit 12. The first distance is the distance from the ship 110 to the near clip plane of the view frustum. The first distance, which is the distance to the near clip plane of the view frustum, is used when generating, as a view frustum, a topographical image indicating the topography of the seabed included in the superimposed image to be superimposed on the landscape image captured by the imaging unit 16. Next, the condition setting unit 22 sets the first distance acquired in the process of S101 as information indicating the first distance to be used when generating the topographical image (S102). The condition setting unit 22 sets the first distance by associating the first distance information acquired in the process of S101 with a predetermined identifier.

[0036] Next, the information acquisition unit 21 acquires display length information indicating a display length, which is the distance between the near clip plane and the far clip plane of the terrain image generated as a view frustum, in response to the operation of the operation unit 13 by the operator 112 (S103). The information acquisition unit 21 stores the acquired display length information in the storage unit 12. Next, the condition setting unit 22 sets the display length acquired in the processing of S103 as information indicating the display length to be used when generating the terrain image (S104). The condition setting unit 22 sets the display length by associating the display length information acquired in the processing of S103 with a predetermined identifier.

[0037] Next, the information acquisition unit 21 acquires tilt information indicating the tilt of the monitoring image generation device 1 from a horizontal plane in response to operation of the operation unit 13 by the operator 112 (S105). When instructed to acquire tilt information via the operation unit 13, the information acquisition unit 21 requests the tilt detection unit 17 to output a tilt signal, and when the tilt signal is input, acquires tilt information indicating the tilt corresponding to the input tilt signal and stores the acquired tilt information in the storage unit 12. Next, the condition setting unit 22 sets the tilt information acquired in the process of S105 as information indicating the tilt to be used when generating a superimposed image (S106). The condition setting unit 22 sets the tilt by associating the tilt information acquired in the process of S105 with a predetermined identifier.

[0038] Next, the information acquisition unit 21 acquires height information indicating the height from sea level of the location where the monitoring image generation device 1 is placed in response to operation of the operation unit 13 by the operator 112 (S107). The information acquisition unit 21 stores the acquired height information in the storage unit 12. Next, the condition setting unit 22 sets the height information acquired in the process of S107 as information indicating the height to be used when generating a superimposed image (S108). The condition setting unit 22 sets the height by associating the height information acquired in the process of S107 with a predetermined identifier. The height set in the process of S108 is the height of the imaging unit 16 from sea level.

[0039] Next, the information acquisition unit 21 acquires draft information indicating the draft of the ship 110 in response to operation of the operation unit 13 by the operator 112 (S109), and stores the acquired draft information in the memory unit 12. Next, the condition setting unit 22 sets the draft information acquired in the processing of S109 as information indicating the draft to be used in the collision determination processing that determines whether or not the ship 110 will collide with the seabed (S110). The condition setting unit 22 sets the draft by associating the draft information acquired in the processing of S109 with a predetermined identifier.

[0040] Fig. 5(a) is a flowchart showing the monitoring image generation process executed by the monitoring image generation device 1. The monitoring image generation process shown in Fig. 5(a) is realized by the processing unit 20 working in cooperation with each component of the monitoring image generation device 1 based on a program stored in the storage unit 12. The monitoring image generation process shown in Fig. 5(a) is executed at a predetermined execution interval.

[0041] First, the image information generation unit 23 starts capturing a scenery image by the imaging unit 16 in response to the operation of the operation unit 13 by the operator 112 (S201). When the operator 112 operates the operation unit 13 to acquire an instruction to start capturing an image via the operation unit 13, the image information generation unit 23 starts the operation of the imaging unit 16 and starts capturing a scenery image in the traveling direction of the ship 110 by the imaging unit 16. The image information generation unit 23 acquires scenery image information indicating the scenery image captured by the imaging unit 16, and outputs the acquired scenery image information to the display unit 14. The display unit 14 displays a scenery image corresponding to the input scenery image information.

[0042] Next, the information acquisition unit 21 acquires current position information indicating the current position of the ship 110 (S202). The information acquisition unit 21 requests the GNSS receiver 101 to output a current position signal via the communication unit 11, and when the current position signal is input to the communication unit 11, acquires current position information indicating the current position corresponding to the input current position signal.

[0043] Next, the information acquisition unit 21 acquires heading information indicating the heading direction of the ship 110 (S203). The information acquisition unit 21 requests the GNSS receiver 101 to output a heading signal via the communication unit 11, and when the heading signal is input to the communication unit 11, acquires heading information indicating the heading corresponding to the input heading signal.

[0044] Next, the information acquisition unit 21 acquires tide level information indicating the tide level at the current position of the ship 110 (S204). The information acquisition unit 21 requests the satellite communication receiver 102 to output a current position signal via the communication unit 11, and when a tide level signal indicating the tide level at the current position of the ship 110 is input to the communication unit 11, the information acquisition unit 21 acquires tide level information indicating the tide level corresponding to the input tide level signal.

[0045] Next, the image information generation unit 23 generates terrain image information showing a point on the ship 110, i.e., a terrain image, which is an image of the seabed terrain in the direction of travel of the ship 110 as seen through the monitoring image generation device 1 (S205), and stores the generated terrain image information in the memory unit 12.

[0046] FIG. 5B is a flowchart showing the process of S205 in more detail.

[0047] First, the image information generation unit 23 sets the range for generating a terrain image (S301) using the first distance, display length, and inclination set in the processes of S102, S104, and S106, and the current position information and traveling direction information acquired in the processes of S202 and S203. The range for generating a terrain image is called the terrain image range.

[0048] 6(a), the image information generation unit 23 sets a quadrangular pyramid R1 including a viewing frustum indicating the range for generating a terrain image from the current position corresponding to the current position information acquired in the processing of S202 and the traveling direction corresponding to the traveling direction information acquired in the processing of S203. The vertex P of the quadrangular pyramid R1 is the latitude and longitude indicating the current position corresponding to the current position information acquired in the processing of S202, and the extension direction of the normal L1 of the base SL of the quadrangular pyramid R1 is the traveling direction corresponding to the traveling direction information acquired in the processing of S203.

[0049] Next, as shown in FIG. 6(b), the image information generation unit 23 sets a plane that is separated from the vertex P of the pyramid R1 by the first distance L1 set in the processing of S102 as the near clip plane S1. The point at which the near clip plane S1 is located is also referred to as the first point. The near clip plane S1 is the end of the terrain image range that is closest to the current position. Next, as shown in FIG. 6(c), the image information generation unit 23 sets a plane that is separated from the near clip plane S1 in the extension direction of the normal line L1 by the display length LD set in the processing of S104 as the far clip plane S2. The point at which the far clip plane S2 is located is also referred to as the second point. The far clip plane S2 is the end of the terrain image range that is farthest from the current position.

[0050] Next, the image information generation unit 23 acquires elevation data included in the near clip plane S1 of the terrain image range determined in the processing of S301 (S302). The image information generation unit 23 references the elevation tile 12a to acquire elevation data of the terrain superimposed on the near clip plane S1 at a position separated by a first distance from the current position in the traveling direction, and stores the acquired elevation data in the storage unit 12 as a first terrain data group.

[0051] Next, the image information generation unit 23 acquires elevation data included in a plane that is separated from the near clip plane S1 by a predetermined plot interval in the normal line L1 direction and is parallel to the near clip plane S1 (S303). Similar to the processing of S302, the image information generation unit 23 references the elevation tile 12a to acquire elevation data of the terrain that is superimposed on the plane that is separated from the near clip plane S1 by the predetermined plot interval, and stores the acquired elevation data in the storage unit 12 as a second terrain data group.

[0052] Next, the image information generation unit 23 determines whether the distance between the acquisition point, which is separated by the plot interval in the direction of the normal L1 from the plane on which the elevation data was acquired in the process of S303, and the near clip plane S1 is longer than the display length LD set in the process of S104 (S304). If it is determined that the distance between the acquisition point and the near clip plane S1 is not longer than the display length LD (S304-NO), the process returns to S303. Thereafter, the processes of S303 and S304 are repeated until it is determined that the distance between the acquisition point and the near clip plane S1 is longer than the display length LD (S304-YES). By repeating the processes of S303 and S304, multiple terrain data groups are stored in the storage unit 12, with the data group numbers being sequentially increased, starting with the third terrain data group.

[0053] When the image information generation unit 23 determines that the distance between the acquisition point and the near clip plane S1 is longer than the display length (S304-YES), it stores the first to Nth terrain data groups acquired by the processes of S302 to S304 in the storage unit 12 as terrain image information (S305). When the image information generation unit 23 stores the terrain image information in the storage unit 12, the process of S205 ends.

[0054] Following the processing of S205, the image information generation unit 23 generates superimposed image information indicating a superimposed image that can be superimposed on a point on the ship 110, i.e., a landscape image captured by the imaging unit 16 (S206), and stores the generated superimposed image information in the memory unit 12.

[0055] FIG. 7 is a flowchart showing the process of S206 in more detail.

[0056] First, the image information generation unit 23 sets an imaging range to be imaged by the imaging unit 16 (S401). As shown in Fig. 8(a), the image information generation unit 23 sets an imaging range R2, which is a quadrangular pyramid indicating the range in which a terrain image is to be generated, from the current position corresponding to the current position information acquired in the processing of S202 and the traveling direction corresponding to the traveling direction information acquired in the processing of S203.

[0057] Next, the image information generation unit 23 arranges the terrain image corresponding to the terrain image information generated in the process of S205 in the imaging range R2 set in the process of S401 (S402). As shown in Fig. 8(b), the image information generation unit 23 acquires the terrain image information stored in the storage unit 12, and arranges, in the imaging range R2, the terrain image I1, which is a point cloud image indicating, as points, each of the elevation data included in the terrain data group corresponding to the acquired terrain image information.

[0058] The image information generation unit 23 extracts structure information including structures included in the imaging range R2 based on the latitude and longitude included in the structure information stored in the storage unit 12. As shown in Fig. 8(c), the image information generation unit 23 arranges, in the imaging range R2, a structure image I2 that shows the shape included in the structure information extracted based on the latitude and longitude.

[0059] Next, the image information generation unit 23 arranges the route information included in the imaging range in the imaging range R2 estimated in the processing of S401 (S404). The route information indicates route lines indicating routes that the ship 110 can travel and areas that are preferably visible when maneuvering the ship 110, including no-entry areas where the ship 110 is prohibited from entering. The image information generation unit 23 extracts route information indicating the area included in the imaging range R2 based on the latitude and longitude included in the route information stored in the storage unit 12. As shown in FIG. 8(d), the image information generation unit 23 arranges an area image I3 indicating the shape included in the route information extracted based on the latitude and longitude in the imaging range R2.

[0060] Next, the image information generation unit 23 places ships included in the imaging range R2 in the imaging range R2 estimated in the processing of S401 (S405). The image information generation unit 23 requests the server that outputs the AIS signal to output the AIS signal, and when the AIS signal is input to the communication unit 11, acquires a signal related to the ship from the input AIS signal. The image information generation unit 23 extracts AIS ships included in the imaging range R2 from the acquired signals related to the ship, and stores the name, captain, speed, and position of the ship from the AIS signal of the extracted AIS ship in the memory unit 12. The image information generation unit 23 also requests the server that transmits the non-AIS signal to output the non-AIS signal, and when the non-AIS signal is input to the communication unit 11, acquires a signal related to the ship from the input non-AIS signal. The image information generation unit 23 extracts non-AIS ships within the imaging range R2 from the acquired ship-related signals, and stores the names, speeds, and positions of the ships from the extracted non-AIS signals in the memory unit 12. As shown in FIG. 8(e), the image information generation unit 23 places images of the ships according to their distance from the ship 110 at the positions of the AIS and non-AIS ships stored in the memory unit 12, and places images including the names and speeds of the AIS and non-AIS ships in association with the ship image I4. Note that the images placed in association with the ship images may include the captains of the AIS ships and the distances between the AIS and non-AIS ships and the ship 110, in addition to the names and speeds of the AIS and non-AIS ships.

[0061] Next, the image information generation unit 23 tilts the imaging range in which the undersea topography and the like are located in the processes of S402 to S405 in accordance with the tilt of the monitoring image generation device 1 so that it is visible at an elevation angle in accordance with the tilt of the monitoring image generation device 1 (S406). As shown in Fig. 8(f), the image information generation unit 23 tilts the imaging range R2 by a tilt θ in accordance with the tilt corresponding to the tilt information set in the process of S106.

[0062] Next, the image information generation unit 23 sets the height of the imaging range R2 from sea level so that the height of the imaging range R2 from sea level matches the height of the monitoring image generation device 1 from sea level (S407). The image information generation unit 23 sets the height of the imaging range R2 so that it matches the height set in the processing of S108.

[0063] Next, the image information generation unit 23 corrects the height of the imaging range R2 set in the process of S407 in accordance with the tide level at the current location (S408). The information acquisition unit 21 corrects the height of the imaging range R2 in accordance with the tide level corresponding to the tide level information acquired in the process of S204.

[0064] Next, the image information generating unit 23 generates an orientation image indicating the orientation (S409). The image information generating unit 23 generates an isosceles triangular image as an orientation image having a slope corresponding to the angle difference between the north direction and the traveling direction corresponding to the traveling direction information acquired in the process of S203, and stores orientation image information indicating the generated orientation image. The orientation image may be an image other than an isosceles triangular image.

[0065] Next, the image information generation unit 23 generates superimposed image information by synthesizing the azimuth image generated in the processing of S406 with the imaging range R2 (S410), and stores the generated superimposed image information in the storage unit 12. When the image information generation unit 23 stores the superimposed image information in the storage unit 12, the processing of S206 ends.

[0066] Following the process of S206, the collision determination unit 24 executes a collision determination process (S207).

[0067] FIG. 9 is a diagram showing more detailed processing of the collision determination processing shown in S207.

[0068] First, the collision determination unit 24 extracts from the elevation tile 12a the elevation of the seabed topography at a point a predetermined distance away in the traveling direction obtained in the processing of S203 (S501), and stores elevation information indicating the extracted elevation in the storage unit 12. For example, the collision determination unit 24 extracts from the elevation tile 12a the elevation of the seabed topography between the near clip plane S1 and the far clip plane S2.

[0069] Next, the collision determination unit 24 calculates the water depth at the point whose elevation was extracted in the process of S501 from the tide level information acquired in the process of S204 and the elevation extracted in the process of S501 (S502). The collision determination unit 24 calculates the water depth by adding the tide level corresponding to the tide level information to the tide level information acquired in the process of S204 and the elevation extracted in the process of S501.

[0070] Next, the collision determination unit 24 calculates the separation distance between the bottom of the ship 110 and the seabed from the draft set in the processing of S110 and the water depth calculated in the processing of S502 (S503). The collision determination unit 24 stores the calculated separation distance in the memory unit 12.

[0071] Next, the collision determination unit 24 determines whether or not there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504). The collision determination unit 24 acquires warning threshold information stored in the memory unit 12. The collision determination unit 24 compares the warning threshold corresponding to the acquired warning threshold information with the separation distance calculated in the processing of S503, and determines whether or not the separation distance calculated in the processing of S503 is equal to or greater than the warning threshold. When the separation distance calculated in the processing of S503 is less than the warning threshold, the collision determination unit 24 determines that there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504-YES). On the other hand, when the separation distance calculated in the processing of S503 is equal to or greater than the warning threshold, the collision determination unit 24 determines that there is no risk that the bottom of the vessel 110 will come into contact with the seabed (S504-NO).

[0072] When the collision determination unit 24 determines that there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504-YES), it outputs a collision warning signal indicating that there is a risk that the bottom of the vessel 110 will come into contact with the seabed to the display unit 14 and the audio output unit 15. The display unit 14 displays a collision area image indicating the area where the bottom of the vessel 110 is likely to come into contact with the seabed by superimposing it on a landscape image. In response to the input of the collision warning signal, the audio output unit 15 outputs an audio signal indicating that there is a risk that the bottom of the vessel 110 will come into contact with the seabed. When the collision determination unit 24 determines that there is no risk that the bottom of the vessel 110 will come into contact with the seabed (S504-NO), the processing of S207 ends. Note that instead of displaying the collision area image, text indicating a warning for each type, such as "Caution: Contact with ground" or "Caution: Entering no-approach area," may be displayed.

[0073] Following the process of S207, the information acquisition unit 21 acquires scenery image information indicating the scenery image captured by the imaging unit 16 (S208). The information acquisition unit 21 requests the imaging unit 16 to output a scenery image signal indicating the scenery image captured by the imaging unit 16, and acquires the scenery image information indicating the scenery image captured by the imaging unit 16.

[0074] Next, the image information generation unit 23 superimposes the superimposed image corresponding to the superimposed image information generated in the processing of S206 on the scenery image corresponding to the scenery image information acquired in the processing of S208 to generate monitoring image information (S209).

[0075] Next, the output unit 25 outputs the monitoring image information generated in the process of S209 to the display unit 14 (S210). In response to the input of the monitoring image information, the display unit 14 displays a monitoring image corresponding to the input monitoring image information.

[0076] FIG. 10 is a diagram showing an example of a monitoring image displayed on the display unit 14. As shown in FIG.

[0077] The monitoring image 120 includes a terrain image 121, a structure image 122, a route image 123, and a ship image 124, and is displayed on the display unit 14 superimposed on a landscape image captured by the imaging unit 16. The terrain image 121 is formed from a point cloud and shows the topography of the seabed. The structure image includes multiple buoys and shows structures located on land and at sea. The route image 123 includes a route line image 125 and a no-entry area image 126. The ship image 124 includes a ship image 127 showing a ship and a ship information image 128 showing information about the ship. In addition, a collision warning area image 129 corresponding to a collision warning signal is superimposed on the monitoring image 120. Furthermore, a warning image 130 showing various warnings is superimposed on the monitoring image 120. The route image 123 may include a navigable width corresponding to the width of the ship sailing toward the ship 110. Furthermore, the monitoring image generating device 1 may output an alarm signal when the ship 110 enters the navigable width included in the route image 123.

[0078] Following the process of S208, the image information generation unit 23 determines whether or not an instruction to end the image generation process has been given (S211). When a process end signal is input in response to operation of the operation unit 13 by the operator 112, the image information generation unit 23 determines that an instruction to end the image generation process has been given (S211-YES). If the image information generation unit 23 determines that an instruction to end the image generation process has not been given (S211-NO), the process returns to S202. Thereafter, the processes of S202 to S211 are repeated until the image information generation unit 23 determines that an instruction to end the image generation process has been given (S211-YES).

[0079] When it is determined that the image information generating unit 23 has instructed to end the image generating process (YES in S211), the capturing of the scenery image by the capturing unit 16 is ended in response to the operation of the operating unit 13 by the operator 112 (S212).

[0080] (Operational effects of the monitoring image generating device according to the embodiment) The monitoring image generating device 1 can reduce the risk of ships and structures on the sea becoming difficult to see by superimposing a topographical image of the seabed between the near clip plane S1 and the far clip plane S2 on the landscape image, rather than a topographical image of the entire seabed included in the landscape image. For example, by making the display length LD very short and positioning the near clip plane S1 and the far clip plane S2 close to each other, it is possible to display the cross-sectional shape of the seabed at a point separated by the first distance L1 from the ship 110 as a topographical image.

[0081] In addition, the monitoring image generating device 1 displays an image of the seabed topography between the near clip plane S1 and the far clip plane S2, which are located in the direction of travel of the ship 110, allowing the operator 112 to easily understand the topography of the seabed in the direction of travel of the ship 110.

[0082] Furthermore, the monitoring image generation device 1 can quickly acquire the seabed topography by using the elevation tiles 12a. Furthermore, the monitoring image generation device 1 generates a topography image by forming the seabed topography as a point cloud, so that the monitoring image generation device 1 can quickly generate and display the seabed topography image.

[0083] Furthermore, the monitoring image generating device 1 corrects the position at which the seabed topography image is superimposed on the landscape image according to the tide level at the current location of the ship 110, so that the seabed topography image can be superimposed on the landscape image with high accuracy regardless of changes in the tide level.

[0084] Furthermore, since the surveillance image generating device 1 generates a superimposed image to include ships and land and sea structures contained in the landscape image, the operator 112 can easily see the structures even when the landscape image captured by the imaging unit 16 is unclear due to nighttime or thick fog, etc.

[0085] Furthermore, since the monitoring image generating device 1 generates the superimposed image information so that it is viewed at an elevation angle corresponding to the tilt of the monitoring image generating device 1, the superimposed image can be superimposed on the landscape image regardless of the tilt of the monitoring image generating device 1.

[0086] Furthermore, the monitoring image generation device 1 outputs an alarm signal when there is a risk that the ship 110 will come into contact with the seabed, allowing the operator 112 to steer the ship 110 so as to reduce the risk of it coming into contact with the seabed.

[0087] In addition, the monitoring image generating device 1 determines whether there is a risk of the ship 110 coming into contact with the seabed based on the tide level at the current location of the ship 110, in addition to the draft of the ship 110 and the elevation of the seabed, so that it can more accurately determine the risk of the ship 110 coming into contact with the seabed.

[0088] (Modification of the monitoring image generating device according to the embodiment) The monitoring image generating device 1 arranges land and sea structures, route information, and other ships in addition to the topographical image in the superimposed image, but in the monitoring image generating device according to the embodiment, only the topographical image may be arranged in the superimposed image. Also, in the monitoring image generating device according to the embodiment, the superimposed image may arrange any of the topographical image and structures, route information, and other ships.

[0089] Furthermore, the surveillance image generating device 1 generates terrain images and superimposed images as a viewing cone, but the surveillance image generating device according to the embodiment may also generate terrain images and superimposed images by performing image processing such as affine transformation and projective transformation on the seabed terrain, etc.

[0090] Furthermore, although the monitoring image generation device 1 generates terrain image information using elevation tiles, the monitoring image generation device according to the embodiment may generate terrain image information using information acquired using a shape detection sensor such as sonar. Furthermore, although the monitoring image generation device 1 generates a terrain image by forming the seabed terrain as a point cloud, the monitoring image generation device according to the embodiment may generate a terrain image by forming the seabed terrain as a line drawing of contour lines.

[0091] Furthermore, the monitoring image generating device 1 corrects the position at which the superimposed image is superimposed on the scenery image in accordance with the tide level at the current position of the ship 110, but the monitoring image generating device according to the embodiment does not have to perform the correction according to the tide level. Furthermore, the monitoring image generating device according to the embodiment performs the collision detection process using the tide level at the current position of the ship 110, but the monitoring image generating device according to the embodiment may perform the collision detection process using the ship's draft and the seabed elevation without using the tide level.

[0092] Furthermore, although the surveillance image generating device 1 has an operation unit 13 to an imaging unit 16, the surveillance image generating device according to the embodiment may not have any of the operation unit 13 to the imaging unit 16, and the operation unit 13 to the imaging unit 16 may be arranged as separate devices from the surveillance image generating device.

[0093] Furthermore, although the surveillance image generation system 100 having the surveillance image generation device 1 superimposes a superimposed image on a scenery image visible from a single ship 110, the surveillance image generation system according to the embodiment may be implemented on a plurality of ships. When the surveillance image generation system according to the embodiment is implemented on a plurality of ships, the scenery image captured by the surveillance image generation system and the superimposed image superimposed on the scenery image may be displayable by the ship monitoring device.

[0094] (Ship monitoring device according to an embodiment) FIG. 11 is a functional block diagram of a vessel monitoring device capable of displaying a scenery image captured by a monitoring image generating system and a superimposed image superimposed on the scenery image.

[0095] The vessel monitoring device 150 has a monitoring communication unit 151, a monitoring memory unit 152, a monitoring operation unit 153, a monitoring display unit 154, a monitoring audio output unit 155, and a monitoring processing unit 160, and displays scenery images and superimposed images captured by a monitoring image generation device 1 mounted on a plurality of vessels 110. Each of the plurality of vessels 110 is equipped with a monitoring image generation device 1. The monitoring communication unit 151, the monitoring audio output unit 155, and the monitoring processing unit 160 are connected via a bus 156.

[0096] The monitoring communication unit 151 is configured to enable communication with the server that outputs AIS signals and non-AIS signals, and the monitoring image generation device 1 via the Internet, and includes a communication interface circuit. The communication interface circuit is, for example, a communication interface circuit for wired LAN, wireless LAN, LTE, or the like. It supplies signals input from the server that outputs AIS signals and non-AIS signals, and the monitoring image generation device 1 to the monitoring processing unit 160, and outputs signals supplied from the monitoring processing unit 160 to the server that outputs AIS signals and non-AIS signals, and to the monitoring image generation device 1.

[0097] The monitoring storage unit 152 is configured to store data and programs, and includes, for example, a semiconductor memory. The monitoring storage unit 152 stores an operating system program, a driver program, an application program, data, and the like used in processing by the monitoring processing unit 160. For example, the monitoring storage unit 152 stores, as an application program, an image display program that causes the monitoring processing unit 160 to execute image display processing that displays monitoring images generated by the monitoring image generation devices 1 installed on multiple ships 110. The image display program is installed into the monitoring storage unit 152 from a computer-readable, non-transitory, portable storage medium such as a CD-ROM or DVD-ROM. The monitoring storage unit 152 also stores various data and information used when executing the image display processing.

[0098] The monitoring operation unit 153 is configured to receive user operations on the vessel monitoring device 150, and includes, for example, a keyboard. The monitoring operation unit 153 generates a signal in response to the user operation, and supplies the generated signal to the monitoring processing unit 160.

[0099] The monitoring display unit 154 is configured to display an image and includes a display such as a liquid crystal display or an organic EL display. The monitoring display unit 154 generates and displays an image based on a signal supplied from the monitoring processing unit 160.

[0100] The monitoring audio output unit 155 is configured to output audio and includes a speaker. The monitoring audio output unit 155 outputs audio based on a signal supplied from the monitoring processing unit 160.

[0101] The monitoring processing unit 160 is configured to comprehensively control the operation of the ship monitoring device 150, and includes one or more processors and their peripheral circuits. The monitoring processing unit 160 includes, for example, a CPU. The monitoring processing unit 160 may also include a DSP, an LSI, an ASIC, an FPGA, etc. The monitoring processing unit 160 executes processing based on a program stored in the monitoring memory unit 152. The monitoring processing unit 160 also controls the operation of each component of the ship monitoring device 150 so that processing is executed appropriately.

[0102] The surveillance processing unit 160 has, as functional blocks, a surveillance information acquisition unit 161, a surveillance determination unit 162, a surveillance image information generation unit 163, and a surveillance output unit 164. These functional blocks are functional modules realized based on a program executed by the surveillance processing unit 160. These functional blocks may be firmware implemented in the surveillance image generation device 1.

[0103] Fig. 12 is a flowchart showing the image display processing executed by the vessel monitoring device 150. The information setting processing is realized by the monitoring processing unit 160 working in cooperation with each component of the vessel monitoring device 150 based on a program stored in the monitoring memory unit 152. The image display processing shown in Fig. 12 is executed at a predetermined execution cycle.

[0104] First, the monitoring information acquisition unit 161 acquires AIS ship information (S501) and stores the acquired AIS ship information in the monitoring memory unit 152. The monitoring information acquisition unit 161 requests the server that outputs the AIS signal to output the AIS signal, and when the AIS signal is input to the monitoring communication unit 151, it acquires AIS ship information including the name and position of the ship from the input AIS signal.

[0105] Next, the monitoring information acquisition unit 161 acquires non-AIS ship information (S502) and stores the acquired non-AIS ship information in the monitoring memory unit 152. The monitoring information acquisition unit 161 requests the server that outputs the non-AIS signal to output the non-AIS signal, and when the non-AIS signal is input to the monitoring communication unit 151, it acquires non-AIS ship information including the name and position of the ship from the input non-AIS signal.

[0106] Next, the monitoring determination unit 162 determines a monitoring target ship to be monitored from the AIS ship information acquired in the processing of S501 and the non-AIS ship information acquired in the processing of S502 (S503). The monitoring determination unit 162 may determine a ship present within the port as a monitoring target ship, or may determine a ship present within a specified area of the port as a monitoring target ship.

[0107] Next, the monitoring information acquisition unit 161 acquires monitoring image information from the ship determined to be the monitoring target ship in the processing of S503 (S504). The monitoring information acquisition unit 161 requests the monitoring image generation device 1 mounted on the ship 110 determined to be the monitoring target ship to output a monitoring image signal, and when the monitoring image signal is input to the monitoring communication unit 151, acquires monitoring image information from the input monitoring image signal. The monitoring image information acquired in the processing of S504 may be a moving image or a still image. When the monitoring image information acquired in the processing of S504 is a still image, the performance and communication speed of the monitoring image generation device 1 and the monitoring image 120 can be lower than when the monitoring image information acquired in the processing of S504 is a moving image.

[0108] Next, the monitoring image information generation unit 163 generates display image information (S505) and stores the generated display image information in the monitoring storage unit 152. Then, the monitoring output unit 164 outputs the display image information generated in the processing of S506 to the monitoring display unit 154. In response to the input of the display image information, the monitoring display unit 154 displays a display image corresponding to the display image information.

[0109] FIG. 13 is a diagram showing an example of a display image displayed on the monitoring display unit 154. As shown in FIG.

[0110] Display image 170 has a first divided image 171, a second divided image 172, a third divided image 173, and a fourth divided image 174. In first divided image 171 to third divided image 173, scenery images and superimposed images acquired from three ships included in the monitored area indicated by arrow A in fourth divided image 174 are displayed together with monitored object information 171a, 172a, and 173a. The monitored object information 171a, 172a, and 173a includes the name of the monitored ship and the time when the displayed image was captured.

[0111] The fourth divided image 174 displays the monitoring target area and the position information of ships located in the vicinity of the monitoring target area in real time as a two-dimensional image viewed in a plane. [Explanation of symbols]

[0112] 1. Surveillance image generation device 21 Information Acquisition Department 22 Condition setting section 23 Image information generation unit 24 Collision determination section

Claims

1. an information acquisition unit that acquires topographical information indicating the topography of the seabed in the direction in which the ship is traveling; an image information generating unit that generates monitoring image information showing a monitoring image in which a superimposed image including a topographical image, which is an image of the topography seen from a point on the ship of the topography between a first point on the topography at sea corresponding to the topographical information and a second point remoter than the first point as viewed from the ship, is superimposed on a scenery image captured from the point; an output unit that outputs the monitoring image information; A monitoring image generating device comprising:

2. the information acquisition unit further acquires current position information indicating a current position of the ship and heading direction information indicating the heading direction; the first point is a point separated from the current position by a first distance in the traveling direction, The monitoring image generating device according to claim 1 , wherein the second point is a point spaced apart from the first point by a predetermined display length in the traveling direction.

3. determining whether or not there is a risk of the vessel coming into contact with the seabed based on the vessel's draft and the elevation of the topography; When it is determined that the vessel is at risk of coming into contact with the seabed, it outputs a warning signal. The monitoring image generating device according to claim 2 , further comprising a collision determination unit.

4. 4. The monitoring image generating device according to claim 3, wherein the collision determination unit determines whether or not there is a risk of the ship coming into contact with the seabed based on the tide level at the current position in addition to the ship's draft and the altitude.

5. an imaging unit that captures the scenery image and outputs scenery image information indicating the scenery image; 5. The surveillance image generating device according to claim 2, wherein the image information generating section generates the surveillance image information by superimposing the superimposed image on the scenery image corresponding to the scenery image information.

6. The information acquisition unit further acquires tide level information indicating a tide level at the current location, The monitoring image generating device according to claim 5 , wherein the image information generating unit corrects the position at which the superimposed image is superimposed on the scenery image in accordance with the tide level.

7. The image information generation unit acquiring structure information indicating structures included in the landscape image; The monitoring image generating device according to claim 5 , wherein a superimposed image showing the superimposed image is generated so as to include the structure.

8. The surveillance image generating device according to claim 5 , wherein the image information generating section generates a superimposed image showing the superimposed image so that the superimposed image can be viewed at an elevation angle corresponding to an inclination of the surveillance image generating device.

9. Obtain topographical information showing the topography of the seabed in the direction the ship is traveling, generating monitoring image information showing a monitoring image in which a superimposed image including a topographical image, which is an image of the topography seen from a point on the ship of the topography between a first point on the topography at sea corresponding to the topographical information and a second point remoter than the first point as viewed from the ship, is superimposed on a scenery image captured from the point; outputting the monitoring image information; A monitoring image generating method comprising:

10. Obtain topographical information showing the topography of the seabed in the direction the ship is traveling, generating monitoring image information showing a monitoring image in which a superimposed image including a topographical image, which is an image of the topography seen from a point on the ship of the topography between a first point on the topography at sea corresponding to the topographical information and a second point remoter than the first point as viewed from the ship, is superimposed on a scenery image captured from the point; outputting the monitoring image information; A monitoring image generating program that causes a computer to execute processing.

Images