Image generation device, image generation method, and image generation program

Abstract

To provide an image generation device that reduces a risk of a landscape image becoming difficult to view when a topographic image is superimposed on the landscape image and displayed.SOLUTION: An image generation device includes: an information acquisition unit 21 that acquires topographic information indicating a topography in a first direction; an image information generating unit 23 that generates image information indicating a superimposed image including a topographic image, which is an image of the topography viewed from one point, superimposed on a landscape image captured from one point, the topography being located between a first point on the topography corresponding to the topographic information and a second point located in the first direction and remote from the first point; and an output unit 25 that outputs the image information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image generating device, an image generating method, and an image generating program. [Background technology]

[0002] Various technologies for supporting the movement of ships, also referred to as navigation systems, have been known for some time. For example, Patent Document 1 describes a technology in which a composite image generated based on display information including the ship's planned route, structures located on the sea such as buoys, and other ships, as well as the position and attitude of the ship, is superimposed on an image captured from the ship and displayed in three dimensions on a display. The technology described in Patent Document 1 generates a composite image to be superimposed on an image captured from the ship according to the attitude of the ship, thereby enabling the acquisition of natural and highly realistic augmented reality images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-184614 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the technology described in Patent Document 1 makes it possible to visually recognize structures and other ships located on the sea, it does not display the seabed topography that is not visible to the operator operating the navigation system. When the technology described in Patent Document 1 overlays an image showing the seabed topography on a landscape image, the topographical image showing the seabed topography is overlaid over the entire surface of the sea, covering the entire displayed screen, which may make it difficult for the operator to visually recognize structures on the sea. Furthermore, when a topographical image showing the topography is overlaid over the entire surface of a landscape image captured from a moving body other than a ship, such as a vehicle, much of the landscape image may be covered by the topographical image, making it difficult for the operator to view the landscape image.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image generating device that reduces the risk of the scenery image becoming difficult to view when a topographical image is displayed superimposed on the scenery image. [Means for solving the problem]

[0006] The image generating device according to the embodiment includes an information acquisition unit that acquires terrain information indicating the terrain in a first direction, an image information generation unit that generates image information indicating an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remoter than the first point, is superimposed on a landscape image captured from a single point, and an output unit that outputs the image information.

[0007] In the image generating device according to the embodiment, the information acquisition unit further acquires current position information indicating the current position and first direction information indicating the first direction, and it is preferable that the first point is a point that is a first distance away from the current position in the direction of travel, and the second point is a point that is a predetermined display length away from the first point in the first direction.

[0008] In the image generating device of the embodiment, it is preferable to determine whether there is a risk that the bottom of the body of a vehicle traveling in a first direction will come into contact with the surface of the terrain based on the vertical distance between the bottom of the body of the vehicle traveling in a first direction and the bottom of the wheels that hold the body movably and the difference in elevation of the terrain, and to output an alarm signal when it is determined that there is a risk that the bottom of the body will come into contact with the surface of the terrain.

[0009] In the image generating device according to the embodiment, it is preferable that the image information generating section generates image information representing an image by superimposing a superimposed image on a landscape image captured by an imaging device arranged at one point.

[0010] In the image generating device according to the embodiment, it is preferable that the image information generating section generates superimposed image information indicating the superimposed image so that it can be viewed at an elevation angle according to the tilt of the imaging device.

[0011] An image generation method according to an embodiment includes acquiring topographical information indicating the topography in a first direction, generating image information indicating a perspective image in which a superimposed image including a topographical image, which is an image of the topography viewed from a single point, is superimposed on a landscape image captured from a single point, showing the topography between a first point on the topography corresponding to the topographical information and a second point located in the first direction and remoter than the first point, and outputting the image information.

[0012] The image generation program of the embodiment acquires topographical information indicating the topography in a first direction, generates image information indicating an image in which a superimposed image including a topographical image, which is an image of the topography between a first point on the topography corresponding to the topographical information and a second point located in the first direction and remoter than the first point, is superimposed on a landscape image captured from a single point, and causes a computer to execute a process of outputting the image information. [Effects of the Invention]

[0013] The image generating device according to the present invention can reduce the risk that the scenery image will become difficult to view when a topographical image is displayed superimposed on the scenery image. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a perspective view of a ship equipped with a monitoring image generation system according to a first embodiment. [Figure 2] 2 is a diagram showing a monitoring image generating device mounted on the ship shown in FIG. 1. FIG. [Figure 3] FIG. 3 is a functional block diagram of the monitoring image generating device shown in FIG. 2. [Figure 4] 3 is a flowchart showing an information setting process executed by the monitoring image generating device shown in FIG. 2. [Figure 5] 3A is a flowchart showing the monitoring image generation process executed by the monitoring image generation device shown in FIG. 2, and FIG. 3B is a flowchart showing more detailed processing of S205 shown in FIG. 3A. [Figure 6](a) is a diagram (part 1) for explaining the processing of S301 shown in Figure 5(b), (b) is a diagram (part 2) for explaining the processing of S301 shown in Figure 5(b), and (c) is a diagram (part 3) for explaining the processing of S301 shown in Figure 5(b). [Figure 7] 6 is a flowchart showing the detailed processing of step S206 shown in FIG. 5(a). [Figure 8] (a) is a diagram for explaining the processing of S401 shown in Figure 7, (b) is a diagram for explaining the processing of S402 shown in Figure 7, (c) is a diagram for explaining the processing of S403 shown in Figure 7, (d) is a diagram for explaining the processing of S404 shown in Figure 7, (e) is a diagram for explaining the processing of S405 shown in Figure 7, and (f) is a diagram for explaining the processing of S406 shown in Figure 7. [Figure 9] 6 is a flowchart showing the detailed processing of step S207 shown in FIG. 5(a). [Figure 10] 4 is a diagram showing an example of a monitoring image displayed on the display unit shown in FIG. 3. FIG. [Figure 11] FIG. 2 is a functional block diagram of the vessel monitoring device. [Figure 12] 12 is a flowchart showing an image display process executed by the vessel monitoring device shown in FIG. 11. [Figure 13] 12 is a diagram showing an example of a display image displayed on the monitoring display unit shown in FIG. 11. FIG. [Figure 14] FIG. 10 is a perspective view of a vehicle equipped with a monitoring image generation system according to a second embodiment. [Figure 15] 15 is a diagram showing a monitoring image generating device mounted on the vehicle shown in FIG. 14. FIG. [Figure 16] FIG. 16 is a functional block diagram of the monitoring image generating device shown in FIG. [Figure 17] 16 is a flowchart showing an information setting process executed by the monitoring image generating device shown in FIG. 15. FIG. [Figure 18]16(a) is a flowchart showing the monitoring image generation process executed by the monitoring image generation device shown in FIG. 15, and FIG. 16(b) is a flowchart showing more detailed processing of S704 shown in FIG. [Figure 19] 18(a) is a flowchart showing the process of S705 shown in FIG. 18(a) in more detail, and FIG. 18(b) is a flowchart showing the process of S706 shown in FIG. 18(a) in more detail. DETAILED DESCRIPTION OF THE INVENTION

[0015] An image generating device, an image generating method, and an image generating program according to the present invention will be described below with reference to the drawings. However, please note that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions set forth in the claims and their equivalents.

[0016] (Configuration and Functions of a Surveillance Image Generation System Having a Surveillance Image Generation Device According to the First Embodiment) FIG. 1 is a perspective view of a ship equipped with a surveillance image generation system according to a first embodiment, FIG. 2 is a diagram showing a surveillance image generation device mounted on the ship shown in FIG. 1, and FIG. 3 is a functional block diagram of the surveillance image generation device shown in FIG. 2.

[0017] The surveillance image generation system 100 has a GNSS receiver 101, a satellite communication receiver 102, and a surveillance image generation device 1 which is an example of an image surveillance device according to the present invention, and is mounted on a ship 110. Based on signals received from the GNSS receiver 101 and the satellite communication receiver 102, the surveillance image generation system 100 displays a scenery image and a superimposed image to be superimposed on the scenery image on a screen visible to an operator 112 on board the ship 110.

[0018] The GNSS receiver 101 receives signals from a satellite positioning system (Global Navigation Satellite System (GNSS)) such as a Global Positioning System (GPS). In response to receiving signals from the GNSS, the GNSS receiver 101 outputs a current position signal indicating the current position of the ship 110 and a heading signal indicating the direction in which the ship 110 is traveling to the surveillance image generation device 1. The satellite communication receiver 102 receives signals from a communication satellite that can be connected to the Internet network via a gateway earth station. The satellite communication receiver 102 receives a tide level signal indicating the tide level at the current position of the ship 110, which is extracted from information on tide levels provided by the Marine Information Department of the Japan Coast Guard via the Internet network, and outputs the received tide level signal to the surveillance image generation device 1. The satellite communications receiver 102 also receives, via the Internet network, AIS signals containing the vessel's identification code, type, position, course, speed, navigation status, and other safety-related information of an AIS vessel that is located near the vessel 110 and is equipped with an AIS signal transceiver, and outputs the received AIS signals to the surveillance image generation device 1. The satellite communications receiver 102 also receives, via the Internet network, non-AIS signals indicating the vessel's position, course, speed, navigation status, and other safety-related information of a non-AIS vessel that is located near the vessel 110 and is not equipped with an AIS signal transceiver, and outputs the received non-AIS signals to the surveillance image generation device 1. A server that outputs the AIS signals and non-AIS signals is connected to the GNSS receiver 101 via the Internet network.

[0019] The monitoring image generating device 1 is disposed inside a cabin 111 of a ship 110 so as to be visible to an operator 112 and so as to enable the built-in imaging unit 16 to capture images of the traveling direction of the ship 110 through a porthole 113. The monitoring image generating device 1 has a communication unit 11, a memory unit 12, an operation unit 13, a display unit 14, an audio output unit 15, the imaging unit 16, a tilt detection unit 17, and a processing unit 20. The communication unit 11, the tilt detection unit 17, and the processing unit 20 are connected via a bus 18.

[0020] The communication unit 11 is configured to enable the monitoring image generation device 1 to communicate with the GNSS receiver 101 to the satellite communication receiver 102, and includes a communication interface circuit. The communication interface circuit is, for example, a communication interface circuit for a wired LAN, a wireless LAN, LTE (Long Term Evolution), or the like. The communication unit 11 supplies signals input from the GNSS receiver 101 to the satellite communication receiver 102 to the processing unit 20, and outputs signals supplied from the processing unit 20 to the GNSS receiver 101 to the satellite communication receiver 102.

[0021] The storage unit 12 is configured to store data and programs and includes, for example, a semiconductor memory. The storage unit 12 stores an operating system program, a driver program, an application program, data, and the like used in processing by the processing unit 20. For example, the storage unit 12 stores, as an application program, an information setting program that causes the processing unit 20 to execute an information setting process that sets various information used when executing an image generation process that generates an image including a topographical image showing the underwater topography. The storage unit 12 also stores, as an application program, an image generation program that causes the processing unit 20 to execute the image generation process. The information setting program and the image generation program are installed into the storage unit 12 from a computer-readable, non-transitory, portable storage medium such as a compact disc (CD)-read-only memory (ROM) or a digital versatile disc (DVD)-ROM. The storage unit 12 also stores various data and information used when executing the information setting process and the image generation process. For example, the storage unit 12 stores elevation data, also referred to as topographical information, that indicates the elevation of the underwater topography as elevation tiles 12a.

[0022] The elevation tile 12a has a 256 x 256 matrix structure, with the coordinates of the seafloor located in each matrix element. The elevation tile defines elevation data at various intervals depending on the zoom level. The structure and function of the elevation tile 12a are well known, so a detailed description will be omitted here.

[0023] The memory unit 12 also stores structure information indicating structures included in the scenery in the direction of travel of the ship 110, including land-based structures such as lighthouses and port facilities, and offshore structures such as buoys and offshore wind power generation facilities. The structure information includes the name of the structure, shape data indicating the shape of the structure, and the latitude and longitude of the structure. The memory unit 12 also stores route information indicating areas that are preferably visible when maneuvering the ship 110, including route lines indicating routes along which the ship 110 can travel, and no-entry areas into which the ship 110 is prohibited from entering. The route information includes the names of areas including the route lines and no-entry areas, shape data indicating the shape of the areas, and the latitude and longitude of the areas. The memory unit 12 also stores alarm threshold information indicating an alarm threshold used to determine whether the bottom of the ship 110 will come into contact with the seabed.

[0024] The operation unit 13 is configured to receive user operations on the surveillance image generation device 1, and includes, for example, a touch panel. The operation unit 13 generates a signal in response to the operation of the operator 112, and supplies the generated signal to the processing unit 20.

[0025] The display unit 14 is configured to display an image, and includes a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 14 generates and displays an image based on a signal supplied from the processing unit 20.

[0026] The audio output unit 15 is configured to output audio and includes a speaker. The audio output unit 15 outputs audio based on a signal supplied from the processing unit 20.

[0027] The imaging unit 16 is configured to capture images of the surroundings of the monitoring image generation device 1 and includes a camera. The camera includes an imaging optical system for forming an image on a light-receiving surface, a photoelectric conversion element, and an image generation circuit for generating an image based on the output of the photoelectric conversion element. The photoelectric conversion elements included in the camera are CMOS (Complementary Metal Oxide Semiconductor) and CCD (Charge Coupled Device) sensors, etc., that are arranged two-dimensionally on the light-receiving surface and output an electrical signal according to the amount of incident light. The imaging unit 16 captures and acquires a landscape image showing the scenery in the direction of travel of the ship 110, and supplies a landscape image signal showing the acquired landscape image to the processing unit 20.

[0028] The tilt detection unit 17 is equipped with, for example, a three-axis acceleration sensor or a gyro sensor, and detects the tilt of the monitoring image generation device based on the indication value of the three-axis acceleration sensor or gyro sensor, and supplies a tilt signal indicating the detected tilt to the processing unit 20.

[0029] The processing unit 20 is configured to comprehensively control the operation of the surveillance image generation device 1, and includes one or more processors and their peripheral circuits. The processing unit 20 includes, for example, a CPU (Central Processing Unit). The processing unit 20 may also include a DSP (Digital Signal Processor), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. The processing unit 20 executes processing based on a program stored in the storage unit 12. The processing unit 20 also controls the operation of each component of the surveillance image generation device 1 so that processing is executed appropriately.

[0030] The processing unit 20 has, as functional blocks, an information acquisition unit 21, a condition setting unit 22, an image information generation unit 23, a collision determination unit 24, and an output unit 25. These functional blocks are functional modules realized based on a program executed by the processing unit 20. These functional blocks may be firmware implemented in the monitoring image generation device 1.

[0031] Fig. 4 is a flowchart showing the information setting process executed by the monitoring image generation device 1. The information setting process shown in Fig. 4 is realized by the processing unit 20 working in cooperation with each component of the monitoring image generation device 1 based on a program stored in the storage unit 12. The information setting process shown in Fig. 4 is executed every time the monitoring image generation device 1 is deployed on the ship 110.

[0032] First, the information acquisition unit 21 acquires first distance information indicating a first distance in response to the operation of the operation unit 13 by the operator 112 (S101), and stores the acquired first distance information in the storage unit 12. The first distance is the distance from the ship 110 to the near clip plane of the view frustum. The first distance, which is the distance to the near clip plane of the view frustum, is used when generating, as a view frustum, a topographical image indicating the topography of the seabed included in the superimposed image to be superimposed on the landscape image captured by the imaging unit 16. Next, the condition setting unit 22 sets the first distance acquired in the process of S101 as information indicating the first distance to be used when generating the topographical image (S102). The condition setting unit 22 sets the first distance by associating the first distance information acquired in the process of S101 with a predetermined identifier.

[0033] Next, the information acquisition unit 21 acquires display length information indicating a display length, which is the distance between the near clip plane and the far clip plane of the terrain image generated as a view frustum, in response to the operation of the operation unit 13 by the operator 112 (S103). The information acquisition unit 21 stores the acquired display length information in the storage unit 12. Next, the condition setting unit 22 sets the display length acquired in the processing of S103 as information indicating the display length to be used when generating the terrain image (S104). The condition setting unit 22 sets the display length by associating the display length information acquired in the processing of S103 with a predetermined identifier.

[0034] Next, the information acquisition unit 21 acquires tilt information indicating the tilt of the monitoring image generation device 1 from a horizontal plane in response to operation of the operation unit 13 by the operator 112 (S105). When instructed to acquire tilt information via the operation unit 13, the information acquisition unit 21 requests the tilt detection unit 17 to output a tilt signal, and when the tilt signal is input, acquires tilt information indicating the tilt corresponding to the input tilt signal and stores the acquired tilt information in the storage unit 12. Next, the condition setting unit 22 sets the tilt information acquired in the process of S105 as information indicating the tilt to be used when generating a superimposed image (S106). The condition setting unit 22 sets the tilt by associating the tilt information acquired in the process of S105 with a predetermined identifier.

[0035] Next, the information acquisition unit 21 acquires height information indicating the height from sea level of the location where the monitoring image generation device 1 is placed in response to operation of the operation unit 13 by the operator 112 (S107). The information acquisition unit 21 stores the acquired height information in the storage unit 12. Next, the condition setting unit 22 sets the height information acquired in the process of S107 as information indicating the height to be used when generating a superimposed image (S108). The condition setting unit 22 sets the height by associating the height information acquired in the process of S107 with a predetermined identifier. The height set in the process of S108 is the height of the imaging unit 16 from sea level.

[0036] Next, the information acquisition unit 21 acquires draft information indicating the draft of the ship 110 in response to operation of the operation unit 13 by the operator 112 (S109), and stores the acquired draft information in the memory unit 12. Next, the condition setting unit 22 sets the draft information acquired in the processing of S109 as information indicating the draft to be used in the collision determination processing that determines whether or not the ship 110 will collide with the seabed (S110). The condition setting unit 22 sets the draft by associating the draft information acquired in the processing of S109 with a predetermined identifier.

[0037] Fig. 5(a) is a flowchart showing the monitoring image generation process executed by the monitoring image generation device 1. The monitoring image generation process shown in Fig. 5(a) is realized by the processing unit 20 working in cooperation with each component of the monitoring image generation device 1 based on a program stored in the storage unit 12. The monitoring image generation process shown in Fig. 5(a) is executed at a predetermined execution interval.

[0038] First, the image information generation unit 23 starts capturing a scenery image by the imaging unit 16 in response to the operation of the operation unit 13 by the operator 112 (S201). When the operator 112 operates the operation unit 13 to acquire an instruction to start capturing an image via the operation unit 13, the image information generation unit 23 starts the operation of the imaging unit 16 and starts capturing a scenery image in the traveling direction of the ship 110 by the imaging unit 16. The image information generation unit 23 acquires scenery image information indicating the scenery image captured by the imaging unit 16, and outputs the acquired scenery image information to the display unit 14. The display unit 14 displays a scenery image corresponding to the input scenery image information.

[0039] Next, the information acquisition unit 21 acquires current position information indicating the current position of the ship 110 (S202). The information acquisition unit 21 requests the GNSS receiver 101 to output a current position signal via the communication unit 11, and when the current position signal is input to the communication unit 11, acquires current position information indicating the current position corresponding to the input current position signal.

[0040] Next, the information acquisition unit 21 acquires heading information indicating the heading direction of the ship 110 (S203). The information acquisition unit 21 requests the GNSS receiver 101 to output a heading signal via the communication unit 11, and when the heading signal is input to the communication unit 11, acquires heading information indicating the heading corresponding to the input heading signal.

[0041] Next, the information acquisition unit 21 acquires tide level information indicating the tide level at the current position of the ship 110 (S204). The information acquisition unit 21 requests the satellite communication receiver 102 to output a current position signal via the communication unit 11, and when a tide level signal indicating the tide level at the current position of the ship 110 is input to the communication unit 11, the information acquisition unit 21 acquires tide level information indicating the tide level corresponding to the input tide level signal.

[0042] Next, the image information generation unit 23 generates terrain image information showing a point on the ship 110, i.e., a terrain image, which is an image of the seabed terrain in the direction of travel of the ship 110 as seen through the monitoring image generation device 1 (S205), and stores the generated terrain image information in the memory unit 12.

[0043] FIG. 5B is a flowchart showing the process of S205 in more detail.

[0044] First, the image information generation unit 23 sets the range for generating a terrain image (S301) using the first distance, display length, and inclination set in the processes of S102, S104, and S106, and the current position information and traveling direction information acquired in the processes of S202 and S203. The range for generating a terrain image is called the terrain image range.

[0045] 6(a), the image information generation unit 23 sets a quadrangular pyramid R1 including a viewing frustum indicating the range for generating a terrain image from the current position corresponding to the current position information acquired in the processing of S202 and the traveling direction corresponding to the traveling direction information acquired in the processing of S203. The vertex P of the quadrangular pyramid R1 is the latitude and longitude indicating the current position corresponding to the current position information acquired in the processing of S202, and the extension direction of the normal L1 of the base SL of the quadrangular pyramid R1 is the traveling direction corresponding to the traveling direction information acquired in the processing of S203.

[0046] Next, as shown in FIG. 6(b), the image information generation unit 23 sets a plane that is separated from the vertex P of the pyramid R1 by the first distance L1 set in the processing of S102 as the near clip plane S1. The point at which the near clip plane S1 is located is also referred to as the first point. The near clip plane S1 is the end of the terrain image range that is closest to the current position. Next, as shown in FIG. 6(c), the image information generation unit 23 sets a plane that is separated from the near clip plane S1 in the extension direction of the normal line L1 by the display length LD set in the processing of S104 as the far clip plane S2. The point at which the far clip plane S2 is located is also referred to as the second point. The far clip plane S2 is the end of the terrain image range that is farthest from the current position.

[0047] Next, the image information generation unit 23 acquires elevation data included in the near clip plane S1 of the terrain image range determined in the processing of S301 (S302). The image information generation unit 23 references the elevation tile 12a to acquire elevation data of the terrain superimposed on the near clip plane S1 at a position separated by a first distance from the current position in the traveling direction, and stores the acquired elevation data in the storage unit 12 as a first terrain data group.

[0048] Next, the image information generation unit 23 acquires elevation data included in a plane that is separated from the near clip plane S1 by a predetermined plot interval in the normal line L1 direction and is parallel to the near clip plane S1 (S303). Similar to the processing of S302, the image information generation unit 23 references the elevation tile 12a to acquire elevation data of the terrain that is superimposed on the plane that is separated from the near clip plane S1 by the predetermined plot interval, and stores the acquired elevation data in the storage unit 12 as a second terrain data group.

[0049] Next, the image information generation unit 23 determines whether the distance between the acquisition point, which is separated by the plot interval in the direction of the normal L1 from the plane on which the elevation data was acquired in the process of S303, and the near clip plane S1 is longer than the display length LD set in the process of S104 (S304). If it is determined that the distance between the acquisition point and the near clip plane S1 is not longer than the display length LD (S304-NO), the process returns to S303. Thereafter, the processes of S303 and S304 are repeated until it is determined that the distance between the acquisition point and the near clip plane S1 is longer than the display length LD (S304-YES). By repeating the processes of S303 and S304, multiple terrain data groups are stored in the storage unit 12, with the data group numbers being sequentially increased, starting with the third terrain data group.

[0050] When the image information generation unit 23 determines that the distance between the acquisition point and the near clip plane S1 is longer than the display length (S304-YES), it stores the first to Nth terrain data groups acquired by the processes of S302 to S304 in the storage unit 12 as terrain image information (S305). When the image information generation unit 23 stores the terrain image information in the storage unit 12, the process of S205 ends.

[0051] Following the processing of S205, the image information generation unit 23 generates superimposed image information indicating a superimposed image that can be superimposed on a point on the ship 110, i.e., a landscape image captured by the imaging unit 16 (S206), and stores the generated superimposed image information in the memory unit 12.

[0052] FIG. 7 is a flowchart showing the process of S206 in more detail.

[0053] First, the image information generation unit 23 sets an imaging range to be imaged by the imaging unit 16 (S401). As shown in Fig. 8(a), the image information generation unit 23 sets an imaging range R2, which is a quadrangular pyramid indicating the range in which a terrain image is to be generated, from the current position corresponding to the current position information acquired in the processing of S202 and the traveling direction corresponding to the traveling direction information acquired in the processing of S203.

[0054] Next, the image information generation unit 23 arranges the terrain image corresponding to the terrain image information generated in the process of S205 in the imaging range R2 set in the process of S401 (S402). As shown in Fig. 8(b), the image information generation unit 23 acquires the terrain image information stored in the storage unit 12, and arranges, in the imaging range R2, the terrain image I1, which is a point cloud image indicating, as points, each of the elevation data included in the terrain data group corresponding to the acquired terrain image information.

[0055] The image information generation unit 23 extracts structure information including structures included in the imaging range R2 based on the latitude and longitude included in the structure information stored in the storage unit 12. As shown in Fig. 8(c), the image information generation unit 23 arranges, in the imaging range R2, a structure image I2 that shows the shape included in the structure information extracted based on the latitude and longitude.

[0056] Next, the image information generation unit 23 arranges the route information included in the imaging range in the imaging range R2 estimated in the processing of S401 (S404). The route information indicates route lines indicating routes that the ship 110 can travel and areas that are preferably visible when maneuvering the ship 110, including no-entry areas where the ship 110 is prohibited from entering. The image information generation unit 23 extracts route information indicating the area included in the imaging range R2 based on the latitude and longitude included in the route information stored in the storage unit 12. As shown in FIG. 8(d), the image information generation unit 23 arranges an area image I3 indicating the shape included in the route information extracted based on the latitude and longitude in the imaging range R2.

[0057] Next, the image information generation unit 23 places ships included in the imaging range R2 in the imaging range R2 estimated in the processing of S401 (S405). The image information generation unit 23 requests the server that outputs the AIS signal to output the AIS signal, and when the AIS signal is input to the communication unit 11, acquires a signal related to the ship from the input AIS signal. The image information generation unit 23 extracts AIS ships included in the imaging range R2 from the acquired signals related to the ship, and stores the name, captain, speed, and position of the ship from the AIS signal of the extracted AIS ship in the memory unit 12. The image information generation unit 23 also requests the server that transmits the non-AIS signal to output the non-AIS signal, and when the non-AIS signal is input to the communication unit 11, acquires a signal related to the ship from the input non-AIS signal. The image information generation unit 23 extracts non-AIS ships within the imaging range R2 from the acquired ship-related signals, and stores the names, speeds, and positions of the ships from the extracted non-AIS signals in the memory unit 12. As shown in FIG. 8(e), the image information generation unit 23 places images of the ships according to their distance from the ship 110 at the positions of the AIS and non-AIS ships stored in the memory unit 12, and places images including the names and speeds of the AIS and non-AIS ships in association with the ship image I4. Note that the images placed in association with the ship images may include the captains of the AIS ships and the distances between the AIS and non-AIS ships and the ship 110, in addition to the names and speeds of the AIS and non-AIS ships.

[0058] Next, the image information generation unit 23 tilts the imaging range in which the undersea topography and the like are located in the processes of S402 to S405 in accordance with the tilt of the monitoring image generation device 1 so that it is visible at an elevation angle in accordance with the tilt of the monitoring image generation device 1 (S406). As shown in Fig. 8(f), the image information generation unit 23 tilts the imaging range R2 by a tilt θ in accordance with the tilt corresponding to the tilt information set in the process of S106.

[0059] Next, the image information generation unit 23 sets the height of the imaging range R2 from sea level so that the height of the imaging range R2 from sea level matches the height of the monitoring image generation device 1 from sea level (S407). The image information generation unit 23 sets the height of the imaging range R2 so that it matches the height set in the processing of S108.

[0060] Next, the image information generation unit 23 corrects the height of the imaging range R2 set in the process of S407 in accordance with the tide level at the current location (S408). The information acquisition unit 21 corrects the height of the imaging range R2 in accordance with the tide level corresponding to the tide level information acquired in the process of S204.

[0061] Next, the image information generating unit 23 generates an orientation image indicating the orientation (S409). The image information generating unit 23 generates an isosceles triangular image as an orientation image having a slope corresponding to the angle difference between the north direction and the traveling direction corresponding to the traveling direction information acquired in the process of S203, and stores orientation image information indicating the generated orientation image. The orientation image may be an image other than an isosceles triangular image.

[0062] Next, the image information generation unit 23 generates superimposed image information by synthesizing the azimuth image generated in the processing of S406 with the imaging range R2 (S410), and stores the generated superimposed image information in the storage unit 12. When the image information generation unit 23 stores the superimposed image information in the storage unit 12, the processing of S206 ends.

[0063] Following the process of S206, the collision determination unit 24 executes a collision determination process (S207).

[0064] FIG. 9 is a diagram showing more detailed processing of the collision determination processing shown in S207.

[0065] First, the collision determination unit 24 extracts from the elevation tile 12a the elevation of the seabed topography at a point a predetermined distance away in the traveling direction obtained in the processing of S203 (S501), and stores elevation information indicating the extracted elevation in the storage unit 12. For example, the collision determination unit 24 extracts from the elevation tile 12a the elevation of the seabed topography between the near clip plane S1 and the far clip plane S2.

[0066] Next, the collision determination unit 24 calculates the water depth at the point whose elevation was extracted in the process of S501 from the tide level information acquired in the process of S204 and the elevation extracted in the process of S501 (S502). The collision determination unit 24 calculates the water depth by adding the tide level corresponding to the tide level information to the tide level information acquired in the process of S204 and the elevation extracted in the process of S501.

[0067] Next, the collision determination unit 24 calculates the separation distance between the bottom of the ship 110 and the seabed from the draft set in the process of S110 and the water depth calculated in the process of S502 (S503). The collision determination unit 24 stores the calculated separation distance in the memory unit 12.

[0068] Next, the collision determination unit 24 determines whether or not there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504). The collision determination unit 24 acquires warning threshold information stored in the memory unit 12. The collision determination unit 24 compares the warning threshold corresponding to the acquired warning threshold information with the separation distance calculated in the processing of S503, and determines whether or not the separation distance calculated in the processing of S503 is equal to or greater than the warning threshold. When the separation distance calculated in the processing of S503 is less than the warning threshold, the collision determination unit 24 determines that there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504-YES). On the other hand, when the separation distance calculated in the processing of S503 is equal to or greater than the warning threshold, the collision determination unit 24 determines that there is no risk that the bottom of the vessel 110 will come into contact with the seabed (S504-NO).

[0069] When the collision determination unit 24 determines that there is a risk that the bottom of the vessel 110 will come into contact with the seabed (S504-YES), it outputs a collision warning signal indicating that there is a risk that the bottom of the vessel 110 will come into contact with the seabed to the display unit 14 and the audio output unit 15. The display unit 14 displays a collision area image indicating the area where the bottom of the vessel 110 is likely to come into contact with the seabed by superimposing it on a landscape image. In response to the input of the collision warning signal, the audio output unit 15 outputs an audio signal indicating that there is a risk that the bottom of the vessel 110 will come into contact with the seabed. When the collision determination unit 24 determines that there is no risk that the bottom of the vessel 110 will come into contact with the seabed (S504-NO), the processing of S207 ends. Note that instead of displaying the collision area image, text indicating a warning for each type, such as "Caution: Contact with the ground" or "Caution: Entering a no-approach area," may be displayed.

[0070] Following the process of S207, the information acquisition unit 21 acquires scenery image information indicating the scenery image captured by the imaging unit 16 (S208). The information acquisition unit 21 requests the imaging unit 16 to output a scenery image signal indicating the scenery image captured by the imaging unit 16, and acquires the scenery image information indicating the scenery image captured by the imaging unit 16.

[0071] Next, the image information generation unit 23 superimposes the superimposed image corresponding to the superimposed image information generated in the processing of S206 on the scenery image corresponding to the scenery image information acquired in the processing of S208 to generate monitoring image information (S209).

[0072] Next, the output unit 25 outputs the monitoring image information generated in the process of S209 to the display unit 14 (S210). In response to the input of the monitoring image information, the display unit 14 displays a monitoring image corresponding to the input monitoring image information.

[0073] FIG. 10 is a diagram showing an example of a monitoring image displayed on the display unit 14. As shown in FIG.

[0074] The monitoring image 120 includes a terrain image 121, a structure image 122, a route image 123, and a ship image 124, and is displayed on the display unit 14 superimposed on a landscape image captured by the imaging unit 16. The terrain image 121 is formed from a point cloud and shows the topography of the seabed. The structure image includes multiple buoys and shows structures located on land and at sea. The route image 123 includes a route line image 125 and a no-entry area image 126. The ship image 124 includes a ship image 127 showing a ship and a ship information image 128 showing information about the ship. In addition, a collision warning area image 129 corresponding to a collision warning signal is superimposed on the monitoring image 120. Furthermore, a warning image 130 showing various warnings is superimposed on the monitoring image 120. The route image 123 may include a navigable width corresponding to the width of the ship sailing toward the ship 110. Furthermore, the monitoring image generating device 1 may output an alarm signal when the ship 110 enters the navigable width included in the route image 123.

[0075] Following the process of S208, the image information generation unit 23 determines whether or not an instruction to end the image generation process has been given (S211). When a process end signal is input in response to operation of the operation unit 13 by the operator 112, the image information generation unit 23 determines that an instruction to end the image generation process has been given (S211-YES). If the image information generation unit 23 determines that an instruction to end the image generation process has not been given (S211-NO), the process returns to S202. Thereafter, the processes of S202 to S211 are repeated until the image information generation unit 23 determines that an instruction to end the image generation process has been given (S211-YES).

[0076] When it is determined that the image information generating unit 23 has instructed to end the image generating process (YES in S211), the capturing of the scenery image by the capturing unit 16 is terminated in response to the operation of the operating unit 13 by the operator 112 (S212).

[0077] (Operational effects of the monitoring image generating device according to the first embodiment) The monitoring image generating device 1 can reduce the risk of ships and structures on the sea becoming difficult to see by superimposing a topographical image of the seabed between the near clip plane S1 and the far clip plane S2 on the landscape image, rather than a topographical image of the entire seabed included in the landscape image. For example, by making the display length LD very short and positioning the near clip plane S1 and the far clip plane S2 close to each other, it is possible to display the cross-sectional shape of the seabed at a point separated by the first distance L1 from the ship 110 as a topographical image.

[0078] In addition, the monitoring image generating device 1 displays an image of the seabed topography between the near clip plane S1 and the far clip plane S2, which are located in the direction of travel of the ship 110, allowing the operator 112 to easily understand the topography of the seabed in the direction of travel of the ship 110.

[0079] Furthermore, the monitoring image generation device 1 can quickly acquire the seabed topography by using the elevation tiles 12a. Furthermore, the monitoring image generation device 1 generates a topography image by forming the seabed topography as a point cloud, so that the monitoring image generation device 1 can quickly generate and display the seabed topography image.

[0080] Furthermore, the monitoring image generating device 1 corrects the position at which the seabed topography image is superimposed on the landscape image according to the tide level at the current position of the ship 110, so that the seabed topography image can be superimposed on the landscape image with high accuracy regardless of changes in the tide level.

[0081] Furthermore, since the surveillance image generating device 1 generates a superimposed image to include ships and land and sea structures contained in the landscape image, the operator 112 can easily see the structures even when the landscape image captured by the imaging unit 16 is unclear due to nighttime or thick fog, etc.

[0082] Furthermore, since the monitoring image generating device 1 generates the superimposed image information so that it is viewed at an elevation angle corresponding to the tilt of the monitoring image generating device 1, the superimposed image can be superimposed on the landscape image regardless of the tilt of the monitoring image generating device 1.

[0083] Furthermore, the monitoring image generation device 1 outputs an alarm signal when there is a risk that the ship 110 will come into contact with the seabed, allowing the operator 112 to steer the ship 110 so as to reduce the risk of it coming into contact with the seabed.

[0084] In addition, the monitoring image generating device 1 determines whether there is a risk of the ship 110 coming into contact with the seabed based on the tide level at the current location of the ship 110, in addition to the draft of the ship 110 and the elevation of the seabed, so that it can more accurately determine the risk of the ship 110 coming into contact with the seabed.

[0085] (Modification of the monitoring image generating device according to the first embodiment) The monitoring image generating device 1 arranges land and sea structures, route information, and other ships in addition to the topographical image in the superimposed image, but in the monitoring image generating device according to the embodiment, only the topographical image may be arranged in the superimposed image. Also, in the monitoring image generating device according to the embodiment, the superimposed image may arrange any of the topographical image and structures, route information, and other ships.

[0086] Furthermore, the surveillance image generating device 1 generates terrain images and superimposed images as a viewing cone, but the surveillance image generating device according to the embodiment may also generate terrain images and superimposed images by performing image processing such as affine transformation and projective transformation on the seabed terrain, etc.

[0087] Furthermore, although the monitoring image generation device 1 generates terrain image information using elevation tiles, the monitoring image generation device according to the embodiment may generate terrain image information using information acquired using a shape detection sensor such as sonar. Furthermore, although the monitoring image generation device 1 generates a terrain image by forming the seabed terrain as a point cloud, the monitoring image generation device according to the embodiment may generate a terrain image by forming the seabed terrain as a line drawing of contour lines.

[0088] Furthermore, the monitoring image generating device 1 corrects the position at which the superimposed image is superimposed on the scenery image in accordance with the tide level at the current position of the ship 110, but the monitoring image generating device according to the embodiment does not have to perform the correction according to the tide level. Furthermore, the monitoring image generating device according to the embodiment performs the collision detection process using the tide level at the current position of the ship 110, but the monitoring image generating device according to the embodiment may perform the collision detection process using the ship's draft and the seabed elevation without using the tide level.

[0089] Furthermore, although the surveillance image generating device 1 has an operation unit 13 to an imaging unit 16, the surveillance image generating device according to the embodiment may not have any of the operation unit 13 to the imaging unit 16, and the operation unit 13 to the imaging unit 16 may be arranged as separate devices from the surveillance image generating device.

[0090] Furthermore, although the surveillance image generation system 100 having the surveillance image generation device 1 superimposes a superimposed image on a scenery image visible from a single ship 110, the surveillance image generation system according to the embodiment may be implemented on a plurality of ships. When the surveillance image generation system according to the embodiment is implemented on a plurality of ships, the scenery image captured by the surveillance image generation system and the superimposed image superimposed on the scenery image may be displayable by the ship monitoring device.

[0091] (Ship monitoring device according to an embodiment) FIG. 11 is a functional block diagram of a vessel monitoring device capable of displaying a scenery image captured by a monitoring image generating system and a superimposed image superimposed on the scenery image.

[0092] The vessel monitoring device 150 has a monitoring communication unit 151, a monitoring memory unit 152, a monitoring operation unit 153, a monitoring display unit 154, a monitoring audio output unit 155, and a monitoring processing unit 160, and displays scenery images and superimposed images captured by a monitoring image generation device 1 mounted on a plurality of vessels 110. Each of the plurality of vessels 110 is equipped with a monitoring image generation device 1. The monitoring communication unit 151, the monitoring audio output unit 155, and the monitoring processing unit 160 are connected via a bus 156.

[0093] The monitoring communication unit 151 is configured to enable communication with the server that outputs AIS signals and non-AIS signals, and the monitoring image generation device 1 via the Internet, and includes a communication interface circuit. The communication interface circuit is, for example, a communication interface circuit for wired LAN, wireless LAN, LTE, or the like. It supplies signals input from the server that outputs AIS signals and non-AIS signals, and the monitoring image generation device 1 to the monitoring processing unit 160, and outputs signals supplied from the monitoring processing unit 160 to the server that outputs AIS signals and non-AIS signals, and to the monitoring image generation device 1.

[0094] The monitoring storage unit 152 is configured to store data and programs, and includes, for example, a semiconductor memory. The monitoring storage unit 152 stores an operating system program, a driver program, an application program, data, and the like used in processing by the monitoring processing unit 160. For example, the monitoring storage unit 152 stores, as an application program, an image display program that causes the monitoring processing unit 160 to execute image display processing that displays monitoring images generated by the monitoring image generation devices 1 installed on multiple ships 110. The image display program is installed into the monitoring storage unit 152 from a computer-readable, non-transitory, portable storage medium such as a CD-ROM or DVD-ROM. The monitoring storage unit 152 also stores various data and information used when executing the image display processing.

[0095] The monitoring operation unit 153 is configured to receive user operations on the vessel monitoring device 150, and includes, for example, a keyboard. The monitoring operation unit 153 generates a signal in response to the user operation, and supplies the generated signal to the monitoring processing unit 160.

[0096] The monitoring display unit 154 is configured to display an image and includes a display such as a liquid crystal display or an organic EL display. The monitoring display unit 154 generates and displays an image based on a signal supplied from the monitoring processing unit 160.

[0097] The monitoring audio output unit 155 is configured to output audio and includes a speaker. The monitoring audio output unit 155 outputs audio based on a signal supplied from the monitoring processing unit 160.

[0098] The monitoring processing unit 160 is configured to comprehensively control the operation of the ship monitoring device 150, and includes one or more processors and their peripheral circuits. The monitoring processing unit 160 includes, for example, a CPU. The monitoring processing unit 160 may also include a DSP, an LSI, an ASIC, an FPGA, etc. The monitoring processing unit 160 executes processing based on a program stored in the monitoring memory unit 152. The monitoring processing unit 160 also controls the operation of each component of the ship monitoring device 150 so that processing is executed appropriately.

[0099] The surveillance processing unit 160 has, as functional blocks, a surveillance information acquisition unit 161, a surveillance determination unit 162, a surveillance image information generation unit 163, and a surveillance output unit 164. These functional blocks are functional modules realized based on a program executed by the surveillance processing unit 160. These functional blocks may be firmware implemented in the surveillance image generation device 1.

[0100] Fig. 12 is a flowchart showing the image display processing executed by the vessel monitoring device 150. The information setting processing is realized by the monitoring processing unit 160 working in cooperation with each component of the vessel monitoring device 150 based on a program stored in the monitoring memory unit 152. The image display processing shown in Fig. 12 is executed at a predetermined execution cycle.

[0101] First, the monitoring information acquisition unit 161 acquires AIS ship information (S501) and stores the acquired AIS ship information in the monitoring memory unit 152. The monitoring information acquisition unit 161 requests the server that outputs the AIS signal to output the AIS signal, and when the AIS signal is input to the monitoring communication unit 151, it acquires AIS ship information including the name and position of the ship from the input AIS signal.

[0102] Next, the monitoring information acquisition unit 161 acquires non-AIS ship information (S502) and stores the acquired non-AIS ship information in the monitoring memory unit 152. The monitoring information acquisition unit 161 requests the server that outputs the non-AIS signal to output the non-AIS signal, and when the non-AIS signal is input to the monitoring communication unit 151, it acquires non-AIS ship information including the name and position of the ship from the input non-AIS signal.

[0103] Next, the monitoring determination unit 162 determines a monitoring target ship to be monitored from the AIS ship information acquired in the processing of S501 and the non-AIS ship information acquired in the processing of S502 (S503). The monitoring determination unit 162 may determine a ship present within the port as a monitoring target ship, or may determine a ship present within a specified area of ​​the port as a monitoring target ship.

[0104] Next, the monitoring information acquisition unit 161 acquires monitoring image information from the ship determined to be the monitoring target ship in the processing of S503 (S504). The monitoring information acquisition unit 161 requests the monitoring image generation device 1 mounted on the ship 110 determined to be the monitoring target ship to output a monitoring image signal, and when the monitoring image signal is input to the monitoring communication unit 151, acquires monitoring image information from the input monitoring image signal. The monitoring image information acquired in the processing of S504 may be a moving image or a still image. When the monitoring image information acquired in the processing of S504 is a still image, the performance and communication speed of the monitoring image generation device 1 and the monitoring image 120 can be lower than when the monitoring image information acquired in the processing of S504 is a moving image.

[0105] Next, the monitoring image information generation unit 163 generates display image information (S505) and stores the generated display image information in the monitoring storage unit 152. Then, the monitoring output unit 164 outputs the display image information generated in the processing of S506 to the monitoring display unit 154. In response to the input of the display image information, the monitoring display unit 154 displays a display image corresponding to the display image information.

[0106] FIG. 13 is a diagram showing an example of a display image displayed on the monitoring display unit 154. As shown in FIG.

[0107] Display image 170 has a first divided image 171, a second divided image 172, a third divided image 173, and a fourth divided image 174. In first divided image 171 to third divided image 173, scenery images and superimposed images acquired from three ships included in the monitored area indicated by arrow A in fourth divided image 174 are displayed together with monitored object information 171a, 172a, and 173a. The monitored object information 171a, 172a, and 173a includes the name of the monitored ship and the time when the displayed image was captured.

[0108] The fourth divided image 174 displays the monitoring target area and the position information of ships located in the vicinity of the monitoring target area in real time as a two-dimensional image viewed in a plane.

[0109] (Configuration and Functions of a Surveillance Image Generation System Having a Surveillance Image Generation Device According to the Second Embodiment) FIG. 14 is a perspective view of a vehicle equipped with a surveillance image generation system according to the second embodiment, FIG. 15 is a diagram showing a surveillance image generation device mounted on the vehicle shown in FIG. 14, and FIG. 16 is a functional block diagram of the surveillance image generation device shown in FIG. 15.

[0110] The surveillance image generation system 200 has a GNSS receiver 201, a wireless communication device 202, an operation device 203, a display device 204, an audio output device 205, an imaging device 206, and a surveillance image generation device 2 which is another example of an image surveillance device according to the present invention, and is mounted on a vehicle 210 which is another example of a moving object. The surveillance image generation system 200 displays a scenery image and a superimposed image superimposed on the scenery image on a screen visible to the driver of the vehicle 210, based on signals received from the GNSS receiver 201, the wireless communication device 202, and the imaging device 206.

[0111] The GNSS receiver 201 receives signals from a satellite positioning system such as a global positioning system. In response to receiving signals from the GNSS, the GNSS receiver 201 outputs a current position signal indicating the current position of the vehicle 210 and a traveling direction signal indicating the traveling direction of the vehicle 210 to the monitoring image generation device 2. The wireless communication device 202 is a communication device that can access the Internet network via the LTE network, and receives signals indicating various information. For example, the wireless communication device 202 receives point cloud signals indicating three-dimensional point cloud data collected by the Ministry of Land, Infrastructure, Transport and Tourism from the webpage of the Japan Digital Road Map Association (https: / / www.drm.jp / pointcloud / ).

[0112] The operation device 203 is configured to receive user operations on the surveillance image generation device 2, and includes, for example, a touch panel. The operation device 203 generates a signal in response to the driver's operation and outputs the generated signal to the surveillance image generation device 2.

[0113] The display device 204 is configured to display an image, and includes a display such as a liquid crystal display or an organic EL display. The display device 204 generates and displays an image based on a signal input from the monitoring image generation device 2.

[0114] The audio output device 205 is configured to output audio and includes a speaker. The audio output device 205 outputs audio based on a signal input from the monitoring image generation device 2.

[0115] The imaging device 206 is configured to capture images of the surroundings of the monitoring image generation device 2, and is equipped with a camera and is arranged so as to be visible to the driver and to be able to capture images in the direction of travel of the vehicle 210. The camera includes an imaging optical system for forming an image on a light-receiving surface, a photoelectric conversion element, and an image generation circuit for generating an image based on the output of the photoelectric conversion element. The photoelectric conversion elements included in the camera are CMOS and CCD sensors or the like that are arranged two-dimensionally on the light-receiving surface and output electrical signals according to the amount of incident light. The imaging device 206 captures and acquires scenery images showing scenery in the direction of travel of the vehicle 210, and outputs scenery image signals showing the acquired scenery images to the monitoring image generation device 2.

[0116] The monitoring image generating device 2 has a communication unit 31, a storage unit 32, and a processing unit 40, and is disposed inside a cabin 213 of a vehicle 210 so as to be visible to the driver. The communication unit 31, the storage unit 32, and the processing unit 40 are communicatively connected via a bus 33.

[0117] The communication unit 31 is configured to enable the monitoring image generation device 2 to communicate with the GNSS receiver 201 to the imaging device 206, and includes a communication interface circuit. The communication interface circuit is, for example, a communication interface circuit for a wired LAN, a wireless LAN, or LTE. The communication unit 31 supplies signals input from the GNSS receiver 201 to the imaging device 206 to the processing unit 40, and outputs signals supplied from the processing unit 40 to the GNSS receiver 201 to the imaging device 206.

[0118] The storage unit 32 is configured to store data and programs and includes, for example, a semiconductor memory. The storage unit 32 stores an operating system program, a driver program, an application program, data, and the like used in processing by the processing unit 40. For example, the storage unit 32 stores, as an application program, an information setting program that causes the processing unit 40 to execute an information setting process that sets information used when executing an image generation process that generates an image including a terrain image showing the terrain of the road on which the vehicle 210 travels. The storage unit 32 also stores, as an application program, an image generation program that causes the processing unit 40 to execute the image generation process. The information setting program and the image generation program are installed in the storage unit 32 from a computer-readable, non-transitory, portable storage medium such as a CD-ROM or DVD-ROM. The storage unit 32 also stores warning threshold information that indicates a warning threshold used when determining whether the body of the vehicle 210 will come into contact with the road. The storage unit 32 also stores various data and information used when executing the information setting process and the image generation process. For example, the storage unit 32 stores point cloud data, also referred to as topographical information, that indicates the topography of roads as point cloud information 32a.

[0119] The point cloud information 32a includes information indicating the latitude, longitude, and altitude of each point included in the point cloud. For example, this information is available from the website of the Japan Digital Road Map Association. The structure and functions of the point cloud information 32a are well known, so a detailed description will be omitted here.

[0120] The processing unit 40 is configured to comprehensively control the operation of the surveillance image generation device 2, and includes one or more processors and their peripheral circuits. The processing unit 40 includes, for example, a CPU. The processing unit 40 may also include a DSP, an LSI, an ASIC, an FPGA, etc. The processing unit 40 executes processing based on a program stored in the storage unit 32. The processing unit 40 also controls the operation of each component of the surveillance image generation device 2 so that processing is executed appropriately.

[0121] The processing unit 40 has, as functional blocks, an information acquisition unit 41, a condition setting unit 42, an image information generation unit 43, a collision determination unit 44, and an output unit 45. These functional blocks are functional modules realized based on a program executed by the processing unit 40. These functional blocks may be firmware implemented in the monitoring image generation device 2.

[0122] 17 is a flowchart showing the information setting process executed by the surveillance image generation device 2. The information setting process is realized by the processing unit 40 working in cooperation with each component of the surveillance image generation device 2 based on a program stored in the storage unit 32. The information setting process shown in FIG. 17 is executed every time the surveillance image generation device 2 is placed in the vehicle 210.

[0123] First, the information acquisition unit 41 acquires first distance information indicating a first distance in response to the driver's operation of the operation device 203 (S601), and stores the acquired first distance information in the storage unit 32. The first distance is the distance from the vehicle 210 to the near clip plane of the viewing frustum of a terrain image indicating the terrain included in a superimposed image to be superimposed on a landscape image captured by the imaging device 206. Next, the condition setting unit 42 sets the first distance acquired in the processing of S601 as information indicating the first distance to be used when generating a terrain image (S602). The condition setting unit 42 sets the first distance by associating the first distance information acquired in the processing of S601 with a predetermined identifier.

[0124] Next, the information acquisition unit 41 acquires display length information indicating a display length, which is the distance between the near clip plane and the far clip plane of the terrain image generated as a view frustum, in response to the driver's operation of the operation device 203 (S603). The information acquisition unit 41 stores the acquired display length information in the storage unit 32. Next, the condition setting unit 42 sets the display length acquired in the processing of S603 as information indicating the display length to be used when generating the terrain image (S604). The condition setting unit 42 sets the display length by associating the display length information acquired in the processing of S603 with a predetermined identifier.

[0125] Next, the information acquisition unit 41 acquires tilt information indicating the tilt of the imaging device 206 from a horizontal plane in response to the driver's operation of the operation device 203 (S605). The information acquisition unit 41 stores the acquired tilt information in the storage unit 32. Next, the condition setting unit 42 sets the tilt information acquired in the processing of S605 as information indicating the tilt to be used when generating a superimposed image (S606). The condition setting unit 42 sets the tilt by associating the tilt information acquired in the processing of S605 with a predetermined identifier.

[0126] Next, the information acquisition unit 41 acquires imaging height information indicating the height from the ground of the location where the imaging device 206 is placed in response to the driver's operation of the operation device 203 (S607). The information acquisition unit 41 stores the acquired imaging height information in the storage unit 32. Next, the condition setting unit 42 sets the imaging height information acquired in the processing of S607 as information indicating the height to be used when generating a superimposed image (S608). The condition setting unit 42 sets the height by associating the imaging height information acquired in the processing of S607 with a predetermined identifier.

[0127] Next, in response to the driver's operation of the operating device 203, the information acquisition unit 41 acquires ground distance information indicating the vertical length between the bottom of the body 211 of the vehicle 210 and the bottom of the wheels 212 that movably support the body 211 (S609). The information acquisition unit 41 stores the acquired ground distance information in the storage unit 32. Next, the condition setting unit 42 sets the ground distance information acquired in the processing of S609 as information indicating the ground distance to be used in the collision determination processing that determines whether or not the body 211 will collide with the ground surface (S610). The condition setting unit 42 sets the ground distance by associating the ground distance information acquired in the processing of S609 with a predetermined identifier.

[0128] Fig. 18(a) is a flowchart showing the monitoring image generation process executed by the monitoring image generation device 2. The monitoring image generation process is realized by the processing unit 40 working in cooperation with each component of the monitoring image generation device 2 based on a program stored in the storage unit 32. The monitoring image generation process shown in Fig. 18(a) is executed at predetermined execution intervals.

[0129] First, the image information generation unit 43 starts capturing a scenery image by the imaging device 206 in response to the driver's operation of the operation device 203 (S701). When the driver operates the operation device 203 and acquires an imaging start instruction via the operation device 203, the image information generation unit 43 starts the operation of the imaging device 206, and starts capturing a scenery image in the traveling direction of the vehicle 210 by the imaging device 206. The image information generation unit 43 acquires scenery image information indicating the scenery image captured by the imaging device 206, and outputs the acquired scenery image information to the display device 204. The display device 204 displays a scenery image corresponding to the input scenery image information.

[0130] Next, the information acquisition unit 41 acquires current position information indicating the current position of the vehicle 210 (S702). The information acquisition unit 41 requests the GNSS receiver 201 to output a current position signal via the communication unit 31, and when the current position signal is input to the communication unit 31, acquires current position information indicating the current position corresponding to the input current position signal.

[0131] Next, the information acquisition unit 41 acquires traveling direction information indicating the traveling direction of the vehicle 210 (S703). The information acquisition unit 41 requests the GNSS receiver 201 to output a traveling direction signal via the communication unit 31, and when the traveling direction signal is input to the communication unit 31, acquires traveling direction information indicating the traveling direction corresponding to the input traveling direction signal.

[0132] Next, the image information generation unit 43 generates terrain image information showing a point on the vehicle 210, i.e., a terrain image which is an image of the ground surface and structures in the direction of travel of the vehicle 210 as viewed from the imaging device 206 (S704), and stores the generated terrain image information in the memory unit 32.

[0133] FIG. 18B is a flowchart showing the process of S704 in more detail.

[0134] First, the image information generation unit 43 sets the range for generating a terrain image (S801) using the first distance, display length, and inclination set in the processes of S602, S604, and S606, and the current position information and traveling direction information acquired in the processes of S702 and S703. The range for generating a terrain image is called the terrain image range. The process of S801 is similar to the process of S301, so a detailed description will be omitted here.

[0135] Next, the image information generation unit 43 acquires information indicating the height of each point included in the near clip plane of the terrain image range determined in the processing of S801 (S802). The image information generation unit 43 refers to the point cloud information 32a to acquire information indicating the height of the terrain superimposed on the near clip plane at a position separated by a first distance from the current position in the traveling direction, and stores the acquired information in the storage unit 32 as a first terrain data group.

[0136] Next, the image information generation unit 43 acquires information indicating the height of each point included in a plane that is separated from the near clip plane by a predetermined plot interval in the travel direction and is parallel to the near clip plane (S803). The image information generation unit 43 references the point cloud information 32a and executes a process similar to the process of S802 to acquire information indicating the height of the terrain superimposed on the plane separated from the near clip plane by the predetermined plot interval, and stores the acquired information in the storage unit 32 as a second terrain data group.

[0137] Next, the image information generation unit 43 determines whether the distance between the acquisition point, which is separated in the normal line L1 direction from the plane for which information was acquired in the processing of S803 by the plot interval, and the near clip plane is longer than the display length set in the processing of S604 (S804). If it is determined that the distance between the acquisition point and the near clip plane is not longer than the display length (S804-NO), the processing returns to S803. Thereafter, the processing of S803 and S804 is repeated until it is determined that the distance between the acquisition point and the near clip plane is longer than the display length (S804-YES). By repeating the processing of S803 and S804, multiple terrain data groups are stored in the storage unit 32, with the data group numbers being sequentially increased, starting with the third terrain data group.

[0138] When the image information generation unit 43 determines that the distance between the acquisition point and the near clip plane is longer than the display length (S804-YES), it stores the first to Nth terrain data groups acquired by the processes of S802 to S804 in the storage unit 32 as terrain image information (S805). When the image information generation unit 43 stores the terrain image information in the storage unit 32, the process of S704 ends.

[0139] Following the processing of S704, the image information generation unit 43 generates superimposed image information indicating a superimposed image that can be superimposed on a point on the vehicle 210, i.e., a landscape image captured by the imaging device 206 (S705), and stores the generated superimposed image information in the memory unit 32.

[0140] FIG. 19A is a flowchart showing the process of S705 in more detail.

[0141] First, the image information generation unit 43 sets an imaging range to be imaged by the imaging device 206 (S901). Next, the image information generation unit 43 arranges a terrain image corresponding to the terrain image information generated in the processing of S704 in the imaging range set in the processing of S901 (S902). Next, the image information generation unit 43 tilts the imaging range in which the terrain is arranged in the processing of S902 in accordance with the tilt of the imaging device 206 so that it is visible at an elevation angle corresponding to the tilt of the imaging device 206 (S903). The processing of S901 to S903 is similar to the processing of S401, S402, and S406, and therefore detailed description thereof will be omitted here.

[0142] Next, the image information generation unit 43 sets the height of the imaging range so that the height of the imaging range from the ground surface matches the height of the imaging device 206 from the ground surface (S904). The image information generation unit 43 sets the height of the imaging range so that it matches the height set in the processing of S608.

[0143] Next, the image information generating unit 43 generates an orientation image indicating the orientation (S905). The image information generating unit 43 generates an isosceles triangular image as an orientation image having a slope according to the angle difference between the north direction and the traveling direction corresponding to the traveling direction information acquired in the process of S703, and stores orientation image information indicating the generated orientation image.

[0144] Next, the image information generation unit 43 generates superimposed image information by combining the azimuth image generated in the processing of S905 with the imaging range (S906), and stores the generated superimposed image information in the storage unit 32. When the image information generation unit 43 stores the superimposed image information in the storage unit 32, the processing of S705 ends.

[0145] Following the process of S705, the collision determination unit 44 executes a collision determination process (S706).

[0146] FIG. 19B is a diagram showing more detailed processing of the collision determination processing shown in S706.

[0147] First, the collision determination unit 44 extracts information indicating the height of the terrain at a point a predetermined distance away in the traveling direction acquired in the processing of S703 from the point cloud information 32a (S1001), and stores the extracted information in the storage unit 32. For example, the collision determination unit 44 extracts information indicating the height of the terrain between the near clip plane and the far clip plane from the point cloud information 32a.

[0148] Next, the collision determination unit 44 calculates the separation distance between the body 211 of the vehicle 210 and the ground surface (S1002). The separation distance is calculated by subtracting the value indicating the vertical direction of the ground surface height information from the value indicating the vertical direction of the coordinates extracted in the process of S1001 from the length corresponding to the ground surface distance information set in the process of S610. The collision determination unit 44 stores the calculated separation distance in the memory unit 32.

[0149] Next, the collision determination unit 44 determines whether or not there is a risk that the body 211 of the vehicle 210 will come into contact with the ground surface (S1003). The collision determination unit 44 acquires warning threshold information stored in the storage unit 32. The collision determination unit 44 compares the warning threshold corresponding to the acquired warning threshold information with the separation distance calculated in the processing of S1002, and determines whether or not the separation distance calculated in the processing of S1002 is equal to or greater than the warning threshold. When the separation distance calculated in the processing of S1002 is less than the warning threshold, the collision determination unit 44 determines that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground surface (S1003-YES). On the other hand, when the separation distance calculated in the processing of S1002 is equal to or greater than the warning threshold, the collision determination unit 44 determines that there is no risk that the body 211 of the vehicle 210 will come into contact with the ground surface (S1003-NO).

[0150] When the collision determination unit 44 determines that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003-YES), it outputs a collision warning signal indicating that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground to the display device 204 and the audio output device 205. The display device 204 displays a collision area image indicating an area where the body 211 of the vehicle 210 is likely to come into contact with the ground by superimposing it on a scenery image. In response to the input of the collision warning signal, the audio output device 205 outputs an audio signal indicating that there is a risk that the body 211 of the vehicle 210 will come into contact with the ground. When the collision determination unit 44 determines that there is no risk that the body 211 of the vehicle 210 will come into contact with the ground (S1003-NO), the processing of S706 ends.

[0151] Following the processing of S706, the information acquisition unit 41 acquires scenery image information indicating a scenery image captured by the imaging device 206 (S707). The information acquisition unit 41 requests the imaging device 206 via the communication unit 31 to output a scenery image signal indicating the scenery image captured by the imaging device 206, and when the scenery image signal is input to the communication unit 31, acquires scenery image information indicating a scenery image corresponding to the input scenery image signal.

[0152] Next, the image information generation unit 43 superimposes the superimposed image corresponding to the superimposed image information generated in the processing of S705 onto the scenery image corresponding to the scenery image information acquired in the processing of S707, thereby generating monitoring image information (S708).

[0153] Next, the output unit 45 outputs the monitoring image information generated in the processing of S708 to the display device 204 (S709). In response to the input of the monitoring image information, the display device 204 displays a monitoring image corresponding to the input monitoring image information.

[0154] Next, the image information generation unit 43 determines whether or not an instruction to end the image generation process has been given (S710). When a processing end signal is input in response to the driver's operation of the operation device 203, the image information generation unit 43 determines that an instruction to end the image generation process has been given (S710-YES). If the image information generation unit 43 determines that an instruction to end the image generation process has not been given (S710-NO), the process returns to S702. Thereafter, the processes of S702 to S710 are repeated until the image information generation unit 43 determines that an instruction to end the image generation process has been given (S710-YES).

[0155] When it is determined that the image information generating unit 43 has instructed to end the image generating process (YES in S710), the image capturing device 206 ends capturing scenery images in response to the driver's operation of the operating device 203 (S711).

[0156] (Operational effects of the monitoring image generating device according to the second embodiment) The surveillance image generating device 2 can reduce the risk of the scenery image becoming difficult to view by superimposing a terrain image of the terrain between the near clip plane and the far clip plane on the scenery image, rather than a terrain image of the terrain included in the scenery image. For example, by very shorting the display length and arranging the near clip plane and the far clip plane close to each other, it is possible to display the cross-sectional shape of the terrain at a point a first distance away from the vehicle 210 as a terrain image.

[0157] In addition, the monitoring image generating device 2 displays a terrain image of the terrain between the near clip plane and the far clip plane, which are located in the direction of travel of the vehicle 210, allowing the driver to easily understand the terrain in the direction of travel of the vehicle 210.

[0158] Furthermore, the monitoring image generation device 2 can acquire the terrain accurately and quickly by using the point cloud information 32a. Furthermore, since the monitoring image generation device 2 forms the terrain using a point cloud to generate a terrain image, it can generate and display the terrain image accurately and quickly.

[0159] Furthermore, since the surveillance image generating device 2 generates superimposed image information so that it is visible at an elevation angle corresponding to the inclination of the imaging device 206, the superimposed image can be superimposed on the landscape image regardless of the inclination of the surveillance image generating device 2.

[0160] In addition, the monitoring image generating device 2 outputs an alarm signal when there is a risk that the body 211 of the vehicle 210 will come into contact with the ground, so that the driver can drive the vehicle 210 in a way that reduces the risk that the body 211 will come into contact with the ground.

[0161] (Modification of the monitoring image generating device according to the second embodiment) The surveillance image generating device 2 generates terrain images and superimposed images as a viewing cone, but the surveillance image generating device according to the embodiment may also generate terrain images and superimposed images by performing image processing such as affine transformation and projective transformation on the terrain.

[0162] In addition, the surveillance image generating device 2 generates terrain image information using point cloud information, but the surveillance image generating device according to the embodiment may also generate terrain image information using information acquired using a shape detection sensor such as a laser scanner.

[0163] Furthermore, although the surveillance image generating device 2 is a device separate from the operation device 203 to the imaging device 206, in the surveillance image generating device according to the embodiment, the operation device 203 to the imaging device 206 may be arranged as a device integrated with the surveillance image generating device.

[0164] Furthermore, although the monitoring image generating devices 1 and 2 generate monitoring image information showing monitoring images, the image generating devices according to the embodiments may generate image information showing images other than monitoring images. For example, the image generating devices according to the embodiments may generate image information showing images including pipes buried underground.

[0165] Furthermore, while the monitoring image generation devices 1 and 2 acquire topographical information of the topography in the traveling direction of the moving object, the image generation device according to the embodiment may acquire topographical information indicating the topography in a desired first direction. Furthermore, the image generation device according to the embodiment may be, for example, a mobile terminal held by a worker performing a predetermined task, or a fixed imaging device that is fixed at one point and can capture an image in a desired direction.

[0166] When an image generating device according to an embodiment acquires topographical information indicating the topography in a first direction, it generates image information indicating an image in which a superimposed image including a topographical image, which is an image of the topography between a first point on the topography corresponding to the topographical information and a second point located in the first direction and remoter than the first point, is superimposed on a landscape image captured from the one point. [Explanation of symbols]

[0167] 1, 2 Surveillance image generation device 21, 41 Information acquisition section 22, 42 Condition setting section 23, 43 Image information generation unit 24, 44 Collision determination section 25, 45 output section

Claims

1. an information acquisition unit that acquires topographical information indicating the topography in the first direction; an image information generating unit that generates image information showing an image obtained by superimposing a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, viewed from a single point on a scenery image captured from the single point; an output unit that outputs the image information, An image generating device characterized in that the superimposed image is generated as a viewing frustum with the first point as a near clip plane and the second point as a far clip plane.

2. the information acquisition unit further acquires current location information indicating a current location and first direction information indicating the first direction; the first point is a point separated from the current location by a first distance in the first direction, The image generating device according to claim 1 , wherein the second point is a point spaced apart from the first point in the first direction by a predetermined display length.

3. determining whether or not there is a risk that the bottom of the body of the vehicle traveling in the first direction will come into contact with the surface of the terrain based on a vertical separation distance between the bottom of the body of the vehicle traveling in the first direction and the bottom of the wheels that movably support the body and a difference in elevation of the terrain; outputting an alarm signal when it is determined that there is a risk that the bottom of the body will come into contact with the surface of the terrain; The image generating device according to claim 2 , further comprising a collision determination unit.

4. The image generating device according to claim 2 , wherein the image information generating section generates image information representing the landscape image captured by the imaging device disposed at the single point by superimposing the superimposed image on the landscape image.

5. The image generating device according to claim 4 , wherein the image information generating section generates superimposed image information indicating the superimposed image so that the superimposed image is viewed at an elevation angle according to an inclination of the imaging device.

6. acquiring topographical information indicating the topography in a first direction; generating image information showing an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a scenery image captured from the one point; outputting the image information; An image generation method, characterized in that the superimposed image is generated as a viewing frustum with the first point as a near clip plane and the second point as a far clip plane.

7. acquiring topographical information indicating the topography in a first direction; generating image information showing an image in which a superimposed image including a terrain image, which is an image of the terrain between a first point on the terrain corresponding to the terrain information and a second point located in the first direction and remote from the first point, is superimposed on a scenery image captured from the one point; outputting the image information; The superimposed image is generated as a viewing frustum with the first point as a near clip plane and the second point as a far clip plane.

Images