Image processing device and control method therefor, imaging device, and imaging system

Abstract

To facilitate monitoring an ambient environment at night.SOLUTION: An image processing device comprises: first acquisition means for acquiring visible light pictures produced at a visible light imaging part for performing imaging in a wavelength range of visible light; second acquisition means for acquiring non-visible light pictures produced at a non-visible light imaging part for performing imaging in a different wavelength range from the visible light; first detection means for detecting at least one periodicity in time change of lamp, that is included in one picture from the visible light pictures and at least the one picture of the visible light pictures; second detection means for detecting one or more lamp colors from the visible light pictures; identification means for identifying at least one lamp type of one or more lamps, based on the periodicity detected by the first detection means, and the one or more colors detected by the second detection means; and overlapping means for overlapping information about at least the one lamp type identified by the identification means with the non-visible light pictures.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for monitoring the surrounding environment at night. [Background technology]

[0002] At sea, navigational aids (such as light beacons and light buoys) are used to show ships their route, and navigation lights are used to show other ships their direction. For example, navigational aids emit lights of a predetermined color and lighting pattern at night to inform surrounding areas of information about their navigational aids (such as port beacons and starboard beacons). Navigation lights are lights of a predetermined color that are installed in predetermined locations on a ship (such as the port side, starboard side, or mast), and by emitting light at night, serve to inform surrounding areas of the direction the ship is heading. Patent Documents 1 and 2 disclose technology in which a ship flashes a light source based on a ship's unique identification code, and an imaging device detects the flashing of the light source to individually recognize the ship. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-61952 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-160626 Summary of the Invention [Problem to be solved by the invention]

[0004] Non-visible light imaging devices, such as thermal cameras, are sometimes used to monitor the surrounding environment at night. However, images captured by non-visible light imaging devices generally have low spatial resolution and do not provide color information. Therefore, users cannot monitor the images captured by the non-visible light imaging device to identify information indicated by the navigational aids and navigation lights. However, there is a problem in that it is cumbersome for users to monitor both the images captured by the visible light imaging device and the images captured by the non-visible light imaging device.

[0005] The present invention has been made in view of the above problems, and aims to provide a technique that makes it easy to monitor the surrounding environment at night. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an image processing device according to the present invention has the following arrangement. a first acquisition means for acquiring a visible light image generated by a visible light imaging unit that captures images in the wavelength range of visible light; a second acquisition means for acquiring a non-visible light image generated by a non-visible light image capturing unit that captures an image in a wavelength range different from that of the visible light; a first detection means for detecting, from at least one of the visible light image and the visible light image, a periodicity of a change over time of one or more lights included in the image; a second detection means for detecting the color of the one or more lights from the visible light image; an identification means for identifying the type of at least one of the one or more lights based on the periodicity detected by the first detection means and the color detected by the second detection means; a superimposing means for superimposing information relating to the type identified by the identifying means on the invisible light image; Equipped with. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique that makes it easy to monitor the surrounding environment at night. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating a functional configuration of the imaging apparatus. [Figure 2] 4 is a flowchart illustrating the operation of the imaging device. [Figure 3] 10 is a detailed flowchart of the navigational aid identification (S206). [Figure 4]FIG. 10 is a diagram showing examples of light colors and lighting patterns according to the types of navigational aids. [Figure 5] FIG. 10 is a diagram illustrating a group of frames when a single flash and a group of flashes are captured. [Figure 6] FIG. 2 is a diagram illustrating the lighting patterns of navigational aids. [Figure 7] 10 is a detailed flowchart of navigation light identification (S207). [Figure 8] This is a diagram that explains how navigation lights appear on a ship moving to the right or left. [Figure 9] FIG. 10 is a diagram illustrating the status of other ships according to the visibility of navigation lights. [Figure 10] FIG. 10 is a diagram showing an example of information superimposition on a navigational aid. [Figure 11] FIG. 10 is a diagram showing an example of information superimposition on a ship. [Figure 12] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a computer device that can be applied to the imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) As a first embodiment of an image processing device according to the present invention, an imaging device that captures images using invisible light will be described below. Note that in the following description, port and starboard beacons will be used as examples of navigational aids, and port / starboard lights and masthead lights will be used as examples of types of navigation lights, but the present invention can also be applied to other types of navigational aids and navigation lights.

[0011] <Device configuration> 1 is a diagram showing the functional configuration of an imaging device. The imaging device includes a visible light imaging unit 101, an invisible light imaging unit 102, a light color detection unit 103, a light detection unit 104, a light identification unit 105, and a light information superimposition unit 106.

[0012] The visible light imaging unit 101 is an imaging unit that captures images in the visible light range (for example, wavelengths of 380 to 780 nm) and acquires images capable of expressing color (such as RGB color images). In images (visible light images) obtained by capturing images of the sea at night with the visible light imaging unit 101, lights such as navigational aids and navigation lights are captured in color, while the structure / shape of the navigational aids and ships are not substantially captured. Note that the exposure setting should be set to reduce the effects of noise and / or prevent the light images from being overexposed and / or reduce the effects of flickering of the light's light source (light bulb, LED), so that the color of the lights can be easily identified.

[0013] The non-visible light imaging unit 102 is an imaging unit that captures images in the non-visible light range (for example, a wavelength range excluding the visible light range) to obtain monochrome images. For example, a thermal camera or the like can be used as the non-visible light imaging unit 102. Images (non-visible light images) obtained by capturing images of the sea at night with the non-visible light imaging unit 102 include illuminants such as navigation aids and navigation lights, as well as the structures and shapes of navigation aids and ships. Furthermore, the spatial resolution of non-visible light images is lower than that of visible light images, and non-visible light images do not contain color information (defined in the visible light range).

[0014] In the following description, it is assumed that the visible light imaging unit 101 and the invisible light imaging unit 102 simultaneously capture images of the same monitored area, and that the same lights in the two images can be associated with each other based on their coordinate positions in the visible light image and the invisible light image. However, it is sufficient that the same lights in the two images can be associated with each other, and at least a partial area of ​​the visible light image and the invisible light image may overlap.

[0015] The light color detection unit 103 detects the color of the light from the visible light image captured by the visible light imaging unit 101 and outputs it as light color information. The light detection unit 104 acquires the light emission cycle, lighting pattern, and light position of the light from the visible light image and / or invisible light image captured by the visible light imaging unit 101 and / or invisible light imaging unit 102, and outputs it as light information. Note that when acquiring the light position, it is preferable to use visible light image, which has a relatively high spatial resolution.

[0016] The light identification unit 105 identifies each light included in the visible light image based on the light color information acquired from the light color detection unit 103 and the light information acquired from the light detection unit 104, and outputs the result as a light identification result. The light identification result includes, for example, information on the type of navigational aid, information on the direction of the ship (based on the combination of navigation lights), etc.

[0017] The light information superimposing unit 106 superimposes information included in the light identification result acquired from the light identification unit 105 on the invisible light image and outputs the superimposed image. Note that the light information superimposing unit 106 itself may have a display unit (such as a liquid crystal organic EL display) and display the superimposed image, or may be configured to transmit the superimposed image to an external display device.

[0018] <Device Operation> 2 is a flowchart showing the operation of the imaging device. The following operation is started when a user instructs the imaging device to perform a monitoring operation.

[0019] In S201, the visible light imaging unit 101 starts acquiring a visible light image. In S202, the light color detection unit 103 detects the light color of the light included in the visible light image acquired in S201.

[0020] In S203, the non-visible light imaging unit 102 starts acquiring non-visible light video. In S204, the light detection unit 104 detects light information about the lights included in the video from the visible light video acquired in S201 and / or the non-visible light video acquired in S203. Here, the light information detected includes information about the light emission cycle, lighting pattern, and light position of each light included in the video.

[0021] In S205, the light detection unit 104 determines whether there is periodicity in the time change of the lighting state (on / off) of each light included in the video, based on the light information detected in S204. If there is periodicity, it is determined to be a navigation light, and the process proceeds to S206. If there is no periodicity (for example, the light is always on), it is determined to be a navigation light, and the process proceeds to S207.

[0022] In S206, the light identification unit 105 identifies the type of navigational aid corresponding to the light based on the light color detected by the light color detection unit 103 and the light information (light emission cycle, light emission pattern) detected by the light detection unit 104.

[0023] In S207, the light identification unit 105 identifies the direction of the ship from the light information (light position) detected by the light detection unit 104 and the light color of each light detected by the light color detection unit 103. Details will be described later with reference to Figures 7 to 9, but in S207, the direction of the ship corresponding to the multiple lights is identified based on the combination of the colors of the multiple lights.

[0024] In S208, the light information superimposing unit 106 superimposes, on the invisible light image, information based on the light identification result obtained by the identification (determination) in S206 and / or S207. Details will be described later with reference to Figs. 10 and 11.

[0025] <Navigational Aids Identification> In the following, first, an overview of the navigational aids will be explained with reference to FIGS. 4 to 6, and then the navigational aid identification (S206) will be explained in detail with reference to FIG.

[0026] Navigational aids come in various forms, such as light beacons, lighted buoys, buoys, and beacons, and each has a type (such as port beacon and starboard beacon, which will be described later). During the day, the type of navigational aid can be identified by its color and shape, and at night, the type can be identified by the color and brightness of its light.

[0027] FIG. 4 is a diagram showing examples of light colors and lighting patterns (lighting patterns) according to the type of navigational beacon. Of the many types of navigational beacons, port and starboard beacons will be used as examples. Note that a port beacon is a navigational beacon that indicates that the position of the beacon is the left end of the route. A starboard beacon is a navigational beacon that indicates that the position of the beacon is the right end of the route.

[0028] The port beacon is painted green, and the starboard beacon is painted red. At night, the starboard beacon emits a green light, while the port beacon emits a red light. The light can be a single flash, a group of flashes, a rapid succession of flashes, or a Morse code signal.

[0029] Figure 5 is a diagram illustrating a group of frames when a single flash and a group of flashes are captured. Figure 5(a) shows one cycle of frames when a single flash is captured, and Figure 5(b) shows one cycle of frames when a group of flashes is captured. The frame rate for capturing visible light video and invisible light video should be set to 30 fps or higher so that group flashes can be properly identified.

[0030] When shooting at a frame rate of 30 fps, a 150 frame cycle can be determined to be a 5-second cycle, and a 180 frame cycle can be determined to be a 6-second cycle. Also, if the lit frames (frames shown in white) are concentrated in one place within one cycle, as in Figure 5(a), it can be determined to be a single flash, and if they are separated into two places, as in Figure 5(b), it can be determined to be a cluster flash.

[0031] Figure 6 is a diagram explaining the lighting patterns (lighting methods) of navigational aids. Note that Figure 6 shows an example of the criteria for a group of alternating flashes, which alternately light up two different colors of light, but similar criteria are set for the time periods of bright and dark intervals for single flashes and group flashes.

[0032] Fig. 6(a) shows an example of the way light is emitted, and Fig. 6(b) shows the criteria for the way light is emitted (criteria for the time duration of each bright and dark period). For example, in the case of group interfacial flashing, there are two bright periods in one cycle, and the criteria state that each bright period must be 0.5 seconds or longer, and one single dark period must be one bright period or longer.

[0033] Therefore, in Figure 6(a), if we set t1 = t3 = 0.5 [s], t2 = 0.5 [s], and t4 = 1.5 [s], the period T = 3 [s]. If this is shot at a frame rate of 30 fps, t1 and t3 will be 15 frames, t2 will be 15 frames, and t4 will be 45 frames, meaning that all bright and dark intervals will be shot in 90 frames.

[0034] 3 is a detailed flowchart of the navigational mark identification (S206). As described above, here, for each light that has periodic lighting states, the type of navigational mark indicated by that light (here, port mark or starboard mark) is determined.

[0035] In S301, the light identification unit 105 acquires the light color of the light currently being focused on. The information acquired in S202 may be reused, or the visible light image may be newly determined and acquired.

[0036] In S302, the light identification unit 105 determines the period of the lighting state of the light currently being focused on. Specifically, the period is determined based on a plurality of consecutive frames included in the visible light video and / or the invisible light video.

[0037] In S303, the light identification unit 105 determines the light emission pattern of the light currently being focused on. Specifically, the light emission pattern is determined based on a plurality of consecutive frames included in the visible light video and / or the invisible light video. Once the light emission pattern is uniquely determined, the corresponding navigational beacon type (here, port beacon or starboard beacon) is identified.

[0038] <Navigation Light Identification> Below, we will first provide an overview of navigation lights with reference to Figures 8 and 9, and then provide details of navigation light identification (S207) with reference to Figure 7. Navigation lights are lights of a specific color that are installed in specific locations on a ship (port side, starboard side, mast, etc.), and their role is to inform surrounding areas (other ships, etc.) of the direction the ship is facing by emitting light at night.

[0039] Figure 8 is a diagram explaining how the navigation lights of a ship proceeding to the right and left appear to an observer. Ships navigate with their navigation lights on when sailing at night (from sunset to sunrise). There are several locations where navigation lights can be installed, but Figure 8 shows a case where a forward mast light 801, aft mast light 802, starboard light 803, and port light 804 are installed.

[0040] Laws and regulations stipulate that the color of the light of the mast lights (forward mast light 801 and aft mast light 802) is "white," the color of the light of the starboard light 803 is "green," and the color of the light of the port light 804 is "red." Furthermore, the lights of the mast lights (forward mast light 801 and aft mast light 802) emit light within a range of 112.5 degrees to the left and right (a total light emission range of 225 degrees) with the bow direction as the reference (0 degrees). Meanwhile, the light of the starboard light 803 emits light within a range of 112.5 degrees to the right with the bow direction as the reference, and the light of the port light 804 emits light within a range of 112.5 degrees to the left with the bow direction as the reference. Therefore, when viewed from another ship, if at least one of the starboard light 803 or the port light 804 is visible, both the forward mast light 801 and the aft mast light 802 will also be visible. Furthermore, the installation position (height from sea level) of the forward mast light 801 is lower than the installation position of the aft mast light 802.

[0041] Figure 9 is a diagram that explains the status of other ships according to how their navigation lights appear (in visible light images) to an observer (own ship). When another ship crosses your ship (the direction of your ship and the direction of the other ship are nearly perpendicular), three lights are visible to your ship. That is, if the other ship is heading left, three lights are visible: the port light (red), forward mast light (white), and aft mast light (white); and if the other ship is heading right, three lights are visible: the starboard light (green), forward mast light (white), and aft mast light (white).

[0042] Furthermore, the closer the distance between the forward mast light and the aft mast light in the captured image, the more parallel the orientation of the other ship and your ship are (however, the other ship is facing in your ship's direction). In other words, the distance between the forward mast light and the aft mast light in the captured image is greatest when they are perpendicular to your ship's direction. When the other ship is facing in your ship's direction, all four lights of the other ship (starboard light (green), port light (red), forward mast light (white), aft mast light (white)) are visible from your ship. Note that when your ship and the other ship are facing the same direction, only the stern light (white) is visible. A stern light is a light that emits light in the stern direction (a beam range of 135 degrees behind the ship).

[0043] As can be seen from Figure 9, the direction of other ships can be determined based on the color and relative positions of the four lights (starboard light (green), port light (red), forward mast light (white), and aft mast light (white)).

[0044] Figure 7 is a detailed flowchart of navigation light identification (S207). The situation (direction) of other ships is determined according to the color combination of multiple lights detected in the visible light image as follows. Note that the light identification unit 105 determines that one or more lights in the image (visible light image or invisible light image) that are less than a predetermined distance from each other are one or more lights installed on the same ship and groups them together. The following process is performed for each group.

[0045] In S701, the light color detection unit 103 determines whether or not both the starboard light and the port light colors (i.e., green and red) are detected in the visible light image. If both the green light and the red light are detected (the state of the third and fourth lines in FIG. 9), the process proceeds to S702, and if at least one of the green light and the red light is not detected (the state of the first, second, and fifth lines in FIG. 9), the process proceeds to S703. In S702, the light identification unit 105 calculates (determines) the direction of the other ship as seen from the ship itself based on the positions of the four lights (starboard light, port light, forward mast light, and aft mast light) in the image.

[0046] In S703, the light color detection unit 103 determines whether the light color of the starboard light (i.e., green light) is detected in the visible light image. If a green light is detected (the state of the second line in FIG. 9), the process proceeds to S704, and if a green light is not detected (the state of the first and fifth lines in FIG. 9), the process proceeds to S705. In S704, the light identification unit 105 calculates (determines) the direction of the other ship as seen from the ship itself based on the positions of the three lights (starboard light, forward mast light, and aft mast light) in the image.

[0047] In S705, the light color detection unit 103 determines whether the light color of the port light (i.e., red light) is detected in the visible light image. If a red light is detected (the state of the first line in FIG. 9), the process proceeds to S706, and if a red light is not detected (the state of the fifth line in FIG. 9), the process proceeds to S707.

[0048] In S706, the light identification unit 105 calculates (determines) the direction of the other ship as seen from the own ship based on the positions of the three lights (port light, forward mast light, and aft mast light) in the image. In addition, in S707, the light identification unit 105 determines that the other ship is facing the same direction as the own ship.

[0049] Small ships are usually equipped with only one mast light, but even in this case, it is possible to calculate (determine) the direction of other ships from the ship's perspective based on the position of at least one of the starboard and port lights and the position of the mast light.

[0050] <Example of information superimposition on invisible light video> An example of information superimposition performed by the light information superimposition unit 106 in S208 will be described below. That is, an example of superimposing information about navigational aids and ships on invisible light video will be described.

[0051] 10 is a diagram showing three examples of information superimposition on images of navigational aids in invisible light video. It is assumed that a ship and a navigational aid 1101 are captured in the invisible light video in FIG.

[0052] Fig. 10(a) shows an example in which a label 1002 of "starboard marking", which is the type of navigational marking of the navigational marking 1001, is superimposed on the navigational marking 1001 in the invisible light image. Fig. 10(b) shows an example in which a lighting pattern 1003 of the light of the navigational marking 1001 is superimposed on the navigational marking 1001 in the invisible light image. Furthermore, Fig. 10(c) shows an example in which the navigational marking image 1001 in the invisible light image is colored with a "paint color" corresponding to the type of navigational marking of the navigational marking 1001.

[0053] In this way, information about a navigational mark obtained from the visible light image is superimposed on the image of the navigational mark in the invisible light image, allowing the user to easily grasp nearby navigational marks on the sea at night by monitoring only the superimposed invisible light image.

[0054] 11 is a diagram showing two examples of information superimposition on a ship image in an invisible light image. As in FIG. 10, the invisible light image in FIG. 11 shows a ship 1101 and navigational aids.

[0055] FIG. 11(a) shows an example in which an arrow 1102 indicating the direction of the ship 1101 is superimposed on the ship 1101. That is, the arrow icon 1102 is superimposed so that the user can intuitively understand that the ship 1101 is facing (navigating) to the "left." FIG. 11(b) shows an example in which navigation lights 1103 to 1105 of the ship 1101 are colored in the light color corresponding to each navigation light. That is, in this example, the navigation lights 1103 and 1104, which are mast lights, are colored "white," and the navigation light 1105, which is a port light, is colored "red." That is, by viewing the superimposed invisible light image, the user can understand the navigation lights of the ship 1101 when viewed in visible light image (color image).

[0056] In this way, information about the navigation lights of a ship obtained from the visible light image is superimposed on the image of the ship in the invisible light image, allowing the user to easily identify nearby ships at sea at night by monitoring only the superimposed invisible light image.

[0057] As described above, according to the first embodiment, information obtained from a visible light image is superimposed on an invisible light image. By monitoring the invisible light image having the superimposed display, the user can intuitively grasp the situation of the surroundings of the monitored object (the range captured by the invisible light image capture unit 102).

[0058] <Modification> Each functional unit of the imaging device shown in FIG. 1 may be implemented by hardware or software (computer program). In the former case, each functional unit may be implemented by hardware such as an ASIC or a programmable logic array (PLA). ASIC stands for Application Specific Integrated Circuit. Note that some of the functional units may be implemented by hardware.

[0059] In the latter case, a computer device capable of executing such a computer program can be applied to an imaging device. An example of the hardware configuration of a computer device applicable to an imaging device will be described using the block diagram of Fig. 12. Such a computer device can be a PC, a tablet terminal device, a smartphone, or the like.

[0060] The CPU 901 executes various processes using computer programs and data stored in the RAM 902 and ROM 903. As a result, the CPU 901 controls the operation of the entire computer device, and also executes or controls the various processes described as processes performed by the imaging device. Note that a programmable processor such as an MPU may be used instead of the CPU 901. CPU stands for Central Processing Unit. MPU stands for Micro-Processing Unit.

[0061] The RAM 902 has an area for storing computer programs and data loaded from the ROM 903 or the storage device 906, and an area for storing computer programs and data received from the outside via the I / F 907. The RAM 902 also has a work area used by the CPU 901 when executing various processes. In this way, the RAM 902 can provide various areas as needed.

[0062] The ROM 903 stores setting data for the computer device, computer programs and data relating to the startup of the computer device, computer programs and data relating to the basic operation of the computer device, and the like.

[0063] The storage device 904 is a large-capacity information storage device such as a hard disk drive. The storage device 904 stores an OS (operating system), computer programs and data for causing the CPU 901 to execute or control the various processes described as processes performed by the imaging device. The computer programs stored in the storage device 904 may also include computer programs for causing the CPU 901 to execute or control the functions of the functional units shown in Fig. 1. Images captured by the visible light imaging unit 101 and the non-visible light imaging unit 102 may be stored in the storage device 904, and the CPU 901 may read out and process the images as needed.

[0064] The I / F 905 is a communication interface for performing data communication with an external device via a network such as a LAN or the Internet.

[0065] The CPU 901, RAM 902, ROM 903, storage device 904, and I / F 905 are all connected to a system bus 906. Note that the hardware configuration of a computer device applicable to the imaging device is not limited to the configuration shown in Fig. 12, and can be modified / changed as appropriate. The system configuration described in each of the above embodiments can be modified or changed as appropriate depending on the specifications of the devices applied to the system and various conditions (usage conditions, usage environment, etc.), and the configurations shown in each of the above embodiments are merely examples.

[0066] The above-described imaging device may be mounted on a ship or placed on land (such as near a coast). The generated invisible light image with a superimposed display may be configured to be displayed on a display unit of the imaging device, or may be configured as an imaging system that displays the image on a display unit of a client device separate from the imaging device. The imaging device and the client device may communicate with each other via any wired or wireless communication path (including a local connection or a remote connection via the Internet).

[0067] The disclosure of this specification includes the following image processing device, imaging device, imaging system, control method, and program. (Item 1) a first acquisition means for acquiring a visible light image generated by a visible light imaging unit that captures images in the wavelength range of visible light; a second acquisition means for acquiring a non-visible light image generated by a non-visible light image capturing unit that captures an image in a wavelength range different from that of the visible light; a first detection means for detecting, from at least one of the visible light image and the visible light image, a periodicity of a change over time of one or more lights included in the image; a second detection means for detecting the color of the one or more lights from the visible light image; an identification means for identifying the type of at least one of the one or more lights based on the periodicity detected by the first detection means and the color detected by the second detection means; a superimposing means for superimposing information relating to the type identified by the identifying means on the invisible light image; An image processing device comprising: (Item 2) The identification means identifies lights detected to have periodicity as lights of navigational beacons, and identifies lights detected to have no periodicity as lights of navigation lights installed on ships. 2. The image processing device according to item 1, (Item 3) The identification means further identifies the type of navigational aid corresponding to the light identified as a navigational aid light based on the light color, the period of time change, and the light emission pattern of the light. 3. The image processing device according to item 2, (Item 4) The superimposing means superimposes a label indicating the type of navigational aid identified by the identifying means onto a navigational aid image included in the invisible light image corresponding to a light identified by the identifying means as a navigational aid light. 4. The image processing device according to item 3, (Item 5) The superimposing means superimposes an image showing the light emission pattern of a navigational mark image included in the invisible light image corresponding to the light identified by the identifying means as a navigational mark light. 4. The image processing device according to item 3, (Item 6) The superimposing means superimposes a color corresponding to the type of navigational aid identified by the identifying means onto a navigational aid image included in the invisible light image corresponding to a light identified by the identifying means as a navigational aid light. 4. The image processing device according to item 3, (Item 7) The identification means has a grouping means for determining and grouping one or more lights that are less than a predetermined distance from each other in the image, as one or more lights installed on the same ship, with respect to the lights that have been identified as navigation lights. 7. The image processing device according to any one of items 2 to 6, wherein: (Item 8) The identification means further identifies the direction of the ship on which the one or more lights are installed based on the light color and positional relationship of each of the one or more lights grouped by the grouping means. 8. The image processing device according to item 7, (Item 9) The superimposing means superimposes an icon indicating the direction of the ship identified by the identifying means on the ship image included in the invisible light image corresponding to the light identified by the identifying means as a navigation light. 9. The image processing device according to item 8, (Item 10) The superimposing means superimposes the color detected by the second detecting means onto each of one or more light positions of the ship image included in the invisible light image corresponding to the light identified as a navigation light by the identifying means. 8. The image processing device according to item 7, (Item 11) An image processing device according to any one of items 1 to 10; the visible light imaging unit; the non-visible light imaging unit; An imaging device comprising: (Item 12) The imaging device according to item 11, a display device that displays the invisible light image acquired from the imaging device and on which information is superimposed by the superimposing means; An imaging system comprising: (Item 13) A control method for an image processing device, comprising: a first acquisition step of acquiring a visible light image generated by a visible light imaging unit that captures images in the wavelength range of visible light; a second acquisition step of acquiring a non-visible light image generated by a non-visible light image capturing unit that captures an image in a wavelength range different from that of the visible light; a first detection step of detecting, from at least one of the visible light image and the visible light image, a periodicity of a time change of one or more lights included in the image; a second detection step of detecting colors of the one or more lights from the visible light image; an identification step of identifying a type of at least one of the one or more lights based on the periodicity detected by the first detection step and the color detected by the second detection step; A control method comprising: (Item 14) Item 14. A program for causing a computer to execute the control method according to Item 13.

[0068] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0069] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0070] 101 Visible light imaging unit; 102 Invisible light imaging unit; 103 Light color detection unit; 104 Light detection unit; 105 Light identification unit; 106 Light information superimposition unit

Claims

1. a first acquisition means for acquiring a visible light image generated by a visible light imaging unit that captures images in the wavelength range of visible light; a second acquisition means for acquiring a non-visible light image generated by a non-visible light image capturing unit that captures an image in a wavelength range different from that of visible light; a first detection means for detecting, from at least one of the visible light image and the visible light image, a periodicity of a time change of one or more lights included in the image; a second detection means for detecting the color of the one or more lights from the visible light image; an identification means for identifying the type of at least one of the one or more lights based on the periodicity detected by the first detection means and the color detected by the second detection means; a superimposing means for superimposing information relating to the type identified by the identifying means on the invisible light image; An image processing device comprising:

2. The identification means identifies lights detected to have periodicity as lights of navigational beacons, and identifies lights detected to have no periodicity as lights of navigation lights installed on ships.

2. The image processing device according to claim 1, wherein:

3. The identification means further identifies the type of navigational aid corresponding to the light identified as a navigational aid light based on the light color, the period of time change, and the light emission pattern of the light.

3. The image processing device according to claim 2.

4. The superimposing means superimposes a label indicating the type of navigational aid identified by the identifying means onto a navigational aid image included in the invisible light image corresponding to a light identified by the identifying means as a navigational aid light.

4. The image processing device according to claim 3.

5. The superimposing means superimposes an image showing the light emission pattern of a navigational mark image included in the invisible light image corresponding to the light identified by the identifying means as a navigational mark light.

4. The image processing device according to claim 3.

6. The superimposing means superimposes a color corresponding to the type of navigational aid identified by the identifying means onto a navigational aid image included in the invisible light image corresponding to a light identified by the identifying means as a navigational aid light.

4. The image processing device according to claim 3.

7. The identification means has a grouping means for determining and grouping one or more lights that are less than a predetermined distance from each other in the image, with respect to the lights that have been identified as navigation lights, as one or more lights installed on the same ship.

3. The image processing device according to claim 2.

8. The identification means further identifies the direction of the ship on which the one or more lights are installed based on the light color and positional relationship of each of the one or more lights grouped by the grouping means.

8. The image processing device according to claim 7,

9. The superimposing means superimposes an icon indicating the direction of the ship identified by the identifying means on the ship image included in the invisible light image corresponding to the light identified by the identifying means as a navigation light.

9. The image processing device according to claim 8,

10. The superimposing means superimposes the color detected by the second detecting means onto one or more light positions of the ship image included in the invisible light image corresponding to the lights identified as navigation lights by the identifying means.

8. The image processing device according to claim 7,

11. The image processing device according to claim 1 ; the visible light imaging unit; the non-visible light imaging unit; An imaging device comprising:

12. The imaging device according to claim 11; a display device that displays the invisible light image acquired from the imaging device and on which information is superimposed by the superimposing means; An imaging system comprising:

13. A control method for an image processing device, comprising: a first acquisition step of acquiring a visible light image generated by a visible light image capturing unit that captures images in a wavelength range of visible light; a second acquisition step of acquiring a non-visible light image generated by a non-visible light image capturing unit that captures an image in a wavelength range different from that of visible light; a first detection step of detecting, from at least one of the visible light image and the visible light image, a periodicity of a time change of one or more lights included in the image; a second detection step of detecting colors of the one or more lights from the visible light image; an identification step of identifying a type of at least one of the one or more lights based on the periodicity detected in the first detection step and the color detected in the second detection step; a superimposing step of superimposing information about the type identified in the identifying step on the invisible light image; A control method comprising:

14. A program for causing a computer to execute the control method according to claim 13.

Images