Night vision image processing system

Abstract

Compared to conventional techniques, this provides a means for enabling objects to be recognized with high accuracy from night vision images. [Solution] The system 1 of the present invention comprises a night vision camera 11, which is a visible light camera that has a night vision correction function and generates a color image by capturing images; a non-night vision camera 12, which is a camera that generates images by sensing electromagnetic waves or sound waves other than visible light; an image processing device 13 that generates an image for display using a first image captured by the night vision camera 11 and a second image captured by the non-night vision camera 12 at the same time as the first image was captured by the night vision camera 11 or after a time interval within a predetermined length of time; and a terminal device 14 that displays the image for display generated by the image processing device 13.

Description

[Technical field]

[0001] The present invention relates to a technique for processing images captured by a night vision camera. [Background technology]

[0002] There is a night vision camera (hereinafter simply referred to as "night vision camera") that is a visible light camera with a night vision correction function. The night vision correction function is a function that lowers the sensitivity of the image sensor during the day and increases the sensitivity of the image sensor at night. Night vision cameras have a highly sensitive image sensor, and the night vision correction function increases the sensitivity at night to detect faint visible light and generate a color image.

[0003] Patent Document 1, for example, is an example of a patent document that describes technology relating to the above-mentioned night vision camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2024-87739 A Summary of the Invention [Problem to be solved by the invention]

[0005] Images taken by night vision cameras at night (hereinafter referred to as "night vision images") are generated by detecting a small amount of visible light, and therefore have lower resolution and color reproduction than images taken by normal visible light cameras or night vision cameras during the day (hereinafter referred to as "bright vision images"). Therefore, it is more difficult to recognize an object from a night vision image than from a bright vision image.

[0006] In view of the above circumstances, the present invention provides a means for enabling objects to be recognized from night vision images with higher accuracy than conventional techniques. [Means for solving the problem]

[0007] The present invention provides a system comprising a night vision camera, which is a visible light camera with a night vision correction function; a non-night vision camera, which is a camera that generates an image by sensing electromagnetic waves or sound waves other than visible light; and an image processing device that generates an image for display using a first image taken by the night vision camera and a second image taken by the non-night vision camera at the same time as the night vision camera took the first image or after a time interval of up to a predetermined length of time. Effect of the Invention

[0008] According to the present invention, objects can be recognized from night vision images with higher accuracy than in the prior art. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of a system according to a first embodiment. [Figure 2A] 3A to 3C are diagrams illustrating examples of images captured by a non-night vision camera according to the first embodiment. [Figure 2B] 2A to 2C are diagrams illustrating examples of images captured by a night-vision camera according to the first embodiment. [Diagram 3] 5A to 5C are diagrams illustrating examples of images displayed by the terminal device according to the first embodiment. [Figure 4] FIG. 13 is a diagram showing the configuration of a system according to a second embodiment. [Figure 5A] 13A and 13B are diagrams illustrating examples of images captured by a non-night vision camera according to the second embodiment. [Figure 5B] 13A to 13C are diagrams illustrating examples of images captured by a night-vision camera according to a second embodiment. [Figure 6] FIG. 11 is a diagram showing the data configuration of a table stored in an image processing apparatus according to the second embodiment. [Figure 7] 13A and 13B are diagrams illustrating examples of images displayed by a terminal device according to the second embodiment. [Figure 8] FIG. 13 is a diagram showing the configuration of a system according to a second embodiment. [Figure 9A] 13A to 13C are diagrams illustrating examples of images captured by a non-night vision camera according to the third embodiment. [Figure 9B]13A to 13C are diagrams illustrating examples of images captured by a night-vision camera according to a third embodiment. [Figure 10] 13A to 13C are diagrams illustrating examples of images displayed by a terminal device according to the third embodiment. [Figure 11] FIG. 13 is a diagram showing the configuration of a system according to a fourth embodiment. [Figure 12A] 13A to 13C are diagrams illustrating examples of images captured by a non-night vision camera according to the fourth embodiment. [Figure 12B] 13A to 13C are diagrams illustrating examples of images captured by a stereo night-vision camera according to a fourth embodiment. [Figure 12C] 13A to 13C are diagrams illustrating examples of images captured by a stereo night-vision camera according to a fourth embodiment. [Figure 13] 13A and 13B are diagrams illustrating examples of images displayed by a terminal device according to the fourth embodiment. [Figure 14] FIG. 13 is a diagram showing the configuration of a system according to a fifth embodiment. [Figure 15] 13A to 13C are diagrams illustrating examples of images captured by a non-night vision camera according to the fifth embodiment. [Figure 16] FIG. 13 is a diagram illustrating an example of an image displayed by a terminal device according to the fifth embodiment. [Figure 17] FIG. 13 is a diagram showing the configuration of a system according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [First embodiment] 1 is a diagram showing the configuration of a system 1 according to a first embodiment of the present invention. The system 1 includes a night vision camera 11, a non-night vision camera 12, an image processing device 13, and a terminal device .

[0011] The night vision camera 11 is a visible light camera with a night vision correction function, and generates a color image by shooting. In an environment where sufficient visible light is available, such as outdoors during the day, the night vision camera 11 lowers the sensitivity of the image sensor to shoot a color image as a normal visible light camera, and in an environment where sufficient visible light is not available, such as at night or in a closed space where visible light from the outside world is blocked, the sensitivity of the image sensor is increased to detect a small amount of visible light and shoot a color image. The image generated by the night vision camera 11 will be referred to as the "first image" below.

[0012] The night vision camera 11 includes a communication interface and transmits image data representing a captured first image (hereinafter referred to as "first image data") to the image processing device 13. Note that instead of the night vision camera 11 having a built-in communication interface, a communication interface that is an external device may be connected to the night vision camera 11.

[0013] The non-night vision camera 12 is a camera that generates an image by sensing electromagnetic waves or sound waves other than visible light. In this embodiment, the non-night vision camera 12 is a thermography camera that generates an image by sensing infrared rays emitted by an object. More specifically, the non-night vision camera 12 generates an image showing the temperature distribution of the object by sensing infrared rays in a wavelength band of approximately 2 to 14 μm, which are emitted with a larger amount of energy as the temperature of the object increases. The image generated by the non-night vision camera 12 is hereinafter referred to as the "second image."

[0014] The non-night vision camera 12 includes a communication interface and transmits image data representing a captured second image (hereinafter referred to as "second image data") to the image processing device 13. Note that instead of the non-night vision camera 12 having a built-in communication interface, a communication interface that is an external device may be connected to the non-night vision camera 12.

[0015] The angle of view of the night vision camera 11 and the angle of view of the non-night vision camera 12 are substantially the same. In this application, the angle of view of the two cameras is substantially the same means that there is an overlapping area between the area captured in the image captured by one camera and the area captured in the image captured by the other camera. However, it is preferable that the shooting directions of the two cameras are parallel, and it is more preferable that the angles of view of the two cameras are the same.

[0016] The image processing device 13 is a data processing device that receives first image data from the night vision camera 11 and second image data from the non-night vision camera 12, generates an image for display using the two images represented by the image data, and transmits image data representing the generated image for display (hereinafter referred to as "image data for display") to the terminal device 14.

[0017] The hardware of the image processing device 13 is, for example, a computer for a server device, and includes a memory for persistently storing various data including programs, a processor for performing various data processing according to the programs stored in the memory, and a communication interface for performing data communication with external devices. That is, the computer, which is the hardware of the image processing device 13, operates as the image processing device 13 that performs the processing described below by executing, by the processor, various data processing according to the programs for the image processing device according to this embodiment that are persistently stored in the memory.

[0018] The terminal device 14 is a data processing device that receives display image data from the image processing device 13 and displays an image represented by the received display image data.

[0019] The hardware of the terminal device 14 is, for example, a computer for the terminal device, and includes a memory for persistently storing various data including programs, a processor for performing various data processing in accordance with the programs stored in the memory, a communication interface for performing data communication with external devices, a display for displaying information to the user, and an input device for accepting information input operations by the user. In other words, the computer, which is the hardware of the terminal device 14, operates as the terminal device 14 that performs the processing described below by executing, by the processor, various data processing in accordance with the programs for the terminal device according to this embodiment that are persistently stored in the memory.

[0020] The type of computer employed as the hardware of the terminal device 14 may be any of a desktop PC (Personal Computer), a laptop PC, a tablet PC (including a smartphone that is a small tablet PC equipped with a calling function via a mobile communication network), etc. Also, instead of the terminal device 14 having at least one of a communication interface, a display, and an input device built therein, these devices may be connected to the terminal device 14 as external devices.

[0021] In this embodiment, the night vision camera 11 and the non-night vision camera 12 capture images at the same time. In this application, "two cameras capture images at the same time" does not mean that images are captured at exactly the same time, but includes images captured substantially at the same time. "Capturing images substantially at the same time" means that images are captured with a time difference that is short enough in light of the rate of change in the relative positions of the cameras and the subject so that a subject captured in an image captured by one camera also appears in an image captured by the other camera.

[0022] In this embodiment, as an example, the night vision camera 11 and the non-night vision camera 12 are synchronized based on a synchronization signal received from the image processing device 13, and each of them continuously captures images at a predetermined time interval. More specifically, for example, each of the night vision camera 11 and the non-night vision camera 12 has a built-in clock, which continuously measures the current time and captures images at substantially the same timing, for example, at intervals of 1 / 10 seconds. The clocks of the night vision camera 11 and the non-night vision camera 12 deviate from each other over time, but the night vision camera 11 and the non-night vision camera 12 periodically correct the clocks based on the synchronization signal received from the image processing device 13. As a result, the images captured by the night vision camera 11 and the non-night vision camera 12 are images captured at substantially the same time.

[0023] Fig. 2A illustrates an example of an image (second image) captured at time t1 by the non-night vision camera 12. Fig. 2B illustrates an example of an image (first image) captured at time t1 by the night vision camera 11.

[0024] The image processing device 13 recognizes an object having a predetermined characteristic from the second image exemplified in Fig. 2A. If there are warm-blooded animals such as birds and mammals, or devices that generate heat when in operation, within the angle of view of the non-night vision camera 12, the second image captured by the non-night vision camera 12 will include images showing the shapes of those objects. For example, if the second image includes an image that occupies an area equal to or larger than a predetermined threshold, the image processing device 13 recognizes the image as an object having a predetermined characteristic.

[0025] Specifically, the image processing device 13 recognizes an object appearing in the area A1 in the second image illustrated in FIG. 2A.

[0026] Next, the image processing device 13 generates, as an image for display, an image representing a portion of the first image that corresponds to the position in the second image of the object recognized from the second image.

[0027] Specifically, the image processing device 13 specifies an area B1 in the first image in FIG. 2B as a portion corresponding to the position of the area A1 in the second image in FIG. 2A.

[0028] Next, the image processing device 13 cuts out the area B1 from the first image in FIG. 2B, and generates an image for display.

[0029] The image processing device 13 transmits display image data representing the image for display generated as described above to the terminal device 14. The terminal device 14 receives the display image data transmitted from the image processing device 13, and displays an image represented by the received display image data on a display. Fig. 3 is a diagram illustrating an example of an image displayed by the terminal device 14.

[0030] The display image shown in FIG. 3 is an image in which a thumbnail image of the entire first image and an image cut out from the first image and enlarged (an image of region B1 of the first image) are displayed side by side.

[0031] A user of the terminal device 14 can easily recognize, for example, living things that are difficult to find because they are hidden in the shade of trees by viewing the display image exemplified in FIG.

[0032] [Second embodiment] A system 2 according to the second embodiment of the present invention will be described below. The system 2 according to the second embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following mainly describes the points where the system 2 differs from the system 1, and the points where the system 2 has in common with the system 1 will be omitted as appropriate.

[0033] 4 is a diagram showing the configuration of the system 2. As shown in FIG. 4, the configuration of the system 2 is the same as the configuration of the system 1.

[0034] The operation of System 2 is different from that of System 1. The operation of System 2 is explained below.

[0035] FIG. 5A is a diagram showing an example of a second image captured by the non-night vision camera 12 of system 2 at time t2, and FIG. 5B is a diagram showing an example of a first image captured by the night vision camera 11 of system 2 at time t2.

[0036] The image processing device 13 of the system 2 recognizes objects from the first image. Any known image recognition technology may be used as a technology by which the image processing device 13 recognizes pigs from the first image. In this case, the image processing device 13 recognizes that each of the images appearing in the area B2 and the area B3 of the first image in FIG. 5B is an image of a pig. For each recognized object, the image processing device 13 stores an image obtained by cutting out a portion of the first image in which the object appears as a new file.

[0037] Next, the image processing device 13 identifies each of the objects recognized from the first image based on the features of the object such as the pattern, shape, etc. Specifically, the image processing device 13 measures the similarity between each of the previously recognized pig images and the newly recognized pig image, for example, in terms of the pattern on the pig's body surface, and identifies those pigs based on the measured similarity.

[0038] In order to allow the image processing device 13 to easily and accurately identify each pig from the first image, for example, an image such as a barcode indicating identification information may be drawn on the body surface of each pig, or a collar, ankle ring, or the like on which an image such as a barcode indicating identification information is drawn may be attached to each pig. In this case, the image processing device 13 identifies the object recognized from the first image by reading and decoding the image such as the barcode shown in the first image to specify the identification information.

[0039] When the image processing device 13 determines that the newly recognized object is not identical to any object previously recognized, it assigns new identification information to the newly recognized object, and generates and stores a data table (hereinafter referred to as "table T") according to the identification information.

[0040] 6 is a diagram showing the data structure of table T stored in image processing device 13. Image processing device 13 stores tables T corresponding to each piece of identification information. Each table T includes the following data fields (hereinafter simply referred to as "fields").

[0041] "Date and time" field: stores data indicating the date and time when the first image (and the second image) was taken. "Image File Name" field: stores data indicating the file name of the image of the object recognized from the first image. "Temperature" field: stores data indicating the body temperature determined from the second image.

[0042] When the image processing device 13 determines that the object newly recognized from the first image is not identical to any object previously recognized, assigns new identification information to the newly recognized object, generates and stores a table T according to the identification information, adds a new data record (hereinafter simply referred to as a "record") to the table T, and stores in the record data indicating the date and time when the first image used to recognize the object was captured, and data indicating the file name of the image of the object cut out from the first image.

[0043] On the other hand, if the image processing device 13 determines that the object newly recognized from the first image is identical to any of the objects previously recognized, it adds a new record to table T corresponding to the identification information of the object, and stores in that record data indicating the date and time when the first image used to recognize the object was captured, and data indicating the file name of the image of the object cut out from the first image.

[0044] Hereinafter, it is assumed that the image processing device 13 has identified that the pig appearing in area B2 is the pig identified by identification information P095, and the pig appearing in area B3 is the pig identified by identification information P143.

[0045] Next, for each object recognized from the first image, the image processing device 13 identifies the temperature indicated by a portion of the second image corresponding to the position of the object. Specifically, the image processing device 13 identifies the temperature indicated by the image of region A2 of the second image corresponding to region B2 of the first image, and the temperature indicated by the image of region A3 of the second image corresponding to region B3 of the first image.

[0046] The image processing device 13 stores data indicating the temperature identified from the area A2 of the second image in the last record of the table corresponding to the identification information P095. Also, the image processing device 13 stores data indicating the temperature identified from the area A3 of the second image in the last record of the table corresponding to the identification information P143.

[0047] The image processing device 13 executes a series of processes, such as recognizing the object, identifying the recognized object, and specifying the body temperature of the identified object, for each pair of a first image represented by the first image data continuously received from the night vision camera 11 and a second image represented by the second image data continuously received from the non-night vision camera 12. As a result, the image processing device 13 stores, for each object within the angle of view of the night vision camera 11 and the non-night vision camera 12, a data string indicating the change over time in the temperature (body temperature) of the object in a table T corresponding to the object.

[0048] When new data indicating a body temperature is stored in table T, the image processing device 13 determines whether the body temperature indicated by the data stored in table T satisfies a predetermined condition indicating an abnormality in the pig's physical condition, and if it determines that the predetermined condition is satisfied, generates an image (display image) to inform the user of terminal device 14 of the abnormality, and transmits display image data representing the generated display image to terminal device 14.

[0049] When the terminal device 14 receives the display image data transmitted from the image processing device 13, it displays an image represented by the received display image data. Fig. 7 is a diagram showing an example of an image displayed by the terminal device 14. For example, the terminal device 14 displays an image including identification information of an object determined to be in an abnormal physical condition and a graph showing the change over time in the temperature (body temperature) of the object.

[0050] By viewing the display image exemplified in FIG. 7, the user can easily learn, for example, about abnormalities in the physical condition of the pig.

[0051] [Third embodiment] A system 3 according to a third embodiment of the present invention will be described below. The system 3 according to the third embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following mainly describes the points where the system 3 differs from the system 1, and the points where the system 3 has in common with the system 1 will be omitted as appropriate.

[0052] Fig. 8 is a diagram showing the configuration of system 3. As shown in Fig. 8, compared to system 1, system 3 includes a non-night vision camera 32 instead of non-night vision camera 12.

[0053] The non-night vision camera 32 included in the system 3 is an infrared camera with an infrared irradiation function that irradiates infrared rays and generates an image by detecting the reflected light of the infrared rays.

[0054] The non-night vision camera 32 can generate clear images by emitting infrared rays even at night or in closed environments where outside light is blocked, but the images generated will be black and white images or images drawn only with shading information.

[0055] Figure 9A is a schematic diagram showing an example of a second image captured by the non-night vision camera 32 at time t3, and Figure 9B is a schematic diagram showing an example of a first image captured by the night vision camera 11 of the system 3 at time t3.

[0056] The image processing device 13 of the system 3 generates an image for display in which an object appearing at the same position in a first image (FIG. 9B) captured by the night vision camera 11 and a second image (FIG. 9A) captured at the same time by the non-night vision camera 32 is represented by a shape based on the second image and a color based on the first image. More specifically, the image processing device 13 identifies color information of the first image and shading information of the second image for each of corresponding pixels in the first and second images, and generates an image that is a collection of pixels having such information.

[0057] Next, the image processing device 13 recognizes an object appearing in the display image from the image generated from the first image and the second image as described above using a known image recognition technique. Then, the image processing device 13 generates a display image including the image generated from the first image and the second image and characters indicating the type of object recognized from the image. The image processing device 13 transmits display image data representing the generated display image to the terminal device 14.

[0058] When the terminal device 14 receives the display image data transmitted from the image processing device 13, it displays an image represented by the received display image data. Fig. 10 is a diagram showing an example of an image displayed by the terminal device 14. The display image displayed by the terminal device 14 includes a clear captured color image and characters indicating the type of object recognized from the color image ("human" in the example of Fig. 10).

[0059] The user can see the situation in a dark environment as if it were a bright environment, by using the display image shown in Fig. 10. In addition, the user can easily see the presence of an object that is difficult to see with the naked eye in a dark environment and the type of the object, by using the display image shown in Fig. 10.

[0060] [Fourth embodiment] A system 4 according to a fourth embodiment of the present invention will be described below. The system 4 according to the fourth embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following mainly describes the points where the system 4 differs from the system 1, and the description of the points where the system 4 and the system 1 have in common will be omitted as appropriate.

[0061] Fig. 11 is a diagram showing the configuration of system 4. As shown in Fig. 11, compared to system 1, system 4 includes a stereo night vision camera 41 instead of the night vision camera 11.

[0062] The stereo night vision camera 41 comprises two night vision cameras (hereinafter referred to as the "first night vision camera" and the "second night vision camera") arranged at a predetermined distance so that their shooting directions are parallel, and transmits first image data representing two images (first images) captured simultaneously by the first night vision camera and the second night vision camera to the image processing device 13.

[0063] In the following description, it is assumed that the system 4 is used by a user who is a participant in a survival game to detect enemy soldiers.

[0064] When the image processing device 13 sequentially receives the second image data continuously transmitted from the non-night vision camera 12 (thermography camera), it determines whether or not there is an area in the image (second image) represented by the received second image data where the temperature is above a predetermined threshold and the area is above a predetermined threshold.

[0065] Fig. 12A is a diagram showing an example of a second image captured at time t4 by the non-night vision camera 12. The image processing device 13 determines that the second image shown in Fig. 12A includes an area A4 that occupies an area equal to or greater than a predetermined threshold and has a temperature equal to or greater than a predetermined threshold.

[0066] When the image processing device 13 determines that there are areas in the second image that satisfy one or more of the above conditions, it performs the following processing for each of those areas using the first image data received from the stereo night vision camera 41 at the same time. The processing performed by the image processing device 13 for the area A4 in Fig. 12A will be described below as an example.

[0067] FIG. 12B is a diagram showing an example of an image captured by the first night vision camera of the stereo night vision camera 41 at time t4, and FIG. 12C is a diagram showing an example of an image captured by the second night vision camera of the stereo night vision camera 41 at time t4.

[0068] The image processing device 13 specifies an area corresponding to the area A4 of the second image for each of the two first images represented by the first image data. In this case, the image processing device 13 specifies an area B4 corresponding to the area A4 for the image of Fig. 12B, and specifies an area C4 corresponding to the area A4 for the image of Fig. 12C.

[0069] Next, the image processing device 13 recognizes objects in the areas specified as described above for each of the two first images by using a known image recognition technique. In this case, the image processing device 13 recognizes that a human is present in the area B4 in Fig. 12B and the area C4 in Fig. 12C.

[0070] Next, the image processing device 13 judges whether or not the object (in this case, a human) recognized from area B4 of Fig. 12B and area C4 of Fig. 12C is an object having a predetermined characteristic. In this example, the image processing device 13 judges whether or not the characteristics of the equipment, such as the clothing, of the human recognized from the first image have the characteristics of the equipment of an enemy soldier. Hereinafter, it is assumed that the image processing device 13 judges the human recognized from area B4 of Fig. 12B and area C4 of Fig. 12C to be an enemy soldier.

[0071] In this case, the image processing device 13 calculates the distance to the enemy soldier, i.e., the distance between the stereo night vision camera 41 and the enemy soldier, by known triangulation based on the position of the image of the enemy soldier in the image of Figure 12B and the position of the image of the enemy soldier in the image of Figure 12C.

[0072] Next, the image processing device 13 generates a display image by adding, for example, the characters "enemy soldier" to the first image captured by the first night-vision camera as an indication that an enemy soldier is captured, and adding, for example, the characters "XX m" as an indication of the distance to the enemy soldier. Note that the image processing device 13 may generate a display image by using the first image captured by the second night-vision camera instead of the first image captured by the first night-vision camera. The image processing device 13 transmits display image data representing the generated display image to the terminal device 14.

[0073] When the terminal device 14 receives the display image data transmitted from the image processing device 13, the terminal device 14 displays an image represented by the received display image data.

[0074] 13, the user can know that there is an enemy soldier in the direction of shooting by the stereo night vision camera 41 and the non-night vision camera 12, and the distance to the enemy soldier. If the enemy soldier is within the range of the user's weapon (such as an airsoft gun), the user fires a bullet (an example of a projectile) at the enemy soldier (a real object).

[0075] As described above, the image processing device 13 recognizes an object having predetermined characteristics from the second image, determines whether or not a portion of the first image corresponding to the position of the object satisfies predetermined conditions, and if it determines that the predetermined conditions are satisfied, notifies the user by transmitting display image data to the terminal device 14.

[0076] In the above example, the user watches the display image displayed by the terminal device 14 and fires a bullet at an enemy soldier, but instead, the firing device may automatically fire a bullet. In that case, when the image processing device 13 recognizes an enemy soldier and measures the distance to the enemy soldier, the firing device determines whether or not the distance is within the firing range of the device itself, and if it is within the firing range, fires a bullet at the actual enemy soldier.

[0077] [Fifth embodiment] A system 5 according to the fifth embodiment of the present invention will be described below. The system 5 according to the fifth embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following mainly describes the points where the system 5 differs from the system 1, and the points where the system 5 has in common with the system 1 will be omitted as appropriate.

[0078] Fig. 14 is a diagram showing the configuration of system 5. As shown in Fig. 14, compared to system 1, system 5 includes a non-night vision camera 52 instead of non-night vision camera 12.

[0079] The non-night vision camera 52 included in the system 5 is a device that transmits sound waves underwater, receives the reflected waves of the sound waves, and generates an image (second image) representing the shape of an object underwater based on the received reflected waves.

[0080] The night vision camera 11 of the system 5 includes a telephoto lens with a zoom function, a drive unit for focusing the telephoto lens and adjusting the focal length, and an electric pan head for adjusting the shooting direction of the night vision camera 11.

[0081] When the image processing device 13 of the system 5 continuously receives the second image data transmitted from the non-night vision camera 52, the image processing device 13 recognizes an object having a predetermined characteristic from the second image represented by the received second image data. Hereinafter, the object having the predetermined characteristic recognized by the image processing device 13 from the second image is assumed to be an underwater organism such as a fish.

[0082] Fig. 15 is a diagram showing an example of a second image captured at time t5 by the non-night vision camera 52. Hereinafter, it is assumed that the image processing device 13 recognizes a fish from the image of Fig. 15 as an object having a predetermined characteristic.

[0083] When the image processing device 13 recognizes an object having a predetermined characteristic from the second image, it determines the direction from the night vision camera 11 toward the object based on the position of the recognized object in the second image as the shooting direction of the night vision camera 11.

[0084] Next, the image processing device 13 instructs the night vision camera 11 to start shooting. This instruction includes data indicating the shooting direction.

[0085] When the night vision camera 11 receives an instruction to start shooting transmitted from the image processing device 13, it adjusts the shooting direction to the specified shooting direction using the electric pan head in accordance with the instruction and starts shooting. The night vision camera 11 sequentially transmits first image data representing the captured images (first images) to the image processing device 13.

[0086] When the image processing device 13 receives the first image data transmitted from the night vision camera 11, it determines whether or not the first image represented by the received first image data is in focus, and if it is not in focus, it instructs the night vision camera 11 to adjust the focus. Note that the technology for adjusting the focus based on the image captured by the camera is known, so a description thereof will be omitted.

[0087] When the image processing device 13 comes into focus on an object captured in the first image represented by the first image data transmitted from the night vision camera 11, the image processing device 13 instructs the night vision camera 11 to adjust the zoom so that the object is captured to fill the entire angle of view. In accordance with the instruction from the image processing device 13, the zoom adjustment is performed and the first image data representing the captured first image is transmitted to the image processing device 13.

[0088] The image processing device 13 generates a display image representing an in-focus image in which an object is captured over the entire angle of view, as represented by the first image data transmitted from the night-vision camera 11, and transmits the generated display image to the terminal device 14.

[0089] When the terminal device 14 receives the display image data transmitted from the image processing device 13, it displays an image represented by the received display image data. Fig. 16 is a diagram showing an example of an image displayed by the terminal device 14. The display image displayed by the terminal device 14 is an image captured by the night vision camera 11 by zooming in on an object recognized from the second image captured by the non-night vision camera 52.

[0090] The user can view the appearance of creatures living in dark water using the display image shown in FIG.

[0091] In the system 5, after an object is recognized from the second image captured by the non-night vision camera 52, the object is captured by the night vision camera 11 to generate a first image. Therefore, the capture time of the second image used to generate the display image is not the same as the capture time of the first image, but the time interval between the capture time of the second image and the capture time of the first image is within a predetermined time length determined as the maximum time required for focus adjustment and zoom adjustment. If this predetermined time length is sufficiently short in light of the speed of change in the relative position between the subject object and the night vision camera 11 and non-night vision camera 52, the system 5 can capture the object captured by the non-night vision camera 52 with the night vision camera 11.

[0092] [Sixth embodiment] A system 6 according to a sixth embodiment of the present invention will be described below. The system 6 according to the sixth embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following will mainly describe the points where the system 6 differs from the system 1, and will appropriately omit a description of the points where the system 6 and the system 1 have in common.

[0093] Fig. 17 is a diagram showing the configuration of a system 6. As shown in Fig. 17, compared to the system 1, the system 6 includes a non-night vision camera 62 instead of the non-night vision camera 12.

[0094] The non-night vision camera 62 included in the system 6 is a device (so-called distance imaging camera) that transmits electromagnetic waves, receives the reflected waves of the electromagnetic waves, and generates an image (second image) showing the shape of an object based on the received reflected waves. Examples of distance imaging cameras that can be used as the non-night vision camera 62 include, but are not limited to, millimeter wave radar that uses millimeter waves and LiDAR (Light Detection And Ranging) that uses laser light.

[0095] The image processing device 13 generates an image for display by mapping, as a texture, the appearance represented by the first image captured by the night vision camera 11 at the same time onto the shape represented by the second image captured by the non-night vision camera 62, and transmits display image data representing the generated display image to the terminal device 14. The terminal device 14 displays the display image represented by the display image data received from the image processing device 13.

[0096] [Variations] Each of the above-described systems 1 to 6 is an embodiment of the system according to the present invention, and may be modified in various ways within the scope of the technical concept of the present invention. Examples of such modifications are shown below. Two or more of the following modifications may be combined as appropriate.

[0097] (1) In the above-described embodiment, the first image and the second image may be appropriately switched.

[0098] For example, in the first embodiment, the image processing device 13 recognizes an object having a predetermined characteristic from the second image, cuts out a portion of the first image corresponding to the position of the object recognized from the second image, and generates an image for display using the cut-out image. Alternatively, the image processing device 13 may recognize an object having a predetermined characteristic from the first image, cut out a portion of the second image corresponding to the position of the object recognized from the first image, and generate an image for display using the cut-out image.

[0099] In addition, in the fourth embodiment, the image processing device 13 recognizes an object having a predetermined characteristic from the second image and determines whether or not a portion of the first image corresponding to the position of the object recognized from the second image satisfies a predetermined condition. Alternatively, the image processing device 13 may recognize an object having a predetermined characteristic from the first image and determine whether or not a portion of the second image corresponding to the position of the object recognized from the first image satisfies a predetermined condition.

[0100] (2) In the above-described embodiment, when the night vision camera 11 (or the stereo night vision camera 41) is a fixed camera, the image processing device 13 may execute a process of correcting the color of the night vision image based on the relationship between the color in the daytime image and the color in the nighttime image of the same object recognized from both an image taken by the night vision camera 11 (or the stereo night vision camera 41) during the day (hereinafter referred to as the "daytime image") and an image taken by the night vision camera 11 (or the stereo night vision camera 41) at night (hereinafter referred to as the "nighttime image"), and generate an image for display using the image that has been corrected.

[0101] Specifically, the image processing device 13 generates a conversion formula or conversion table that brings the color of the nighttime image closer to the color of the daytime image based on the correspondence relationship between color information of the same object at the same position in both the daytime image and the nighttime image. The image processing device 13 converts the color information of each pixel of the nighttime image according to the conversion formula or conversion table, and generates a color-corrected nighttime image, which is a collection of pixels with converted color information, as an image for display.

[0102] (3) The image processing device 13 and the night vision camera 11 (or the stereo night vision camera 41) may be directly connected by a cable or the like, or may be connected via a communication network. In the latter case, the image processing device 13 receives the first image from the night vision camera 11 (or the stereo night vision camera 41) via the communication network.

[0103] Furthermore, the image processing device 13 and the non-night vision camera 12 (or the non-night vision camera 32, the non-night vision camera 52, or the non-night vision camera 62) may be directly connected by a cable or the like, or may be connected via a communication network. In the latter case, the image processing device 13 receives the second image from the non-night vision camera 12 (or the non-night vision camera 32, the non-night vision camera 52, or the non-night vision camera 62) via the communication network.

[0104] (4) The image processing device 13 may be configured as a single device, or may be configured as a group of multiple devices (for example, a group of server devices) that operate in cooperation with each other.

[0105] In addition, in the above-described embodiment, part of the processing performed by the image processing device 13 may be performed by the night vision camera 11 (or the stereo night vision camera 41), the non-night vision camera 12 (or the non-night vision camera 32, the non-night vision camera 52, the non-night vision camera 62), or the terminal device 14.

[0106] For example, in the above-described embodiment, if the terminal device 14 executes part of the processing (e.g., object recognition processing) that is to be performed by the image processing device 13, the image processing device 13 and the terminal device 14 constitute an image processing device. [Explanation of symbols]

[0107] 1...system, 2...system, 3...system, 4...system, 5...system, 6...system, 11...night vision camera, 12...non-night vision camera, 13...image processing device, 14...terminal device, 32...non-night vision camera, 41...stereo night vision camera, 52...non-night vision camera, 62...non-night vision camera.

Claims

[Claim 1] A night vision camera which is a visible light camera having a night vision correction function; A non-night vision camera is a camera that generates an image by detecting electromagnetic waves or sound waves other than visible light; an image processing device that generates an image for display using a first image captured by the night vision camera and a second image captured by the non-night vision camera at the same time as the first image captured by the night vision camera or at a time interval of within a predetermined length of time; Equipped with The image processing device executes a process of correcting the color of the image taken by the night vision camera at night based on a relationship between the color in the image taken during the day and the color in the image taken at night of the same object recognized from both the image taken by the night vision camera during the day and the color in the image taken at night, and generates the image for display using the image that has been subjected to the correction process. system.

Images