System for processing night vision image

Abstract

The present invention provides means allowing to recognize an object from a night vision image with higher accuracy compared to prior art. A system 2 according to the present invention includes: a night vision camera 11 which is a visible light camera having a night vision correction function and generating a color image by photographing; a non-night vision camera 12 which is a thermography camera for generating an image by sensing infrared rays; and an image processing device 13 for recording data indicating a temporal change in the temperature of the same object by repeating the processes of identifying an object from the image photographed by the night vision camera 11, specifying and storing the temperature of the object from the image photographed by the non-night vision camera 12.

Description

Night vision image processing system

[0001] The present invention relates to a technique for processing images captured by a night vision camera.

[0002] There are night vision cameras (hereinafter simply referred to as "night vision cameras") that are visible light cameras with night vision correction functions. The night vision correction function reduces the sensitivity of the image sensor during the day and increases the sensitivity of the image sensor at night. Night vision cameras have a highly sensitive image sensor, and the night vision correction function increases the sensitivity at night to detect faint visible light and generate color images.

[0003] Patent Document 1, for example, is a patent document that describes the technology related to the above-mentioned night vision camera.

[0004] JP 2024-87739 A

[0005] Images taken by night vision cameras at night (hereinafter referred to as "night vision images") are generated by detecting a small amount of visible light, and therefore have lower resolution and color reproduction than images taken by normal visible light cameras or night vision cameras during the day (hereinafter referred to as "bright vision images"). Therefore, it is more difficult to recognize objects from night vision images than from bright vision images.

[0006] In view of the above circumstances, the present invention provides a means for enabling objects to be recognized from night vision images with higher accuracy than conventional techniques.

[0007] The present invention provides a system comprising a night vision camera, which is a visible light camera with a night vision correction function, a thermographic camera that senses infrared rays and generates an image, and an image processing device, wherein an image taken by the night vision camera is defined as a first image, and an image taken by the thermographic camera at the same time as the night vision camera took the first image is defined as a second image corresponding to the first image, and when the image processing device determines that the second image contains a first area that is an area with a temperature above a predetermined threshold and occupies an area above a predetermined threshold, the system identifies an area in the first image corresponding to the second image that corresponds to the position of the first area in the second image as a second area, recognizes an object captured in the second area, and determines whether the object has predetermined characteristics.

[0008] According to the present invention, objects can be recognized from night vision images with higher accuracy than in the prior art.

[0009] FIG. 1 is a diagram showing the configuration of a system according to a first embodiment. FIG. 2 is a diagram showing an example of an image captured by a non-night-vision camera according to the first embodiment. FIG. 3 is a diagram showing an example of an image captured by a night-vision camera according to the first embodiment. FIG. 4 is a diagram showing an example of an image displayed by a terminal device according to the first embodiment. FIG. 5 is a diagram showing the configuration of a system according to a second embodiment. FIG. 6 is a diagram showing an example of an image captured by a non-night-vision camera according to the second embodiment. FIG. 7 is a diagram showing an example of an image captured by a night-vision camera according to the second embodiment. FIG. 8 is a diagram showing the data configuration of a table stored in an image processing device according to the second embodiment. FIG. 9 is a diagram showing an example of an image displayed by a terminal device according to the second embodiment. FIG. 10 is a diagram showing the configuration of a system according to a third embodiment. FIG. 11 is a diagram showing an example of an image captured by a non-night-vision camera according to the third embodiment. FIG. 12 is a diagram showing an example of an image captured by a night-vision camera according to the third embodiment. FIG. 13 is a diagram showing an example of an image displayed by a terminal device according to the third embodiment. FIG. 14 is a diagram showing the configuration of a system according to a fourth embodiment. FIG. 15 is a diagram showing an example of an image captured by a non-night-vision camera according to the fourth embodiment. FIG. 16 is a diagram showing an example of an image captured by a stereo night-vision camera according to the fourth embodiment. FIG. 17 is a diagram showing an example of an image captured by a stereo night-vision camera according to the fourth embodiment. FIG. 18 is a diagram showing an example of an image displayed by a terminal device according to the fourth embodiment. FIG. 19 is a diagram showing the configuration of a system according to a fifth embodiment. 13 is a diagram illustrating an example of an image captured by a non-night vision camera according to a fifth embodiment. FIG. 14 is a diagram illustrating an example of an image displayed by a terminal device according to a fifth embodiment. FIG. 15 is a diagram illustrating the configuration of a system according to a sixth embodiment.

[0010] 1 is a diagram showing the configuration of a system 1 according to a first embodiment of the present invention. The system 1 includes a night vision camera 11, a non-night vision camera 12, an image processing device 13, and a terminal device 14.

[0011] The night vision camera 11 is a visible light camera with night vision correction function that generates color images by capturing images. In an environment where sufficient visible light is available, such as outdoors during the day, the night vision camera 11 reduces the sensitivity of the image sensor and captures color images as a normal visible light camera. In an environment where sufficient visible light is not available, such as at night or in an enclosed space where visible light from the outside world is blocked, the sensitivity of the image sensor is increased to detect even small amounts of visible light and capture color images. The image generated by the night vision camera 11 will be referred to as the "first image" hereinafter.

[0012] The night vision camera 11 has a communication interface and transmits image data representing the captured first image (hereinafter referred to as "first image data") to the image processing device 13. Note that instead of the night vision camera 11 having a built-in communication interface, an external communication interface may be connected to the night vision camera 11.

[0013] The non-night vision camera 12 is a camera that generates images by sensing electromagnetic waves or sound waves other than visible light. In this embodiment, the non-night vision camera 12 is a thermography camera that generates images by sensing infrared rays emitted by an object. More specifically, the non-night vision camera 12 generates an image showing the temperature distribution of an object by sensing infrared rays in a wavelength band approximately around 2 to 14 μm, which are emitted with greater energy the higher the temperature of the object. The image generated by the non-night vision camera 12 will be referred to below as the "second image."

[0014] The non-night vision camera 12 is equipped with a communication interface and transmits image data representing the captured second image (hereinafter referred to as "second image data") to the image processing device 13. Note that instead of the non-night vision camera 12 having a built-in communication interface, an external communication interface may be connected to the non-night vision camera 12.

[0015] The angle of view of the night vision camera 11 and the angle of view of the non-night vision camera 12 are substantially the same. In this application, "the angles of view of the two cameras are substantially the same" means that there is an overlapping area between the area captured in the image captured by one camera and the area captured in the image captured by the other camera. However, it is preferable that the shooting directions of the two cameras are parallel, and it is even more preferable that the angles of view of the two cameras are the same.

[0016] The image processing device 13 is a data processing device that receives first image data from the night vision camera 11 and second image data from the non-night vision camera 12, generates an image for display using the two images represented by the image data, and transmits image data representing the generated image for display (hereinafter referred to as ``display image data'') to the terminal device 14.

[0017] The hardware of the image processing device 13 is, for example, a computer for a server device, and includes a memory that continuously stores various data including programs, a processor that processes various data in accordance with the programs stored in the memory, and a communication interface that communicates data with external devices. That is, the computer that is the hardware of the image processing device 13 operates as the image processing device 13 that performs the processing described below by using the processor to execute various data processes in accordance with the programs for the image processing device according to this embodiment that are continuously stored in the memory.

[0018] The terminal device 14 is a data processing device that receives display image data from the image processing device 13 and displays an image represented by the received display image data.

[0019] The hardware of the terminal device 14 is, for example, a computer for the terminal device, and includes a memory that persistently stores various data including programs, a processor that processes various data in accordance with the programs stored in the memory, a communication interface that communicates data with external devices, a display that displays information to the user, and an input device that accepts information input operations by the user. In other words, the computer that is the hardware of the terminal device 14 operates as the terminal device 14 that performs the processing described below by using the processor to execute various data processes in accordance with the programs for the terminal device according to this embodiment that are persistently stored in the memory.

[0020] The type of computer employed as the hardware of the terminal device 14 may be any of a desktop PC (Personal Computer), a laptop PC, a tablet PC (including a smartphone, which is a small tablet PC equipped with a function for making calls via a mobile communication network), etc. Also, instead of the terminal device 14 having at least one of a communication interface, a display, and an input device built in, these devices may be connected to the terminal device 14 as external devices.

[0021] In this embodiment, the night vision camera 11 and the non-night vision camera 12 capture images simultaneously. In this application, "two cameras capture images simultaneously" does not necessarily mean capturing images at exactly the same time, but also includes capturing images substantially simultaneously. "Capturing images substantially simultaneously" means capturing images with a time difference that is sufficiently short in light of the rate of change in the relative positions of the cameras and the objects so that an object captured by one camera also appears in an image captured by the other camera.

[0022] In this embodiment, for example, the night vision camera 11 and the non-night vision camera 12 are synchronized based on a synchronization signal received from the image processing device 13, and each camera continuously captures images at a predetermined time interval. More specifically, for example, each of the night vision camera 11 and the non-night vision camera 12 has a built-in clock that continuously measures the current time using that clock, and captures images at substantially the same timing, for example, at intervals of 1 / 10 seconds. The clocks of the night vision camera 11 and the non-night vision camera 12 gradually become out of sync over time, but the night vision camera 11 and the non-night vision camera 12 periodically correct their clocks based on the synchronization signal received from the image processing device 13. As a result, the images captured by the night vision camera 11 and the non-night vision camera 12 are images captured at substantially the same time.

[0023] 2A illustrates an example of an image (second image) captured by the non-night vision camera 12 at time t1, and FIG. 2B illustrates an example of an image (first image) captured by the night vision camera 11 at time t1.

[0024] The image processing device 13 recognizes an object having a predetermined characteristic from the second image exemplified in Fig. 2A. If there are warm-blooded animals such as birds or mammals, or devices that generate heat when in operation, within the field of view of the non-night vision camera 12, the second image captured by the non-night vision camera 12 will include images representing the shapes of those objects. For example, if the second image includes an image that occupies an area equal to or greater than a predetermined threshold, the image processing device 13 recognizes the image as an object having a predetermined characteristic.

[0025] Specifically, the image processing device 13 recognizes an object appearing in the area A1 in the second image illustrated in FIG. 2A.

[0026] Next, the image processing device 13 generates, as a display image, an image representing a portion of the first image that corresponds to the position in the second image of the object recognized from the second image.

[0027] Specifically, the image processing device 13 identifies an area B1 in the first image of FIG. 2B as a portion corresponding to the position of the area A1 in the second image of FIG. 2A.

[0028] Next, the image processing device 13 cuts out the area B1 from the first image in FIG. 2B and generates an image for display.

[0029] The image processing device 13 transmits display image data representing the display image generated as described above to the terminal device 14. The terminal device 14 receives the display image data transmitted from the image processing device 13 and displays the image represented by the received display image data on a display. Figure 3 is a diagram illustrating an example of an image displayed by the terminal device 14.

[0030] The display image illustrated in FIG. 3 is an image in which a thumbnail image of the entire first image and an enlarged image cut out from the first image (an image of area B1 of the first image) are displayed side by side.

[0031] The user of the terminal device 14 can easily recognize creatures that are difficult to find, for example, hidden in the shade of trees, by viewing the display image exemplified in FIG.

[0032] [Second Embodiment] A system 2 according to a second embodiment of the present invention will be described below. The system 2 according to the second embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following will mainly describe the points where the system 2 differs from the system 1, and will omit a description of the points where the system 2 has in common with the system 1 as appropriate.

[0033] 4 is a diagram showing the configuration of the system 2. As shown in FIG. 4, the configuration of the system 2 is the same as the configuration of the system 1.

[0034] The operation of System 2 differs from that of System 1. The operation of System 2 will be described below.

[0035] Figure 5A is a diagram showing an example of a second image captured by the non-night vision camera 12 of system 2 at time t2, and Figure 5B is a diagram showing an example of a first image captured by the night vision camera 11 of system 2 at time t2.

[0036] The image processing device 13 of the system 2 recognizes objects from the first image. Any known image recognition technology may be used as the technology by which the image processing device 13 recognizes pigs from the first image. In this case, the image processing device 13 recognizes that each of the images appearing in areas B2 and B3 of the first image in FIG. 5B is an image of a pig. For each recognized object, the image processing device 13 cuts out the portion of the first image in which the object appears, and stores the image as a new file.

[0037] Next, the image processing device 13 identifies each of the objects recognized from the first image based on the features of the object, such as its pattern, shape, etc. Specifically, the image processing device 13 measures the similarity between each of the previously recognized pig images and the newly recognized pig image, for example, in terms of the pattern on the pig's body surface, and identifies the pigs based on the measured similarity.

[0038] In order to enable the image processing device 13 to easily and accurately identify each pig from the first image, for example, an image such as a barcode indicating identification information may be drawn on the body surface of each pig, or each pig may be fitted with a collar, ankle bands, or the like on which an image such as a barcode indicating identification information is drawn. In this case, the image processing device 13 reads the image such as the barcode shown in the first image, decodes it, and specifies the identification information, thereby identifying the object recognized from the first image.

[0039] If the image processing device 13 determines that the newly recognized object is not identical to any of the objects recognized in the past, it assigns new identification information to the newly recognized object, and generates and stores a data table (hereinafter referred to as "table T") corresponding to the identification information.

[0040] 6 is a diagram showing the data structure of table T stored in image processing device 13. Image processing device 13 stores a table T corresponding to each piece of identification information. Each table T includes the following data fields (hereinafter simply referred to as "fields").

[0041] "Date and time" field: stores data indicating the date and time when the first image (and second image) was taken. "Image file name" field: stores data indicating the file name of the image of the object recognized from the first image. "Body temperature" field: stores data indicating the body temperature identified from the second image.

[0042] When the image processing device 13 determines that the newly recognized object from the first image is not identical to any of the objects recognized in the past, assigns new identification information to the newly recognized object, generates and stores a table T according to the identification information, adds a new data record (hereinafter simply referred to as a "record") to the table T, and stores in the record data indicating the date and time the first image used to recognize the object was taken and data indicating the file name of the image of the object cut out from the first image.

[0043] On the other hand, if the image processing device 13 determines that the object newly recognized from the first image is identical to one of the objects recognized in the past, it adds a new record to table T corresponding to the object's identification information, and stores in that record data indicating the date and time the first image used to recognize the object was taken and data indicating the file name of the image of the object cut out from that first image.

[0044] Hereinafter, it is assumed that the image processing device 13 has identified that the pig in area B2 is the pig identified by identification information P095, and the pig in area B3 is the pig identified by identification information P143.

[0045] Next, for each object recognized from the first image, the image processing device 13 identifies the temperature indicated by a portion of the second image corresponding to the position of the object. Specifically, the image processing device 13 identifies the temperature indicated by the image of region A2 of the second image corresponding to region B2 of the first image, and the temperature indicated by the image of region A3 of the second image corresponding to region B3 of the first image.

[0046] The image processing device 13 stores data indicating the temperature identified from area A2 of the second image in the last record of the table corresponding to the identification information P095. The image processing device 13 also stores data indicating the temperature identified from area A3 of the second image in the last record of the table corresponding to the identification information P143.

[0047] The image processing device 13 executes the series of processes of recognizing the object, identifying the recognized object, and specifying the body temperature of the identified object for each pair of a first image represented by the first image data continuously received from the night vision camera 11 and a second image represented by the second image data continuously received from the non-night vision camera 12. As a result, the image processing device 13 stores, in a table T corresponding to each object, a data string indicating the change in temperature (body temperature) of the object over time for each object within the field of view of the night vision camera 11 and the non-night vision camera 12.

[0048] When new data indicating a body temperature is stored in table T, image processing device 13 determines whether the body temperature indicated by the data stored in table T satisfies predetermined conditions indicating an abnormality in the pig's physical condition, and if it determines that the predetermined conditions are satisfied, generates an image (display image) to inform the user of terminal device 14 of the abnormality, and transmits display image data representing the generated display image to terminal device 14.

[0049] Upon receiving the display image data transmitted from the image processing device 13, the terminal device 14 displays an image represented by the received display image data. Fig. 7 is a diagram illustrating an example of an image displayed by the terminal device 14. For example, the terminal device 14 displays an image including identification information of an object determined to be in an abnormal physical condition and a graph showing the change in temperature (body temperature) of the object over time.

[0050] The user can easily know, for example, if there is an abnormality in the physical condition of the pig by viewing the display image exemplified in FIG.

[0051] [Third Embodiment] A system 3 according to a third embodiment of the present invention will be described below. The system 3 according to the third embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following will mainly describe the points where the system 3 differs from the system 1, and will omit a description of the points where the system 3 has in common with the system 1 as appropriate.

[0052] Fig. 8 is a diagram showing the configuration of System 3. As shown in Fig. 8, System 3 is different from System 1 in that it includes a non-night vision camera 32 instead of the non-night vision camera 12.

[0053] The non-night vision camera 32 included in the system 3 is an infrared camera with an infrared irradiation function that irradiates infrared light, senses the reflected light of the infrared light, and generates an image.

[0054] The non-night vision camera 32 can generate clear images by irradiating infrared rays even at night or in an enclosed space where outside light is blocked, but the images generated are images that are drawn only with shading information, such as black and white images.

[0055] Figure 9A is a schematic diagram showing an example of a second image captured by the non-night vision camera 32 at time t3, and Figure 9B is a schematic diagram showing an example of a first image captured by the night vision camera 11 of system 3 at time t3.

[0056] The image processing device 13 of the system 3 generates a display image that represents an object that appears at the same position in a first image (FIG. 9B) taken by the night vision camera 11 and a second image (FIG. 9A) taken at the same time by the non-night vision camera 32, using a shape based on the second image and a color based on the first image. More specifically, the image processing device 13 identifies the color information of the first image and the shading information of the second image for each of the corresponding pixels in the first and second images, and generates an image that is a collection of pixels having this information.

[0057] Next, the image processing device 13 recognizes objects appearing in the display image from the image generated from the first image and the second image as described above using a known image recognition technique. The image processing device 13 then generates a display image including the image generated from the first image and the second image and text indicating the type of object recognized from the image. The image processing device 13 transmits display image data representing the generated display image to the terminal device 14.

[0058] When the terminal device 14 receives the display image data transmitted from the image processing device 13, it displays an image represented by the received display image data. Fig. 10 is a diagram showing an example of an image displayed by the terminal device 14. The display image displayed by the terminal device 14 includes a clear captured color image and text indicating the type of object recognized from the color image ("human" in the example of Fig. 10).

[0059] The display image shown in Fig. 10 allows the user to understand the situation in a dark environment as if it were a bright environment. Furthermore, the display image shown in Fig. 10 allows the user to easily understand the existence of an object that is difficult to see with the naked eye in a dark environment and the type of that object.

[0060] [Fourth Embodiment] A system 4 according to a fourth embodiment of the present invention will be described below. The system 4 according to the fourth embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following will mainly describe the points where the system 4 differs from the system 1, and will omit a description of the points where the system 4 has in common with the system 1 as appropriate.

[0061] Fig. 11 is a diagram showing the configuration of System 4. As shown in Fig. 11, System 4 is different from System 1 in that it includes a stereo night-vision camera 41 instead of the night-vision camera 11.

[0062] The stereo night vision camera 41 comprises two night vision cameras (hereinafter referred to as the "first night vision camera" and the "second night vision camera") arranged at a predetermined distance so that their shooting directions are parallel, and transmits to the image processing device 13 first image data representing two images (first images) captured simultaneously by the first night vision camera and the second night vision camera.

[0063] In the following description, it is assumed that the system 4 is used by a user who is a participant in a survival game to detect enemy soldiers.

[0064] When the image processing device 13 sequentially receives the second image data continuously transmitted from the non-night vision camera 12 (thermography camera), it determines whether there is an area in the image (second image) represented by the received second image data where the temperature is above a predetermined threshold and the area is above a predetermined threshold.

[0065] 12A is a diagram showing an example of a second image captured at time t4 by the non-night vision camera 12. The image processing device 13 determines that the second image shown in FIG. 12A contains an area A4 that occupies an area equal to or greater than a predetermined threshold and has a temperature equal to or greater than a predetermined threshold.

[0066] When the image processing device 13 determines that there are areas in the second image that satisfy one or more of the above conditions, it performs the following processing for each of those areas using the first image data received from the stereo night-vision camera 41 at the same time. The processing performed by the image processing device 13 for area A4 in Fig. 12A will be described below as an example.

[0067] Figure 12B is a diagram showing an example of an image captured by the first night vision camera of the stereo night vision camera 41 at time t4, and Figure 12C is a diagram showing an example of an image captured by the second night vision camera of the stereo night vision camera 41 at time t4.

[0068] The image processing device 13 identifies, for each of the two first images represented by the first image data, an area corresponding to area A4 of the second image. In this case, the image processing device 13 identifies area B4 corresponding to area A4 for the image in Fig. 12B, and identifies area C4 corresponding to area A4 for the image in Fig. 12C.

[0069] Next, the image processing device 13 recognizes objects in the identified areas of each of the two first images using a known image recognition technique. In this case, the image processing device 13 recognizes that a person is present in area B4 in Fig. 12B and area C4 in Fig. 12C.

[0070] Next, the image processing device 13 determines whether the object (in this case, a human) recognized from area B4 of Fig. 12B and area C4 of Fig. 12C is an object having predetermined characteristics. In this example, the image processing device 13 determines whether the characteristics of the equipment, such as the clothing, of the human recognized from the first image have the characteristics of the equipment of an enemy soldier. Hereinafter, it is assumed that the image processing device 13 has determined that the human recognized from area B4 of Fig. 12B and area C4 of Fig. 12C is an enemy soldier.

[0071] In this case, the image processing device 13 calculates the distance to the enemy soldier, i.e., the distance between the stereo night vision camera 41 and the enemy soldier, using known triangulation based on the position of the image of the enemy soldier in the image of Figure 12B and the position of the image of the enemy soldier in the image of Figure 12C.

[0072] Next, the image processing device 13 generates a display image by adding, for example, the words "enemy soldier" to the first image captured by the first night vision camera to indicate that an enemy soldier is captured, and adding, for example, the words "XX m" to indicate the distance to the enemy soldier. Note that the image processing device 13 may generate a display image using the first image captured by the second night vision camera instead of the first image captured by the first night vision camera. The image processing device 13 transmits display image data representing the generated display image to the terminal device 14.

[0073] When the terminal device 14 receives the display image data transmitted from the image processing device 13, it displays the image represented by the received display image data. Fig. 13 is a diagram showing an example of an image displayed by the terminal device 14.

[0074] 13, the user can know that there is an enemy soldier in the direction of shooting by the stereo night vision camera 41 and the non-night vision camera 12, and the distance to that enemy soldier. If the enemy soldier is within the range of the user's weapon (such as an airsoft gun), the user can fire a bullet (an example of a projectile) at the enemy soldier (a real object).

[0075] As described above, the image processing device 13 recognizes an object having predetermined characteristics from the second image, determines whether the part of the first image corresponding to the position of the object satisfies predetermined conditions, and if it determines that the predetermined conditions are satisfied, notifies the user by sending display image data to the terminal device 14.

[0076] In the above example, the user watches the display image displayed by the terminal device 14 and fires a bullet at an enemy soldier, but instead, the firing device may automatically fire the bullet. In this case, when the image processing device 13 recognizes an enemy soldier and measures the distance to the enemy soldier, the firing device determines whether or not the distance is within the firing range of the device itself, and if it is within the firing range, fires a bullet at the actual enemy soldier.

[0077] Fifth Embodiment A system 5 according to a fifth embodiment of the present invention will be described below. The system 5 according to the fifth embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the points where the system 5 differs from the system 1, and descriptions of the points where the system 5 has in common with the system 1 will be omitted as appropriate.

[0078] Fig. 14 is a diagram showing the configuration of System 5. As shown in Fig. 14, System 5 is different from System 1 in that it includes a non-night vision camera 52 instead of non-night vision camera 12.

[0079] The non-night vision camera 52 provided in the system 5 is a device that transmits sound waves into water, receives the reflected waves of those sound waves, and generates an image (second image) representing the shape of an object underwater based on the received reflected waves.

[0080] The night vision camera 11 provided in the system 5 is equipped with a telephoto lens with a zoom function, a drive unit for focusing the telephoto lens and adjusting the focal length, and an electric pan head for adjusting the shooting direction of the night vision camera 11.

[0081] When the image processing device 13 of the system 5 continuously receives the second image data transmitted from the non-night vision camera 52, it recognizes an object having a predetermined characteristic from the second image represented by the received second image data. Hereinafter, the object having a predetermined characteristic recognized by the image processing device 13 from the second image is assumed to be an aquatic organism such as a fish.

[0082] Fig. 15 is a diagram showing an example of a second image captured at time t5 by the non-night vision camera 52. Hereinafter, it is assumed that the image processing device 13 recognizes a fish as an object having a predetermined characteristic from the image in Fig. 15 .

[0083] When the image processing device 13 recognizes an object having a predetermined characteristic from the second image, it determines the direction from the night vision camera 11 toward the object based on the position of the recognized object in the second image as the shooting direction of the night vision camera 11.

[0084] Next, the image processing device 13 instructs the night vision camera 11 to start shooting. This instruction includes data indicating the shooting direction.

[0085] When the night vision camera 11 receives an instruction to start shooting transmitted from the image processing device 13, it adjusts the shooting direction to the specified shooting direction using the electric camera platform in accordance with the instruction and then starts shooting. The night vision camera 11 sequentially transmits first image data representing the captured images (first images) to the image processing device 13.

[0086] When the image processing device 13 receives the first image data transmitted from the night vision camera 11, it determines whether the first image represented by the received first image data is in focus, and if it is not in focus, it instructs the night vision camera 11 to adjust the focus. Note that the technology for adjusting the focus based on the image captured by the camera is known, and therefore a description thereof will be omitted.

[0087] When the image processing device 13 comes into focus on an object in the first image represented by the first image data transmitted from the night vision camera 11, it instructs the night vision camera 11 to adjust the zoom so that the object is captured to fill the angle of view. The image processing device 13 performs zoom adjustment in accordance with the instruction from the image processing device 13 and transmits the first image data representing the captured first image to the image processing device 13.

[0088] The image processing device 13 generates a display image representing an in-focus image in which an object is captured over the entire angle of view, as represented by the first image data transmitted from the night-vision camera 11, and transmits the generated display image to the terminal device 14.

[0089] When the terminal device 14 receives the display image data transmitted from the image processing device 13, it displays the image represented by the received display image data. Fig. 16 is a diagram showing an example of an image displayed by the terminal device 14. The display image displayed by the terminal device 14 is an image captured by the night vision camera 11 by zooming in on an object recognized from the second image captured by the non-night vision camera 52.

[0090] The user can view the appearance of creatures living in dark water using the display image shown in FIG.

[0091] In system 5, after an object is recognized from the second image captured by non-night vision camera 52, the object is captured by night vision camera 11 to generate the first image. Therefore, although the capture times of the second image and the first image used to generate the display image are not the same, the time interval between the capture times of the second image and the first image is within a predetermined time length determined as the maximum time required for focus adjustment and zoom adjustment. If this predetermined time length is sufficiently short in light of the rate of change in the relative position of the subject object and the night vision camera 11 and non-night vision camera 52, system 5 can capture the object captured by non-night vision camera 52 with night vision camera 11.

[0092] Sixth Embodiment A system 6 according to a sixth embodiment of the present invention will be described below. The system 6 according to the sixth embodiment has many points in common with the system 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the points where the system 6 differs from the system 1, and descriptions of the points where the system 6 has in common with the system 1 will be omitted as appropriate.

[0093] Fig. 17 is a diagram showing the configuration of System 6. As shown in Fig. 17, System 6 is different from System 1 in that it includes a non-night vision camera 62 instead of the non-night vision camera 12.

[0094] The non-night vision camera 62 included in the system 6 is a device (so-called distance imaging camera) that transmits electromagnetic waves, receives reflected waves of the electromagnetic waves, and generates an image (second image) representing the shape of an object based on the received reflected waves. Examples of distance imaging cameras that can be used as the non-night vision camera 62 include, but are not limited to, millimeter wave radar that uses millimeter waves and LiDAR (Light Detection and Ranging) that uses laser light.

[0095] The image processing device 13 generates an image for display by mapping, as a texture, the appearance represented by the first image captured by the night vision camera 11 at the same time onto the shape represented by the second image captured by the non-night vision camera 62, and transmits display image data representing the generated display image to the terminal device 14. The terminal device 14 displays the display image represented by the display image data received from the image processing device 13.

[0096] [Modifications] Each of the above-described systems 1 to 6 is an embodiment of the system according to the present invention, and may be modified in various ways within the scope of the technical concept of the present invention. Examples of such modifications are shown below. Two or more of the modifications shown below may be combined as appropriate.

[0097] (1) In the above-described embodiment, the first image and the second image may be switched as appropriate.

[0098] For example, in the first embodiment, the image processing device 13 recognizes an object having a predetermined characteristic from the second image, cuts out a portion of the first image corresponding to the position of the object recognized from the second image, and generates an image for display using the cut-out image. Alternatively, the image processing device 13 may recognize an object having a predetermined characteristic from the first image, cut out a portion of the second image corresponding to the position of the object recognized from the first image, and generate an image for display using the cut-out image.

[0099] Furthermore, in the fourth embodiment, the image processing device 13 recognizes an object having predetermined characteristics from the second image and determines whether or not a portion of the first image corresponding to the position of the object recognized from the second image satisfies predetermined conditions. Alternatively, the image processing device 13 may recognize an object having predetermined characteristics from the first image and determine whether or not a portion of the second image corresponding to the position of the object recognized from the first image satisfies predetermined conditions.

[0100] (2) In the above-described embodiment, when the night vision camera 11 (or the stereo night vision camera 41) is a fixed camera, the image processing device 13 may perform a process of correcting the color of the night vision image based on the relationship between the color in the daytime image and the color in the night time image of the same object recognized from both the image taken by the night vision camera 11 (or the stereo night vision camera 41) during the day (hereinafter referred to as the “daytime image”) and the image taken by the night vision camera 11 (or the stereo night vision camera 41) at night (hereinafter referred to as the “night time image”), and may generate an image for display using the corrected image.

[0101] Specifically, the image processing device 13 generates a conversion formula or conversion table that brings the color of the nighttime image closer to the color of the daytime image based on the correspondence between color information of the same object at the same position in both the daytime image and the nighttime image. The image processing device 13 converts the color information of each pixel of the nighttime image according to the conversion formula or conversion table, and generates a color-corrected nighttime image, which is a collection of pixels with converted color information, as an image for display.

[0102] (3) The image processing device 13 and the night vision camera 11 (or the stereo night vision camera 41) may be directly connected by a cable or the like, or may be connected via a communication network. In the latter case, the image processing device 13 receives the first image from the night vision camera 11 (or the stereo night vision camera 41) via the communication network.

[0103] Furthermore, the image processing device 13 and the non-night vision camera 12 (or the non-night vision camera 32, the non-night vision camera 52, or the non-night vision camera 62) may be directly connected by a cable or the like, or may be connected via a communications network. In the latter case, the image processing device 13 receives the second image from the non-night vision camera 12 (or the non-night vision camera 32, the non-night vision camera 52, or the non-night vision camera 62) via the communications network.

[0104] (4) The image processing device 13 may be configured as a single device, or may be configured as a group of multiple devices (for example, a group of server devices) that operate in cooperation with each other.

[0105] In addition, in the above-described embodiment, part of the processing performed by the image processing device 13 may be performed by the night vision camera 11 (or the stereo night vision camera 41), the non-night vision camera 12 (or the non-night vision camera 32, the non-night vision camera 52, or the non-night vision camera 62), or the terminal device 14.

[0106] For example, in the above-described embodiment, if the terminal device 14 performs part of the processing (e.g., object recognition processing) that is to be performed by the image processing device 13, the image processing device 13 and the terminal device 14 will constitute an image processing device.

[0107] 1...system, 2...system, 3...system, 4...system, 5...system, 6...system, 11...night vision camera, 12...non-night vision camera, 13...image processing device, 14...terminal device, 32...non-night vision camera, 41...stereo night vision camera, 52...non-night vision camera, 62...non-night vision camera.

Claims

1. A system for continuously identifying and storing the temperature of each of a plurality of objects moving within a space in a dark environment, comprising: a night vision camera, which is a visible light camera with night vision correction function; a thermographic camera that generates images by sensing infrared light; and an image processing device, wherein an image of the space photographed by the night vision camera is defined as a first image, and an image of the space photographed by the thermographic camera at the same time as the first image is defined as a second image corresponding to the first image; the image processing device recognizes a plurality of objects having predetermined characteristics from the first image, and identifies, for each of the plurality of objects, the temperature indicated by a portion of the second image corresponding to the first image that corresponds to the position of the object having the predetermined characteristics recognized from the first image; when the image processing device recognizes an object having the predetermined characteristics from a new first image, it identifies the previously recognized object and the newly recognized object based on the image of the object having the predetermined characteristics recognized from the previous first image and the image of the object having the predetermined characteristics recognized from the new first image; When the image processing device determines that the object having the predetermined characteristic recognized from a new first image is not identical to any of the objects having the predetermined characteristic recognized from past first images, it assigns identification information to the newly recognized object and stores, in association with the identification information, data indicating the date and time the new first image was taken and data indicating the temperature indicated by a portion of the second image corresponding to the new first image that corresponds to the position of the newly recognized object; when the image processing device determines that the object having the predetermined characteristic recognized from a new first image is identical to any of the objects having the predetermined characteristic recognized from past first images, it stores, in association with the identification information assigned to the object that is identical to the newly recognized object, data indicating the date and time the new first image was taken and data indicating the temperature indicated by a portion of the second image corresponding to the new first image that corresponds to the position of the newly recognized object.

Images