System for processing night vision images

Abstract

This technology provides a means to recognize objects from night vision images with higher accuracy compared to conventional technologies. [Solution] The system 1 according to the present invention comprises a night vision camera 11, which is a visible light camera having a night vision correction function and generating a color image by taking a picture; a non-night vision camera 12, which is a camera that senses electromagnetic waves or sound waves other than visible light and generates an image; an image processing device 13 that generates a display image using a first image taken by the night vision camera 11 and a second image taken by the non-night vision camera 12 at the same time as when the night vision camera 11 took the first image, or at a time interval of a predetermined length or less; and a terminal device 14 that displays the display image generated by the image processing device 13.

Description

Technical Field

[0001] The present invention relates to a technique for processing an image captured by a night vision camera.

Background Art

[0002] There is a night vision camera (hereinafter simply referred to as a "night vision camera"), which is a visible light camera having a night vision correction function. The night vision correction function is a function of reducing the sensitivity of an image sensor during the day and increasing the sensitivity of the image sensor at night. The night vision camera has a high-sensitivity image sensor, and by the night vision correction function, the sensitivity is increased at night to sense a small amount of visible light and generate a color image.

[0003] As a patent document describing a technique related to the above night vision camera, for example, there is Patent Document 1.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] An image captured by a night vision camera at night (hereinafter referred to as a "night vision image") is an image generated by sensing a small amount of visible light. Therefore, compared with an image captured by a normal visible light camera or a night vision camera during the day (hereinafter referred to as a "daylight image"), the resolution and color reproducibility are low. Therefore, it is more difficult to recognize an object from a night vision image than from a daylight image.

[0006] In view of the above circumstances, the present invention provides means for enabling recognition of an object with high accuracy from a night vision image as compared with the prior art.

Means for Solving the Problems

[0007] The present invention provides a system comprising: a night vision camera which is a visible light camera having a night vision correction function; a non-night vision camera which is a camera that senses electromagnetic waves or sound waves other than visible light and generates an image; and an image processing device which generates a display image using a first image captured by the night vision camera and a second image captured by the non-night vision camera at the same time as when the night vision camera captured the first image, or at a time interval of a predetermined length or less. [Effects of the Invention]

[0008] According to the present invention, objects can be recognized from night vision images with higher accuracy compared to conventional technology. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing the configuration of the system according to the first embodiment. [Figure 2A] A diagram illustrating an image captured by a non-night vision camera according to the first embodiment. [Figure 2B] A diagram illustrating an image captured by a night vision camera according to the first embodiment. [Figure 3] A diagram illustrating an image displayed by a terminal device according to the first embodiment. [Figure 4] A diagram showing the configuration of the system according to the second embodiment. [Figure 5A] A diagram illustrating an image captured by a non-night vision camera according to the second embodiment. [Figure 5B] A diagram illustrating an image captured by a night vision camera according to the second embodiment. [Figure 6] A diagram showing the data structure of a table stored by the image processing device according to the second embodiment. [Figure 7] A diagram illustrating an image displayed by a terminal device according to the second embodiment. [Figure 8] A diagram showing the configuration of the system according to the second embodiment. [Figure 9A] A diagram illustrating an image captured by a non-night vision camera according to the third embodiment. [Figure 9B]A diagram illustrating an image captured by the night vision camera according to the third embodiment. [Figure 10] A diagram illustrating an image to be displayed by the terminal device according to the third embodiment. [Figure 11] A diagram showing the configuration of the system according to the fourth embodiment. [Figure 12A] A diagram illustrating an image captured by the non-night vision camera according to the fourth embodiment. [Figure 12B] A diagram illustrating an image captured by the stereo night vision camera according to the fourth embodiment. [Figure 12C] A diagram illustrating an image captured by the stereo night vision camera according to the fourth embodiment. [Figure 13] A diagram illustrating an image to be displayed by the terminal device according to the fourth embodiment. [Figure 14] A diagram showing the configuration of the system according to the fifth embodiment. ​​​​​​​​​​​​​​​​​​​​​The night vision camera 11 is a visible light camera with a night vision correction function, and generates color images through shooting. In environments where sufficient visible light illumination is available, such as outdoors during the day, the night vision camera 11 lowers the sensitivity of its image sensor and captures color images as a normal visible light camera. In environments where sufficient visible light illumination is not available, such as at night or in enclosed spaces where visible light from the outside is blocked, the night vision camera 11 increases the sensitivity of its image sensor to detect even slight visible light and capture color images. The image generated by the night vision camera 11 will be referred to as the "first image" below.

[0012] The night vision camera 11 is equipped with a communication interface and transmits image data representing the first captured image (hereinafter referred to as "first image data") to the image processing device 13. Alternatively, instead of the night vision camera 11 having a built-in communication interface, an external communication interface may be connected to the night vision camera 11.

[0013] The non-night vision camera 12 is a camera that generates an image by sensing electromagnetic waves or sound waves other than visible light. In this embodiment, the non-night vision camera 12 is a thermographic camera that generates an image by sensing infrared radiation emitted by an object. More specifically, the non-night vision camera 12 generates an image representing the temperature distribution of an object by sensing infrared radiation in the wavelength band of approximately 2 to 14 μm, which the higher the temperature of the object, the greater the amount of energy emitted. The image generated by the non-night vision camera 12 will be referred to below as the "second image".

[0014] The non-night vision camera 12 is equipped with a communication interface and transmits image data representing the captured second image (hereinafter referred to as "second image data") to the image processing device 13. Alternatively, instead of the non-night vision camera 12 having a built-in communication interface, an external communication interface may be connected to the non-night vision camera 12.

[0015] The field of view of the night vision camera 11 and the field of view of the non-night vision camera 12 are substantially the same. In this application, the substantially same field of view of the two cameras means that there is an overlapping area between the area captured in the image taken by one camera and the area captured in the image taken by the other camera. However, it is desirable that the shooting directions of the two cameras are parallel, and it is even more desirable that the field of view of the two cameras are identical.

[0016] The image processing device 13 is a data processing device that receives first image data from the night vision camera 11, receives second image data from the non-night vision camera 12, generates a display image using the two images represented by these image data, and transmits the generated display image data (hereinafter referred to as "display image data") to the terminal device 14.

[0017] The hardware of the image processing device 13 is, for example, a computer for a server device, and includes a memory for persistently storing various data including programs, a processor for performing various data processing according to the programs stored in the memory, and a communication interface for data communication with external devices. In other words, the computer, which is the hardware of the image processing device 13, operates as the image processing device 13 that performs the processing described later by having the processor execute various data processing according to the programs for the image processing device according to this embodiment that are persistently stored in the memory.

[0018] Terminal device 14 is a data processing device that receives display image data from image processing device 13 and displays the image represented by the received display image data.

[0019] The hardware of the terminal device 14 is, for example, a computer for the terminal device, and includes a memory that persistently stores various data including programs, a processor that performs various data processing according to the programs stored in the memory, a communication interface for data communication with external devices, a display that displays information to the user, and an input device that accepts information input operations from the user. In other words, the computer, which is the hardware of the terminal device 14, operates as the terminal device 14 that performs the processing described later by executing various data processing according to the programs for the terminal device according to this embodiment that are persistently stored in the memory using the processor.

[0020] The type of computer used as the hardware for the terminal device 14 may be any of the following: a desktop PC (Personal Computer), a laptop PC, or a tablet PC (including a smartphone, which is a small tablet PC equipped with calling capabilities via a mobile communication network). Alternatively, instead of the terminal device 14 having at least one of the communication interface, display, and input device built-in, these devices may be connected to the terminal device 14 as external devices.

[0021] In this embodiment, the night vision camera 11 and the non-night vision camera 12 take pictures at the same time. In this application, "two cameras taking pictures at the same time" is not limited to cases where they take pictures at exactly the same time, but also includes cases where they take pictures substantially at the same time. "Taking pictures substantially at the same time" means taking pictures with a sufficiently short time difference in light of the rate of change of the relative position between the camera and the subject, so that the subject captured in the image taken by one camera is also captured in the image taken by the other camera.

[0022] In this embodiment, for example, the night vision camera 11 and the non-night vision camera 12 are synchronized based on a synchronization signal received from the image processing device 13, and each continuously takes pictures at predetermined time intervals. More specifically, for example, each of the night vision camera 11 and the non-night vision camera 12 has a built-in clock, which continuously measures the current time, and takes pictures at substantially the same timing, for example, at 1 / 10 second intervals. The clocks of the night vision camera 11 and the non-night vision camera 12 drift over time, but the night vision camera 11 and the non-night vision camera 12 periodically correct their clocks based on the synchronization signal received from the image processing device 13. As a result, the images taken by the night vision camera 11 and the non-night vision camera 12 are images taken at substantially the same time.

[0023] Figure 2A shows an example of an image (second image) taken by the non-night vision camera 12 at time t1. Figure 2B shows an example of an image (first image) taken by the night vision camera 11 at time t1.

[0024] The image processing device 13 recognizes an object having predetermined characteristics from the second image illustrated in Figure 2A. If there are warm-blooded animals such as birds and mammals, or devices that generate heat during operation, within the field of view of the non-night vision camera 12, the second image captured by the non-night vision camera 12 will include images representing the shapes of those objects. For example, if the second image contains an image that occupies an area greater than or equal to a predetermined threshold, the image processing device 13 recognizes that image as an object having predetermined characteristics.

[0025] Specifically, the image processing device 13 recognizes an object in region A1 with respect to the second image illustrated in Figure 2A.

[0026] Next, the image processing device 13 generates an image for display that represents the portion of the first image corresponding to the position of the object recognized in the second image within the second image.

[0027] Specifically, the image processing device 13 identifies region B1 as the part of the first image in Figure 2B that corresponds to the position of region A1 in the second image in Figure 2A.

[0028] Next, the image processing device 13 extracts the region B1 from the first image in Figure 2B and generates an image for display.

[0029] The image processing device 13 transmits the display image data, which represents the display image generated as described above, to the terminal device 14. The terminal device 14 receives the display image data transmitted from the image processing device 13 and displays the image represented by the received display image data on its display. Figure 3 is an example of an image displayed by the terminal device 14.

[0030] The example display image in Figure 3 shows a thumbnail image of the entire first image and an enlarged image cropped from the first image (the image of region B1 of the first image) displayed side by side.

[0031] The user of terminal device 14 can view the display image exemplified in Figure 3 and easily recognize, for example, organisms that are difficult to find hidden in the shade of trees, etc.

[0032] [Second Embodiment] The following describes System 2 according to the second embodiment of the present invention. System 2 according to the second embodiment shares many similarities with System 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the differences between System 2 and System 1, and will omit explanations of the similarities between System 2 and System 1 as appropriate.

[0033] Figure 4 shows the configuration of System 2. As shown in Figure 4, the configuration of System 2 is the same as that of System 1.

[0034] How does System 2 differ from System 1 in its operation? The operation of System 2 is described below.

[0035] Figure 5A shows an example of a second image taken by the non-night vision camera 12 of System 2 at time t2, and Figure 5B shows an example of a first image taken by the night vision camera 11 of System 2 at time t2.

[0036] The image processing device 13 of System 2 recognizes an object from the first image. Any known image recognition technique may be used by the image processing device 13 to recognize a pig from the first image. In this case, the image processing device 13 recognizes that each of the images in regions B2 and B3 of the first image in Figure 5B is an image of a pig. For each recognized object, the image processing device 13 stores an image extracted from the first image containing that object as a new file.

[0037] Next, the image processing device 13 identifies each of the objects recognized from the first image based on the characteristics of the object, such as its pattern and shape. Specifically, the image processing device 13 measures the similarity between each of the previously recognized images of pigs and the newly recognized image of pigs, for example, the pattern on the surface of the pigs' bodies, and identifies the pigs based on the measured similarity.

[0038] Furthermore, in order to enable the image processing device 13 to easily and accurately identify each pig from the first image, for example, an image such as a barcode indicating identification information may be drawn on the body surface of each pig, or collars, leg bands, etc., on which an image such as a barcode indicating identification information is drawn may be attached to each pig. In that case, the image processing device 13 reads the barcode or other image in the first image, decodes it to identify the identification information, and thereby identifies the object recognized from the first image.

[0039] If the image processing device 13 determines that a newly recognized object is not identical to any object previously recognized, it assigns new identification information to the newly recognized object, generates a data table (hereinafter referred to as "table T") corresponding to that identification information, and stores it.

[0040] Figure 6 shows the data structure of table T stored by the image processing device 13. The image processing device 13 stores table T corresponding to each piece of identification information. Each piece of table T contains the following data fields (hereinafter simply referred to as "fields").

[0041] "Date and Time" field: Stores data indicating the date and time the first image (and the second image) were taken. "Image File Name" field: Stores data indicating the file name of the image of the object recognized from the first image. "Body Temperature" field: Stores data indicating the body temperature identified from the second image.

[0042] The image processing device 13 determines that the object newly recognized from the first image is not identical to any object previously recognized, assigns new identification information to the newly recognized object, generates and stores a table T corresponding to that identification information, and adds a new data record (hereinafter simply referred to as "record") to the table T. The record stores data indicating the date and time the first image used to recognize the object was taken, and data indicating the file name of the image of the object extracted from the first image.

[0043] On the other hand, if the image processing device 13 determines that an object newly recognized from the first image is identical to any object previously recognized, it adds a new record to the table T corresponding to the identification information of that object. This record stores data indicating the date and time the first image used to recognize the object was taken, and data indicating the file name of the image of the object extracted from the first image.

[0044] Hereinafter, the image processing device 13 will be assumed to have identified the pig in area B2 as the pig identified by identification information P095, and the pig in area B3 as the pig identified by identification information P143.

[0045] Next, the image processing device 13 identifies the temperature shown in the portion of the second image corresponding to the location of each object recognized from the first image. Specifically, the image processing device 13 identifies the temperature shown in the image of region A2 of the second image corresponding to region B2 of the first image, and the temperature shown in the image of region A3 of the second image corresponding to region B3 of the first image.

[0046] The image processing device 13 stores the temperature data identified from region A2 of the second image in the last record of the table corresponding to the identification information P095. The image processing device 13 also stores the temperature data identified from region A3 of the second image in the last record of the table corresponding to the identification information P143.

[0047] The image processing device 13 performs a series of processes for each pair of images: a first image represented by the first image data continuously received from the night vision camera 11, and a second image represented by the second image data continuously received from the non-night vision camera 12. These processes include object recognition, identification of the recognized object, and determination of the body temperature of the identified object. As a result, for each object within the field of view of the night vision camera 11 and the non-night vision camera 12, a data sequence showing the change in the temperature (body temperature) of that object over time is stored in a table T corresponding to that object.

[0048] When new data indicating body temperature is stored in table T, the image processing device 13 determines whether the body temperature indicated by the data stored in table T meets predetermined conditions indicating an abnormal condition in the pig. If it determines that the predetermined conditions are met, it generates an image (display image) to notify the user of terminal device 14 of the abnormality, and transmits the display image data representing the generated display image to terminal device 14.

[0049] When the terminal device 14 receives display image data transmitted from the image processing device 13, it displays the image represented by the received display image data. Figure 7 is an example of an image displayed by the terminal device 14. For example, the terminal device 14 displays an image that includes identification information of an object determined to be in a state of abnormal health, and a graph showing the change in the object's temperature (body temperature) over time.

[0050] By viewing the example display image in Figure 7, users can easily identify, for example, any health abnormalities in a pig.

[0051] [Third Embodiment] The following describes System 3 according to the third embodiment of the present invention. System 3 according to the third embodiment shares many similarities with System 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the differences between System 3 and System 1, and will omit explanations of the similarities between System 3 and System 1 as appropriate.

[0052] Figure 8 shows the configuration of System 3. As shown in Figure 8, System 3 is equipped with a non-night vision camera 32 in place of the non-night vision camera 12, compared to System 1.

[0053] The non-night vision camera 32 in System 3 is an infrared camera with an infrared illumination function that emits infrared light and generates an image by sensing the reflected infrared light.

[0054] The non-night vision camera 32 can generate clear images by emitting infrared light even in environments such as at night or in enclosed spaces where external light is blocked, but these images are black and white images, or images composed only of grayscale information.

[0055] Figure 9A is a schematic diagram showing an example of a second image taken by the non-night vision camera 32 at time t3, and Figure 9B is a schematic diagram showing an example of a first image taken by the night vision camera 11 of system 3 at time t3.

[0056] The image processing device 13 of System 3 generates a display image that represents an object in the same position in the first image (Figure 9B) captured by the night vision camera 11 and the second image (Figure 9A) captured by the non-night vision camera 32 at the same time, using the shape based on the second image and the color based on the first image. More specifically, the image processing device 13 identifies the color information of the first image and the grayscale information of the second image for each corresponding pixel in the first and second images, and generates an image which is a collection of pixels that possess that information.

[0057] Next, the image processing device 13 recognizes objects in the display image from the image generated from the first image and the second image as described above, using known image recognition techniques. Then, the image processing device 13 generates a display image that includes the image generated from the first image and the second image, and characters indicating the type of object recognized from that image. The image processing device 13 transmits the display image data representing the generated display image to the terminal device 14.

[0058] When the terminal device 14 receives display image data transmitted from the image processing device 13, it displays the image represented by the received display image data. Figure 10 is an example of an image displayed by the terminal device 14. The display image displayed by the terminal device 14 includes a clear color image that has been captured and text indicating the type of object recognized from that color image (in the example of Figure 10, it is "human").

[0059] The user can perceive the situation in a dark environment as if they were viewing it in a bright environment, using the example display image in Figure 10. Furthermore, the user can easily identify the presence of objects that are difficult to see with the naked eye in a dark environment, and the type of those objects, using the example display image in Figure 10.

[0060] [Fourth Embodiment] The following describes System 4 according to the fourth embodiment of the present invention. System 4 according to the fourth embodiment shares many similarities with System 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the differences between System 4 and System 1, and will omit explanations of the similarities between System 4 and System 1 as appropriate.

[0061] Figure 11 shows the configuration of System 4. As shown in Figure 11, System 4, compared to System 1, is equipped with a stereo night vision camera 41 instead of the night vision camera 11.

[0062] The stereo night vision camera 41 comprises two night vision cameras (hereinafter referred to as the "first night vision camera" and the "second night vision camera") arranged at a predetermined distance apart so that their shooting directions are parallel, and transmits first image data representing two images (first images) simultaneously captured by the first night vision camera and the second night vision camera to the image processing device 13.

[0063] In the following description, System 4 will be used by a user who is a participant in a survival game to locate enemy soldiers.

[0064] When the image processing device 13 sequentially receives the second image data that is continuously transmitted from the non-night vision camera 12 (thermography camera), it determines from the image represented by the received second image data (second image) whether or not there is a region with a temperature above a predetermined threshold that occupies an area above a predetermined threshold.

[0065] Figure 12A shows an example of a second image captured by the non-night vision camera 12 at time t4. The image processing device 13 determines that, with respect to the second image illustrated in Figure 12A, there is a region A4 that occupies an area greater than or equal to a predetermined threshold and has a temperature greater than or equal to a predetermined threshold.

[0066] If the image processing device 13 determines from the second image that there is one or more regions that satisfy the above conditions, it performs the following processing for each of those regions using the first image data received from the stereo night vision camera 41 at the same time. The following explanation will use the processing performed by the image processing device 13 with respect to region A4 in Figure 12A as an example.

[0067] Figure 12B shows an example of an image taken by the first night vision camera of the stereo night vision camera 41 at time t4, and Figure 12C shows an example of an image taken by the second night vision camera of the stereo night vision camera 41 at time t4.

[0068] The image processing device 13 identifies a region corresponding to region A4 of the second image for each of the two first images represented by the first image data. In this case, the image processing device 13 identifies region B4 corresponding to region A4 for the image in Figure 12B, and identifies region C4 corresponding to region A4 for the image in Figure 12C.

[0069] Next, the image processing device 13 recognizes objects in the regions identified above for each of the two first images using known image recognition techniques. In this case, the image processing device 13 recognizes that a human being is depicted in region B4 in Figure 12B and region C4 in Figure 12C.

[0070] Next, the image processing device 13 determines whether the object (in this case, a human) recognized from region B4 in Figure 12B and region C4 in Figure 12C is an object with predetermined characteristics. In this example, the image processing device 13 determines whether the characteristics of the clothing and other equipment of the human recognized from the first image are those of enemy soldier equipment. Hereafter, it will be assumed that the image processing device 13 has determined the human recognized from region B4 in Figure 12B and region C4 in Figure 12C to be an enemy soldier.

[0071] In that case, the image processing device 13 calculates the distance to the enemy soldier, that is, the distance between the stereo night vision camera 41 and the enemy soldier, based on the position of the enemy soldier's image in the image of Figure 12B and the position of the enemy soldier's image in the image of Figure 12C, using known triangulation.

[0072] Next, the image processing device 13 generates a display image by adding the text "Enemy Soldier" to the first image captured by the first night vision camera as an indication that an enemy soldier is present, and adding the text "○○m" to indicate the distance to the enemy soldier. The image processing device 13 may also generate the display image using the first image captured by the second night vision camera instead of the first image captured by the first night vision camera. The image processing device 13 then transmits the display image data representing the generated display image to the terminal device 14.

[0073] When the terminal device 14 receives display image data transmitted from the image processing device 13, it displays the image represented by the received display image data. Figure 13 is an example of an image displayed by the terminal device 14.

[0074] The user can view the example display image in Figure 13 to determine that there are enemy soldiers in the direction captured by the stereo night vision camera 41 and the non-night vision camera 12, and to determine the distance to those enemy soldiers. If the enemy soldier is within the range of the user's weapon (airsoft gun, etc.), the user fires a projectile (an example of a projectile) at the enemy soldier (a real person).

[0075] As described above, the image processing device 13 recognizes an object having predetermined characteristics from the second image, determines whether the portion of the first image corresponding to the position of that object satisfies predetermined conditions, and if it determines that the predetermined conditions are met, it notifies the user by transmitting display image data to the terminal device 14.

[0076] In the example described above, the user views the display image shown by the terminal device 14 and fires a bullet at the enemy soldier. However, instead, the firing device may automatically fire the bullet. In that case, the firing device will determine whether the distance is less than or equal to its firing range when the image processing device 13 recognizes the enemy soldier and measures the distance to that soldier. If it is less than or equal to the firing range, the firing device will fire a bullet at the actual enemy soldier.

[0077] [Fifth Embodiment] The following describes System 5 according to the fifth embodiment of the present invention. System 5 according to the fifth embodiment shares many similarities with System 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the differences between System 5 and System 1, and will omit explanations of the similarities between System 5 and System 1 as appropriate.

[0078] Figure 14 shows the configuration of System 5. As shown in Figure 14, System 5 is equipped with a non-night vision camera 52 in place of the non-night vision camera 12, compared to System 1.

[0079] The non-night vision camera 52 in System 5 is a device that transmits sound waves into the water, receives the reflected sound waves, and generates an image (second image) representing the shape of an object in the water based on the received reflected waves.

[0080] Furthermore, the night vision camera 11 in System 5 is equipped with a telephoto lens with a zoom function, a drive unit that adjusts the focus and focal length of the telephoto lens, and an electric pan / tilt head that adjusts the shooting direction of the night vision camera 11.

[0081] When the image processing device 13 of System 5 receives second image data continuously transmitted from the non-night vision camera 52, it recognizes an object having predetermined characteristics from the second image represented by the received second image data. Hereinafter, the object having predetermined characteristics that the image processing device 13 recognizes from the second image will be an aquatic organism such as a fish.

[0082] Figure 15 shows an example of a second image captured by the non-night vision camera 52 at time t5. Hereafter, it will be assumed that the image processing device 13 recognized a fish as an object with predetermined characteristics from the image in Figure 15.

[0083] When the image processing device 13 recognizes an object having predetermined characteristics from the second image, it determines the direction from the night vision camera 11 toward the object, based on the position of the recognized object in the second image, as the shooting direction of the night vision camera 11.

[0084] Next, the image processing device 13 instructs the night vision camera 11 to start shooting. This instruction includes data indicating the shooting direction.

[0085] When the night vision camera 11 receives an instruction to start shooting from the image processing device 13, it adjusts the shooting direction to the specified direction using the motorized pan / tilt head in accordance with the instruction, and then starts shooting. The night vision camera 11 sequentially transmits the first image data representing the captured image (first image) to the image processing device 13.

[0086] When the image processing device 13 receives the first image data transmitted from the night vision camera 11, it determines whether the first image represented by the received first image data is in focus or not. If it is not in focus, it instructs the night vision camera 11 to adjust the focus. Note that the technique of adjusting the focus based on the image captured by the camera is known, so its explanation will be omitted.

[0087] When the image processing device 13 finds that an object in the first image represented by the first image data transmitted from the night vision camera 11 is in focus, it instructs the night vision camera 11 to adjust the zoom so that the object fills the entire field of view. Following the instructions from the image processing device 13, the zoom is adjusted and the first image data representing the captured first image is transmitted back to the image processing device 13.

[0088] The image processing device 13 generates a display image representing a focused image of an object filling the entire field of view, which is represented by the first image data transmitted from the night vision camera 11, and transmits the generated display image to the terminal device 14.

[0089] When the terminal device 14 receives display image data transmitted from the image processing device 13, it displays the image represented by the received display image data. Figure 16 is an example of an image displayed by the terminal device 14. The display image displayed by the terminal device 14 is an image taken by the night vision camera 11, zoomed in on an object recognized from a second image taken by the non-night vision camera 52.

[0090] The user can view the appearance of organisms living in dark water using the example display image shown in Figure 16.

[0091] In System 5, after an object is recognized from the second image captured by the non-night vision camera 52, the night vision camera 11 captures the object to generate the first image. Therefore, the capture times of the second image and the first image used to generate the display image are not the same, but the time interval between the capture times of the second image and the first image is within a predetermined time length defined as the maximum time required for focus adjustment and zoom adjustment. If this predetermined time length is sufficiently short in light of the rate of change in the relative position between the subject object and the night vision camera 11 and the non-night vision camera 52, System 5 can capture the object captured by the non-night vision camera 52 with the night vision camera 11.

[0092] [Sixth Embodiment] The system 6 according to the sixth embodiment of the present invention will be described below. The system 6 according to the sixth embodiment shares many similarities with the system 1 according to the first embodiment described above. Therefore, the following description will mainly focus on the differences between system 6 and system 1, and will omit explanations of the similarities between system 6 and system 1 as appropriate.

[0093] Figure 17 shows the configuration of System 6. As shown in Figure 17, System 6 is equipped with a non-night vision camera 62 in place of the non-night vision camera 12, compared to System 1.

[0094] The non-night vision camera 62 in System 6 is a device (a so-called depth image camera) that transmits electromagnetic waves, receives the reflected waves of those electromagnetic waves, and generates an image (a second image) representing the shape of an object based on the received reflected waves. Examples of depth image cameras that can be used as the non-night vision camera 62 include, but are not limited to, millimeter-wave radar using millimeter waves or LiDAR (Light Detection And Ranging) using laser light.

[0095] The image processing device 13 maps the appearance represented by the first image captured by the night vision camera 11 at the same time as a texture onto the shape represented by the second image captured by the non-night vision camera 62 to generate a display image, and transmits the display image data representing the generated display image to the terminal device 14. The terminal device 14 displays the display image represented by the display image data received from the image processing device 13.

[0096] [Differentiation] Each of the above-described Systems 1 to 6 is an embodiment of the system according to the present invention and may be modified in various ways within the scope of the technical idea of ​​the present invention. Examples of such modifications are shown below. Two or more of the following modifications may be combined as appropriate.

[0097] (1) In the embodiment described above, the first image and the second image may be swapped as appropriate.

[0098] For example, in the first embodiment, the image processing device 13 recognizes an object having predetermined characteristics from the second image, cuts out a portion of the first image corresponding to the position of the object recognized from the second image, and generates a display image using the cut-out image. Alternatively, the image processing device 13 may recognize an object having predetermined characteristics from the first image, cut out a portion of the second image corresponding to the position of the object recognized from the first image, and generate a display image using the cut-out image.

[0099] Furthermore, in the fourth embodiment, the image processing device 13 recognizes an object having predetermined characteristics from the second image and determines whether the portion of the first image corresponding to the position of the object recognized from the second image satisfies predetermined conditions. Alternatively, the image processing device 13 may recognize an object having predetermined characteristics from the first image and determine whether the portion of the second image corresponding to the position of the object recognized from the first image satisfies predetermined conditions.

[0100] (2) In the embodiments described above, if the night vision camera 11 (or stereo night vision camera 41) is a fixed-point camera, the image processing device 13 may perform a process to correct the color of the night image based on the relationship between the color in the daytime image and the color in the night image of the same object recognized from both the daytime image (hereinafter referred to as the "daytime image") and the nighttime image (hereinafter referred to as the "nighttime image") of the night vision camera 11 (or stereo night vision camera 41), and generate a display image using the corrected image.

[0101] Specifically, the image processing device 13 generates a conversion formula or conversion table that approximates the color of the night image to the color of the day image based on the correspondence of color information of the same object at the same position in both the daytime image and the nighttime image. For each pixel of the nighttime image, the image processing device 13 converts the color information of that image according to the conversion formula or conversion table, and generates a color-corrected nighttime image, which is a collection of pixels with the converted color information, as an image for display.

[0102] (3) The image processing device 13 and the night vision camera 11 (or stereo night vision camera 41) may be directly connected by a cable or the like, or they may be connected via a communication network. In the latter case, the image processing device 13 receives the first image from the night vision camera 11 (or stereo night vision camera 41) via the communication network.

[0103] Furthermore, the image processing device 13 and the non-night vision camera 12 (or non-night vision camera 32, non-night vision camera 52, non-night vision camera 62) may be directly connected by a cable or the like, or they may be connected via a communication network. In the latter case, the image processing device 13 receives a second image from the non-night vision camera 12 (or non-night vision camera 32, non-night vision camera 52, non-night vision camera 62) via the communication network.

[0104] (4) The image processing device 13 may consist of a single device or a group of devices that work in coordination (for example, a group of server devices).

[0105] Furthermore, in the embodiments described above, some of the processing that the image processing device 13 is supposed to perform may be performed by the night vision camera 11 (or stereo night vision camera 41), the non-night vision camera 12 (or the non-night vision camera 32, non-night vision camera 52, non-night vision camera 62), or the terminal device 14.

[0106] For example, in the embodiment described above, if the terminal device 14 performs a part of the processing that the image processing device 13 would normally perform (for example, object recognition processing), then the image processing device 13 and the terminal device 14 will constitute an image processing device. [Explanation of Symbols]

[0107] 1...System, 2...System, 3...System, 4...System, 5...System, 6...System, 11...Night vision camera, 12...Non-night vision camera, 13...Image processing device, 14...Terminal device, 32...Non-night vision camera, 41...Stereo night vision camera, 52...Non-night vision camera, 62...Non-night vision camera.

Claims

1. A night vision camera is a visible light camera with a night vision correction function, A non-night vision camera is a camera that generates images by sensing electromagnetic waves or sound waves other than visible light, An image processing apparatus that generates a display image using a first image captured by the night vision camera and a second image captured by the non-night vision camera at the same time as the night vision camera captured the first image, or at a time interval of a predetermined length or less. A system equipped with these features.

2. The image processing device recognizes an object having predetermined characteristics from one of the first image and the second image, and generates an image representing the portion of the other of the first image and the second image corresponding to the position of the object recognized in the recognition process as the display image. The system according to claim 1.

3. The aforementioned non-night vision camera is a thermographic camera that detects infrared radiation emitted by an object and generates an image. The image processing device recognizes an object having predetermined characteristics from the second image and generates the display image representing the portion of the first image corresponding to the position of the object recognized from the second image. The system according to claim 1.

4. The aforementioned non-night vision camera is a thermographic camera that detects infrared radiation emitted by an object and generates an image. The image processing device recognizes an object having predetermined characteristics from the first image, identifies the recognized object and assigns identification information to it, identifies the temperature indicated by the portion of the second image corresponding to the location of the recognized object, and generates the display image showing the identification information and the identified temperature. The system according to claim 1.

5. The aforementioned non-night vision camera is an infrared camera equipped with an infrared illumination function that emits infrared light, detects the reflected light of the infrared light, and generates an image. The image processing device generates a display image that represents an object appearing in the same position in the first image and the second image, using the shape based on the second image and the color based on the first image. The system according to claim 1.

6. The aforementioned night vision camera is a stereo night vision camera. The image processing device performs a process to determine the distance from the night vision camera to the object recognized in the recognition process based on a plurality of first images simultaneously captured by the night vision camera, and generates the display image indicating that distance. The system according to claim 1.

7. The image processing device recognizes an object having predetermined characteristics from one of the first image and the second image, and determines whether the portion of the other image corresponding to the position of the recognized object satisfies predetermined conditions. The system according to claim 1.

8. The image processing device shall notify the user if it determines that predetermined conditions are met in the determination. The system according to claim 7.

9. The image processing device includes a launcher that, when it determines that predetermined conditions are met in the determination, fires a projectile at the actual object recognized by the image processing device. The system according to claim 7.

10. The image processing device recognizes an object having predetermined characteristics from the image generated by the non-night vision camera. The night vision camera captures an object recognized by the image processing device from the image generated by the non-night vision camera and generates the first image. The system according to claim 1.

11. The non-night vision camera transmits sound waves into the water, receives the reflected sound waves, and generates the second image representing the shape of an object in the water based on the received reflected waves. The system according to claim 10.

12. The non-night vision camera transmits electromagnetic waves, receives the reflected waves of said electromagnetic waves, and generates the corresponding second image representing the shape of an object based on the received reflected waves. The image processing device maps the first image to the shape represented by the second image to generate the display image. The system according to claim 1.

13. The image processing device performs a process to correct the color of the image taken by the night vision camera at night, based on the relationship between the color of the same object recognized in both the image taken by the night vision camera during the day and the image taken by the night vision camera at night, and generates the display image using the image after the correction process. The system according to claim 1.

14. The image processing device receives the first image and the second image via a communication network. The system according to claim 1.

Images