Image generation device, image generation method, and image generation program

Abstract

To obtain an image which represents an object in a pseudo manner even when a camera image is in poor condition.SOLUTION: An image generation apparatus 1 includes a camera image acquisition unit 11, a sensor information acquisition unit 12, a trained model storage unit 13, and an image generation unit 10. The camera image acquisition unit 11 acquires a camera image 6 which is visual information of a subject. The sensor information acquisition unit 12 acquires sensor information 7 obtained by an object sensor detecting the subject. The trained model storage unit 13 stores a trained model 13a for generating a pseudo-camera image 14 which is a synthesized image obtained by superimposing the camera image 6 with a sensor image which is data representing the visual information of the subject in a pseudo manner, from the camera image 6 and the sensor information 7. The image generation unit 10 outputs the pseudo-camera image 14 using the trained model 13a, from the camera image 6 acquired in the camera image acquisition unit 11 and the sensor information 7 acquired in the sensor information acquisition unit 12.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device, an image generation method, and an image generation program that generate an image from a camera image and information from an object sensor. [Background technology]

[0002] In recent years, in surveillance systems for people, vehicles, etc. that use camera images, the accuracy of recognizing monitored objects decreases when visibility is poor due to rain or snow. To compensate for this, sensor fusion technology is being used to obtain advanced recognition results by combining data obtained from other sensors with camera image data.

[0003] For example, Patent Document 1 discloses a technology for improving the accuracy of object recognition by using a camera with excellent horizontal angular resolution and a millimeter-wave radar with excellent recognition accuracy in bad weather to recognize the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO2020-116194 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 identifies the position of the object being monitored by surrounding it with a frame in the camera image, so there is a problem that, for example, when the quality of the camera image is poor, there is extremely little visual information, making it difficult to determine the type and condition of the object being monitored, making effective monitoring difficult.

[0006] The present disclosure has been made to solve the above-mentioned problems, and aims to obtain an image that pseudo-represents an object even when the quality of the camera image is poor. [Means for solving the problem]

[0007] In order to solve the above-described problems, an image generation device according to the present disclosure includes a camera image acquisition unit, a sensor information acquisition unit, a trained model storage unit, and an image generation unit. The camera image acquisition unit acquires camera images, which are visual information of a subject to be photographed. The sensor information acquisition unit acquires sensor information obtained by detecting the subject to be photographed using an object sensor. The trained model storage unit stores a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and the sensor image, which is visual information that simulates the subject to be photographed, from the camera image and the sensor information. The image generation unit outputs the pseudo camera image using the trained model from the camera image acquired by the camera information acquisition unit and the sensor information acquired by the sensor information acquisition unit. The sharpness determining unit determines the sharpness of the camera image acquired by the camera image acquiring unit. The output switching unit outputs either the camera image or the pseudo camera image based on the sharpness. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to obtain an image that artificially represents an object even when the quality of the camera image is poor. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an overall diagram showing the configuration of a monitoring system according to a first embodiment. [Figure 2] 1 is a diagram illustrating a configuration of an image generating device according to a first embodiment. [Figure 3] 3 is a diagram showing a simulation of generation of a pseudo camera image according to the first embodiment. FIG. [Figure 4] FIG. 4 is a diagram illustrating the generation of a pseudo camera image according to the first embodiment, in which learning is insufficient. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of a learning unit according to the first embodiment. [Figure 6] 4 is a diagram illustrating an example of a data flow in a learning unit according to the first embodiment. FIG. [Figure 7]4 is a diagram illustrating an example of a data flow when a learning unit according to the first embodiment causes a discrimination unit to learn. FIG. [Figure 8] 4 is a diagram illustrating an example of a data flow when a learning generation unit is trained in a learning unit according to the first embodiment. FIG. [Figure 9] 4 is a flowchart showing the processing operation of the image generating device according to the first embodiment. [Figure 10] 1 is a diagram illustrating an example of a hardware configuration of an image generating device according to a first embodiment. [Figure 11] FIG. 4 is a diagram illustrating another example of the configuration of the image generating device according to the first embodiment. [Figure 12] FIG. 3 is an overall view showing another example of the configuration of the monitoring system according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating a configuration of an image generating device according to a second embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of a data flow in a learning unit according to the second embodiment. [Figure 15] 10 is a flowchart showing the processing operation of the image generating device according to the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating another example of the configuration of the image generating device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiment 1 The following describes an image generation device 1 according to Embodiment 1. Fig. 1 shows an example of a monitoring system 100 including an image generation device 1 according to Embodiment 1.

[0011] The monitoring system 100 comprises an image generating device 1, a monitoring unit 2, and a display unit 3.

[0012] The monitoring unit 2 includes a camera 4 and an object sensor 5, and outputs a camera image 6 and sensor information 7 obtained by detecting a monitoring area including subjects such as people and cars using the camera 4 and the object sensor 5 to the image generating device 1. The camera 4 and the object sensor 5 are positioned so that at least a portion of their detection ranges overlap, and the overlapping detection ranges form the monitoring area of ​​the monitoring system 100.

[0013] The camera 4 may be, for example, a digital camera or a video camera. The object sensor 5 is a non-contact sensor that uses electromagnetic waves, ultrasonic waves, or the like. In the following description, a millimeter-wave sensor is used as the object sensor 5. Millimeter-wave sensors have excellent environmental resistance, such as the ability to detect objects regardless of poor visibility due to weather or backlighting due to lighting. Therefore, by combining the millimeter-wave sensor with information from the camera image and performing sensor fusion, information on the object being photographed can be obtained even in poor environments when the camera image is captured. Another example of the object sensor 5 is LiDAR (Laser Imaging Detection and Ranging). The image generation device 1 receives the camera image 6 and sensor information 7 from the monitoring unit 2, generates a pseudo camera image 14 (described below), and outputs it to the display unit 3.

[0014] The display unit 3 displays the pseudo camera image 14 generated by the image generation device 1. The display unit 3 is, for example, a liquid crystal display or an organic EL (organic electroluminescence) display. A user of the surveillance system 100 can grasp the situation of the surveillance area by, for example, visually checking the pseudo camera image 14 displayed on the screen of the display unit 3.

[0015] The following describes the configuration of the image generation device 1. Fig. 2 is a diagram showing the configuration of the image generation device 1 according to embodiment 1. The image generation device 1 includes an image generation unit 10, a camera image acquisition unit 11, a sensor information acquisition unit 12, and a trained model storage unit 13.

[0016] The camera image acquisition unit 11 acquires a camera image 6, which is visual information obtained by capturing an image of a person, vehicle, or other object with a camera. Visual information is information presented in a form that can be visually recognized by a user of the surveillance system 100, and particularly refers to information presented as an image. By visually recognizing the visual information displayed on the display unit 3, the user can identify the shape, color, etc. of the object.

[0017] The sensor information acquisition unit 12 acquires sensor information 7 obtained by detecting a person, vehicle, or other object to be photographed using the object sensor 5. The sensor information 7 is, for example, information indicating the position of the detected object, and specifically may be information on the direction, distance, and coordinates within the detection range, or may be information obtained by processing this information into visual information and turning it into an image. Note that although the photographing by the camera 4 and the detection by the object sensor 5 are performed simultaneously, a time or spatial delay may occur as long as it does not affect the processing of the image generation device 1. The camera image 6 and the sensor information 7 are time-synchronized and positioned based on the time or spatial delay information acquired in advance.

[0018] The image generation unit 10 outputs a pseudo camera image 14, which is a pseudo representation of the object being photographed, as a composite image from the camera image 6 acquired by the camera image acquisition unit 11 and the sensor information 7 acquired by the sensor information acquisition unit 12 using a trained model 13a described below. The image generation unit 10 can output the pseudo camera image 14 even if the camera image 6 is an image in which some or all of the visual information about the object being photographed is missing. An image in which some or all of the visual information about the object being photographed is an image in which the object is unclear and it is difficult to distinguish between the object and the background due to being photographed under adverse conditions such as insufficient ambient illumination, backlight, rain, or fog, and in which it is difficult to identify the outline, color, etc. of the object from the image alone.

[0019] The trained model storage unit 13 stores a trained model 13a that has been trained in advance to output a pseudo camera image 14, which is a composite image that simulates the object being photographed, from the camera image 6 and the sensor information 7. The trained model 13a is a trained model that generates a sensor image, which is visual information that simulates the object being photographed in the shooting range of the camera image 6, from the camera image 6 and the sensor information 7. The sensor image is, for example, an image that represents the characteristics of the object, such as its outer shape, estimated from the visual information of the object remaining in the camera image 6 and the information indicating the position of the object contained in the sensor information 7. Note that although the sensor information 7 does not provide recognizable visual information such as the outer shape of the object, by combining and training the sensor information 7 with a camera image 6 that captures the same object relatively clearly, a trained model 13a that generates a sensor image that simulates the visual information of the object can be obtained. The pseudo camera image 14 is an image in which the camera image 6 and the above-mentioned sensor image are superimposed.

[0020] FIG. 3 is a diagram illustrating the generation of a pseudo camera image 14 when the subject captured in the camera image 6 is unclear due to lighting or other factors. When the camera image 6 and sensor information 7 at this time are input into the image generation device 1, the pseudo camera image 14 is generated. Object A in the camera image 6 represents the person being photographed. Multiple points B in the sensor information 7 are information detected by the object sensor 5 and represent the positions where the person being photographed is detected. The image generation device 1 uses these two pieces of information and the trained model 13a to output the pseudo camera image 14, which simulates the appearance of a person. Object C in the pseudo camera image 14 is a sensor image created from the sensor information 7 using the trained model 13a, and the pseudo camera image 14 is generated by superimposing this on the camera image 6.

[0021] While FIG. 3 shows an example in which part of the subject captured in the camera image 6 cannot be recognized, it is also possible that the subject cannot be seen or is faintly visible in the camera image 6 due to reasons such as darkness around the area. In this case, the trained model 13a needs to be trained using the camera image 6 and sensor information 7 according to each situation. Furthermore, while the pseudo camera image 14 in FIG. 3 shows a clear human figure, if the trained model 13a has not trained sufficiently or if the learning content and the situation in the camera image 6 differ, the image may have only the contours emphasized, as shown in FIG. 4.

[0022] The trained model 13a can be trained using, in particular, Generative Adversarial Networks (GAN), a well-known technology, to output a pseudo camera image 14 that allows the user to more clearly recognize the subject. The following describes the case where the trained model 13a is obtained using GAN.

[0023] 5 is a diagram showing the configuration of a learning unit 20 that obtains a trained model 13a using a GAN. The learning unit 20 includes a discrimination unit 21, a training data generation unit 22, a discriminative model storage unit 23, a generative model storage unit 24, a training data acquisition unit 25, and a degradation processing unit 26. In the present disclosure, the discrimination unit 21 trains as a "Discreminator" neural network in the GAN, and the training data generation unit 22 trains as a "Generator" neural network in the GAN.

[0024] The learning data acquisition unit 25 acquires, from an external storage unit (not shown), learning image data including a learning camera image 27, which is a camera image in which the subject is clearly captured, and learning sensor information 28, which is sensor information that detects the subject simultaneously with capturing the learning camera image 27. The learning image data is used for learning in the learning unit 20.

[0025] The degradation processor 26 degrades the image quality of the training camera image 27 included in the training image data to generate a degraded training camera image 27a. This process degrades an image of a target captured in good condition, thereby generating an image simulating an image captured under adverse conditions. The image quality degradation process may be, for example, a process of reducing brightness, a process of reducing resolution, a process of reducing contrast, or a process of adding random noise. The image quality degradation process is not particularly limited as long as it removes at least a portion of the visual information related to the target in the training camera image 27, making the target less recognizable to the user. The image quality degradation process may be performed on the entire training camera image 27 or on a region of the target. The degraded training camera image 27a and the training sensor information 28 generated by the degradation processor 26 are output to the training generation unit 22, and the training camera image 27 is output to the identification unit 21.

[0026] Identification unit 21 classifies training camera image 27, which is a relatively good camera image, as "true," and training pseudo camera image 29 output by training generation unit 22 as "false." Training generation unit 22 generates training pseudo camera image 29 from degraded training camera image 27a, which is obtained by degrading training camera image 27, and training sensor information 28.

[0027] The discriminative model storage unit 23 stores discriminative model 23a that the discrimination unit 21 uses to discriminate between training camera image 27 and training pseudo camera image 29 generated by training generation unit 22. Furthermore, the generative model storage unit 24 stores generative model 24a that is used to generate a training sensor image, which is pseudo visual information that simulates the visual information of the subject that was lost in the process of degrading the image quality of training camera image 27, from degraded training camera image 27a and training sensor information 28, and to output training pseudo camera image 29, which is a composite image obtained by superimposing the training sensor image on degraded training camera image 27a.

[0028] Next, a description will be given of a learning process in which the learning unit 20 generates the trained model 13a using GAN. In the GAN learning process, the classification unit 21, which is a "Discreminator," and the training generation unit 22, which is a "Generator," are trained competitively.

[0029] In the training of the discrimination unit 21, the discrimination model 23a is trained to improve the accuracy of discriminating between the training camera images 27 and the training pseudo camera images 29. In the following description, the training camera images 27 are assumed to be "true" and the training pseudo camera images 29 are assumed to be "false", and the discrimination model 23a is trained to discriminate whether an input image is "true" or "false". Note that the discrimination model 23a may be configured so that the training camera images 27 and the training pseudo camera images 29 are input simultaneously and the discrimination model 23a discriminates which is "true" and which is "false".

[0030] In the training of the learning generation unit 22, the learning generation unit 22 trains the generative model 24a so as to improve the accuracy of generating training pseudo camera images 29 that would be mistakenly identified as "true" by the identification unit 21.

[0031] FIG. 6 is a diagram illustrating a data flow when the learning unit 20 performs GAN learning. The learning unit 20 improves the accuracy with which the classification unit 21 distinguishes between training camera images 27, which are images in good condition, and training pseudo camera images 29, and also improves the accuracy with which the training pseudo camera images 29 output by the training generation unit 22 cause the classification unit 21 to misjudge and classify them as "correct." Image degradation is handled by a degradation processing unit 26. By having the classification unit 21 and the training generation unit 22 learn competitively in this way, the training generation unit 22 can generate a generative model 24a that outputs training pseudo camera images 29 that are closer to good camera images.

[0032] Competitive learning between the identification unit 21 and the learning generation unit 22 is performed, for example, by training either the identification model 23a or the generation model 24a depending on the judgment result of the identification unit 21 on the learning pseudo camera image 29.

[0033] FIG. 7 shows the data flow when the discrimination unit 21 determines that the training pseudo camera image 29 is "correct." If the discrimination unit 21 determines that the training pseudo camera image 29 generated by the training generation unit 22 from the degraded training camera image 27a and the training sensor information 28 is "correct," this means that the discrimination unit 21 has made an incorrect judgment, while the training generation unit 22 has successfully misidentified the discrimination unit 21. Therefore, in this case, the discrimination model 23a of the discrimination unit 21 that made the incorrect judgment is trained. For example, the training is performed by labeling the training camera image 27, which is a camera image in good condition, as "correct," and labeling the training pseudo camera image 29 output by the training generation unit 22 as "false," thereby reflecting this in the discrimination model 23a of the discrimination unit 21.

[0034] FIG. 8 shows the flow of data when the classifier 21 determines that the training pseudo camera image 29 is “false.” If the classifier 21 determines that the training pseudo camera image 29 generated by the training generator 22 from the degraded training camera image 27a and the training sensor information 28 is “false,” this means that the classifier 21 made a correct determination, while the training generator 22 failed to misidentify the image. Therefore, in this case, the generative model 24a of the training generator 22 that failed to misidentify the image is trained. For example, the training pseudo camera image 29 output by the training generator 22 is input to the classifier 21 with fixed weights, and the classifier 21 learns to generate a training pseudo camera image 29 that is closer to the training camera image 27, aiming to be determined as “correct.” The learning results are then reflected in the generative model 24a of the training generator 22.

[0035] As described above, either the discriminative model 23a or the generative model 24a is trained based on the determination of the discriminator 21 for the training pseudo camera image 29. By repeating this process, the discriminator 21 and the training generation unit 22 are trained competitively. As a result, the training generation unit 22 creates a generative model 24a from the degraded training camera image 27a and the training sensor information 28 so as to output a training pseudo camera image 29 that is closer to the training camera image 27. The image generation device 1 is configured by applying this generative model 24a to the trained model 13a. In other words, the trained generation model 24a is stored in the trained model storage unit 13 as the trained model 13a and is used as the trained model 13a in the image generation device 1.

[0036] In the above explanation, either discriminative model 23a or generative model 24a is trained based on the judgment of training pseudo camera image 29, but it is also possible to input both training camera image 27 and training pseudo camera image 29 into discrimination unit 21, have it judge which image is "correct," and train either discriminative model 23a or generative model 24a based on the result.

[0037] In the above description, the training generation unit 22 generates training pseudo camera images 29 from the degraded training camera images 27a and the training sensor information 28 to train the generative model 24a, which is a training model. However, the training camera images 27 may also be used as training data. For example, the training camera images 27 are stored in the generative model storage unit 24 as part of the generative model 24a. Hereinafter, the training camera images 27 stored as training data will be referred to as stored training camera images 27b. The training generation unit 22 generates the training pseudo camera images 29 using the degraded training camera images 27a, the training sensor information 28, and the stored training camera images 27b stored in the generative model storage unit 24. As a result, if the stored training camera images 27b similar to the captured subject exist in the generative model storage unit 24, the generative model 24a can be trained to generate clearer training pseudo camera images 29.

[0038] When training camera images 27 are used as training data, generative model 24a is trained to generate training pseudo camera images 29 using degraded training camera images 27a, training sensor information 28, and stored training camera images 27b. The generative model storage unit 24, which stores stored training camera images 27b and has stored therein generative model 24a that uses stored training camera images 27b for image generation, is reflected in the trained model storage unit 13 and is used as trained model 13a that uses stored training camera images 27b for generating pseudo camera images 14 in the image generation device 1.

[0039] Next, the processing operations in the image generating device 1 according to embodiment 1, from when the camera image acquisition unit 11 acquires the camera image 6 and when the sensor information acquisition unit 12 acquires the sensor information 7, until when the image generating unit 10 outputs the pseudo camera image 14, will be explained using the flowchart of FIG.

[0040] Because the camera 4 and the object sensor 5 acquire the camera image 6 and the sensor information 7 at predetermined intervals, the camera image acquisition unit 11 and the sensor information acquisition unit 12 repeatedly confirm the acquisition of data until the camera image 6 and the sensor information 7 acquired by the camera 4 and the object sensor 5 are input (S11: No). When the camera image acquisition unit 11 receives the camera image 6 and the sensor information acquisition unit 12 receives the sensor information 7 (S10: YES), the camera image 6 and the sensor information 7 are input to the image generation unit 10 (S11). Thereafter, the image generation unit 10 uses the trained model 13a to create a pseudo camera image 14 from the camera image 6 and the sensor information 7 (S12). The pseudo camera image 14 is then output to the display unit 3 (S13).

[0041] Next, a description will be given of an example of the hardware configuration of the image generation device 1 according to embodiment 1. Fig. 10 is a diagram showing an example of the hardware configuration of the image generation device 1. The image generation device 1 includes a computer including a processor 101 and a memory 102.

[0042] The processor 101 and the memory 102 can transmit and receive information to and from each other via, for example, a bus 103. The processor 101 executes functions such as an image generation unit 10, a learning unit 20, a learning generation unit 22, and a degradation processing unit 26 by reading and executing a program stored in the memory 102. The processor 101 is, for example, an example of a processing circuit, and includes one or more of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an LSI (Large Scale Integration), and a GPU (Graphics Processing Unit).

[0043] The memory 102 includes one or more of a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The memory 102 also includes a recording medium on which a computer-readable program is recorded. Such recording medium includes non-volatile or volatile semiconductor memory, a magnetic disk, an optical disk, and the like. The image generating device 1 may also include an integrated circuit such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0044] As described above, the image generating device 1 according to the first embodiment includes a camera image acquiring unit 11, a sensor information acquiring unit 12, a trained model storage unit 13, and an image generating unit 10. The camera image acquiring unit 11 acquires a camera image 6, which is visual information of a photographed object. The sensor information acquiring unit 12 acquires sensor information 7 obtained by detecting the photographed object using an object sensor. The trained model storage unit 13 stores a trained model 13a for generating a pseudo camera image 14, which is a composite image obtained by superimposing the camera image 6 and the sensor image, which is visual information that simulates the visual information of the photographed object, from the camera image 6 and the sensor information 7. The image generating unit 10 outputs the pseudo camera image 14 using the trained model 13a from the camera image 6 acquired by the camera image acquiring unit 11 and the sensor information 7 acquired by the sensor information acquiring unit 12. This makes it possible to obtain an image that simulates an object even when the camera image 6 is missing. In addition, by displaying a pseudo camera image 14 that simulates the visual information of the subject on the display unit 3, the user can determine the type and condition of the subject, allowing for effective monitoring.

[0045] Although the first embodiment has been described with reference to an example in which the camera image 6 contains one photographed object, the camera image 6 may contain multiple photographed objects. For example, the image generating unit 10 may determine the number of photographed objects contained in the camera image 6 using the sensor information 7, generate pseudo camera images using the camera images 6 divided so that each image contains one photographed object, and output the multiple generated pseudo camera images 14 by combining them into a single image. Furthermore, the first embodiment has been described with reference to a configuration in which the degradation processing unit 26 generates the degradation learning camera image 27a, but the learning data acquiring unit 25 may also be configured to acquire the learning camera image 27 and the degradation learning camera image 27a that are prepared in advance.

[0046] Below, we will explain a modified example of the image generating device 1 according to embodiment 1. The image generating device 1a according to the modified example differs from the image generating device 1 in that it includes a sharpness determination unit 15 that determines the sharpness of the subject captured in the camera image 6, and an output switching unit 16 that outputs either the camera image 6 or the pseudo camera image 14 to the display unit 3 based on the sharpness. Below, we will mainly explain the differences from embodiment 1.

[0047] FIG. 11 is a diagram showing the configuration of image generating device 1a. Image generating device 1a is composed of image generating unit 10, camera image acquiring unit 11, sensor information acquiring unit 12, learned model storage unit 13, sharpness determining unit 15, and output switching unit 16. Sharpness determining unit 15 calculates the sharpness of camera image 6 acquired by camera image acquiring unit 11. The sharpness may be, for example, the high-frequency components or edge components of the entire camera image 6. Alternatively, the sharpness may be the high-frequency components of a region of camera image 6 corresponding to the position where an object was detected in sensor information 7. Furthermore, whether or not the captured object can be recognized as the target, i.e., whether or not object detection was successful, may also be added to the sharpness determination.

[0048] Output switching unit 16 switches the image to be output to display unit 3 based on the clarity. If the clarity satisfies a predetermined condition, it determines that the subject captured in camera image 6 is clear, and outputs camera image 6 sent from camera image acquisition unit 11 to display unit 3. On the other hand, if the clarity does not satisfy the predetermined condition, it determines that the subject captured in camera image 6 is unclear, and outputs pseudo camera image 14 generated by image generation unit 10 to display unit 3. As a result, in surveillance system 100, if the subject is not clearly captured in camera image 6, pseudo camera image 14, which is a pseudo composite of the subject, is displayed on display unit 3, and if the subject is clearly captured, camera image 6 can be displayed as is on display unit 3. As a result, it is possible to display either camera image 6 or pseudo camera image 14, whichever image makes it easier to recognize the subject.

[0049] 12, the sharpness determination unit 15 and the output switching unit 16 may be configured to be provided outside the image generating device 1, and when the sharpness determination unit 15 determines that the subject captured in the camera image 6 is sharp, the sharpness determination unit 15 outputs the camera image 6 and the sensor information 7 to the image generating unit 10, and when it determines that the subject is not sharp, the sharpness determination unit 15 outputs the camera image 6 directly to the display unit 3. This makes it possible to determine whether or not to output the pseudo camera image 14 depending on the sharpness of the subject, without changing the design of the image generating device 1.

[0050] Embodiment 2 The image generation device according to the second embodiment differs from the image generation device 1 according to the first embodiment in that the image generation device includes a learning unit that learns a trained model. In the following, components having the same functions as those in the first embodiment are denoted by the same reference numerals and their explanations are omitted, and the explanation will focus on the differences from the image generation device 1 according to the first embodiment.

[0051] The configuration of an image generating device 1b according to the second embodiment will be described. FIG. 13 is a diagram showing the configuration of the image generating device 1b according to the second embodiment. The image generating device 1b includes a learning unit 20a that trains a trained model 13a used by the image generating unit 10 to create a pseudo camera image 14. As a result, when the state of the camera image 6 is poor, the image generating device 1b can use visual information immediately before the state of the image deteriorates as a training model, thereby achieving the effect of being able to generate a pseudo camera image 14 that is closer to the actual captured object. Furthermore, the image generating device 1b outputs the pseudo camera image 14 while training the trained model 13a in real time, thereby achieving the effect of being able to perform efficient learning suited to the environment.

[0052] The learning unit 20a included in the image generating device 1b in the second embodiment differs from the learning unit 20 that generates the trained model 13a in the first embodiment in that the learning data acquiring unit 25 acquires the camera images 6 acquired by the camera image acquiring unit 11 and the sensor information 7 acquired by the sensor information acquiring unit 12 in the image generating device 1b, and uses these for learning. The learning unit 20a may also be configured to apply a model trained in advance by the learning unit 20 according to the first embodiment or the like, and to additionally train a part or all of the model.

[0053] As described above, the acquired camera image 6 and sensor information 7 are used to output the pseudo camera image 14 in the image generation unit 10, and at the same time, the learned model 13a is cumulatively learned by the learning unit 20a. However, since the camera images 6 used for learning need to be clear, not all acquired camera images 6 and sensor information 7 are learned, but for example, the learning unit 20a trains the learned model 13a only when a camera image 6 that is deemed to be sufficiently clear has been acquired.

[0054] Whether or not a camera image 6 that is deemed sufficiently clear is used for learning may be determined by, for example, setting a threshold for parameters such as image brightness, high frequency components, or edge components, or by switching depending on the time of day. Alternatively, a method in which the learning state and the non-learning state are switched at the discretion of the operator may also be used. Furthermore, whether or not the photographed subject can be recognized as the relevant subject, i.e., whether or not object detection is successful, may also be added to the determination of clarity.

[0055] Similarly, learning using GAN is effective in the second embodiment. FIG. 14 is a diagram showing the flow of data in learning by the learning unit 20a when GAN is used. The identification unit 21 judges the camera image 6 acquired by the camera image acquisition unit 11 as "true" and the training pseudo camera image 29 output by the training generation unit 22 as "false." The training generation unit 22 also receives as input a degraded camera image 6a obtained by degrading the camera image 6 acquired by the camera image acquisition unit 11 and sensor information 7 acquired by the sensor information acquisition unit 12, and generates the training pseudo camera image 29 using the generative model 24a. The learning method is the same as that of the learning unit 20 according to the first embodiment.

[0056] Furthermore, the learning unit 20a is more effective in using the camera image 6 as learning data in addition to the degraded camera image 6a and the sensor information 7 than in the first embodiment. This is because, when the condition of the camera image 6 of the subject deteriorates, the camera image 6 taken immediately before that and in which the subject is clearly visible can be used to generate the pseudo camera image 14.

[0057] The processing of the learning unit 20a when using a camera image 6 as training data will be described. A clear camera image 6 and sensor information 7 are acquired by the training data acquisition unit 25, and a degraded camera image 6a is generated by the degradation processing unit 26. The degraded camera image 6a and sensor information 7 are then input to the training data generation unit 22, which further stores the camera image 6 in the generative model storage unit 24 of the training data generation unit 22 as training data. Hereinafter, the camera image 6 stored as training data will be referred to as a stored camera image 6b. The training data generation unit 22 generates a training pseudo camera image 29 using the degraded camera image 6a, the sensor information 7, and the stored camera image 6b and generative model 24a stored in the generative model storage unit 24. This makes it possible to generate a clearer training pseudo camera image 29 based on the past camera image 6.

[0058] Using camera image 6 as learning data is particularly effective when stored camera image 6b that captures the same subject as the subject captured in camera image 6 acquired by camera image acquisition unit 11 exists in generative model storage unit 24. This is because the generative model storage unit 24 that stores camera image 6 that captured the same subject is reflected in trained model storage unit 13, and camera image 6 that captures the same subject can be used to generate pseudo camera image 14. As a result, if the condition of camera image 6 suddenly deteriorates, information on camera image 6 immediately before the image condition deteriorates is stored in trained model storage unit 13, and this can be used to generate pseudo camera image 14, making it possible to generate pseudo camera image 14 that is closer to the actual subject.

[0059] Alternatively, the pseudo camera image 14 may be generated using the corresponding stored camera image 6b only when the trained model storage unit 13 contains a stored camera image 6b that depicts the same subject as the subject in the shooting range of the camera image 6 acquired by the camera image acquisition unit 11. In this case, for example, during learning by the learning generation unit 22, the camera image 6 and the sensor information 7 acquired at the same time are associated and stored in the generative model storage unit 24. Hereinafter, the sensor information 7 stored in the generative model storage unit 24 will be referred to as stored sensor information 7b. The image generation unit 10 generates the pseudo camera image 14 using the stored camera image 6b when the trained model storage unit 13 contains a stored camera image 6b depicting the same subject, taking into account the positional continuity, shooting time, etc. between the stored sensor information 7b and the sensor information 7 acquired by the sensor information acquisition unit 12.

[0060] However, when generating a pseudo camera image 14 using the corresponding stored camera image 6b only if the learned model memory unit 13 contains a stored camera image 6b that captures the same subject as the subject in the shooting range of the camera image 6 acquired by the camera image acquisition unit 11, when training the learning generation unit 22, it is necessary to train the learning generation unit 22 to both generate a training pseudo camera image 29 using the stored camera image 6b and generate a training pseudo camera image 29 without using the stored camera image 6b.

[0061] Next, in the image generating device 1b according to embodiment 2, the processing operations from when the camera image acquiring unit 11 acquires the camera image 6 and when the sensor information acquiring unit 12 acquires the sensor information 7, to when the learning unit 20a reflects the learning in the trained model 13a, and further until the image generating unit 10 outputs the pseudo camera image 14 will be explained using the flowchart of FIG.

[0062] Because the camera 4 and the object sensor 5 acquire the camera image 6 and the sensor information 7 at predetermined intervals, the camera image acquisition unit 11 and the sensor information acquisition unit 12 repeatedly confirm data acquisition until the camera image 6 and the sensor information 7 acquired by the camera 4 and the object sensor 5 are input (S20: NO). When the camera image acquisition unit 11 receives the camera image 6 and the sensor information acquisition unit 12 receives the sensor information 7 (S20: YES), the camera image 6 and the sensor information 7 are input to the image generation unit 10 (S21). After receiving the camera image 6 and the sensor information 7, the image generation unit 10 creates a pseudo camera image 14 using the trained model 13a (S22). Thereafter, the pseudo camera image 14 is output to the display unit 3 (S23).

[0063] The learning unit 20a also determines whether the acquired camera image 6 is clear (S30). If it is determined that the camera image 6 is clear (S30: YES), the learning unit 20a trains a model using the camera image 6 and the sensor information 7 (S31). If a GAN is used for training, the recognition unit 21 and the training generation unit 22 are trained here. Next, the training results are reflected in the trained model 13a (S32), and training ends. If a GAN is used for training, the trained generative model 24a of the training generation unit 22 is reflected in the trained model 13a. Note that if it is determined that the camera image 6 is not clear, the learning unit 20a simply ends training without training (S30: NO).

[0064] Here, we have explained that the image generation unit 10 creates the pseudo camera image 14 and then the learning unit 20a performs the learning process, but the learning unit 20a may perform the learning first and then the image generation unit 10 creates the pseudo camera image 14, or learning and image generation may be performed simultaneously.

[0065] An example of the hardware configuration of image generation device 1b according to embodiment 2 is the same as the hardware configuration of image generation device 1 shown in the figure. Processor 101 can execute the functions of image generation unit 10 and learning unit 20a by reading and executing a program stored in memory 102.

[0066] As described above, the image generating device 1b according to the second embodiment further includes a learning unit 20a that trains the trained model 13a in addition to the configuration of the image generating device 1 according to the first embodiment. As a result, when the condition of the camera image 6 suddenly deteriorates due to a malfunction of the surrounding lighting or the like, the device can use training data obtained by training the camera image 6 of the subject captured in good condition immediately before the malfunction, thereby achieving the effect of outputting a pseudo camera image 14 that is closer to the real thing. Furthermore, the image generating device 1a can train the trained model 13a in real time while operating the image generating unit 10. As a result, the device can learn information about people and vehicles in the environment of the monitored location, thereby achieving the effect of outputting a pseudo camera image 14 that has information closer to the camera image 6 captured in good condition.

[0067] The following describes a first modification of the image generating device 1b according to the second embodiment. The image generating device 1c according to the first modification is different from the image generating device 1b in that it includes a sharpness determination unit 15, an output switching unit 16, and an input switching unit 17 that inputs the camera image 6 and the sensor information 7 to either the learning unit 20a or the image generating unit 10 based on the sharpness. The following mainly describes the differences from the second embodiment. FIG. 11 is a diagram showing the configuration of the image generating device 1a. The image generating device 1a is composed of an image generating unit 10, a camera image acquiring unit 11, a sensor information acquiring unit 12, a trained model storage unit 13, a sharpness determination unit 15, and an output switching unit 16.

[0068] 16 is a diagram showing the configuration of image generation device 1c. Image generation device 1c is composed of image generation unit 10, camera image acquisition unit 11, sensor information acquisition unit 12, trained model storage unit 13, sharpness determination unit 15, output switching unit 16, input switching unit 17, and learning unit 20a. Camera image acquisition unit 11 outputs camera image 6 to the sharpness determination unit, input switching unit 17, and output switching unit 16, and sensor information acquisition unit 12 inputs sensor information 7 to input switching unit 17.

[0069] The input switching unit 17 switches the object to which the camera image 6 and the sensor information 7 are input based on the sharpness. If the sharpness satisfies a predetermined condition, it determines that the object captured in the camera image 6 is sharp, and inputs the camera image 6 and the sensor information 7 to the learning unit 20a. At this time, the learning unit 20a trains the trained model 13a. Furthermore, since the camera image 6 and the sensor information 7 are not input to the image generation unit 10, the pseudo camera image 14 is not generated. On the other hand, if the sharpness does not satisfy the predetermined condition, it determines that the object captured in the camera image 6 is unclear, and inputs the camera image 6 and the sensor information 7 to the image generation unit 10. At this time, the image generation unit 10 generates the pseudo camera image 14. Furthermore, since the camera image 6 and the sensor information 7 are not input to the learning unit 20a, the trained model 13a is not trained.

[0070] Output switching unit 16 is the same as in the modification according to embodiment 1. As a result, in monitoring system 100, when the subject is not clearly captured in camera image 6, pseudo camera image 14, which is a synthetic image of the subject, is displayed on display unit 3, and when the subject is clearly captured, camera image 6 is displayed as is on display unit 3, while trained model 13a can be trained using the clear camera image 6. As a result, it is possible to display the image between camera image 6 and pseudo camera image 14, whichever is easier to recognize the subject, and simultaneously train trained model 13a efficiently.

[0071] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention. [Explanation of symbols]

[0072] 1, 1a, 1b, 1c image generation device, 2 monitoring unit, 3 display unit, 4 camera, 5 object sensor, 6 camera image, 6a degraded camera image, 6b stored camera image, 7 sensor information, 7b stored sensor information, 10 image generation unit, 11 camera image acquisition unit, 12 sensor information acquisition unit, 13 trained model storage unit, 13a trained model, 14 pseudo camera image, 15 sharpness determination unit, 16 output switching unit, 17 input switching unit, 20, 20a learning unit, 21 recognition unit, 22 training generation unit, 23 discrimination model storage unit, 23a discrimination model, 24 generative model storage unit, 24a generative model, 25 training data acquisition unit, 26 degradation processing unit, 27 training camera image, 27a degraded training camera image, 27b stored training camera image, 28 training sensor information, 29 Pseudo camera image for training,100,100a surveillance system,101 processor,102 memory,103 bus.

Claims

1. a camera image acquisition unit that acquires a camera image, which is visual information of a subject; a sensor information acquisition unit that acquires sensor information obtained by detecting the object to be photographed using an object sensor; a trained model storage unit that stores a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and a sensor image that is visual information that simulates visual information of the subject, from the camera image and the sensor information; and an image generation unit that outputs the pseudo camera image using the trained model from the camera image acquired by the camera image acquisition unit and the sensor information acquired by the sensor information acquisition unit; a sharpness determination unit that determines the sharpness of the camera image acquired by the camera image acquisition unit; an output switching unit that outputs either the camera image or the pseudo camera image based on the sharpness; An image generating device comprising:

2. The trained model storage unit trains an identification model, which is a learning model of the identification unit, so that the identification unit, which identifies the training camera image, which is an image in which the subject is clearly captured, as positive and the pseudo camera image, as false, improves the accuracy of the identification, and trains a generative model, which is a learning model of the training generation unit, so that the training generation unit, which is a generator for learning that outputs the pseudo camera image from the camera image and the sensor information, generates the pseudo camera image that misleads the identification unit, and stores the trained generative model as the trained model in a generative adversarial network.

2. The image generating device according to claim 1.

3. The learning generation unit generates the pseudo camera image from a degraded learning camera image obtained by processing the learning camera image so that at least a part of visual information of the photographed object included in the learning camera image is missing, and the sensor information.

3. The image generating device according to claim 2.

4. The image processing device further includes a learning unit that acquires the camera image acquired by the camera image acquisition unit and the sensor information acquired by the sensor information acquisition unit as learning data, and uses the learning data to train the trained model for generating the pseudo camera image.

2. The image generating device according to claim 1.

5. The learning unit an identification unit that identifies the camera images acquired as the learning data as genuine and the pseudo camera images as fake; a learning generation unit that is a learning generator that outputs the pseudo camera image from the camera image acquired as the learning data and the sensor information; A learning device using a generative adversarial network comprising: the identification unit trains an identification model, which is a learning model of the identification unit, so as to improve the accuracy of the identification; the generation unit for learning trains a generation model that is a learning model of the generation unit for learning so as to generate the pseudo camera image that causes the identification unit to misjudge the identification; The trained generative model is stored in the trained model storage unit as the trained model.

5. The image generating device according to claim 4.

6. The learning generation unit generates the pseudo camera image from the sensor information and a degraded camera image obtained by processing the camera image acquired as the learning data so that at least a part of visual information of the subject included in the camera image is lost.

6. The image generating device according to claim 5.

7. the learning unit stores the camera images acquired as the learning data in a generative model storage unit that stores the generative model as camera images for image generation; The learning generation unit trains the generative model to generate the pseudo camera image from the learning data and the camera image for image generation.

7. The image generating device according to claim 5 or 6.

8. The image processing device further includes an input switching unit that inputs the camera image and the sensor information to either the image generating unit or the learning unit based on the sharpness, the image generation unit generates the pseudo camera image when the input switching unit inputs the camera image and the sensor information to the image generation unit; the learning unit causes the trained model to learn when the input switching unit inputs the camera image and the sensor information to the learning unit; The output switching unit outputs a pseudo camera image when the input switching unit inputs the camera image and the sensor information to the image generation unit, and outputs the camera image when the input switching unit inputs the camera image and the sensor information to the learning unit.

8. An image generating device according to claim 4, wherein the image generating device is a digital camera.

9. The sensor information acquisition unit acquires millimeter wave sensor information obtained by detecting the object to be photographed using a millimeter wave sensor.

9. An image generating device according to claim 1, wherein the image generating device is a digital camera.

10. A computer-implemented image generation method comprising: a camera image acquisition step of acquiring a camera image which is visual information of the subject; a sensor information acquisition step of acquiring sensor information obtained by detecting the object to be photographed using an object sensor; a step of outputting the pseudo camera image from the camera image acquired in the camera image acquiring step and the sensor information acquired in the sensor information acquiring step, using a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and a sensor image, which is data that simulates visual information of the subject, from the camera image and the sensor information; a clarity determination step of determining clarity of the camera image acquired in the camera image acquisition step; an output switching step of outputting either the camera image or the pseudo camera image based on the sharpness; An image generating method comprising:

11. a camera image acquisition step of acquiring a camera image which is visual information of the subject; a sensor information acquisition step of acquiring sensor information obtained by detecting the object to be photographed using an object sensor; a step of outputting the pseudo camera image from the camera image acquired in the camera image acquiring step and the sensor information acquired in the sensor information acquiring step, using a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and a sensor image, which is data that simulates visual information of the subject, from the camera image and the sensor information; a clarity determination step of determining clarity of the camera image acquired in the camera image acquisition step; an output switching step of outputting either the camera image or the pseudo camera image based on the sharpness; An image generating program that causes a computer to execute the above.

Images