Image generation device, image concealment system, and image concealment method
The image generation apparatus addresses the issue of FPN in imaging devices by storing and removing noise in a memory unit, reducing costs and power consumption while ensuring secure image transmission.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- MITSUBISHI ELECTRIC CORP
- Filing Date
- 2024-12-27
- Publication Date
- 2026-07-09
AI Technical Summary
Image data captured by imaging devices often includes fixed pattern noise (FPN) unique to each sensor, which interferes with accurate monitoring or surveillance, and existing solutions require costly and power-consuming circuits for noise reduction and encryption.
An image generation apparatus with a storage unit to store FPN and a noise reduction unit to remove FPN from captured image data, using a memory unit and processing circuits to restore the image data without the need for large-scale encryption circuits.
Reduces the cost and power consumption required to ensure image data confidentiality by storing FPN in a memory unit and removing it before transmission, allowing for secure and efficient image processing.
Smart Images
Figure 2026115061000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image generation device, an image anonymization system, and an image anonymization method.
Background Art
[0002] Conventionally, a configuration for encrypting and transferring image data captured by an imaging device is known. For example, Japanese Unexamined Patent Application Publication No. 2011-130402 (Patent Document 1) discloses a surveillance camera that encrypts image data captured by an imaging device and transfers it to the Internet so that the image data is not stolen by a third party on the transmission path. According to the surveillance camera, by pseudo-encrypting the image data captured by the imaging unit, the security on the transmission path between the imaging unit and the signal processing unit for the surveillance camera is ensured, and leakage prevention and privacy protection of the image data can be achieved.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] Image data (imaging data) captured by an imaging device may include fixed pattern noise (FPN) unique to each individual imaging sensor. Since image data containing FPN cannot accurately grasp the state of the target, it cannot be used for monitoring or surveillance of the target. Therefore, it is necessary to remove FPN from the image data before encryption.
[0005] In the surveillance camera described in Patent Document 1, the image sensor performs calculations related to noise reduction and encryption of image data before transmitting the image data to the internet. Thus, according to the surveillance camera described in Patent Document 1, a circuit is required to perform these calculations, which may increase the manufacturing cost and power consumption of the surveillance camera.
[0006] This disclosure was made to address the aforementioned issues, and its purpose is to reduce the costs required to ensure the confidentiality of image data. [Means for solving the problem]
[0007] The image generation apparatus according to this disclosure comprises a storage unit and a noise reduction unit. The storage unit stores fixed pattern noise of the image sensor. The noise reduction unit removes the fixed pattern noise from first image data captured by the image sensor and restores second image data. [Effects of the Invention]
[0008] According to the image generation apparatus, image concealment system, and image concealment method described herein, the cost required to ensure the confidentiality of image data can be reduced because fixed pattern noise is stored in a memory unit in advance. [Brief explanation of the drawing]
[0009] [Figure 1] This block diagram shows an example of the configuration of the image concealment system according to Embodiment 1. [Figure 2] This figure shows an example of thermal image data of a subject when FPN is not superimposed on the thermal image data in Figure 1. [Figure 3] This figure shows the FPN specific to the image sensor in Figure 1. [Figure 4] This flowchart shows an example of the image concealment process performed in an image concealment system. [Figure 5] This block diagram shows an example of the configuration of the image concealment system according to Embodiment 2. [Figure 6] Figure 5 shows three examples of variability models: a table data model, a multiple regression model corresponding to multiple regression analysis, and a predictive model corresponding to sequential optimization. [Figure 7] This flowchart shows an example of the image concealment process performed in an image concealment system. [Figure 8] This is a block diagram showing an example of the configuration of the security system according to Embodiment 3. [Modes for carrying out the invention]
[0010] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and their descriptions will not be repeated in principle.
[0011] Embodiment 1. Figure 1 is a block diagram showing an example of the configuration of the image concealment system 1 according to Embodiment 1. As shown in Figure 1, the image concealment system 1 comprises an imaging device 304, an image generation device 322, and a display device 326. The imaging device 304 includes an image sensor 308 and a transmission unit 312. The image sensor 308 receives infrared light emitted by a subject 302 and captures thermal image data 310 (first image data) of the subject 302. The image sensor 308 includes, for example, an infrared element. The thermal image data 310 has an FPN specific to the image sensor 308 superimposed on it.
[0012] Figure 2 shows an example of thermal image data 311 of a subject 302 when FPN is not superimposed on the thermal image data 310 of Figure 1. As shown in Figure 2, the part Rg1 of the subject 302 (human body) with a relatively high temperature has pixels with relatively high brightness and is displayed in white. On the other hand, the temperature of the part Rg2, which is not part of the human body, is lower than the temperature of part Rg1, so the brightness of part Rg2 is lower than that of part Rg1. As a result, part Rg2 is displayed in black. Consequently, the outline of the subject 302 can be identified in the thermal image data 311, and it is possible to determine whether or not the subject 302 is captured in the thermal image data 311.
[0013] Figure 3 shows the FPN specific to the image sensor 308 in Figure 1. The FPN has a random shape and is almost always different for each image sensor. During the generation of thermal image data 310, the FPN in Figure 3 is superimposed on Figure 2, making it impossible to identify the contour of the subject 302 in the thermal image data 310. As a result, it is impossible to determine whether or not the subject 302 is captured in the thermal image data 310.
[0014] Referring again to Figure 1, the transmitting unit 312 incorporates an ADC (Analog to Digital Converter). The thermal image data 310 is converted from an analog signal to a digital signal by the ADC, and then converted into data packets or the like with a predetermined data format. The thermal image data 310 converted in this way is transferred to the image generation device 322 via a wireless or wired telecommunications line 316 and network NW. Even if an attacker illegally infiltrates the telecommunications line 316 and intercepts the thermal image data 310, the intercepted thermal image data 310 has an FPN unique to the image sensor 308 superimposed on it, so the attacker cannot determine whether or not the subject 302 is captured in the thermal image data 310.
[0015] The image generation device 322 includes an FPN storage memory 318 (storage unit), an imaging data storage memory 320 (storage unit), and an FPN removal unit 321. The FPN storage memory 318 may be configured using a flash memory capable of writing the FPN 317 unique to the image sensor 308 only once at the time of shipment from the factory, or a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a fuse ROM (Read Only Memory). Furthermore, to ensure confidentiality, the FPN storage memory 318 is configured so that the written FPN 317 cannot be read from an external communication function such as a telecommunications line 316. On the other hand, the imaging data storage memory 320 is configured using a read-write memory such as an SRAM (Static Random Access Memory) or SDRAM (Synchronous Dynamic Random Access Memory). The imaging data storage memory 320 stores one or more frames of thermal image data 310 transferred via the telecommunications line 316.
[0016] The FPN removal unit 321 removes the FPN 317, which is specific to the image sensor 308 and stored in the FPN storage memory 318, from the thermal image data 310 stored in the imaging data storage memory 320, by a predetermined calculation (e.g., subtraction), and restores (decodes) the image data 324 (second image data) (FPN removal process). The restored image data 324 is displayed on the display device 326 connected to the image generation device 322. The FPN removal unit 321 may be implemented by a hardware circuit or processing circuit (e.g., a CPU (Central Processing Unit)) that executes an image generation program.
[0017] Even if the imaging device 304 transmits the thermal image data 310 to the image generation device 322 via a wireless or wired electrical communication line 316 without performing encryption, the fixed pattern noise (FPN) unique to the imaging element 308 is superimposed on the thermal image data 310. Thus, by deliberately not removing the FPN of the imaging element 308 in the imaging device 304, confidentiality regarding what is captured in the thermal image data 310 passing through the electrical communication line 316 is ensured, similar to the case where the thermal image data 310 is encrypted.
[0018] According to the image anonymization system 1, it is not necessary to provide a large-scale encryption circuit in the imaging device 304, nor is it necessary to provide a large-scale decryption circuit in the image generation device 322. As a result, the cost required to ensure the confidentiality of the thermal image data 310 transmitted via the electrical communication line 316 can be reduced.
[0019] FIG. 4 is a flowchart showing an example of the flow of the image anonymization process performed in the image anonymization system 1. Hereinafter, steps will be simply referred to as S. As shown in FIG. 4, the imaging device 304 captures the thermal image data 310 in S101 and transmits the thermal image data 310 to the image generation device 322 via the electrical communication line 316 in S102. The image generation device 322 removes the FPN from the thermal image data 310 to restore the image data 324 and transmits the image data 324 to the display device 326 in S103. The display device 326 displays the image data 324 in S104 and ends the process.
[0020] As described above, according to the image generation device, the image anonymization system, and the image anonymization method according to Embodiment 1, the cost required to ensure the confidentiality of an image can be reduced.
[0021] Embodiment 2. Figure 5 is a block diagram showing an example of the configuration of the image concealment system 2 according to Embodiment 2. Of the configurations shown in Figure 5, those having the same reference numerals as those shown in Figure 1 are the same as those in Embodiment 1, so the explanation of similar configurations will not be repeated. In Embodiment 2, a configuration that performs correction for temperature changes of the image sensor 308 (shutterless correction) in the FPN removal process will be described.
[0022] As shown in Figure 5, the imaging device 404 further includes a temperature sensor 402 and a temperature measuring unit 410 in addition to the transmission unit 312. The temperature sensor 402 is positioned adjacent to the image sensor 308. The temperature sensor 402 may be an existing element such as a thermistor or semiconductor temperature sensor, positioned in close contact with the outside of the image sensor 308, or it may be configured to be integrated with the image sensor 308 and positioned inside the image sensor 308. When thermal image data 310 is captured (during imaging), the temperature signal of the image sensor 308 (module temperature) obtained by the temperature sensor 402 is converted from an analog signal to a digital signal by an ADC built into the temperature measuring unit 410. This digital signal is converted to a predetermined data format as module temperature information 412 associated with the thermal image data 310, and then transferred to the image generation device 422 via the telecommunication line 316.
[0023] The image generation device 422 further includes a module temperature storage memory 423 (storage unit) and an FPN removal unit 421, in addition to the FPN storage memory 318 and the imaging data storage memory 320. The module temperature storage memory 423 stores module temperature information 412 transferred via the telecommunication line 316 for one frame or more. In the FPN removal process, the FPN removal unit 421 restores temperature-corrected image data 424 (second image data) based on the module temperature information 412 stored in the module temperature storage memory 423, using a variation model 500 corresponding to a predetermined calculation.
[0024] The module temperature storage memory 423 may be configured as the same memory area as the imaging data storage memory 320. Furthermore, since the module temperature storage memory 423 has a much smaller data capacity than the imaging data, it may be implemented as a memory-mapped register to facilitate access from the processing circuit implementing the FPN removal unit 421.
[0025] Figure 6 shows three examples of the variability model 500 from Figure 5: table data 501, a multiple regression model 502 corresponding to multiple regression analysis, and a prediction model 503 corresponding to sequential optimization. In the FPN removal process using the variability model 500, three variables are referenced: module temperature T, FPN(x,y) at the coordinates (x,y) of each pixel included in the thermal image data 310, and the pixel value (x,y) of the thermal image data 310.
[0026] As shown in Figure 6, the table data 501 includes FPN(x,y) corresponding to the module temperature T. The table data 501 is stored in the memory built into the FPN removal unit 421. When using the table data 501, it is necessary that the memory holds an amount of FPN(x,y) equal to the range of module temperatures guaranteed by the image sensor 308 divided by the resolution.
[0027] Next, we will describe the multiple regression model 502 that corresponds to the multiple regression analysis. The multiple regression model 502 is generated by performing a multivariate multiple regression analysis using FPN(x,y) data at multiple module temperatures acquired in advance before the image sensor 308 is shipped from the factory. The multiple regression model 502 predicts FPN(x,y) (dependent variable) at a desired module temperature T (explanatory variable).
[0028] The multiple regression model 502 can be generated before the image sensor 308 is shipped from the factory. Therefore, the calculations required for the FPN removal process are sum-of-products calculations based on the coefficients and intercepts of a three-variable linear equation obtained by predictions for three variables: module temperature T, and the x and y coordinates of the image data. Accordingly, the computational resources that the FPN removal unit 421 should incorporate can be limited to a multiplier and an adder that realize the sum-of-products calculation. When using the multiple regression model 502, the module temperature T measured by the temperature measurement unit 410 must be included in the module temperature range of FPN(x,y) that was acquired in advance before the image sensor 308 was shipped from the factory.
[0029] Furthermore, we will describe the prediction model 503 that corresponds to sequential optimization. Sequential optimization includes, for example, Bayesian optimization using Gaussian process regression. In Gaussian process regression, a probabilistic model or deterministic model is used to fit the FPN(x,y) at multiple module temperatures that have been acquired in advance before the image sensor 308 is shipped from the factory, and the FPN(x,y) at module temperatures other than those of the previously acquired FPN(x,y) is predicted.
[0030] Bayesian optimization is applied to improve the accuracy of the prediction of FPN(x,y) for module temperatures that fall between two previously acquired FPN(x,y) module temperatures. First, a prediction model 503 is constructed from FPN(x,y) at multiple module temperatures acquired in advance before the image sensor 308 leaves the factory. By determining search points from the expected value and variance of the prediction for the prediction model 503, recommended points for newly acquired FPN(x,y) are obtained. The prediction model 503 is updated by actually observing FPN(x,y) at these recommended points. This type of machine learning can improve the accuracy of the prediction of FPN(x,y) at the search points. The computing resources that the FPN removal unit 421 should incorporate may include a relatively high-performance floating-point arithmetic unit, a parallel computer, and a relatively large amount of memory.
[0031] According to the image anonymization system 2, by using module temperature information 412 and variation model 500 in the decoding process, noise caused by the influence of the FPN and ambient temperature specific to the imaging device 404 can be removed. Furthermore, a shutterless correction function that automatically corrects for ambient temperature during imaging can be realized.
[0032] Figure 7 is a flowchart illustrating an example of the image anonymization process performed in the image anonymization system 2. As shown in Figure 7, the imaging device 404 captures thermal image data 310 and measures the module temperature in S201, then proceeds to S202. In S202, the imaging device 404 transmits the thermal image data 310 and module temperature information 412 to the image generation device 422 via the telecommunication line 316. In S203, the image generation device 422 removes the FPN corresponding to the module temperature information 412 from the thermal image data 310 to restore the image data 424, and transmits the image data 424 to the display device 326. In S204, the display device 326 displays the image data 424 and terminates the process.
[0033] As described above, the image generation apparatus, image concealment system, and image concealment method according to Embodiment 2 can reduce the cost required to ensure the confidentiality of images.
[0034] Embodiment 3. Embodiment 3 describes an image concealment system configured as a security system or monitoring system using the imaging device 404 and image generation device 422 shown in Figure 5.
[0035] Figure 8 is a block diagram showing an example of the configuration of the image concealment system 3 according to Embodiment 3. Of the configurations shown in Figure 8, those having the same reference numerals as those shown in Figure 5 are the same as those in Embodiment 2, so the description of similar configurations will not be repeated.
[0036] As shown in Figure 8, the image concealment system 3 is configured as a security system including a security camera 601. The security camera 601 includes an imaging device 404 and a visible light camera (not shown). The security camera 601 can capture images using both visible light and infrared light. When it is difficult to capture images with the visible light camera due to strong glare during the day or darkness at night, the security camera 601 can capture the silhouette of the suspicious person 302A as thermal image data 610 (first image data) using the imaging device 404.
[0037] The thermal image data 610 is transmitted to the NVR (Network Video Recorder) 602 via the telecommunications line 316. The NVR 602 includes an image generation device 422. The image generation device 422 outputs the temperature-corrected and restored image data 624 (second image data) to the display device 326 and the storage device 603. The image data 424 is displayed on the display device 326 connected to the NVR 602 and recorded in the storage device 603. The storage device 603 is configured, for example, as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).
[0038] According to the image concealment system 3, shutterless correction can be achieved when imaging with infrared light. Therefore, compared to the case of an infrared imaging device with a shutter, it is possible to eliminate periods when imaging is impossible due to the time interval during shutter operation. As a result, a security system that enables continuous monitoring can be realized. Furthermore, compared to the case of an infrared imaging device with a shutter, a security system can be realized in which the presence of the security camera 601 is not noticed by suspicious individuals due to the sound of the shutter operating.
[0039] Next, we will explain how to ensure the confidentiality and privacy of thermal image data captured by security camera 601 if security camera 601 is modified by a malicious eavesdropper, or if a security vulnerability is intentionally created in it, and if it is replaced with a fake security camera that closely resembles security camera 601 in appearance.
[0040] The fake security camera uses a different image sensor than the security camera 601 to capture images. Therefore, the FPN required to restore the thermal image data captured by the fake security camera is different from the FPN specific to the image sensor 308 stored on the FPN storage memory 318. Since the image generation device 422 uses the FPN specific to the image sensor 308 to restore the thermal image data, the thermal image data captured by the fake security camera is not restored correctly. As a result, an image whose content cannot be determined is displayed on the display device 326.
[0041] In such a case, the administrator of the image concealment system 3 can notice an anomaly, such as a malfunction in the security camera 601, or the possibility that the security camera 601 has been replaced with a fake security camera. Furthermore, since the eavesdropper cannot obtain the FPN unique to the image sensor built into the fake security camera, they cannot recover the thermal image data captured by the fake security camera.
[0042] Next, we will explain how the image concealment system 3 can be configured as a monitoring system. By removing the visible light camera from the security camera 601 and making the security camera 601 a monitoring camera capable of imaging only with infrared, the image concealment system 3 can be configured as a monitoring system. Since this monitoring camera can achieve shutterless correction, it is possible to eliminate time periods in the time interval during which imaging is not possible, compared to the case of an infrared imaging device with a shutter. As a result, a monitoring system that enables continuous monitoring can be realized. In addition, compared to the case of an infrared imaging device with a shutter, it is possible to prevent causing discomfort to the person being monitored due to the sound of the shutter operating. Furthermore, by incorporating the image generation device 422 into the NVR 602, the viewers of the monitoring images can be limited to users who can use the FPN unique to the imaging device 404. As a result, the reliability of ensuring the confidentiality of the monitoring system and protecting privacy can be improved.
[0043] As described above, the image generation apparatus, image concealment system, and image concealment method according to Embodiment 3 can reduce the cost required to ensure the confidentiality of image data.
[0044] The various aspects of this disclosure are summarized below as an appendix. [Configuration 1] A memory unit in which the fixed pattern noise of the image sensor is stored, An image generation apparatus comprising: a noise reduction unit that removes the fixed pattern noise from first image data captured by the image sensor and restores second image data.
[0045] [Configuration 2] The storage unit further stores information regarding the temperature of the image sensor when the first image data was captured. The image generation apparatus according to configuration 1, wherein the noise reduction unit removes the fixed pattern noise corresponding to the temperature from the first image data.
[0046] [Configuration 3] The memory unit further stores a variation model that associates the temperature with the fixed pattern noise. The image generation apparatus according to configuration 2, wherein the noise reduction unit acquires the fixed pattern noise corresponding to the temperature using the variation model.
[0047] [Structure 4] The image generation device described in any one of configurations 1 to 3, The system comprises an imaging device connected to the image generation device, The imaging device is The aforementioned imaging sensor, An image concealment system comprising a transmission unit that transmits the first image data to the image generation device.
[0048] [Composition 5] The imaging device further includes a temperature measuring unit that measures the temperature of the image sensor and outputs information related to the temperature, The image concealment system according to configuration 4, wherein the transmitting unit further transmits the temperature information to the image generation device.
[0049] [Composition 6] The image concealment system according to configuration 4 or 5, further comprising a security camera including the imaging device and a visible light camera.
[0050] [Composition 7] The image concealment system according to any one of configurations 4 to 6, further comprising a display device for displaying the second image data.
[0051] Each embodiment disclosed herein is intended to be implemented in appropriate combinations, to the extent that they do not contradict each other. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope equivalent to the claims are intended. [Explanation of symbols]
[0052] 1-3 Image concealment system, 302 Subject, 302A Suspicious person, 304,404 Imaging device, 308 Image sensor, 310,311,610 Thermal image data, 312 Transmitter, 316 Telecommunication line, 317 FPN, 318 Memory for FPN storage, 320 Memory for imaging data storage, 321,421 FPN removal unit, 322,422 Image generation device, 324,424,624 Image data, 326 Display device, 402 Temperature sensor, 410 Temperature measurement unit, 412 Module temperature information, 423 Memory for module temperature storage, 500 Variation model, 501 Table data, 502 Multiple regression model, 503 Prediction model, 601 Security camera, 603 Storage device, NW Network, Rg1,Rg2 Part, T Module temperature.
Claims
1. A memory unit in which the fixed pattern noise of the image sensor is stored, An image generation apparatus comprising: a noise reduction unit that removes the fixed pattern noise from first image data captured by the image sensor and restores second image data.
2. The storage unit further stores information regarding the temperature of the image sensor when the first image data was captured. The image generation apparatus according to claim 1, wherein the noise reduction unit removes the fixed pattern noise corresponding to the temperature from the first image data.
3. The memory unit further stores a variation model that associates the temperature with the fixed pattern noise. The image generation apparatus according to claim 2, wherein the noise reduction unit acquires the fixed pattern noise corresponding to the temperature using the variation model.
4. An image generation apparatus according to any one of claims 1 to 3, The system comprises an imaging device connected to the image generation device, The imaging device is The aforementioned imaging sensor, An image concealment system comprising a transmission unit that transmits the first image data to the image generation device.
5. The imaging device further includes a temperature measuring unit that measures the temperature of the image sensor and outputs information related to the temperature, The image concealment system according to claim 4, wherein the transmitting unit further transmits information regarding the temperature to the image generating device.
6. The image concealment system according to claim 4, further comprising a security camera including the imaging device and a visible light camera.
7. The image concealment system according to claim 4, further comprising a display device for displaying the second image data.
8. The steps include: the image sensor capturing the first image data, The steps include transmitting the first image data from the imaging device including the image sensor to the image generation device according to any one of claims 1 to 3, An image concealment method comprising the step of a processing circuit included in the image generation device removing the fixed pattern noise from the first image data to restore the second image data.