Method for authenticating image sensor

Abstract

The invention relates to a computer-implemented method (100) for verifying an image sensor, a system, a computer program product, a computer-readable storage medium and a motor vehicle. In a first step (101), a camera for a motor vehicle is provided, the camera having an image sensor. Subsequently, a first noise feature of the image sensor is determined (102) and stored (103) on a data memory. In a further step (104), a second noise characteristic of the image sensor of the camera is determined. Subsequently, the stored first noise feature is compared (105) with the determined second noise feature, and the image sensor is verified (106) if a predetermined consistency can be determined when the stored first noise feature is compared (105) with the determined second noise feature.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for verifying an image sensor, a system, a computer program product, a computer-readable storage medium, and a motor vehicle. Background Art

[0002] For the use of cameras and their image sensors on the control devices of motor vehicles, it is important that the control device can determine whether the correct camera is connected to the control device. This is especially a requirement for network security. Here, the image sensor should be verified. Basically, the risk that the controller uses an unauthorized camera, especially an unverified image sensor, should be reduced, especially minimized. Therefore, the controller should be able to confirm the identity of the camera, especially the image sensor, in order to allow the use of the image sensor.

[0003] EP 4 149 054 A1 discloses a camera system in a motor vehicle, in which it is possible to determine whether the image sensor of the camera system has been replaced by means of an encrypted common secret between the camera system of the motor vehicle and the control unit.

[0004] WO 2022 / 058345 A1 discloses a method for a camera system of a motor vehicle. Here, the camera system has a first camera module with a first identification code, a second camera module with a second identification code, and an electronic control unit. Hash values are generated based on the first identification code and the second identification code respectively.

[0005] However, as always, for example, the identification codes of cameras are copied or forged, thereby increasing the risk that unauthorized cameras, especially their image sensors, are used in the motor vehicle controller. Summary of the Invention

[0006] Therefore, the object of the present invention is to improve the verification of cameras, especially the image sensors of cameras, by means of a control device.

[0007] This object is achieved by a computer-implemented method for verifying an image sensor, a system, a computer program, a computer-readable storage medium, and a motor vehicle.

[0008] The first aspect relates to a computer-implemented method for verifying an image sensor. In a first step, a camera for a motor vehicle is provided, the camera having an image sensor. Subsequently, a first noise characteristic / distribution (Rauschprofil) of the image sensor is determined and stored in a data memory. In another step, a second noise characteristic of the image sensor of the camera (i.e., the image sensor assumed to be the same as the image sensor from which the first noise characteristic is derived) is determined. Subsequently, the stored first noise characteristic is compared with the determined second noise characteristic, and if a predetermined consistency can be determined when comparing the stored first noise characteristic with the determined second noise characteristic, the image sensor is verified.

[0009] The image sensor converts incident light into a measurable electrical signal. A digital photograph thus consists of a plurality of individual image points, the image points being so-called pixels. Each pixel corresponds to a unit on the camera sensor. The number of pixels or units is typically counted in millions of pixels (megapixels). In each unit, the amount of light incident thereon is measured by means of a voltage. This is because the incident photons are proportional to the generated voltage.

[0010] Under ideal conditions, each sensor unit generates the same signal when the amount of incident light is the same. However, this is not the case in practice: for example, the photosensitive material of the sensor may include impurities, so that only a smaller part of the light is converted in the affected pixels. Another factor is the manufacturing-determined size difference of the individual sensor units. In the individual sensor units, fluctuations in the nanometer range result in differences in pixel brightness.

[0011] Larger units receive more light and thus produce brighter pixels, and conversely, smaller units produce darker pixels. These inaccuracies are superimposed on each other, resulting in a fixed pattern in the small brightness differences occurring in the image, and thus forming a noise characteristic. This noise characteristic is generated during sensor manufacturing and hardly changes even after the camera has been operating for several years. That is to say, after several years, each new image produced by the camera will also embed this noise characteristic curve into the image content.

[0012] These two factors are particularly interesting because they occur randomly during manufacturing. Even if two image sensors are manufactured directly one after the other in the factory, these two image sensors do not have the same noise characteristic: each image sensor has its own unique pattern. The noise characteristic is usually referred to as a fingerprint because, like a human fingerprint, the noise characteristic does not change over time and is unique, so the noise characteristic can be used to identify the device.

[0013] Here, the present invention is based on the concept of establishing a fingerprint from the noise distribution of the pixels of an image sensor and writing this fingerprint into a data memory (such as the storage area of a camera, in particular an image sensor). This can be carried out during vehicle production or during maintenance during camera learning.

[0014] The noise characteristics are manifested as slight differences in adjacent pixels, but these differences are not visible to the naked eye. However, the noise characteristics can be determined by means of digital signal processing. Since the noise characteristics are too weak, several images are required to reliably extract the noise characteristics. Briefly speaking, the average value of the pixels over several images is considered. If in several images, a pixel is slightly brighter or darker than the average value of all pixels, this can be attributed to the above-mentioned manufacturing differences. Since the brightness difference is very low, a single pixel in a single image may deviate from the actual pattern. However, a stable fingerprint, i.e., the noise characteristics, is still obtained by averaging over several images.

[0015] If the image sensor is now (maliciously) replaced and disguised as the original image sensor, the noise fingerprint of the image sensor can be determined and compared with the stored noise fingerprint. If the image sensor has been replaced, the stored noise fingerprint and the determined noise fingerprint will differ from each other to a substantially pre-determined extent, and it can be inferred that there is a certain probability that the currently used image sensor has been replaced.

[0016] Therefore, hardware for an encryption solution can be saved because it can be implemented by software. No dedicated hardware unit of the camera is required, for example, no hardware unit integrated in the image sensor. It is mainly implemented in software. In relation to the camera, in particular, only a data memory is required on the image sensor, preferably in the form of an OTP (One-Time Pad, a storage area on the sensor that can be written once). Alternatively or additionally, it can also be provided that a data memory, in particular in the form of an EEPROM, is used, and the data memory is arranged on or in the camera. In addition, the noise characteristics can also be stored on a permanent memory, which is arranged on or in the control device of the motor vehicle.

[0017] The second aspect relates to a system having means for performing the method according to the first aspect.

[0018] The third aspect relates to a computer program product which, when the program is executed by a computer, causes the computer to perform the method according to the first aspect.

[0019] The fourth aspect relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to the first aspect.

[0020] The fifth aspect relates to a motor vehicle having means for carrying out the method according to the first aspect, a system according to the second aspect, a computer program according to the third aspect or a computer-readable storage medium according to the fourth aspect.

[0021] According to a preferred embodiment, the image sensor is verified in the presence of at least 60%, preferably 70%, more preferably 80%, most preferably 90% agreement. By predefining how high the agreement is when comparing the stored first noise signature with the determined second noise signature, a range can be given according to the camera type, in particular the image sensor, within which the deviation of the second noise signature from the first noise signature is tolerable, or within which it can be concluded that the same image sensor is present.

[0022] According to another preferred embodiment, the control device for the motor vehicle has a data memory. In the case where the control device has a data memory, the comparison of the two noise signatures can be carried out particularly simply. For this purpose, preferably, the control device requests the second noise signature, i.e., the control device at least partially controls the camera such that the noise signature can be determined based on the camera characteristics. In particular, it can be provided that the control device receives data from the camera, in particular the camera image sensor, and determines the noise signature in the control device based on the received data.

[0023] According to another preferred embodiment, the steps of comparison and verification are carried out by the control device. In this way, on the one hand, a computationally intensive device - as is usually the case for the control device of a motor vehicle - carries out the comparison of the noise signatures. Thus, the camera only requires fewer computing resources since no costly computing operations for comparing the noise signatures are required. In addition, based on the verification carried out by the control device, it can be achieved that the authenticity of the image sensor is confirmed by a basically trustworthy entity, namely the camera and in particular its image sensor. Thus, the possibility of tampering with the image sensor of the camera can be reduced, in particular minimized.

[0024] According to another preferred embodiment, the camera has a data memory. In this case, it is possible that, although the image sensor of the camera can be replaced. However, the data memory usually remains in the camera since it is in particular fixedly mounted or connected to the camera, thereby increasing the anti-counterfeiting security of the data memory. In addition, the noise signature can be stored particularly simply and in a targeted manner.

[0025] According to another preferred embodiment, if the step of storing the determined noise signature on the data memory of the camera is completed, the data memory of the camera cannot be overwritten. This in particular reduces the risk of the noise signature being tampered with or overwritten.

[0026] According to another preferred embodiment, the first noise feature is encrypted using a predetermined key, and the key for decrypting the encrypted first noise feature is provided to the control device. By encrypting the first noise feature, the risk of the noise feature being tampered with can be reduced, especially minimized. For example, it is thus less likely that the first noise feature is maliciously changed or adapted to, in particular, the second noise feature. Therefore, the probability of performing the verification step based on correct data can be increased.

[0027] According to another preferred embodiment, the first noise feature is stored encrypted. If the first noise feature is also stored encrypted, the probability of performing the verification step based on correct data can be additionally increased. In addition, the first noise feature is also more difficult to tamper with.

[0028] According to another preferred embodiment, the first noise feature and the second noise feature are determined based on substantially the same process. Thus, it can be substantially achieved that the first noise feature can be compared with the second noise feature particularly effectively.

[0029] According to another preferred embodiment, the first noise feature and the second noise feature are determined based on the Photo Response Non-Uniformity of the image sensor. Photo Response Non-Uniformity (PRNU) is a representation of the uniformity of the camera's response to light. Therefore, depending on the camera, the image sensor, and / or the environment around the camera, it makes sense to create noise features based on PRNU.

[0030] According to another preferred embodiment, the first noise feature and the second noise feature are determined based on the Dark Signal Non-Uniformity of the image sensor. DSNU provides a rough numerical indication of the background image quality regarding patterns or structures (which may sometimes exist). Therefore, depending on the camera, the image sensor, and / or the environment around the camera, it also makes sense to create noise features based on DSNU.

[0031] According to another preferred embodiment, the first noise feature and / or the second noise feature can be obtained based on substantially black pixels. Here, black pixels represent pixels of the image sensor that have substantially not reached ambient light or have reached too little ambient light. Based on this, it is preferably possible to create particularly distinct noise features.

[0032] For use cases or application scenarios that may arise in this method and are not explicitly described herein, it can be stipulated that an error report and / or a request for inputting user feedback and / or adjustment of standard settings and / or a predetermined initial state are output according to this method.

[0033] The invention further includes a control device for a motor vehicle. The control device may have a data processing device or a processor device, which is configured to execute an embodiment of the method according to the invention. For this purpose, the processor device may have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit) or an NPU (Neural Processing Unit) may be used as the microprocessor respectively. In addition, the processor device may have a program code, which is configured to implement an embodiment of the method according to the invention when executed by the processor device. The program code may be stored in the memory of the processor device. The processor device may be based on, for example, at least one circuit board and / or at least one SoC (System on Chip).

[0034] The invention further includes an improvement of the method according to the invention, which has the features as already described in connection with the improvement of the motor vehicle according to the invention. For this reason, the corresponding improvements of the method according to the invention are not described herein again.

[0035] The motor vehicle according to the invention is preferably designed as an automobile, in particular a passenger car or a commercial vehicle, or as a bus or a motorcycle.

[0036] As another solution, the present invention also includes a computer-readable storage medium, which includes program code that, when executed by a computer or a computer cluster, causes the computer or the computer cluster to implement an embodiment of the method according to the present invention. The storage medium can be at least partially provided as a non-volatile data memory (such as flash memory and / or SSD (solid state drive)) and / or at least partially provided as a volatile data memory (such as RAM (random access memory)). The storage medium can be arranged in a computer or a computer cluster. However, the storage medium can also operate in the Internet, for example, as a so-called Appstore server and / or a cloud server. A processor circuit having, for example, at least one microprocessor can be provided by the computer or the computer cluster. The program code can be provided as binary code and / or assembly code and / or source code of a programming language (such as the C language) and / or a program script (such as Python).

[0037] The present invention also includes combinations of the features of the described embodiments. Thus, the present invention also includes implementations that respectively have combinations of the features of multiple embodiments in the described embodiments, as long as these features are not described as mutually exclusive. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Embodiments of the present invention are described below. For this purpose, it is shown that:

[0039] Figure 1 A flowchart showing a computer-implemented method for verifying an image sensor is shown. DETAILED DESCRIPTION

[0040] The embodiments described below are preferred embodiments of the present invention. In the embodiments, the described parts of the embodiments are respectively features of the present invention that can be considered independently of each other, and these features also improve the present invention independently of each other. Therefore, the present disclosure also includes combinations of the features of the embodiments other than the shown combinations. In addition, the described embodiments can also be supplemented by other features of the present invention that have been described.

[0041] In the drawings, the same reference numerals respectively denote elements having the same functions.

[0042] Figure 1 A flowchart showing a computer-implemented method 100 for verifying an image sensor is shown.

[0043] In a first step 101, a camera for a motor vehicle is provided, where the camera has an image sensor. The camera can be configured, for example, as a camera for a driver assistance system, that is, for observing the environment of the motor vehicle or the interior space of the motor vehicle.

[0044] In another step 102, a first noise characteristic of the image sensor of the camera is determined. Here, the noise characteristic is associated with the image noise of the camera, in particular the image sensor of the camera. Here, the first noise characteristic can be created during the manufacturing process of the image sensor or the camera. Alternatively, the noise characteristic of the image sensor can also be determined at a first moment. Here, the determination of the noise characteristic can also be understood as determining, that is, invoking, the already stored noise characteristic of the image sensor.

[0045] Next, the first noise characteristic is stored 103 in a data memory. Here, the data memory can be part of the camera. This is advantageous for the case where the noise characteristic has been determined during the manufacturing process of the image sensor. Then, the determined noise characteristic can be directly stored in the data memory of the camera. Preferably, the data memory of the camera cannot be overwritten, that is, the data can only be stored in the data memory once. Thus, it can be substantially ensured that the noise characteristic of the image sensor is also stored in the data memory. Alternatively or additionally, the data memory can also be arranged in a control device.

[0046] Preferably, in order to store 103 the first noise characteristic, the first noise characteristic is encrypted with a predetermined key, and further preferably, the key for decrypting the encrypted first noise characteristic is provided to the control device.

[0047] In another step 104, a second noise characteristic of the image sensor is determined. The second noise characteristic is preferably mainly based on the same basis as for establishing the first noise characteristic, that is, using the same parameters and calculations as those also used for determining the first noise characteristic.

[0048] In particular, if there is a specific time interval between the determination of the first noise characteristic 102 and the determination of the second noise characteristic 104, a malicious replacement of the image sensor may have occurred, and such a malicious replacement is detectable. To achieve this, the second noise characteristic of the image sensor arranged in the camera and used by the camera is determined.

[0049] To determine the first noise characteristic 102 and / or to determine the second noise characteristic 104, the noise characteristic can be determined based on the dark signal non-uniformity (DSNU) of the image sensor.

[0050] Dark signal non-uniformity is a measure of the degree of time-independent fluctuations in the background of a camera image. This dark signal non-uniformity provides a rough numerical indication of the background image quality in terms of patterns or structures that may sometimes be present. In low-light imaging, the background quality of the camera can be an important factor.

[0051] When no photons are incident on the camera, the detected image typically does not indicate a pixel value of 0 gray level (ADU). Typically, there is an "offset" value, such as 100 gray levels, which indicates the positive or negative impact of the camera's noise on the measurement when there is no light. However, without careful calibration and correction, pixel-to-pixel variations can occur at this fixed offset value. This variation is called "Fixed Pattern Noise". DNSU basically represents the magnitude of this fixed pattern noise. That is, it represents the standard deviation of the pixel offset values measured in electrons.

[0052] Alternatively or additionally, in order to determine the first noise characteristic 102 and / or in order to determine the second noise characteristic 104, the noise characteristics can be determined based on the photo-response non-uniformity (PRNU) of the image sensor.

[0053] Photo-response non-uniformity is a manifestation of the uniformity of the image sensor's response to light, which is important in high-light applications. When the image sensor detects light, the number of photoelectrons captured by each pixel during the exposure is measured and stored specifically as a digital gray value (ADU). This conversion from electrons to ADU follows a specific ratio of ADU to electrons (preferably called the conversion gain) plus a fixed offset value (usually 100 ADU). These values are determined by the analog-to-digital converter and amplifier used for the conversion.

[0054] For example, a CMOS camera achieves the speed and low noise of a CMOS camera by the parallel operation of one or more analog-to-digital converters on each column of the camera and an amplifier for each pixel. This can be a possibility for representing small fluctuations in terms of offset and gain between pixels. In the case of weaker light, the fluctuations in this offset value can result in a fixed pattern noise that can be presented. PRNU represents all the fluctuations in the gain, which is the ratio of the detected electrons to the displayed ADU. PRNU represents the standard deviation of the pixel gain values. Since the difference in the resulting intensity values depends on the magnitude of the signal, it can be expressed as a percentage.

[0055] Next, the first noise characteristic and the second noise characteristic are compared with each other. Here, for example, the difference in the values can be determined by analyzing the two noise characteristics. For this purpose, the degree of consistency - especially that which can be expressed as a percentage - can be determined.

[0056] In another step 106, this percentage value is used to weigh whether the image sensor can be verified. In particular, if the consistency between the first noise feature and the second noise feature is at least 60%, the image sensor can be verified. Preferably, this value is 70% or greater, thereby increasing the likelihood that the image sensor is not replaced. To further increase the probability, the value of the consistency is greater than 80%, preferably even greater than 90%.

[0057] In summary, the example illustrates how to provide a method for verifying an image sensor.

Claims

1. A computer-implemented method (100) for validating an image sensor, comprising the steps of: - providing (101) a camera for a motor vehicle, the camera having an image sensor; - determining (102) a first noise characteristic of the image sensor; - storing (103) the first noise signature on a data memory; - determining (104) a second noise characteristic of the image sensor; - comparing the stored first noise signature with the determined second noise signature (105); If a predetermined correspondence is determined when comparing the stored first noise signature with the determined second noise signature ( 105 ), the image sensor is authenticated ( 106 ).

2. The method (100) according to claim 1, characterized in that: In case there is at least 60%, preferably 70%, more preferably 80%, most preferably at least 90% consistency, the image sensor is validated.

3. The method (100) according to any one of the preceding claims, characterized in that A control unit for a motor vehicle has a data memory.

4. The method (100) according to claim 3, characterized in that: The comparison (105) step and the verification (106) step are performed by the control device.

5. The method (100) according to claim 1 or 2, characterized in that: The camera has a data memory.

6. The method (100) according to claim 5, characterized in that: Upon completion of the step of storing (103) the determined noise characteristics on a data memory of the camera, the data memory of the camera cannot be overwritten.

7. The method (100) according to claim 1 or 2, characterized in that: The first noise feature is encrypted using a predetermined key, and the key for decrypting the encrypted first noise feature is provided to the control device.

8. The method (100) according to any one of the preceding claims, characterized in that The first noise signature is stored in an encrypted form.

9. The method (100) according to any one of the preceding claims, characterized in that The first noise characteristic and the second noise characteristic are determined based on substantially the same process.

10. The method (100) according to any one of the preceding claims, characterized in that The first noise characteristic and the second noise characteristic are determined based on the light response non-uniformity of the image sensor.

11. The method (100) according to any one of claims 1 to 9, characterized in that: The first noise characteristic and the second noise characteristic are determined based on dark signal non-uniformity of the image sensor.

12. The method (100) according to any one of the preceding claims, characterized in that The determination of the first noise characteristic (102) and / or the determination of the second noise characteristic (104) is performed based on black pixels, which indicate that no ambient light arrives or too little ambient light arrives, in particular the ambient light arrives below a predetermined threshold.

13. A system having components for carrying out the method (100) according to any one of claims 1 to 12.

14. A computer program product which, when the program is executed by a computer, causes the computer to execute the method (100) according to any one of claims 1 to 12.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method (100) according to any one of claims 1 to 12.

16. A motor vehicle (1) having means for carrying out the method (100) according to any one of claims 1 to 12, a system according to claim 13, a computer program product according to claim 14 or a computer-readable storage medium according to claim 15.

Images