Electronic devices and endoscopes

Abstract

The device includes: an offset subtraction unit; an image sensor, the image sensor receives a corresponding image signal obtained during a corresponding exposure time of the image sensor for each of a plurality of bright frames and transmits the corresponding image signal to the offset subtraction unit, and receives a corresponding image signal obtained during a corresponding exposure time of the image sensor for a dark frame and transmits the corresponding image signal to the offset subtraction unit; a control unit, the control unit ensures that the image sensor alternately transmits a plurality of bright frames and one dark frame to the offset subtraction unit, wherein the amount of light generated by the corresponding image signal for each of the bright frames is greater than the amount of light generated by the corresponding image signal for the dark frame; the offset subtraction unit obtains an offset based on the image signal of the dark frame, and subtracts the offset from the signal based on the image signal of one of the bright frames to obtain a calibration signal, the dark frame and the plurality of bright frames constituting a frame sequence that directly follows each other in time.

Description

Technical Field

[0001] The present invention relates to a method for calibrating image sensor data, in particular in conjunction with an endoscope. Background Art

[0002] Current state of technology

[0003] Optical sensors often exhibit spatial non-uniformities in signal offsets (black level, dark level). These non-uniformities can change over time and may also depend on the operating environment. These non-uniformities can originate from the sensor itself, its electronics, and / or the wiring that transmits the signal from the sensor to the analog-to-digital converter. In the latter case, the spatial structure is often independent of the sensor cells, resulting in a vertically striped pattern. In some cases, the non-uniformity is of such intensity that compensation is required. Compensating for such errors typically involves a calibration step, in which the non-uniformity is detected in a controlled environment and stored in the sensor device. This data is then used to compensate for the spatial structure of the non-uniformity during recording and image processing.

[0004] This calibration approach exhibits several disadvantages.

[0005] (1.) For production, additional steps are necessary, which leads to increased cost per sensor device.

[0006] (2.) The calibration data must be stored in the sensor device. Unless such a storage unit is provided, space is required for this purpose and the cost per unit will increase.

[0007] (3.) Calibration in a controlled environment does not take into account every possible operating environment, or every possible combination of different hardware with sensors, circuits, processing units, and corresponding terminals.

[0008] (4.) Repeating calibration at specific time intervals requires dedicated logistics and infrastructure. Summary of the Invention

[0009] The object of the present invention is to improve the prior art.

[0010] The present invention provides:

[0011] A device comprising

[0012] offset subtraction unit;

[0013] an image sensor configured to receive, for each of a plurality of bright frames, a corresponding image signal obtained by optically imaging at least a portion of a field of view of an imaging device during a corresponding exposure time of the image sensor, and to transmit the corresponding image signal to an offset subtraction unit at a first frame rate, and to receive, for at least one dark frame, a corresponding image signal obtained by optically imaging at least a portion of a field of view of the imaging device during a corresponding exposure time of the image sensor, and to transmit the corresponding image signal to the offset subtraction unit at a second frame rate;

[0014] a control unit configured to ensure that the image sensor transmits a plurality of bright frames and at least one dark frame alternately to the offset subtraction unit, wherein

[0015] the control unit being configured to ensure that, assuming that a scene of at least a portion of a field of view of the imaging device is identical in an exposure time of the at least one dark frame and in a respective exposure time of each of the light frames, a respective amount of light detected by the image sensor at the respective exposure time and generated by the respective image signal for each of the light frames is greater than a respective amount of light detected by the image sensor at the respective exposure time and generated by the respective image signal for the at least one dark frame;

[0016] an offset subtraction unit configured to obtain an offset based on an image signal of at least one dark frame, to obtain a calibration signal by subtracting the offset from a signal based on an image signal of one of the bright frames, and to provide the calibration signal for further processing,

[0017] The at least one dark frame and the plurality of bright frames constitute a frame sequence that directly follows each other in time.

[0018] In this way, at least one of the following advantages will be achieved:

[0019] No additional steps are required to produce the sensor device;

[0020] Calibration is appropriate for the operating environment and hardware; and

[0021] No logistics and / or infrastructure required for repeated calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 An image recording of a scene is shown on the left, and an image recording of a dark image according to the prior art (intensified by a factor of 10 and averaged over a number of frames) is shown on the right;

[0023] Figure 2On the left, an image recording of a scene is shown, and on the right, an image recording of a dark image (intensified by a factor of 10 and averaged over a plurality of frames) after offset compensation according to one exemplary embodiment of the present invention is shown;

[0024] Figure 3 According to one embodiment of the present invention, a recording sequence of bright and dark frames is shown;

[0025] Figure 4 The terminology of frame and exposure time is shown;

[0026] Figure 5 An apparatus is shown according to one embodiment of the present invention; and

[0027] Figure 6 An apparatus is shown which may be used in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] Although the following detailed description describes a method, it also relates to an apparatus configured to perform the method. Similarly, although the following detailed description describes an apparatus configured to perform the method, it also relates to the method itself. The method may exclude use in surgical or therapeutic applications on the human or animal body, or diagnostic procedures performed on the human or animal body. However, several apparatuses according to the present invention may be suitable for such applications or procedures.

[0029] To address the aforementioned issues, embodiments of the present invention provide a "live calibration" approach. In live calibration, frames are used to record dark images at regular intervals or due to specific events, rather than images of the corresponding scene ("light frames"). These frames ("dark frames"), or an average of several such dark frames, are used to calibrate the light frames.

[0030] Here, a “scene” is understood to be the object space recorded by the sensor (ie in particular the objects, their arrangement and the background). The term also includes the lighting of the object space, unless this is controlled by the device according to several embodiments of the invention.

[0031] Thus, there are three main variants: The first main variant of the present invention is primarily useful when the image sensor is used in a dark environment. An example of this is an endoscope, which is inserted, for example, into a cavity in the human or animal body or into a circuit. In such an environment, it is dark, so the endoscope carries its own light source for illuminating the scene. The light source can be, for example, one or more LEDs or the outlet end of a glass fiber, which is connected to a light source at the proximal end of the endoscope, which is at the end of the endoscope at its distal end. According to this main variant, during the exposure time of the dark frame, the illumination is turned off or at least reduced compared to the light frame. The sensor thus records only the dark image. The latter is used for calibration in several embodiments of the present invention.

[0032] While the first main variant requires a dark scene (the light intensity detected by the sensor during the exposure time of the dark frame should be at least substantially smaller than the light intensity detected by the sensor during the light frame), according to the second and third main variants the environment can also be brighter.

[0033] Figure 4 The terms frame (or, respectively, frame rate and inverse frame rate) and exposure time are explained. Figure 4 The term is illustrated using the example of a pixel of a sensor. If the image sensor has multiple pixels, the times of the other pixels may coincide with the time of one pixel or may be shifted relative to one pixel.

[0034] The sensor transmits its corresponding image signal once for each frame. The time interval from one transmission to the next is the inverse frame rate. Its reciprocal is the frame rate. Therefore, the exposure time of each frame must be distinguished. The exposure time is the time from the reset of the pixel to the readout of the signal from the pixel. Unless the sensor stores the readout signal immediately, it is immediately transmitted for further evaluation (e.g. A / D conversion). In this case, the time of readout coincides with the time of transmission. If intermediate storage occurs, the transmission takes place after the readout, e.g. Figure 4 shown.

[0035] In bright frames, for example, the exposure time may be only insignificantly shorter than the inverse frame rate. For example, for a frame rate of 30 images per second, it may be about 33 milliseconds. However, the exposure time may also be significantly reduced compared to the inverse frame rate. It may be half the inverse frame rate (for example, for moving pictures). In endoscopy, where motion blur should be avoided, it may be even shorter (in the range of 1 / 5 of the inverse frame rate or in the order of a few milliseconds). The minimum exposure time is a characteristic of the respective sensor.

[0036] According to a second main variant of the invention, the exposure time of the dark frame is reduced compared to the exposure time of the light frame. For example, the exposure time of the dark frame can be 2 times, preferably 5 times, even more preferably 10 times, even more preferably 50 times, and even more preferably 500 times shorter than the exposure time of the normal frame. The maximum factor can be 10000, preferably 5000, even more preferably 1000, since the amount of light detected by the sensor will become smaller. The exposure time of the dark frame can even be 0. For this purpose, assuming that the scene (including its lighting) has not changed, the amount of light detected by the sensor during the exposure time of the dark frame is clearly less than the amount of light detected by the sensor during the exposure time of the light frame. Therefore, several embodiments of the invention utilize such dark frames to calibrate non-uniformities of the sensor and / or its electronics and / or circuitry.

[0037] According to a third main variant of the present invention, when the scene remains unchanged (including the scene's illumination remains unchanged), the amount of light detected by the sensor per time unit is controlled by a variable aperture. This aperture can be, for example, a mechanical aperture. However, due to the short frame rate, a mechanical aperture may be too slow in many cases. Alternatively, for example, a stroboscope aperture (rotating at a moderate speed) or an electronic aperture can be used. Electronic apertures can be composed, for example, of parallel semiconductor light valves, as are used in LCD screens.

[0038] During the dark frame, the diaphragm aperture is smaller than during the bright frame. During the dark frame, the diaphragm is preferably closed. That is, in this case, the camera shutter is considered a special case of the diaphragm.

[0039] The first to third main variations can be arbitrarily combined with each other to improve the relative darkness of the dark frame.

[0040] According to an embodiment of the present invention, the device records at least one dark frame and then a plurality of light frames. However, it is also possible to record a plurality of dark frames or to record one dark frame and one light frame alternately. Typically, one or more dark frames are recorded periodically. In addition, or alternatively, one or more dark frames may also be recorded due to a predetermined event. For example, one or more dark frames may be recorded due to an operator input, or when the scene (in particular, its (external) lighting) is found to have changed by more than a fixed threshold, or when the multiplier of a signal amplifier, and thus the intensity of the offset, has changed.

[0041] Figure 3 An example of a sequence of frames (light frame 101 and dark frame 102) is shown according to one embodiment of the present invention. In this example, a dark frame 102 and then a specific number of light frames 101 (here, five light frames) are generated directly after each other. This sequence repeats periodically, possibly interrupted by dark frame recording due to predetermined events. The offset of subsequent light frames 101 is based at least on the signal of the preceding dark frame 102.

[0042] The dark frame is not shown during image processing and respective display, but is used only for calibration. In one example, the intensity of the dark frame is subtracted from the intensity of the subsequent normal frame.

[0043] If the image sensor comprises a plurality of pixels, calibration can be performed pixel by pixel or pixel by pixel group. Pixel by pixel means that, for each pixel, a separate offset is calculated and subtracted from the pixel value of the corresponding pixel of the bright frame. Pixel by pixel group means that the pixels are divided into a plurality of groups and the same offset is subtracted from the corresponding pixel value of each pixel of a group of bright frames. The offset can be obtained by averaging all pixels within a dark frame group. The groups can be arranged on the sensor surface in any order. For example, a column of pixels can form a group, or the pixels can be divided into a plurality of groups of n*m adjacent pixels each (e.g., 2*2 pixels, or 2*3 pixels, or 3*2 pixels, etc.) in a checkerboard pattern. In an extreme example, a group can also include all pixels of the image sensor.

[0044] In another example, multiple dark frames are recorded. These dark frames are then processed together, for example, by simple averaging or weighted averaging, where the last dark frame is typically weighted more heavily than earlier dark frames. This can help reduce noise in the calibration data. Furthermore, by comparing the various dark frames, it is possible to detect when interfering signals are present (such as, for example, changing illumination from an external light source); in such cases, the corresponding dark frame will either not be used at all or will be used only with a small weight. A moving average approach allows the calibration to automatically adapt to the respective operating environment.

[0045] In some embodiments, particularly when compensating for fringe patterns involving an image sensor comprising a plurality of pixels arranged in rows and clusters, averaging is performed on several adjacent pixels per column (or row) of the same frame. This averaging can be combined with averaging over multiple frames.

[0046] One example of an application of the present invention is an endoscope. However, the present invention is not limited to endoscopes for insertion into the human body. For example, it can also be used for endoscopes inserted into pipes. The present invention is also not limited to endoscopes but can be applied to essentially all types of image sensors.

[0047] For example, in many endoscopes the A / D converter (analog to digital converter) may be provided on the same chip as the image sensor in the endoscope tip at the distal end of the endoscope; whereas in other endoscopes it is provided at the proximal end of the endoscope. Especially in the latter case, signal interference is more significant due to the large distance of the image sensor from the A / D converter and the control unit. The main source of interference (both for endoscopes comprising an A / D converter at the distal end and for endoscopes comprising an A / D converter at the proximal end) is the clock, whose signal interferes with the image signal (resistive and / or capacitive coupling), which requires a column-dependent offset that results in a vertical stripe pattern in the image (see Figure 1 ). Figure 1 On the left, an image recording of a scene (bright frame) is shown, and on the right, an image recording of a dark image (intensified by a factor of 10 and averaged over a number of dark frames) is shown. The stripe pattern is evident both in the scene recording and in the dark image.

[0048] Therefore, according to an embodiment of the present invention, dark frames are recorded periodically, as explained above. This allows the signal offset to be estimated and then compensated during image processing, resulting in an image in which only the minimum fringe pattern is visible (see Figure 2 ). Figure 2 Corresponding to Figure 1 ,The difference is that in both the scene image on the left and the dark frame on the right, the offset is compensated by the moving average method over multiple dark frames. Figure 1 The striped pattern in Figure 2 is actually no longer visible in the image and, therefore, the recording quality of the scene is significantly improved.

[0049] In this example, a dark frame can be recorded by turning off the endoscope's light source for the duration of the dark frame's exposure. Alternatively, the dark frame's exposure time can be set to be substantially shorter than that of a normal frame. In this example, the exposure time is set to a minimum value of less than 100 μs, whereas the exposure time of a bright frame is several milliseconds. It is preferred to minimize the sensor's exposure time and turn off the light source during the dark frame.

[0050] When the sensor (the distal end of the endoscope) is located inside the human or animal body, or, for example, inside a pipeline, no external light source is present, so a dark image is considered to be recorded in the dark frame. When the endoscope is located in a position where an external light source may be present, the signal is considered dark because the exposure time of the dark frame is significantly shorter than the exposure time of the light frame; and / or, if an iris diaphragm is present, the iris aperture during the exposure time of the dark frame is substantially smaller than the iris aperture during the exposure time of the light frame.

[0051] As described above, regarding calibration, a moving average algorithm for a plurality of dark frames and / or a number of pixels within a pixel group can be used to obtain calibration data calib. In the following, an example of such an algorithm will be described, which operates on integers.

[0052] If the pixel value of the dark frame is less than the threshold, its value is added to the existing calibration data using a moving average method. The pixel value is multiplied by 2^globalShift to achieve higher accuracy. In the programming language C, the averaging algorithm is as follows:

[0053] if(pixel <threshold)

[0054] {

[0055] calib*=(1< <avgShift)-1;

[0056] calib+=(pixel< <globalShift)+(1<<(avgShift-1));

[0057] calib>>=avgShift;

[0058] }

[0059] Preferably, the value of the threshold "threshold" should be chosen to be Blacklevel + MaxFP, where Blacklevel is the black level of the A / D converter and MaxFP is the maximum value that the fringe pattern can assume. Ignoring pixel values ​​above the threshold ensures that the calibration data is such that the maximum error in the compensated frame is less than 2*MaxFP when the value exceeds the threshold.

[0060] In this example, the calibration data is calculated pixel by pixel. However, it is also possible to perform an initial smoothing of the pixel data inside each dark frame.

[0061] The calibration data is added to the image data which is transferred via USB, for example in a predetermined row (for example the first row).Before the actual image processing, the calibration data is subtracted from the image data and the first row comprising the calibration data is removed.

[0062] The present invention is not limited to vertical stripe patterns formed by the readout principles of typical CCD or CMOS sensors. It can be applied to any offset pattern. Preferably, the offset pattern changes only slowly over time, so that it is constant (or substantially constant) over multiple dark frames. For example, it should not deviate by more than 10% over ten dark frames. A deviation of less than 5% or even less than 2% is more preferred.

[0063] Typically, the frame rate is constant for all light frames and dark frames during a specific time period, and is identical for all of the light frames and dark frames. However, for embodiments of the present invention, it is not necessary that the frame rate is constant or identical for light frames and dark frames. For example, some light frames or dark frames may be transmitted at a higher frame rate, and other light frames or dark frames may be transmitted at a lower frame rate. For example, all light frames may be transmitted at a (preferably smaller) frame rate during a time period, which is different from the frame rate of all dark frames. However, it is preferred that the inverse frame rate of the light frames (i.e., the time interval between two image signal transmissions of the sensor) should be high compared to the minimum exposure time of the sensor, unless it can be ensured that the sensor is indeed in a dark environment when reading the dark frames.

[0064] Depending on the embodiment, calibration (i.e., subtracting the offset from the image data) may be performed before, during, or after A / D conversion of the sensor signal of the light frame. When calibration is performed after A / D conversion of the sensor signal of the light frame, calibration values ​​may be established from multiple dark frames before and after A / D conversion of corresponding sensor signals of the multiple dark frames, as appropriate.

[0065] According to an embodiment of the present invention, assuming that the scene recorded by the image sensor (including optional external lighting) has not changed, the corresponding amount of light detected by the image sensor and generated by the corresponding image signal for each of the light frames is greater than the corresponding amount of light detected by the image sensor and generated by the corresponding image signal for at least one dark frame. Preferably, the amount of light in the light frame will be twice as large as that in the dark frame, even more preferably 5 times as large, even more preferably 10 times as large, and even more preferably 100 times as large. The amount of light detected in the dark frame may also be 0.

[0066] Figure 5 An apparatus according to an embodiment of the present invention is shown, comprising an offset subtraction unit 10 , an image sensor 20 and a control unit 30 .

[0067] For each of the plurality of bright frames, the image sensor 20 receives a corresponding image signal obtained during a corresponding exposure time of the image sensor through optical imaging of the scene, and transmits the corresponding image signal at a first frame rate to the offset subtraction unit 10. In addition, the image sensor 20 receives a corresponding image signal obtained during a corresponding exposure time of the image sensor through optical imaging of the scene for at least one dark frame, and transmits the corresponding image signal to the offset subtraction unit 10 at a second frame rate.

[0068] The image sensor may be, for example, a CCD sensor or a CMOS sensor. The number of pixels of the sensor is not limited.

[0069] The control unit 30 ensures that the image sensor transmits a first number of light frames and at least one dark frame alternately to the offset subtraction unit 10. In addition, the control unit ensures that, assuming that the scene (including its (external) illumination) has not changed, the respective amount of light detected by the image sensor 20 during the respective exposure time and the respective image signal generated for each of the light frames is greater than the respective amount of light detected by the image sensor 20 during the respective exposure time and the respective image signal generated for the at least one dark frame. This can be achieved by the control unit, for example, by controlling the internal light source illuminating the scene, the respective exposure time, and / or the aperture size of the iris diaphragm.

[0070] The offset subtraction unit 10 receives an offset based on the image signal of at least one dark frame, subtracts the offset from the signal based on the image signal of one of the light frames to obtain a calibration signal, and provides the calibration signal for further processing. The signal can be the image signal of the light frame of the image sensor 10, or can be a signal derived from the image signal of the light frame of the image sensor 10, for example, through A / D conversion and / or other processing steps. The offset can equally be the corresponding image signal of the dark frame of the image sensor 10, or can be a signal derived from the corresponding image signal of the dark frame of the image sensor 10, for example, through A / D conversion and / or other processing steps (e.g., averaging of multiple dark frames).

[0071] The control unit and the offset subtraction unit and the possible image processing unit can be in the form of hardware and / or software. They can be implemented in one or more hardware units and / or software units. They can equally be implemented in the cloud. Each of them can also be implemented in one or more computers (e.g. laptops). Figure 6 As shown, they may even be implemented by a processor 810 comprising a memory 820 storing program instructions which, when executed by the processor 810 , ensure that the processor 810 carries out the method according to the invention.

[0072] It is assumed that, generally speaking, the greater the amount of light detected by the image sensor, the greater the amount of light incident on the image sensor. For example, the amount of light detected may be proportional to the amount of incident light, or may show gradual saturation with increasing light amount.

Claims

1. An electronic device comprising offset subtraction unit; an image sensor configured to receive, for each of a plurality of bright frames, a corresponding image signal obtained by optical imaging of at least a portion of a field of view of an imaging device during a corresponding exposure time of the image sensor, and to transmit the corresponding image signal to the offset subtraction unit at a first frame rate, and to receive, for at least one dark frame, a corresponding image signal obtained by optical imaging of the at least a portion of the field of view of the imaging device during a corresponding exposure time of the image sensor, and to transmit the corresponding image signal to the offset subtraction unit at a second frame rate; a control unit configured to ensure that the image sensor transmits a plurality of bright frames and at least one dark frame alternately to the offset subtraction unit; a lighting unit configured to illuminate a scene; wherein, the control unit being configured to ensure that, assuming that a scene of the at least a portion of the field of view of the imaging device is identical in the exposure time of the at least one dark frame and in the respective exposure time of each of the light frames, a respective amount of light detected by the image sensor at the respective exposure time and generated by the respective image signal for each of the light frames is greater than a respective amount of light detected by the image sensor at the respective exposure time and generated by the respective image signal for the at least one dark frame; the offset subtraction unit being configured to obtain an offset based on the image signal of the at least one dark frame, to obtain a calibration signal by subtracting the offset from a signal based on the image signal of one of the light frames, and to provide the calibration signal for further processing, wherein the offset subtraction unit is configured to obtain the offset by averaging the image signals of a plurality of dark frames; wherein the at least one dark frame and the plurality of bright frames constitute a sequence of frames that directly follow each other in time; The control unit is configured to control the lighting unit so that the lighting unit illuminates the scene with a first intensity during the corresponding exposure time of the bright frame, and illuminates the scene with a second intensity during the exposure time of the at least one dark frame, or the control unit is turned off, and the second intensity is less than the first intensity.

2. The electronic device according to claim 1, wherein The control unit is configured to set the exposure time of the at least one dark frame such that the exposure time of the at least one dark frame is shorter than an exposure time of each of the light frames.

3. The electronic device according to claim 1 , comprising: a variable aperture, the amount of light incident on the image sensor per time unit is set by the variable aperture, wherein The control unit is configured to control the variable iris so that the variable iris is in a corresponding first open state for each of the bright frames, and is configured to control the variable iris so that the variable iris is in a corresponding second open state for the at least one dark frame, the variable iris having a larger aperture in each of the first open states than in at least one second open state. The electronic device according to claim 3 , wherein: The variable aperture is closed in the at least one second opening state.

5. The electronic device according to claim 3, wherein: The variable aperture is opened in the at least one second opening state so that the corresponding image signal of the at least one dark frame is obtained by the optical imaging of the scene. The electronic device according to claim 1 , wherein: The control unit is configured to ensure that the sequences of frames that directly follow one another in time are repeated periodically, each of the sequences directly following in time a corresponding preceding sequence of the sequence, and The offset subtraction unit is configured to obtain a corresponding offset for each of the sequences based on the image signal of at least one dark frame of the corresponding sequence, to obtain a calibration signal by subtracting the offset from the signal based on the image signal of one of the bright frames of the corresponding sequence, and to provide the calibration signal for further processing.

7. The electronic device according to claim 1, wherein The control unit is configured to ensure that at least one dark frame occurs when a predetermined event occurs, and The offset subtraction unit is configured to obtain an offset for at least one bright frame subsequent to the dark frame based on the image signal of the at least one dark frame, the offset occurring due to a predetermined event, to subtract the offset from the signal based on the image signal of the at least one bright frame to obtain a calibration signal, and to provide the calibration signal for further processing.

8. The electronic device according to claim 1, wherein The image sensor has a plurality of pixels, and The offset subtraction unit is configured to obtain a corresponding offset for each of the pixels based on the image signal of the corresponding pixel of the at least one dark frame.

9. The electronic device according to claim 8, wherein: The pixels are divided into groups of one or more pixels, and The offset subtraction unit is configured to obtain a corresponding offset for each of the pixels in the group, and is configured to subtract the information corresponding offset of each of the pixels in the group from the signal of the corresponding pixel based on the image signal of one of the bright frames to obtain a calibration signal of the corresponding pixel.

10. The electronic device according to claim 9, wherein: The offset subtraction unit is configured to obtain the corresponding offset of each group by averaging the image signals of the pixels of the corresponding groups of the at least one dark frame.

11. An endoscope comprising the electronic device according to any one of claims 1 to 10, wherein: The image sensor is located at the tip of the endoscope at the distal end of the endoscope, and the offset subtraction unit is connected to the proximal end of the endoscope.

Images