METHOD FOR PLAYING BACK IMAGE SEQUENCES AND IMAGE PROCESSING UNIT AND COMPUTER PROGRAM THEREFOR

Description

[0001] The invention relates to a method for reproducing image sequences with flickering light information, wherein an image sequence comprises a sequence of images of an image scene captured one after the other in time according to independent claim 1.

[0002] The invention further relates to an image processing unit and a computer program for carrying out the method according to independent claims 6 and 8.

[0003] For encoding image data, it is known from ISO / IEC Standard 11172-2, "Information technology - Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mbit / s - Part 2: Video" to perform motion estimation by calculating motion vectors. This improves the efficiency of predicting PEL values. For this purpose, a motion estimator is placed in the encoder after image regrouping. In principle, to estimate motion vectors, it is checked which image blocks in consecutive images of an image sequence match each other best. To do this, the smallest error when comparing two image blocks of consecutive images is sought, and the spatial offset between the best-matching image blocks is assumed to be the motion vector. To reduce computational effort, a search space is defined. These motion estimation methods are described in detail in Chapter D.6.2 of ISO / IEC standard 11172-2.

[0004] The MPEG-4 standard ISO / IEC 14496-10: "Coding of audiovisual objects - Part 10: Advanced Video Coding" describes the coding of objects, whereby motion estimation of the luminance information is also carried out.

[0005] Other motion estimation methods are known from the fields of phase correlation, optical flow (e.g., Lucas Kanade method), and temporal-spatial recursive search (3DRS). The goal of such motion estimation methods is to estimate the movements of all image content, down to individual pixels, from one image to another in the forward and / or backward direction as accurately as possible.

[0006] US 8,922,706 B2 describes an image recording system with an image sensor, a light source, and a data processing unit. To eliminate flickering caused by changes in ambient light brightness, synchronization of the illumination from the light source with the image recording is provided. The recording of successive images occurs at intervals corresponding to the variation in ambient light brightness. This requires active illumination of the environment being recorded.

[0007] From FORBIN, G. and VLACHOS, T.: "Nonlinear Flicker Compensation for Archived Film Sequences Using Motion-Compensated Graylevel Tracing," IEEE Transactions on Circuits and Systems for Video Technology, Institute of Electrical and Electronics Engineers, Vol. 18, No. 6, June 2008, pages 803-816, a method for restoring film sequences archived in film archives is known that utilizes motion compensation. This method is used to eliminate or mitigate image defects that can be caused by uneven exposure of the film material, multiple copies of the film material, dust, or aging processes of the film material.

[0008] US 2010 / 103089 A1 discloses a method for compensating brightness fluctuations in the backlight of an LCD display.

[0009] US 2008 / 068359 A1 discloses a method for compensating for motion blur, which inherently occurs when displaying on LCD displays, plasma displays and similar displays.

[0010] When recording image sequences containing light emitted by LEDs, images alternate between containing and excluding information from the LED light source. This is due to the high frequency with which LED light sources are switched on and off. If the image recording unit is not synchronized with the light source, the captured and displayed image information may be incomplete.

[0011] This is particularly critical when an electronic camera acts as a rearview mirror in a motor vehicle, and the recorded image is displayed on a screen after image processing. In this case, a lack of synchronization between the image sequence recording on the camera and the ambient light sources can result in, for example, a vehicle's turn signal not being reflected in the surroundings. This is due to the fact that images are recorded during the turn signal's illumination phase, during the turn signal's blanking intervals, which are invisible to the human eye but still present.

[0012] However, this is just one example of the problems caused by flickering light information in connection with the use of cameras in motor vehicles. Cameras that are integrated into motor vehicles, i.e. vehicle cameras, are used in motor vehicles for a variety of tasks. For example, they serve as front cameras, rear cameras or reversing cameras or as side cameras for the driver to observe the vehicle's surroundings. However, in the form of mono cameras or stereo cameras, they function as optical sensors that are used to collect sensor data for driver assistance systems such as emergency braking assistance systems, distance and lane keeping assistance systems and other assistance systems. In connection with such cameras integrated in motor vehicles, the incomplete and / or incorrect reproduction of image sequences that contain flickering light information can lead to an increased risk of accidents.

[0013] The object of the present invention is therefore to provide an improved method for reproducing image sequences with flickering light information, in which the light information is reliably contained in the image sequence to be reproduced without synchronizing the image recording with a light source.

[0014] The object is achieved by the method having the features of claim 1 as well as by the image processing unit designed to carry out the method having the features of claim 6 and the computer program having the features of claim 8. Advantageous embodiments are described in the subclaims.

[0015] To reproduce image sequences with flickering light information, it is proposed to generate an output image by mixing the image data of the first input image with the image data of the intermediate images, whereby for the luminance pixels the corresponding output image pixels are used as the mean value of the maximum value of the intermediate images and the first input image and the median of the pixels of these images.

[0016] For the chrominance pixels, the corresponding output image pixels can be calculated as the average of the corresponding pixels from the intermediate images and the first input image.

[0017] Flickering light information means that the light source periodically changes its brightness, as is the case with the control of pulsed LED light sources.

[0018] The motion vectors for image data for a recorded first input image are estimated with respect to this image data in a temporally preceding and subsequent input image. This first input image is then blended with the image data of the temporally preceding or subsequent input image, motion-compensated using the estimated motion vectors, to generate an output image that matches the time of the first input image.

[0019] The captured input images of an image sequence are combined with motion compensation to create an output image from multiple input images. This involves performing combined forward and backward motion estimation for the temporally preceding and subsequent input images.

[0020] This achieves a good balance of brightness fluctuations between images (deflicker performance), thus reducing artifacts.

[0021] The image data for which motion vectors are estimated can, for example, be image blocks within an image. A motion vector then describes the spatial offset of an anchor point for an image block in a temporally preceding or subsequent image frame of the image sequence.

[0022] As already explained at the beginning, flickering light information in image sequences generated by vehicle cameras can pose safety risks, e.g. due to an increased risk of accidents, if the image sequences are displayed incorrectly or incompletely in the vehicle due to the flickering light information.

[0023] The method according to the invention can therefore advantageously be carried out in a motor vehicle.

[0024] In an advantageous embodiment, the method according to the invention can be carried out, in particular, by an image processing unit integrated into a motor vehicle. The method according to the invention can be carried out, in particular, by a data processing unit that is a component of the image processing unit or is connected to it. The image sequence can, in particular, be an image sequence generated by a camera integrated into the motor vehicle. The camera integrated into the motor vehicle is a component of the image processing unit or is connected to the image processing unit.

[0025] Such a camera integrated into the motor vehicle is also referred to as a vehicle camera. Such a camera integrated into the vehicle can in particular be a front camera, a rear camera, a reversing camera, a side camera or another camera integrated into the vehicle. Such a camera integrated into the vehicle can in particular be a camera that allows the driver to observe the vehicle's surroundings. Such a camera integrated into the vehicle can in particular also be a mono camera or a stereo camera, which serves as an optical sensor for determining sensor data for one or more driver assistance systems of the motor vehicle.

[0026] Such an embodiment of the method according to the invention is particularly advantageous because it can reduce or even completely eliminate the aforementioned safety risks associated with the use of cameras in motor vehicles. This is particularly important since LEDs, which cause flickering light information, are becoming increasingly widespread in the vehicle environment.

[0027] The flickering light information can therefore in particular be flickering light information caused by light emitted by light-emitting diodes.

[0028] The image processing unit has a data memory for storing image data and a data processing unit. The data processing unit is then configured to carry out the method, for example, using suitable programming or in hardware as an ASIC, FPGA, or similar. For this purpose, the image data stored in the data memory is accessed.

[0029] The image processing unit can additionally include or be connected to a camera that supplies the digital image data of an image sequence for further processing. The image processing unit can also be connected to a monitor to display the generated output images of an image sequence.

[0030] In an advantageous embodiment of the invention, the image processing unit and / or a camera which is part of the image processing unit or is connected to the image processing unit is integrated in a motor vehicle.

[0031] In this way, it is advantageously possible to use the method according to the invention in the motor vehicle and to reduce or even completely eliminate the above-mentioned safety risks in the context of the use of cameras in motor vehicles.

[0032] The invention is explained in more detail below using exemplary embodiments with the accompanying drawings. They show: Figure 1 - Sketch of a method not according to the invention for reproducing image sequences with flickering light information; Figure 2 - Sketch of a first embodiment of the method; Figure 3 - Sketch of a second embodiment of the method; Figure 4 - Block diagram of an image processing unit for carrying out the method.

[0033] Figure 1 shows a sketch of a method for reproducing image sequences with flickering light information, not according to the invention. It can be seen that an image sequence with input images E 1 , E 2 , E 3 , E 4 , ..., E n acquired sequentially at time t is present. These input images can, for example, have been captured by a digital camera and temporarily stored in a data memory.

[0034] For the purposes of the present invention, "chronologically consecutive" does not necessarily mean that two images of an image sequence must be recorded immediately one after the other. It is conceivable that intermediate images that are not further processed may also be present. Only a relatively short period of time must exist behind the images of an image sequence so that the image sequence can still be clearly reproduced with the actual image movements and image information.

[0035] From these input images E 1 , ..., E n of the image sequence, output images A 1 , A 2 , A 3 , A 4 , ..., A n are generated, which are displayed on a screen directly or after further processing.

[0036] In the first embodiment of the method, it is provided that a motion estimation ME is carried out on image information of an output image A i with i = 1, 2, 3, 4, n relative to a temporally offset input image E i+1 (or possibly also E i-1 ) of the input image sequence. Known methods of motion estimation for image information are used to determine the extent to which this information is spatially offset from the output image A i to the comparison image E i+1 . For this purpose, for example, the spatial displacement of objects detected as light information can be checked using the luminance information in order to determine L motion vectors for these light points. It is also conceivable, however, to determine motion vectors for image blocks of the images.

[0037] For motion estimation, it is checked which image information in the comparison image best matches the image information of the source image.

[0038] With the at least one determined motion vector, a motion compensation of the output image A i is then carried out in order to actually calculate a real intermediate image B i+1 or to mix this as a virtual intermediate image directly with the subsequent step of mixing this intermediate image B i+1 with the corresponding input image E i+1.

[0039] It can be seen that such a temporally offset output image A i+1 is generated by mixing the motion-compensated previous output image A i with the temporally subsequent input image E i+1.

[0040] Such a generated sequence of output images A i with i = 1 ..., n can then be directly displayed on a screen.

[0041] It is clear that through mixing, the initially still fully preserved light information L from the output image after motion estimation is mixed (interpolated) with the light information L present in the subsequent input image E i+1. The light information in the temporally subsequent output image A i+1 is then somewhat attenuated, but still present. This is particularly evident in the third output image A 3, where the attenuated light information L is shown shaded gray. The already somewhat attenuated light information L from the previous output image A 2 is mixed with the third input image E 3, in which the light information is not visible due to flickering. Nevertheless, the light information in the image sequence is always at least partially recognizable and optically visible due to the sequence of images in the image sequence with their actual movement in the image.

[0042] This provides a simple way to deflicker the flickering light information L that is partially invisible in the input images of the image sequence.

[0043] The method is fundamentally based on the principle of motion estimation between the last output image and the subsequent new input image. Motion compensation then converts the last output image into the time frame of the subsequent input image. This motion compensation can be integrated directly into the motion estimation or performed simultaneously during the blending step. However, it can also be performed as a separate step.

[0044] Motion estimation and motion compensation can be performed for whole pels or pixels, or with partial pixel accuracy (half-pel or quarter-pel). The image quality achieved by considering partial pixels in motion estimation and motion compensation is better than with full-pel methods.

[0045] Mixing can be performed by direct interpolation or filtered with weighting factors. Mixing can be the same for the luminance and chrominance pixels or, preferably, different. If the luminance pixels of the input image have a higher brightness value than the corresponding luminance pixels in the motion-compensated intermediate image B i, an output image can also contain only the luminance value of the input image without mixing in the corresponding luminance information from the motion-compensated intermediate image B i.

[0046] Otherwise, it is advantageous if the luminance image information of the motion-compensated intermediate image B i , weighted with a higher weighting factor, is mixed with the corresponding image data of the temporally offset input image E i+1 , weighted with a lower weighting factor, e.g., by addition. This results in a first-order IIR low-pass filter.

[0047] For the chrominance pixels, it is advantageous if the corresponding chrominance pixels of the motion-estimated intermediate image B i+1 are weighted and mixed with the corresponding input image E i+1. The difference in the weighting factors should be smaller here than when mixing the luminance pixels to reduce artifacts.

[0048] Figure 2shows a sketch of a first embodiment of the method for reproducing image sequences with flickering light information L. Three temporally successive input images E i-1 ; E i , E i+1 are used to generate an associated output image A i. A forward estimation is carried out from the temporally preceding input image E i-1 to the temporally subsequent input image E i and a backward estimation is carried out from the temporally subsequent input image E i+1 to the temporally preceding input image E i . With this bidirectional motion estimation, two motion vectors are determined for each image data item, one for the forward estimation FW and one for the backward estimation BW. This results in motion compensation for the temporally preceding image E i-1 with the forward motion vector FW and the negative backward motion vector BW in the time frame of the middle input image E i .In addition, the temporally subsequent image E i+1 is motion compensated using the backward motion vector BW and the negative forward motion vector FW in the time frame of the middle input image E i . These motion-compensated intermediate images L i , R i from the temporally preceding and subsequent input images E i-1 , E i+1 are then mixed with the middle input image E i to determine an output image A i from these three images.

[0049] Again, motion compensation can be performed directly alongside motion estimation or directly during the blending step, eliminating the need to generate and save a complete intermediate image (L i , R i ). These intermediate images (L i , R i ) can also be used as virtual images during the motion estimation or blending steps. Here, too, full pixels (full pel) or partial pixels (e.g., half pel, quarter pel) can be used for motion estimation and motion compensation.

[0050] Mixing is performed by filtering, where the corresponding output image pixel is used for the luminance pixels as the mean of the maximum value of the left and right intermediate images L i , R i and the input image E i , and the median of the pixels of these images. It is also conceivable that the median of the intermediate images is calculated separately for forward and backward compensation.

[0051] For the chrominance pixels, only the mean value of the corresponding pixels from the intermediate images L i , R i and the corresponding input image E i can be calculated.

[0052] Figure 3 shows a sketch of a second embodiment of the method for reproducing image sequences with flickering light information. This proposes a recursive application of motion-compensated interpolation in several stages. Three intermediate stages are shown, although a different number of intermediate stages, such as two, four, five, or more, is also conceivable. The more intermediate stages there are, the greater the complexity of the method.

[0053] In principle, an intermediate image Z1 i+0.5 is generated from two temporally successive input images E i E i+1 after a motion estimation between these input images E i , E i+1 and a motion compensation. The motion estimation is carried out as a forward estimation from the previous input image E i to the subsequent input image E i+1 and by backward estimation from the subsequent input image E i+1 to the previous input image E i . Using the motion vectors determined in this way, these input images E i , E i+1 are then transformed into the time frame of the intermediate image Z i+0.5 and the motion-compensated image information from the two input images E i , E i+1 is mixed together. Here, too, the mixing for the luminance and chrominance information can be different or, preferably, the same.In this case, averaging the image data from the intermediate images generated by forward estimation and the intermediate images generated by backward estimation is recommended. Artifacts can be further reduced by using a median filter to blend the median of this average with the corresponding image information from forward compensation and backward compensation. However, the median of the image pixels can also be used to generate the intermediate images.

[0054] However, weighted links of the corresponding image data (pixels) from the motion-compensated input images E i , E i+1 are also conceivable, with the weighting factors being 1 in the simplest case.

[0055] The intermediate images Z1 i+0.5 thus determined in the first stage are then merged again. Two temporally consecutive intermediate images Z i+0.5 and Z i+1.5 are again merged based on the previously determined motion vectors or after further motion estimation. This process is repeated in several stages to generate an output image A i.

[0056] The intermediate images Z i+k with k = 0.5, 1.5, ... can be used as initial images for display or to generate a final initial image A i. The intermediate images thus already represent quasi-initial images.

[0057] Motion estimation can be performed with full pixels (full pel) or with partial pixels.

[0058] Interpolation can be performed identically in individual stages. However, it is also conceivable to treat the even and odd stages differently, with forward and backward motion estimation and compensation occurring in the even stages, including the first stage, and forward compensation from the temporally preceding frame to the current frame and backward compensation from the temporally succeeding frame to the current frame in the odd stages. Motion estimation is then performed for the temporally intermediate frame relative to the current frame using the partial pixel method.

[0059] For all of the aforementioned methods, motion estimation is preferably performed using the luminance component of two temporally consecutive images (image frames) of an image sequence (video sequence). One of these images then serves as the reference image and the other as the search image. For the image estimation algorithm, it is not important whether the reference image is the temporally preceding image and the search image the temporally subsequent image, or vice versa. The order only needs to be maintained throughout the entire image sequence. For motion estimation, the reference image is usually divided into rectangular blocks of pixels. The block size is an essential parameter of the motion estimation method. The block size can, for example, be selected as 16x16 pixels.

[0060] Motion estimation determines a motion vector for each pixel block in the reference image. This motion vector points to a position in the search image that best matches the reference block and thus follows the direction of motion. The coordinates of the motion vectors are relative to the position of the reference block.

[0061] The motion vectors are determined from a number of candidate vectors using a search strategy. For this purpose, the absolute differences between the reference block and the search block can be calculated for the candidate vectors. If necessary, a candidate-specific factor can be added to calculate a cost factor for the candidates. The candidate vector with the lowest cost (deviation) is then selected as the optimal motion vector for the respective block.

[0062] To calculate the sum of the absolute differences of a block, a block is read from the search image at the position designated by the assumed candidate vector. Bilinear interpolation can also be used to calculate a search block from the search block image. The sum of the absolute differences between the pixels of the reference block and the search block is then calculated.

[0063] The motion vectors can be calculated by examining the blocks in a meandering order. The blocks are first examined from left to right in an image matrix row and then in the adjacent columns of the image matrix.

[0064] The candidate vectors can be determined in different ways. For example, the vectors can be used for motion vectors already identified as suitable in the vicinity. Motion vectors from the neighborhood of the current block, which were determined as optimal motion vectors for the previous frame of the image sequence, can also be assumed as candidate vectors. The motion vector field determined in the previous step must be temporarily stored for this purpose. Candidate vectors can also be determined from the motion vectors determined for neighboring blocks by adding a pseudo-random offset.

[0065] A candidate vector can also be a zero vector, where no displacement of the image block is assumed.

[0066] Figure 4shows a sketch of an image processing unit V with a data processing unit µC and a data memory MEM for storing image data. The data processing unit is, for example, a suitably programmed microprocessor, a specialized image processing processor, an FPGA (Field Programmable Gate Array), an ASIC (Application-Specific Integrated Circuit), or similar. The data processing unit µC accesses the data memory MEM for reading and writing. This can be fragmented or consist of multiple storage units.

[0067] Input image data of an image sequence are recorded by a camera K, which is directly or indirectly connected to the image processing unit V. The input image data of the image sequence can then be temporarily stored in the data memory MEM.

[0068] After processing this input data in the manner described above by the data processing unit µC, the output images are then transmitted to a screen (display D) for display. This screen D is connected directly or indirectly to the image processing unit V.

[0069] It is also conceivable that the output image data stream is stored on a suitable data carrier or transmitted via a remote data connection.

[0070] The image processing unit V is preferably integrated into a motor vehicle. The camera K and the display D replace or complement conventional rearview mirrors. The camera K is aligned with the field of view to be observed, and the display D is positioned within the driver's field of vision.

Claims

1. A method for reproducing image sequences with flickering light information (L) with a data processing unit (µC) by accessing image files stored in a data memory (MEM), wherein an image sequence comprises a sequence of input images Ei of an image scene captured one after the other in time, comprising the steps: - estimating (ME) motion vectors for image data of an input image Ei to describe the spatial displacement of an image datum in a first input image Ei to the same image datum in a temporally offset preceding third input image Ei-1 and a temporally offset subsequent second input image Ei+1, wherein a forward estimation is carried out in each case from the temporally preceding input image Ei-1; Ei to the temporally subsequent input image Ei; Ei+1 and a backward estimation is carried out in each case from the temporally subsequent input image Ei+1; Ei to the temporally preceding input image Ei; Ei-11, in order to determine two motion vectors for each image datum, once for the forward estimation FW and once for the backward estimation BW, characterized by - generating motion-compensated intermediate images Li, Ri in the time frame of the first input image Ei, wherein a left intermediate image Li is generated from the temporally preceding third input image Ei-1 with the associated motion vectors determined for the forward estimation and the negative motion vector for the backward estimation, and a right intermediate image Ri is generated from the temporally subsequent second input image Ei+1 with the associated motion vectors determined for the forward estimation and the negative motion vector for the backward estimation, - generating (MIX) an output image Ai by mixing the image data of the first input image Ei with the image data of the left and right intermediate images Li, Ri, wherein, for the luminance pixels, the corresponding output image pixels are used as the mean value of the maximum value of the intermediate images Li, Ri and the first input image Ei and the median of the pixels of these images by calculating the mean of the maximum value and the median.

2. The method according to claim 1, characterized in that, for the chrominance pixels, the corresponding output image pixels are calculated as the mean value of the corresponding pixels from the intermediate images Li, Ri and the first input image Ei.

3. The method according to claim 1 or 2, characterized in that the image data for which motion vectors are estimated are image blocks in an image.

4. The method according to any one of the preceding claims, characterized in that the method is carried out by an image processing unit which is integrated in a motor vehicle and the image sequence is an image sequence generated by a camera integrated in the motor vehicle which is a component of the image processing unit or is connected to the image processing unit.

5. The method according to any one of the preceding claims, characterized in that the flickering light information is flickering light information caused by light emitted by light-emitting diodes.

6. An image processing unit with a data memory for storing image data and with a data processing unit, characterized in that the data processing unit is set up to carry out the method according to any one of the preceding claims by accessing image data stored in the data memory.

7. The image processing unit according to claim 6, characterized in that the image processing unit and / or a camera which is a component of the image processing unit or is connected to the image processing unit is integrated in a motor vehicle.

8. A computer program with program code means which are configured to carry out the method according to any one of claims 1 to 5 when the computer program is executed by a data processing unit.