Device for image frame detection, frozen image frame detection, frozen image frame detection systems, method and vehicle

Abstract

Device (1, 1') for image frame detection, the device (1, 1') comprising: at least one image sensor (3, 3') designed to capture at least one image frame of an overall optical signal illuminating at least one area of ​​the image sensor (3, 3'), wherein the overall optical signal comprises at least a first optical signal comprising an optical signal of a scene to be captured, and at least one second optical signal; and at least one optical pattern signal generator (19, 19') designed to generate at least the second optical signal, wherein the second optical signal comprises an optical pattern signal, wherein the at least one region of the image sensor (3, 3') comprises at least a first region (5, 5') and at least a second region (7, 7'), wherein • the at least one first region (5, 5') of the image sensor (3, 3') is illuminated by the first optical signal, wherein the first region (5, 5') of the image sensor (3, 3') is designed to detect at least light of the visible, infrared and / or ultraviolet light spectrum, • the at least one second region (7, 7') of the image sensor (3, 3') is illuminated by the second optical signal, wherein the second region (7, 7) of the image sensor (3, 3') is designed to at least partially detect light of the visible, infrared and / or ultraviolet light spectrum, and • the first region (5, 5') and the second region (7, 7') are arranged relative to one another in such a way that the second region (7, 7') lies completely outside the first region (5, 5') wherein the pattern signal (i) is dynamic or changes over time, (ii) comprises, encodes, represents and / or displays at least one predefined piece of information and / or at least one visual pattern, (iii) at least partially comprises light of the visible, infrared and / or ultraviolet light spectrum, and (iv) changes synchronously with the frame rate of the image sensor (3, 3').

Description

[0001] The present invention relates to a device for image frame detection, a system comprising such a device for image frame detection and a device for providing detection of frozen image frames within at least one video stream, a method and a vehicle, at least comprising such a device for image frame detection.

[0002] Advanced mirror replacement systems require the system to operate in a known safety state. With regard to the safety of the vehicle and its occupants, a phenomenon known as "frozen frame" is of particular concern and is cataloged as a serious hazard. A frozen frame can occur, for example, in situations where delays occur along the signal chain from the acquisition unit to the display unit of the mirror replacement system. This can be caused by the presence of buffer memories along the signal chain or other system-dependent latencies. To avoid dangerous situations associated with outdated frame information, frozen frames must be reliably and fail-safely detected.

[0003] Various systems are currently known for detecting normal video operation.

[0004] Document US 2017 / 0339375 A1 relates to a rearview video system for a vehicle, including a display and a rearview camera that generates a raw video signal. A video modification module receives the raw video signal and generates a modified video signal based on it. A video analysis module analyzes whether the modified video signal is valid or invalid. A switch outputs the modified video signal to the display when the vehicle is reversing and the modified video signal is valid, and outputs the raw video signal to the display when the vehicle is reversing and the modified video signal is invalid.

[0005] Document US 2016 / 0039341 A1 relates to a vehicle display that displays camera video with an indicator. The indicator is static on the display while the camera video is frozen on the display, informing a viewer that the camera video is frozen, and the indicator is dynamic on the display while the camera video is live on the display.

[0006] From the generic document DE 10 2016 202 620 A1, a device for monitoring a spatial area, in particular in the surroundings or within a vehicle, preferably a rail vehicle, is known. The device comprises a detection device for detecting the spatial area, which is designed to generate a series of images, wherein images of the image series are each representative of the spatial area at a time of detection, a display device for displaying the generated image series, and a dynamic means designed to generate a visually perceptible temporal change. The detection device is designed to interact with the dynamic means such that the images each contain a state of temporal change. Furthermore, a control unit is provided, which is designed to control the temporal change by means of an irregular change pattern.

[0007] DE 10 2014 213 456 A1 relates to a system for transmitting a signal containing optical information from a receiving device to an output device and for monitoring the transmission of the signal. For this purpose, the system comprises a receiving device, a signal transmission device, and an output device. For monitoring the signal transmission, the system comprises a signal monitoring device, which in turn comprises a reference signal generating device for generating a reference signal in the signal generated by the receiving device. Furthermore, the signal monitoring device comprises a detection device for detecting at least part of a signal transmitted to the output device and / or at least part of the signal output by the output device. Furthermore, a monitoring device of the signal monitoring device is designed to monitor whether the signal has been detected by the detection device.

[0008] DE 10 2013 215 756 A1 relates to a device for monitoring the functionality of a transmission link, in particular a camera, comprising a camera, in particular a digital camera, for detecting a detection area and for generating corresponding first signals, a display unit for displaying the detection area detected by the camera using the first signals, a light source arrangement arranged in the detection area of ​​the camera, a detection unit with an optical measuring sensor arrangement for detecting the display of the display unit and for generating corresponding second signals, a monitoring unit which controls the light source arrangement by means of a predefinable pattern and checks whether the second signals detected by the detection unit contain the pattern when evaluating the display of the display unit, and a transmission unit for transmitting the first and second signals and the predefinable pattern.Furthermore, the invention relates to a corresponding method and a corresponding computer program.

[0009] DE 100 47 896 A1 relates to a method for checking the functionality of an optical sensor device, in which a changeable pattern is generated which is detected by the sensor device and evaluated by means of an evaluation unit for malfunctions in the sensor device.

[0010] The known systems have the disadvantage that they are difficult and costly to implement. Furthermore, it is quite difficult to efficiently and reliably automate the detection of frozen image frames with these systems. However, reliable display and / or detection of frozen image frames is of great importance. Otherwise, dangerous situations can arise, for example, in which a driver reverses without realizing that the video stream from the rearview camera system is currently frozen. The driver may then collide with objects or—even worse—people behind the vehicle, since the objects or people are not displayed to the driver due to the frozen image phenomenon.

[0011] The object of the invention is to further develop the generic device in such a way that it overcomes the disadvantages of the prior art. The invention aims to provide schemes that enable the provision of reliable information about the operating status of a video stream, particularly with regard to the occurrence of frozen image frames. At the same time, the implementation and operation of the schemes should be possible in a cost-effective and, above all, reliable and secure manner.

[0012] The invention solves the problem according to a first aspect by the features of the characterising part of claim 1.

[0013] Preferred devices of the invention are described in claims 2 to 17.

[0014] Accordingly, a device for image frame detection is provided, the device comprising: at least one image sensor configured to detect at least one image frame of an overall optical signal illuminating at least one region of the image sensor, the overall optical signal comprising at least a first optical signal and at least one second optical signal; and at least one optical pattern signal generator configured to generate at least the second optical signal.

[0015] The first optical signal comprises an optical signal of a scene to be captured and / or the second optical signal comprises an optical pattern signal. Furthermore, the second optical signal and / or the optical pattern signal change synchronously with the frame rate of the image sensor. For example, the second optical signal and / or the optical pattern can comprise and / or represent at least one roller donut.

[0016] Furthermore, the at least one region of the image sensor comprises at least a first region and at least one second region, wherein the at least one first region of the image sensor is illuminated by the first optical signal, the at least one second region of the image sensor is illuminated by the second optical signal, and the first region and the second region are arranged relative to one another such that the second region lies completely outside the first region of the image sensor.

[0017] In one embodiment, the first optical signal and / or the optical signal of the scene to be captured comprises at least (a) light reflected, emitted and / or outgoing from at least one object, wherein in particular the object is present in the scene, (b) light reflected in the direction of the image sensor, (c) at least partially light of the visible, infrared and / or ultraviolet light spectrum.

[0018] The second optical signal and / or the optical pattern signal is dynamic and / or changes over time, (ii) the second optical signal and / or the optical pattern signal comprises, encodes, represents and / or displays at least one predefined piece of information and / or at least one visual pattern, (iii) the second optical signal and / or the optical pattern signal at least partially comprises light of the visible, infrared and / or ultraviolet light spectrum. In yet another embodiment, at least one first optical path is defined by the propagation path taken by the first optical signal, in particular from the scene to be captured and / or in the direction of the image sensor, and / or at least one second optical path is defined by the propagation path taken by the second optical signal, in particular from the generator of an optical pattern signal and / or in the direction of the image sensor.

[0019] In one embodiment, the device further comprises at least one first optical system which is designed to (i) project and / or direct the first optical signal onto and / or in the direction of the image sensor, in particular onto and / or in the direction of the first region, and / or (ii) illuminate the image sensor, in particular the first region, with the first optical signal.

[0020] In a further embodiment, the first optical system is placed in the first optical path and / or is placed between the scene to be captured and the image sensor.

[0021] In yet another embodiment, the optical pattern signal generator comprises at least one illumination unit for generating, transmitting, and / or emitting the second optical signal and / or the optical pattern signal, wherein the illumination unit emits light in particular in the visible, infrared, and / or ultraviolet light spectrum, and at least one monitor, in particular operating in the visible, infrared, and / or ultraviolet light spectrum. For example, the illumination unit could be an LED or an array of LEDs, wherein the LEDs can emit light in the same wavelength range or in different wavelength ranges.

[0022] In one embodiment, the device and / or the optical pattern signal generator comprises / comprise at least one second optical system, which is in particular placed in the second optical path, is optically placed between the optical pattern signal generator and the image sensor and / or is placed within the optical pattern signal generator.

[0023] In a further embodiment, the first optical system and / or the second optical system comprises / comprise at least one lens and / or at least one aperture.

[0024] In yet another embodiment, the image sensor further comprises means configured to generate at least one video stream signal from successive captured image frames.

[0025] In one embodiment, the device further comprises at least one mirror element, in particular placed in the first optical path, placed in the second optical path, placed optically between the generator of an optical pattern signal and the image sensor, placed optically between the first optical system and the image sensor, placed optically between the scene to be captured and the first optical system, placed optically between the generator of an optical pattern signal and the second optical system, placed optically between the second optical system and the image sensor and / or placed optically between the scene to be captured and the image sensor.

[0026] In yet another embodiment, the mirror element is designed, at least in certain regions, as a semi-transparent mirror or as a cold light mirror, which are transmissive for a specific wavelength range and reflective for other spectral ranges, wherein in particular (i) the first optical signal and / or the second optical signal passes / passes the mirror element through the semi-transparent side of the mirror element and / or (ii) the first optical signal and / or the second optical signal is / are reflected on the reflective side of the mirror element.

[0027] In one embodiment, the first optical path comprises at least one first optical path and at least one second first optical path, wherein in particular the first first optical path extends between the first optical system and / or the scene to be captured on the one hand and the mirror element on the other hand, and / or the second first optical path extends between the mirror element on the one hand and the image sensor on the other hand.

[0028] In a further embodiment, the second optical path comprises at least a first second optical path and at least a second second optical path, wherein in particular the first second optical path extends between the generator of an optical pattern signal and / or the second optical system on the one hand and the mirror element on the other hand, and / or the second second optical path extends between the mirror element and the image sensor.

[0029] In yet another embodiment, the first optical path, the second first optical path, the first second optical path and / or the second second optical path on the one hand and the first first optical path, the second first optical path, the first second optical path and / or the second second optical path on the other hand extend perpendicularly and / or parallel to one another, at least substantially and / or in sections.

[0030] In one embodiment, the first optical path and / or the first second optical path on the one hand and the second first optical path and / or the second second optical path on the other hand extend along a common straight line, at least substantially and / or in sections.

[0031] In a further embodiment, the generator of an optical pattern signal is attached to the image sensor and / or arranged such that it illuminates the image sensor, in particular the second region, with the second optical signal, preferably (i) directly, wherein in particular (a) the direction of the second optical path is not changed by one or more optical elements, in particular by mirror elements and / or the mirror element, and / or (b) one or more optical elements, in particular mirror elements and / or the mirror element, is / are not placed in the second optical path and / or optically between the generator of an optical pattern signal and the image sensor, and / or (ii) indirectly, in particular via the mirror element and / or by reflecting the second optical signal on the reflective side of the mirror element.

[0032] The invention solves the problem according to a second aspect by a system for providing detection of frozen image frames in at least one video stream, the system comprising: at least one first device, comprising at least one device according to the first aspect of the invention, for image frame detection and generation of the video stream from consecutive detected image frames; at least one second device, comprising at least one device for detecting at least one frozen image frame within the video stream; and / or at least one display device for displaying the video stream.

[0033] In one embodiment of the system, a device for detecting at least one frozen image frame within at least one video stream of image frames successively captured by a device of the first aspect of the invention comprises: at least one receiving unit for receiving the video stream; and at least one image processing unit, comprising: at least one extraction element configured to extract at least one first image frame from the video stream, to extract at least one first pattern mixed within the at least one first image frame, to extract at least one second image frame from the video stream, and / or to extract at least one second pattern mixed within the at least one second image frame;and at least one comparator element configured to compare the first pattern extracted from the first image frame with the second pattern extracted from the second image frame and to identify whether a frozen image frame is present, wherein a frozen image frame is identified as present if the first pattern is equal to the second pattern and / or if the first pattern and / or the second pattern is / are not equal to an expected first pattern or an expected second pattern, respectively.;

[0034] In one embodiment of the system, it is preferred that the first image frame is a current image frame, the second image frame is a previous image frame, the second image frame is temporally immediately before the first image frame and / or the second frame and the first frame are temporally separated from one another by at least one, preferably by a plurality of intermediate image frames and / or that the expected first pattern and / or the expected second pattern is / are expected based on the knowledge of the pattern dynamics.

[0035] In a further embodiment of the system, the device further comprises at least one notification unit which is designed to output a notification signal and / or to manipulate the video stream when the comparator element detects that a frozen image frame is present, wherein in particular the manipulation of the video stream comprises mixing at least one item of driver information perceivable by the human observer of the video stream, preferably on at least one display device, within at least one frame of the video stream, wherein the notification signal and / or the driver information comprises, encodes, represents and / or displays at least one item of information relating to the detection of a frozen image frame.

[0036] In yet another embodiment of the system, the first pattern and / or the second pattern can be represented at least in one instance of time by an optical signal, in particular the second optical signal and / or an optical signal generated by an optical pattern signal generator, preferably the optical pattern signal generator comprised by the apparatus of the first aspect of the invention.

[0037] The invention solves the problem according to a third aspect by a method for image frame detection, carried out on a device of the first aspect of the invention, wherein the method comprises the following steps: illuminating at least a first region of an image sensor with a first optical signal, wherein the first optical signal preferably comprises, represents and / or is an optical signal of a scene to be detected; generating, with an optical pattern signal generator, at least a second optical signal, wherein the second optical signal preferably comprises, represents and / or is an optical pattern signal; illuminating at least a second region of the image sensor with the second optical signal;and capturing at least one frame of the total optical signal illuminating at least a region of the image sensor, the total optical signal comprising at least the first optical signal and at least the second optical signal, the region comprising at least the first region and at least the second region;

[0038] In one embodiment, the method comprises a further method for detecting at least one frozen image frame within at least one video stream of image frames successively acquired by a device of the first aspect of the invention, carried out by the method of the second aspect of the invention, the method comprising the following steps: receiving the video stream; extracting at least a first image frame from the video stream, extracting at least one first pattern comprising the at least one first image frame, extracting at least one second image frame from the video stream and / or extracting at least one second pattern comprising the at least one second image frame;Comparing the first pattern extracted from the first image frame with the second pattern extracted from the second image frame, and identifying whether a frozen image frame is present, wherein a frozen image frame is identified as present if the first pattern is equal to the second pattern and / or if the first pattern and / or the second pattern is / are not equal to an expected first pattern or an expected second pattern, respectively;and preferably outputting a notification signal and / or manipulating the video stream when a frozen image frame is identified as being present, wherein in particular the manipulation of the video stream comprises mixing at least one piece of driver information perceivable by the human observer of the video stream, preferably on at least one display device, within at least one frame of the video stream, wherein the notification signal and / or the driver information comprises, encodes, represents and / or displays at least one piece of information relating to the detection of a frozen image frame;

[0039] The invention solves the problem according to a fourth aspect by a vehicle comprising at least one device according to the first aspect of the invention for image frame detection, or at least one system according to the second aspect of the invention for providing detection of frozen image frames within at least one video stream.

[0040] It has thus been surprisingly found that by mixing a first optical signal, e.g. the optical signal of the scene to be captured, with a second optical signal, e.g. the optical pattern signal, which preferably changes over time, and thus forming an overall optical signal and illuminating at least one area of ​​an image sensor with this overall optical signal, the image frames captured by the image sensor inherently comprise a pattern, in particular a visual pattern, according to the second optical pattern signal.

[0041] Due to the mechanism used to mix the two optical signals, the visual pattern is present in the captured image frame from the very beginning. Delays that occur after the capture of the image frames in the subsequent transmission and / or processing of the signal of the subsequently captured image frames therefore affect the captured first and second optical signals in the same way. Thus, the second optical signal, particularly in the form of a visual pattern comprising the captured image frames, represents a reliable indicator for detecting frozen image frames. In particular, it is not possible for only the captured first optical signal to be affected by delays in the video stream. Thus, there is no risk of false positives (i.e., the pattern does not change even though the video stream is functioning properly) or, even worse, false negatives (i.e.,, the pattern still changes even though the video stream is in a frozen frame state.) At the same time, the invention can be implemented in a simple manner while the structure is still quite robust.

[0042] In this regard, the inventors have further discovered that the mixing mechanism, i.e., the mixing of the two optical signals, can be achieved in a variety of ways. Advantageously, the second optical signal can be projected directly or indirectly onto the image sensor to illuminate it. The best choice can be made depending on the design requirements of the device.

[0043] Frozen image frames can be detected very conveniently and reliably by a person observing the visual pattern within the video stream on a display on which the video stream is shown. As soon as the pattern stops changing, the person knows that a frozen image frame is present. It has been shown that automated detection of frozen image frames can also be implemented efficiently and reliably when the second optical signal, and thus also the visual pattern present in the captured image frames, changes synchronously with the frame rate of the image sensor. If the patterns are identical in two consecutive frames received for display on the display, a frozen image frame is present.Alternatively or additionally, if the optical signal is known in advance, it is also possible to compare the patterns of one or more frames of the video stream with the expected patterns. A frozen image frame can then be identified if the actual patterns do not match the expected ones, and appropriate warnings can be issued.

[0044] The invention thus enables the cost-effective and simple implementation of reliable and effective schemes for detecting image frames containing a pattern, as well as for detecting frozen image frames. This allows for the introduction of certainty about the actual state of the video stream for the purpose of processing and / or displaying the image frames. This, in turn, enables more reliable algorithms for detecting frozen image frames without introducing excessive complexity, either in the implementation or in the processing algorithms.

[0045] As noted, if the second optical signal changes at the frame rate, each captured frame will have a unique pattern, particularly within a period of the pattern. The frame rate is the rate at which the image sensor captures subsequent image frames.

[0046] For the purposes of the present invention, a first element is "optically disposed between" a first second element and a second second element if an optical signal propagates across the first element along an optical path from the first second element to the second second element. Thus, a first element may be optically disposed between two second elements even though the first element is not spatially disposed between the two second elements, i.e., the three elements are not sandwiched.

[0047] The following drawings show aspects of the invention for a better understanding of the invention in conjunction with some exemplary representations, wherein the Fig. 1a, Fig. 1b shows a schematic representation of an apparatus for image frame detection according to a first embodiment of the first aspect of the invention; the Fig. 2a, Fig. 2b shows a schematic representation of an apparatus for image frame detection according to a second embodiment of the first aspect of the invention; Fig. 3 shows a block diagram of the principle of the mixing mechanism according to an embodiment of the first aspect of the invention; Fig. 4 shows a schematic representation of an embodiment of an apparatus for detecting frozen image frames; Fig. 5 shows a flowchart of image signal processing according to an embodiment; Fig. 6 shows a schematic representation of a system for providing detection of a frozen image frame within at least one video stream according to an embodiment of the second aspect of the invention; Fig. 7 shows a functional diagram of a method according to an embodiment of the third aspect of the invention; and Fig. 8 shows a functional diagram of a method according to an embodiment.

[0048] In Fig. 1a shows a device 1 for image frame detection according to a first embodiment of the first aspect of the invention. The device 1 comprises an image sensor 3. In Fig. 1b the image sensor is positioned along the Fig. 1a. Consequently, like reference numerals in the Fig. 1a and Fig. 1b on the same characteristics.

[0049] The image sensor is designed to capture at least one image frame of an overall optical signal that illuminates at least one area of ​​the image sensor. This area comprises a first area 5 and a second area 7. As in particular Fig. As can be seen from Figure 1b, the first region and the second region are arranged relative to each other such that the first and second regions meet at a single corner of the respective regions. The overall optical signal comprises a first optical signal illuminating the first region 5 of the image sensor 3 and a second optical signal illuminating the second region 7 of the image sensor 3.

[0050] The first optical signal represents an optical signal of a scene to be captured. In the Fig. 1a, the scene to be captured comprises an object 9. The first optical signal (which in this case corresponds to the optical signal of the scene to be captured) therefore comprises at least light that is reflected and / or emitted by the object 9 present in the scene in the direction of the image sensor 3. To illustrate the first optical signal, which propagates symmetrically to the optical axis 15 in the direction of the image sensor 3, Fig. 1a shows edge rays 11, 13 emanating from the object 9. The device 1 further comprises a first optical system 17 configured to project the first optical signal onto the first region 5 in order to illuminate the first region 5 of the image sensor 3 with the first optical signal.

[0051] The second optical signal represents an optical pattern signal generated by an optical pattern signal generator 19 included in the device 1. The second optical signal (which in this case corresponds to the optical pattern signal) comprises at least partially light of the visible, infrared, or ultraviolet spectrum and represents an optical pattern that changes in a predefined manner over time. Thus, if the pattern is known at one instance of time, it is also known for all other instances of time. In particular, the pattern changes synchronously with the frame rate of the image sensor 3.

[0052] To illustrate the second optical signal, which propagates along a second optical axis 25 in the direction of the image sensor 3, Fig. 1a shows two edge rays 21, 23 emanating from the optical pattern signal generator 19. The device 1 further comprises a mirror element 27, which is optically positioned in the second optical axis 25 between the optical pattern signal generator 19 and the image sensor 3. The second optical axis 25 comprises a first second optical path 29 and a second second optical path 31. The first second optical path 29 extends between the optical pattern signal generator on the one hand and the mirror element 27 on the other. The second second optical path 31 extends between the mirror element 27 and the image sensor 3. The mirror element 27 reflects the second optical signal by 90° and in the direction of the image sensor 3, where it illuminates the second area 7 and simultaneously transmits the first optical signal through it in the direction of the image sensor 3.Consequently, the first second optical path 29 and the second second optical path 31 extend perpendicular to each other, and the first optical path 15 and the second second optical path 31 extend parallel to each other.

[0053] As further stated in Fig. As shown in Figure 1a, the mirror element 27 is also positioned between the first optical system 17 and the image sensor 3. Since the mirror element 27 is designed here as a semi-transparent mirror or a wavelength-dependent cold mirror that is transmissive for one spectral range and reflective for other spectral ranges, the first optical signal passes through the mirror element 27 through the semi-transparent side of the mirror element 27, and the second optical signal is reflected by the mirror element via the reflective side. In other words, the first optical path, in contrast to the second optical path, is a straight line.

[0054] Due to the described design of device 1, the image frame captured by image sensor 3 includes both the scene to be captured (first optical signal) and the visual pattern (second optical signal). The pattern is generated by the optical pattern signal generator 19 and changes synchronously with the frame rate according to a specific, known sequence (e.g., a rolling donut). As described below, a frozen image frame in a video stream comprising subsequent captured image frames can be easily identified based on the pattern.

[0055] Those skilled in the art will, of course, recognize that it would also be possible to rearrange the structure of device 1 and thus swap the course of the first and second optical paths. This means that the first optical signal is reflected by mirror element 27, and the second optical signal is transmitted along a straight line through mirror element 27.

[0056] The device of Fig. 1a, Fig. 1b is therefore based on an optical overlap mechanism.

[0057] In the Fig. 2a shows a device 1' for image frame detection according to a second embodiment of the first aspect of the invention. Elements of the second embodiment of the device 1' that largely correspond functionally to those of the first embodiment of the device 1 are provided with the same, but primed, reference numerals. Fig. 2b, the image sensor 3' is positioned along a Fig. 2a. Since the functionality of the second embodiment of the device 1' largely corresponds to the first embodiment, only differences between the first and second embodiments will be discussed below. The explanations for Fig. 1a and Fig. 1b also apply to the second embodiment and the Fig. 2a and Fig. 2b.

[0058] While in device 1, the optical pattern signal generator 19 is arranged such that it indirectly illuminates the second region 7 of the image sensor 3 with the second optical signal via the mirror element 27, the optical pattern signal generator 19' in device 1' is arranged such that it directly illuminates the second region 7' ​​of the image sensor 3' with the second optical signal. Thus, the direction of the second optical path is not altered by one or more mirrors. In other words, the optical pattern signal generator 19' is attached to the image sensor 3' and projects the second optical signal directly onto the second region 7', thus illuminating it. Consequently, the first optical axis 15' and the second optical axis 25' extend parallel to one another.

[0059] The device of Fig. 2a, Fig. 2b is therefore based on a direct overlap mechanism.

[0060] Fig. Figure 3 shows a block diagram of the principle of the mixing mechanism according to an embodiment of the first aspect of the invention within a device according to the first aspect of the invention, such as device 1 or 1'. The device may in particular be a camera. A first optical signal, which is a scene to be captured and in Fig. 3 is denoted by S(n), and a second optical signal, which is a known (in particular visual) pattern and in Fig. 3 denoted by P(n), are summed (or, in other words, mixed) to form a total signal, which is Fig. 3 is denoted by I(n) = S(n) + P(n). This total signal I(n) is in turn detected by an image sensor, in particular the image sensor 3 or the image sensor 3'. The manner of adding or mixing the first and second optical signals may vary and any suitable mechanism may be applied. For example, the optical overlap mechanism, as described above with respect to the Fig. 1a and Fig. 1b, or the direct overlap mechanism as discussed above with respect to the Fig. 2a and Fig. 2b discussed above.

[0061] Since the first optical signal is present in the captured image frame in the form of the optical pattern signal from the outset, the same delays experienced by the captured first optical signal (i.e., the scene to be captured) are also experienced by the captured second optical signal in the same way. Thus, due to the design of the device 1 or 1', a failure in the transmission of the video stream, e.g., due to buffers or other delay sources, can be easily detected by observing and / or evaluating the optical pattern in the received video stream signal. This can, for example, be based on the driver's perception when viewing the video stream on a display or the like. The driver can then decide for themselves that the image frame is frozen if they recognize that the pattern, e.g., a rolling donut, does not change. For this to work, the pattern must, of course, be within the active area of ​​the image sensor 3, 3', i.e., within the area displayed to the driver. This can also be done, for example, on the basis of image processing such as a software algorithm. In this case, the pattern could also overlap outside the active area of ​​the image sensor 3, 3', i.e., outside the area displayed to the driver. Such automation of the detection of frozen images is described below with reference to . Fig. 4 described in more detail.

[0062] Fig. Fig. 4 shows a schematic representation of a device 101 for detecting at least one frozen image frame within at least one video stream, in particular a video stream of image frames which are successively captured by a device according to the first aspect of the invention, such as the device 1 or 1'.

[0063] The device 101 comprises a receiving unit 103 for receiving the video stream.

[0064] The device 101 further comprises an image processing unit 105. The image processing unit 105 in turn comprises an extraction element (in Fig. 4 not shown separately) configured to extract a first pattern mixed in a first image frame of the video stream and to extract a second pattern mixed in a second image frame of the video stream. The image processing unit 105 further comprises a comparator element (in Fig. 4 (not shown separately) configured to compare the first pattern extracted from the first image frame with the second pattern extracted from the second image frame and to identify whether a frozen image frame is present, identifying that a frozen image frame is present if the first pattern is equal to the second pattern. The image processing unit 105 may be implemented as a "system on a chip" (SoC) and / or may comprise an image signal processor (ISP).

[0065] Fig. 5 shows a flowchart 111 of image signal processing, which can be executed in particular by the image processing unit 105. In a step 113, the extraction element extracts a first pattern P(n) that is mixed into a first image frame of the video stream. In a step 115, a decision is made as to whether the first pattern P(n) extracted from the first image frame is equal to the second pattern P(n-1) extracted from the second image frame. The second image frame is an image frame immediately preceding the first image frame. If the first and second patterns are equal, the comparator element identifies in a step 117 that a frozen image frame is present. This could trigger further actions. Alternatively and / or additionally, the next pattern is extracted and compared with the previous one (“No” branch in Fig. 5).

[0066] The Fig. The device 101 shown in Figure 4 further comprises a notification unit 107 configured to output a notification signal when the comparator element identifies that a frozen image is present, wherein the notification signal comprises, encodes, represents and / or displays at least one item of information relating to the detection of a frozen image.

[0067] Fig. 6 is a schematic representation of a system 121 for providing detection of a frozen image frame within at least one video stream according to an embodiment of the second aspect of the invention.

[0068] The system 121 comprises a first device 123, in particular a device according to the first aspect of the invention, such as device 1 or 1', for image frame capture and generating the video stream from successive captured image frames. The system 121 further comprises a second device 125, such as device 101, for detecting frozen image frames within the video stream. The system 121 further comprises a display device 127 for displaying the video stream, in particular for a driver. On the display device 127, either a scene S(n) to be captured can be displayed alone, or the scene S(n) to be captured mixed with a pattern P(n), i.e., the image of an overall optical signal I(n) = S(n) + P(n), can be displayed. For illustration purposes, Fig. 6 also shows a flowchart for detecting frozen image frames like the one above with respect to Fig. 5. The steps of the flowchart are executed by the second device 125. However, reference is made to the above explanations Fig. 5, which also apply here mutatis mutandis. Furthermore, the system 121 also comprises at least one connection means 129 for transmitting the video stream from the first device 123 to the second device 125 and / or the display device 127. The connection means 129 can comprise at least one cable and / or at least one wireless interface. Since delays in the video stream signal influence the captured overall optical signal, the pattern P(n) is a reliable reference for detecting frozen image frames.

[0069] Fig. 7 shows a functional diagram of a method 200 for image frame detection, in particular carried out on a device according to the first aspect of the invention, such as the device 1 or 1', according to an embodiment of the third aspect of the invention.

[0070] In a step 201, at least a first region, in particular the first region 5 or 5', of an image sensor, in particular the image sensor 3 or 3', is illuminated with a first optical signal, wherein the first optical signal comprises, represents and / or is an optical signal of a scene to be captured, in particular the object 9'.

[0071] In a step 203, at least one second optical signal is generated, wherein the second optical signal comprises, represents and / or is an optical pattern signal.

[0072] In a step 205, at least a second region, in particular the second region 7 or 7' of the image sensor, is illuminated with the second optical signal.

[0073] In a step 207, at least one frame of the overall optical signal illuminating at least one area of ​​the image sensor is captured, wherein the overall optical signal comprises at least the first optical signal and at least the second optical signal, wherein the area comprises at least the first area and at least the second area.

[0074] Fig. 8 shows a functional diagram of a method 300 for detecting at least one frozen image frame within at least one video stream, in particular a video stream of image frames successively acquired by a device according to the first aspect of the invention, such as device 1 or 1', executed on the device such as device 101, and / or the image frames acquired by the method such as method 200, according to an embodiment of the invention.

[0075] In a step 301 the video stream is received.

[0076] In a step 303, at least one first pattern comprising at least one first image frame and at least one second pattern comprising at least one second image frame are extracted.

[0077] In a step 305, the first pattern extracted from the first image frame is compared with the second pattern extracted from the second image frame, and it is identified whether a frozen image frame is present, wherein a frozen image frame is detected if the first pattern is equal to the second pattern.

[0078] In a step 307, a notification signal is output when the comparator element identifies that a frozen image frame is present, wherein the notification signal comprises, encodes, represents and / or indicates at least one piece of information relating to the detection of a frozen image frame.

[0079] The features disclosed in the claims, the description and the drawings may be essential for various embodiments of the claimed invention, both individually and in any combination with one another.

Claims

[1] Device (1, 1') for image frame detection, wherein the device (1, 1') comprises: at least one image sensor (3, 3') configured to detect at least one image frame of an optical total signal illuminating at least one area of ​​the image sensor (3, 3'), wherein the optical total signal comprises at least a first optical signal comprising an optical signal of a scene to be detected, and at least a second optical signal; and at least one generator of an optical pattern signal (19, 19') configured to generate at least the second optical signal, wherein the second optical signal comprises an optical pattern signal, wherein the at least one area of ​​the image sensor (3, 3') comprises at least a first area (5, 5') and at least a second area (7, 7'), wherein • the at least one first area (5, 5') of the image sensor (3, 3') is illuminated by the first optical signal, wherein the first area (5, 5') of the image sensor (3, 3') is configured to detect at least light of the visible, infrared and / or ultraviolet light spectrum, • the at least one second area (7, 7') of the image sensor (3, 3') is illuminated by the second optical signal, wherein the second area (7, 7) of the image sensor (3, 3') is configured to detect at least partially light of the visible, infrared and / or ultraviolet light spectrum, and • the first area (5, 5') and the second area (7, 7') are arranged in such a way that the second area (7, 7') lies completely outside the first area (5, 5') wherein the pattern signal (i) is dynamic or changes over time, (ii) includes, encodes, represents and / or displays at least one predefined piece of information and / or at least one visual pattern, (iii) includes at least some light from the visible, infrared and / or ultraviolet light spectrum, and (iv) changes synchronously with the frame rate of the image sensor (3, 3'). [2] Device (1, 1') according to claim 1, characterized by , that the first optical signal includes at least the following: (a) light reflected, emitted and / or outgoing from at least one object (9, 9'), wherein the object (9, 9') is present in the scene, (b) light reflected towards the image sensor (3, 3'), (c) at least partially light from the visible, infrared and / or ultraviolet light spectrum. [3] Device (1, 1') according to any one of the preceding claims, characterized by, that at least a first optical path (15, 15') is defined by the propagation path taken by the first optical signal, in particular from the scene to be detected and / or in the direction of the image sensor (3, 3'), and at least a second optical path (25, 25') is defined by the propagation path taken by the second optical signal, in particular from the generator of an optical pattern signal (19, 19') and / or in the direction of the image sensor (3, 3'). [4] Device (1, 1') according to any one of the preceding claims, characterized by , that the device (1, 1') further comprises at least one first optical system (17, 17') configured to (i) project and / or direct the first optical signal onto and / or in the direction of the image sensor (3, 3'), in particular onto and / or in the direction of the first area (5, 5'), and / or (ii) illuminate the image sensor (3, 3'), in particular the first area (5, 5'), with the first optical signal. [5] Device (1, 1') according to claim 4, characterized by , that the first optical system (17, 17') is placed in the first optical path (15, 15') and between the scene to be captured and the image sensor (3, 3'). [6] Device (1, 1') according to any one of the preceding claims, characterized by , that the generator of the optical pattern signal (19, 19') comprises at least one illumination unit for generating, transmitting and / or emitting the second optical signal and / or the optical pattern signal, wherein the illumination unit emits light in the visible, infrared and / or ultraviolet light spectrum. [7] Device (1, 1') according to any one of the preceding claims, characterized by, that the device (1, 1') and / or the generator of an optical pattern signal (19, 19') comprises / comprise at least a second optical system which is placed in the second optical path (25, 25') and / or is optically placed between the generator of the optical pattern signal (19, 19') and the image sensor (3, 3'). [8] Device (1, 1') according to any one of the preceding claims, characterized by , that the first optical system (17, 17') and / or the second optical system comprises at least one lens and / or at least one aperture. [9] Device (1, 1') according to any one of the preceding claims, characterized by , that the image sensor (3, 3') further comprises means configured to generate at least one video stream signal from successive captured image frames. [10] Device (1, 1') according to any one of the preceding claims, characterized by, that the device (1, 1') further comprises at least one mirror element (27) placed in the first optical path (15, 15'), placed in the second optical path (25, 25'), placed optically between the generator of the optical pattern signal (19, 19') and the image sensor (3, 3'), placed optically between the first optical system (17, 17') and the image sensor (3, 3'), placed optically between the scene to be detected and the first optical system (17, 17'), placed optically between the generator of the optical pattern signal (19, 19') and the second optical system, placed optically between the second optical system and the image sensor (3, 3') and / or placed optically between the scene to be detected and the image sensor (3, 3'). [11] Device (1, 1') according to claim 10, characterized by, that the mirror element (27) extends through at least one focal point (33, 33') of the first and / or second optical system, wherein in particular the at least one focal point (33, 33') is located optically and / or spatially between the first and / or second optical system on the one hand and the image sensor (3, 3') on the other. [12] Device (1, 1') according to one of claims 10 or 11 , characterized by , that the mirror element (27) is designed as a semi-transparent mirror or as a cold light mirror at least in certain areas, in particular (i) the first optical signal and / or the second optical signal pass through the semi-transparent side of the mirror element (27) and / or (ii) the first optical signal and / or the second optical signal is reflected at the reflecting side of the mirror element. [13] Device (1, 1') according to any one of the preceding claims, characterized by, that the first optical path (15, 15') comprises at least one first optical path and at least one second first optical path, wherein in particular the first first optical path extends between the first optical system (17, 17') and / or the scene to be detected on the one hand and the mirror element (27) on the other hand, and / or the second first optical path extends between the mirror element (27) on the one hand and the image sensor (3, 3') on the other hand. [14] Device (1, 1') according to any one of the preceding claims, characterized by, that the second optical path (25, 25') comprises at least a first second optical path (29) and at least a second second optical path (31), wherein in particular the first second optical path (29) extends between the generator of the optical pattern signal (19, 19') and / or the second optical system on the one hand and the mirror element (27) on the other hand, and / or the second second optical path (31) extends between the mirror element (27) and the image sensor (3, 3'). [15] Device (1, 1') according to one of claims 13 to 14, characterized by, that the first first optical path, the second first optical path, the first second optical path (29) and / or the second second optical path (31) on the one hand and the first first optical path, the second first optical path, the first second optical path (29) and / or the second second optical path (31) on the other hand extend perpendicularly and / or parallel to each other, at least substantially and / or sectionally. [16] Device (1, 1') according to any one of claims 13 to 15, characterized by , that the first optical path and / or the first second optical path (29) on the one hand and the second optical path and / or the second second optical path (31) on the other hand extend along a common straight line, at least substantially and / or piecewise. [17] Device (1, 1') according to any one of the preceding claims, characterized by, that the generator of the optical pattern signal (19, 19') is attached to the image sensor (3, 3') and / or arranged such that it illuminates the image sensor (3, 3'), in particular the second area (7, 7'), with the second optical signal, preferably (i) directly, wherein in particular (a) the direction of the second optical path (25, 25') is not changed by one or more optical elements, in particular by mirror elements and / or the mirror element (27), and / or (b) one or more optical elements, in particular mirror elements and / or the mirror element (27), are not placed in the second optical path (25, 25') and / or optically between the generator of an optical pattern signal (19, 19') and the image sensor (3, 3'), and / or (ii) indirectly, in particular via the mirror element (27) and / or by reflecting the second optical signal at the reflective side of the mirror element (27). [18] System (121) for providing detection of frozen frames in at least one video stream, wherein the system (121) comprises: at least one first device (123), comprising at least one device (1, 1') according to any one of claims 1 to 17, for capturing image frames and generating the video stream from successive captured image frames; and at least one second device (125) comprising at least one device (101) for detecting at least one frozen frame within the video stream; and / or at least one display device for displaying the video stream. [19] System according to claim 18, wherein the device (101) for detecting at least one frozen frame within at least one video stream of frames comprises: at least one receiving unit (103) for receiving the video stream; and at least one image processing unit (105), comprising: at least one extraction element configured to extract at least one first frame from the video stream, to extract at least one first pattern mixed within the at least one first frame, to extract at least one second frame from the video stream, and / or to extract at least one second pattern mixed within the at least one second frame; and at least one comparator element configured to compare the first pattern extracted from the first image frame with the second pattern extracted from the second image frame and to identify whether a frozen image frame is present, wherein a frozen image frame is identified as present if the first pattern is equal to the second pattern and / or if the first pattern and / or the second pattern is / are not equal to an expected first pattern or an expected second pattern. [20] System (121) according to claim 19, characterized bythat the first image frame is a current image frame, the second image frame is a previous image frame, the second image frame is immediately prior to the first image frame in time, and / or the second frame and the first frame are separated in time by at least one, preferably by a plurality of, intermediate image frames, and / or that, based on knowledge of the pattern dynamics, the expected first pattern and / or the expected second pattern is / are expected. [21] System (121) according to one of claims 19 or 20, characterized by, that the device (101) for detecting at least one frozen frame within at least one video stream of frames further comprises at least one notification unit (107) configured to output a notification signal and / or manipulate the video stream when the comparator element detects that a frozen frame is present, wherein in particular the manipulation of the video stream comprises mixing at least one driver information perceptible to the human observer of the video stream, preferably on at least one display device, within at least one frame of the video stream, wherein the notification signal and / or the driver information comprises, encodes, represents and / or displays at least one piece of information relating to the detection of a frozen frame. [22] System (121) according to any one of claims 19 to 21, characterized by, that the first pattern and / or the second pattern can be represented at least in one instance of time by an optical signal, in particular the second optical signal and / or an optical signal generated by a generator of an optical pattern signal (19, 19'), preferably the generator of the optical pattern signal (19, 19') comprising the device (1, 1') according to one of claims 1 to 21. [23] Method (200) for image frame detection, carried out on a device according to any one of claims 1 to 17, wherein the method (200) comprises the following steps: Illuminating (201) at least one first area (5, 5') of an image sensor (3, 3') with a first optical signal, wherein the first optical signal comprises, represents and / or is an optical signal of a scene to be detected; Generating (203) with a generator of an optical pattern signal (19, 19'), at least one second optical signal, wherein the second optical signal comprises, represents and / or is an optical pattern signal; Illuminate (205) at least one second area (7, 7') of the image sensor (3, 3') with the second optical signal; and Capturing (207) at least one frame of the total optical signal illuminating at least one region of the image sensor (3, 3'), wherein the total optical signal includes at least the first optical signal and at least the second optical signal, wherein the region includes at least the first region (5, 5') and at least the second region (7, 7'), wherein the first region (5, 5') and the second region (7, 7') are arranged relative to each other such that the second region (7, 7') lies completely outside the first region (5, 5'); and where the pattern signal (i) is dynamic or changes over time, (ii) includes, encodes, represents and / or displays at least one predefined piece of information and / or at least one visual pattern, (iii) includes at least some light from the visible, infrared and / or ultraviolet light spectrum, and (iv) changes synchronously with the frame rate of the image sensor (3, 3'). [24] Method according to claim 23, further comprising a method (300) for detecting at least one frozen frame within at least one video stream of frames successively detected by a device (1, 1') according to any one of claims 1 to 17, performed on a device (101) in a system according to any one of claims 18 to 22, and / or of the frames detected by the method according to claim 23, wherein the method (300) for detecting at least one frozen frame within at least one video stream comprises the following steps: Receiving the video stream; (301) Extracting at least one first frame from the video stream, extracting at least one first pattern comprising the at least one first frame, extracting at least one second frame from the video stream, and / or extracting at least one second pattern comprising the at least one second frame; (303)Comparing the first pattern extracted from the first image frame with the second pattern extracted from the second image frame, and identifying whether a frozen image frame is present, whereby a frozen image frame is identified as present if the first pattern is equal to the second pattern and / or if the first pattern and / or the second pattern is not equal to an expected first pattern or...an expected second pattern is / are; (305) and preferably outputting a notification signal and / or manipulating the video stream when a frozen frame is identified as being present, wherein in particular the manipulation of the video stream comprises mixing at least one driver information perceptible to the human observer of the video stream, preferably on at least one display device, within at least one frame of the video stream, wherein the notification signal and / or driver information comprises, encodes, represents and / or displays at least one piece of information relating to the detection of a frozen frame. (307) [25] Vehicle comprising at least one device according to any one of claims 1 to 17 for image frame detection or at least one system according to any one of claims 18 to 22 for providing a detection of frozen image frames within at least one video stream.

Images