METHOD FOR REAL-TIME VIDEO IMAGES READING AND PROCESSING

Description

Technical field

[0001] The present invention relates to a method for reading and processing video images in real time, in which one or more video cameras provide images, wherein the images are digitized and stored in a memory, after which the images are supplied to at least one monitor. State of the art

[0002] In television broadcasting, a distinction is made between (video) images and video signals. This separation is necessary to differentiate, on the one hand, a finished video frame from an electronically defined video signal for transmission, and on the other hand, to separate the different functions of the image and the video signal. An image refers to a complete picture as it appears on the screen, while the video signal is necessary to transmit the image electronically.

[0003] Traditional image transmission systems were analog, meaning the image captured by the video camera was transmitted analogously and displayed immediately on a screen. Any malfunctions were instantly visible on the screen, whether caused by a camera problem or a problem along the transmission chain. On the other hand, a time delay between camera and screen was virtually impossible – apart from the minimal signal propagation delay.

[0004] Modern image transmission solutions are designed to produce the highest possible image quality and suppress interference. This includes automatic adjustment processes in the video camera, the transmission link, and the processing of the image(s) before they reach the screen. Given the requirements of entertainment television, no errors should be visible; ideally, a stored image should be displayed on the screen during any error state. For this reason, all modern transmission systems feature rather complex electronic circuits with image storage, starting in the video camera, possibly extending into the transmission link, and finally into the image processing unit.However, as soon as digital image processing and storage come into play, a time lag between camera and monitor is unavoidable; in television broadcasts, it is very common, as can be observed when the same live event is broadcast by different television stations. Delays of a few seconds are the norm here.

[0005] Displaying multiple video images simultaneously on a single monitor has been known for decades. Typical examples are surveillance monitors in supermarkets, where, for instance, the images from four cameras are displayed on one screen. The person monitoring the images can at any time display the image from one camera in full screen or switch to other cameras on the monitor should they detect something suspicious.

[0006] Equally well-known are the surveillance monitors in the Vienna subway, which allow the driver to see when all passengers have boarded and alighted, enabling him to dispatch the train and close the doors. Sometimes, images from two cameras are displayed on a single monitor. In both of these applications, the individual images are simply shown side-by-side and / or superimposed without any transitions.

[0007] Analogously, this is only possible if all video cameras are synchronized with each other, and even then only with specially adapted video cameras and a great deal of circuitry.

[0008] The current state of the art therefore involves digitizing and storing the four incoming individual images, then creating the new, composite image from the digitally available, stored images and transmitting it to the screen. If the transmission to the screen is analog, the necessary synchronization signals must be regenerated.

[0009] EP2341710 discloses a method in which the latency of a camera signal has an effect only on the display of the image on the monitor assigned to the camera and no effect on the display of the other images in the other cameras.

[0010] Furthermore, so-called electronic rearview mirrors are known, in which three camera images (corresponding to the left exterior mirror, right exterior mirror, and interior mirror) are seamlessly combined ("stitching") and displayed to the driver on a monitor, allowing the driver to see the entire traffic situation behind them at a glance, without any blind spots. An example can be found online at https: / / www.macnica.eu / de / node / 219.

[0011] A disadvantage of these solutions is that significant delays cannot be ruled out as a result of digitalization. This makes them unusable for monitoring traffic systems, as the time lag cannot be perceived by the person watching the screen. In two seconds, for example, a departing train might already have traveled a meter, while a critical situation on the platform remains undetectable on the screen. For a vehicle traveling at 100 km / h, this would amount to over 50 meters covered without visual contact. Unfortunately, such a time lag has already resulted in a fatality.

[0012] Even without image processing (such as stitching), images are often stored in memory, read from it, and displayed on a monitor. An example is the aforementioned display of four camera images on one monitor. The video signals from all cameras are stored digitally, and then only every second pixel of each line from each camera is sent to the monitor, so that the image occupies only one quadrant. Here, too, disruptive time delays can occur, as it's always possible that a camera could fail and stop transmitting a signal. In that case, the last image remains in memory and is displayed continuously as a still image. Brief description of the invention

[0013] The object of the present invention is to provide a method of the type mentioned above in which excessively delayed playback of video images is prevented.

[0014] This problem is solved according to the invention by a method of the type mentioned at the outset in that all video images are provided with a timestamp which is stored in memory together with the video images, that afterwards the timestamp of the video image or - in the case of simultaneous playback of several video images - the timestamp of all played video images is read out and compared with the current time and the signal is only supplied to the at least one monitor if the timestamp is at most a predetermined time period ago or if all timestamps are at most a predetermined time period ago.

[0015] The invention is based on the understanding that it is better to display no image at all than an outdated image, because this immediately reveals the lack of usable information. For example, if the specified time interval is set to 100 ms, the maximum time delay is significantly below typical human reaction times and therefore does not cause any disturbance. If the time delay is greater, then no meaningful image is displayed (e.g., only black, only white, or a pattern), making the malfunction immediately apparent.

[0016] The term "timestamp" is to be understood generally. It doesn't matter which point in time is considered "zero"—for example, the system being switched on or a reset signal. It also doesn't matter which unit of measurement is used. The unit could be, for example, the frame rate (usually 40 ms), meaning the digitized images are simply numbered sequentially. The "actual" time the image was captured can then be derived from the image number and the switch-on time. However, this time doesn't actually need to be calculated. It suffices to simply run a counter that is reset to zero when the system is switched on or upon receiving a reset signal and increments by 1 every 40 ms. If this counter is compared with the image number, an image can be identified as outdated if there is a deviation of, for example, 3 or 5 images (corresponding to 120 ms or 200 ms, respectively).

[0017] If image processing takes place, the stored images should be processed after being time-stamped, the timestamp read after processing, and then the processed images displayed on at least one monitor. This way, the time required for processing is also recorded.

[0018] According to a first embodiment of the invention, the one or more video cameras deliver the images as an analog signal, and the processed images are converted into an analog signal before being fed to the at least one monitor. In this case, the incoming video images are time-stamped, and the timestamp of the video image, or—in the case of simultaneous playback of several video images—the timestamp of all displayed video images, is read and compared during the conversion of the processed images into an analog signal.

[0019] In the first embodiment, the aim is to work with analog signals as much as possible. The video camera delivers analog signals (without prior digital processing), so no delay can occur. The monitor also operates analogously, displaying the incoming video signal directly. Therefore, no delay can occur here either.

[0020] As mentioned previously, it is difficult to manage entirely without digitization, especially when images from multiple video cameras are to be displayed on a monitor. The invention addresses this by assigning a timestamp to each image during digitization, and by ensuring that stored images are only used if the timestamp is not too far in the past. Therefore, should an unexpected delay occur (e.g., because an image remains in memory and is not overwritten by new images—for example, due to a defect in the analog-to-digital converter), this is detected, and the image is not displayed at all.

[0021] According to a second embodiment of the invention, the one or more video cameras deliver the images as a digital signal. In this case, it is safest if the video images are already time-stamped in the one or more video cameras. However, if the one or more video cameras deliver the images as SDI signals, a time delay is practically impossible even in the case of digital signals: The CCD sensor digitizes the pixels sequentially with an A / D converter, and these digital values ​​are output in the SDI protocol, either via a coaxial cable or a two-wire line, and independently of the receiving devices, so that in this case, too, it is possible without any security risk to time-stamp only the incoming video images.(There is therefore hardly any difference to analog signals; only the serial processing of the pixels results in a small but constant time offset on the order of 20 ms.) The advantage of the second solution is that conventional SDI video cameras can be used.

[0022] In contrast, IP cameras transmit data via network cables and are subject to network congestion; and since networks can be overloaded and / or disrupted, unpredictable delays can occur. Therefore, when using IP cameras, the timestamp should absolutely be generated directly within the camera.

[0023] It is advantageous to delete the video image from memory as soon as it has been processed and fed to a monitor. This provides an additional safety measure to prevent outdated images from being displayed. Even if a timestamp is mistakenly recognized as current, although it is outdated, such an outdated image can only be displayed once at most, which is hardly noticeable (the usual refresh rate is 25 Hz). In the first embodiment, the video image is deleted from memory as soon as it has been converted into an analog signal.

[0024] For analog signals, at least two memory locations should be used in this case, with the video signals being written alternately to these two locations, switching between them after each image is erased. The video signals are always written to the most recently erased memory location. If memory location 1 is erased at time x, then at time x + 40 ms, complete images are present in memory location 1, and the process of reading and converting them to an analog signal can begin. At this time, memory location 2 is erased, and the remaining video signals are written to memory location 2. Since the transmission to the monitor also takes 40 ms, memory location 1 can be erased at time x + 80 ms. In the next 40 ms, memory location 2 is then converted to an analog signal and transmitted to the monitor, and the video signals are again stored in memory location 1.

[0025] When digital signals are transmitted to the monitor, this transmission is extremely fast compared to the refresh rate, so that a single memory is sufficient: Because the image transmission and deletion occur almost instantaneously, practically the entire 40 ms is available between two image transmissions in which the video signals can be stored, so that even when using only one memory, complete images are always available.

[0026] Additionally or alternatively, it is possible to delete the video images from the memory at regular intervals, for example every 200 ms.

[0027] In both cases, security can be further increased by retrieving the video image after deletion and deleting it again if it was not completely erased. This drastically reduces the possibility of erroneous deletions.

[0028] Aside from an undesirable time delay, displaying multiple camera images on one monitor also carries the risk of showing images from the wrong camera. For example, the Vienna subway system uses one camera for the front half of the train and a second camera for the rear. If, due to a switching error, the image for the rear half of the train is accidentally displayed twice on the monitor (or even worse, the image of a train on the opposite platform), this is not immediately obvious, and there is a risk that the driver will close the doors while passengers are still boarding or alighting.

[0029] To prevent this, a further feature of the invention provides that the signals are provided with an identification code and that this identification code is checked to ensure that the correct image is displayed in full-screen mode or that the individual images are displayed in the correct position when the images are displayed together on a monitor.

[0030] In the case of analog technology, i.e., in the case of the first embodiment of the invention, it is customary for the analog signals arriving from the video cameras to first be fed to the inputs of a video matrix, which then forwards the analog signals to specific outputs depending on the control input. In this way, it is possible to display any video camera on the monitor, or the combined image from several video cameras, which is also fed to this video matrix. However, such circuits, which switch analog signals depending on digital control input, can malfunction, especially in the presence of strong electromagnetic interference.It can therefore happen that the driver wants to switch to the video camera for the rearmost part of the train because he has noticed in the combined image that a door there might be blocked, but after switching, he actually sees a different part of the train and mistakenly assumes that all doors are properly closed. To prevent such a malfunction, one embodiment of the invention provides that this identification code is detected after the video matrix to check whether the correct signal has been switched on.

[0031] The safest method is to modulate the identification codes onto the analog signals and demodulate them after the video matrix. Alternatively, the identification codes could be superimposed onto the video signal, ideally in a different frequency range, but this carries the risk of the identification codes being filtered out.

[0032] If the identification code is already modulated into the video cameras, then it is also possible to detect if the video camera cables are connected incorrectly. However, standard cameras can then no longer be used.

[0033] If the video signal of the video images is a standard video signal with a black shoulder, the identification code of the black shoulder of the video signal can be imprinted, preferably after the color burst (chroma burst) in the case of color images.

[0034] If an identification code is modulated, the timestamp can also be modulated along with it and decoded together. This can potentially reduce the circuit complexity.

[0035] In the case of digital technology, i.e., the second implementation, it is advantageous to overlay the timestamp and, if applicable, the identification code onto the image as a steganogram. This is a well-known technique; for example, the LSB (least significant bit) of some color pixels can be used to store the timestamp and the identification code. The slightly increased image noise resulting from this is only noticeable with extremely high-quality cameras and optimal lighting conditions. Brief description of the drawing figures

[0036] The present invention will be explained in more detail using an exemplary embodiment. The drawings show... Fig. 1 a schematic diagram according to a first embodiment with which the method according to the invention can be carried out; Fig. 2 shows an analog television signal with an imprinted identification code that can be used in the first embodiment; and Fig. 3shows a schematic diagram of a second embodiment. Description of the execution types

[0037] Several video cameras 11 (four in this example) are connected to a video matrix 13. This video matrix 13 is a digitally controlled electronic switch that can forward a signal from any input to each output. At least one monitor 16 (two in this example) is connected to this video matrix 13. The video matrix 13 is controlled by a controller 14. It can be controlled, for example, so that the first monitor 16 displays the image from the fourth video camera 11 and the second monitor 16 displays the image from the first video camera 11. However, both monitors 16 can also display the same image.

[0038] Both the video cameras 11 and the monitors 16 are analog devices; that is, the video cameras 11 deliver analog signals and the monitors 16 process analog signals without the images ever being digitized in these devices. Therefore, when a monitor 16 displays the image from a video camera 11, there can be no time delay.

[0039] However, it is often desirable to display the images from four video cameras 11 simultaneously in four quadrants on a monitor 16 (or the images from two video cameras 11 in two halves). To achieve this, the analog signals from the video matrix 13 are routed to an analog-to-digital converter 17, where they are digitized and fed to an FPGA 18 (FPGA = Field Programmable Gate Array).

[0040] Using fast Field Programmable Gate Arrays, or FPGAs, response times in the nanosecond range are achieved. This rapid response is impossible with conventional signal processors because they process incoming signals sequentially, according to the program's instructions. FPGAs, on the other hand, are programmable hardware; up to 100 processes can be routed through the chip in parallel, independent of the individual steps, with processing speeds up to 500 times faster than in a signal processor.

[0041] This FPGA 18 now stores the data supplied by the analog-to-digital converter 17 in a memory (RAM) 19, so that four images from four video cameras 11 are stored. From these images, the FPGA 18 assembles new images (in this example, two images), e.g., the images from all four video cameras 11 in the four quadrants of one new image and the images from two of the four video cameras 11 in the two halves of another new image. These new images, which are stored digitally in the FPGA, are then converted back into analog signals in a digital-to-analog converter 20 and fed to the video matrix 13. Thus, each monitor 16 can display either a four-quadrant image, a two-half image, or the image from each video camera 11 as a full frame, depending on which signals the video matrix 13 sends to the corresponding output.

[0042] The invention consists in the fact that, when saving the images, the FPGA 18 also stores a timestamp with each image and checks this timestamp when assembling the new image. If the timestamp is too far in the past, the stored image is ignored; that is, the corresponding part of the new image (a quadrant or half) remains dark. (Of course, a color or pattern can also be displayed.) This makes it impossible for the new image to be assembled based on outdated data.

[0043] The invention prevents outdated information from being displayed. However, it is understood that this should only occur rarely. The entire system is therefore designed so that delays are normally very minimal.

[0044] As an additional safety measure, the FPGA 18 deletes the data in memory 19 as soon as it has been read. This means that each image from each video camera 11 can only be used once to construct a new image.

[0045] Another safety risk is posed by the video matrix 13, as there have been reports of such a video matrix incorrectly wiring its inputs and outputs when strong interference fields occurred. This can result in the driver seeing the image from a different video camera 11 on monitor 16 than the one he has selected.

[0046] To eliminate this security risk as well, modulators 12 are located before the video matrix 13, and demodulators 15 are located after the video matrix 13. The modulators 12 imprint an identification code onto the video signal, which can be decoded by the demodulators 15. The controller 14 then informs each demodulator 15 which identification code is to be expected based on the control signal of the video matrix 13. If the identification code does not match the target value, the demodulator 15 interrupts the video signal. In this case, too, no display occurs in the event of an error, which is considerably better than an incorrect display.

[0047] The overlaying of additional signals onto a video signal is widespread. All color television systems use this method to transmit a color burst for synchronization. Similar solutions are described in large numbers in patent literature (group H04N7 / 025 of the International Classification).

[0048] Based on Fig. 2 The process of imprinting the identification code is explained. This identification code is imprinted on the black shoulder of the video signal. A classic composite video signal (CVBS) in line mode has a line sync pulse 21, which has a leading black shoulder 22 and a lagging black shoulder 23. Black shoulders 22 and 23 are used as a reference for a black image; their values ​​are used in the display for the so-called clamping function. The image content is encoded at position 24. A color burst 25, which carries the reference signals for color modulation, is located on the trailing black shoulder 23. The identification code 26 is preferably written into the CVBS signal after the color burst 25. The identification code 26 can consist of a digital encoding in the nanosecond range because modern components can easily process these speeds.

[0049] In one specific solution, a modern, high-performance FPGA 18 was used, which reads the digital data from all four video cameras 11 in parallel and timestamps all signals. During the reading process, the images are also prepared for their later position on the screen, unnecessary image content is removed, and the image from each video camera 11 is written to a corresponding area of ​​the memory 19.

[0050] A circuit group in FPGA 18 then restores the necessary data for a combined image, provided the timestamps are not too far in the past. To increase safety and to flag malfunctions, the output memory is immediately cleared after reading and creating the combined image(s), so that no static images can be displayed.

[0051] The measures described make it possible to limit and verify the transmission time of video signals within the FPGA to a maximum of 100 ms (milliseconds) and to check the correct wiring in the video matrix 13. The transmission time of 100 ms results from the asynchronous operating state: An analog video signal requires 40 ms for the entire image transmission (at 25 Hz). Since the video cameras 11 are not synchronized, it can take a maximum of 80 ms until the data from all video cameras 11 is stored. The image may remain in memory 19 for this time, with a safety margin of 20 ms. Anything that remains in memory 19 for longer is considered outdated and may no longer be displayed. This time of 100 ms is at most half the reaction time of an attentive person. Therefore, the transmission is considered sufficiently timely to perform control operations.

[0052] To further prevent image fragments from previous image data from remaining in memory 19, memory 19, in which the FPGA 18 assembles the images into a binary or quadrant image, is erased after being read out, written with "0". This zero value corresponds to a black screen when converted to a video signal. In an immediately subsequent step, the memory is read out again to ensure that its contents have been erased. This check readout is processed only internally; no information may remain. In the event of a malfunction, i.e., a second readout as an image, a black image would be transmitted to the screen, making the error recognizable even to a normal observer.

[0053] In contrast to the first embodiment, which is based on the Figs. 1 and 2As explained, and in which analog technology is used wherever possible, the second embodiment, which is based on the Fig. 3 It is explained that digital technology is used.

[0054] Fig. 3 This illustrates the principle of an electronic rearview mirror, in the simplest case of which three cameras 11 are provided, corresponding to the left exterior mirror, the right exterior mirror and the interior mirror. These three images are combined and displayed on a monitor 16.

[0055] The basis for digital image transmission is the "SDI" interface, in particular HD-SDI (SMPTE 292M 1.485 Gbit / s and 1.485 / 1.001 Gbit / s, 720p, 1080i (HDTV)) and 12G UHD-SDI (SMPTE ST 2082 12 Gbit / s, 4K). This means that digital video data is transmitted from the camera sensors at up to 12 Gbit / s via coaxial cable with 75 Ohm impedance or two-wire twisted-pair cable (CAT7).

[0056] The values ​​of the image matrix are converted into a digital serial data stream, whereby the brightness value and color value are transmitted as digital values ​​for each pixel; normally 20 bits (10 bits for brightness and 10 bits for color) are used for this purpose.

[0057] According to the invention, the LSB (least significant bit, i.e., the bit with the lowest value) of some of these values ​​is not used to provide information about brightness or color, but rather for other information, namely a timestamp and, optionally, an identification code. Since 10 bits can represent 1024 values, this reduces the possible information to 9 bits, or 512 values. However, this is still more than sufficient in practice. Because the LSB used for the timestamp is "random" with respect to brightness or color, minimal brightness or color noise results, which is not perceptible under normal viewing conditions. In most cases, the sensor's noise is greater anyway. (Note: The values ​​0-3 and 1020-1023 are reserved and may not be used for brightness or color values. Strictly speaking, there are not 1024, but only 1016 values ​​available.)

[0058] According to the invention, a timestamp and, optionally, an identification code are imprinted onto the images in the cameras in this way; since this is not recognizable under normal viewing conditions, it is called steganography. The advantage is that the normal SDI protocol remains usable without the slightest modification, and that this information is automatically preserved during deserialization and storage of the image.

[0059] The digital data from several cameras are now deserialized in the image processing unit, which is implemented by an FPGA 18, and stored in a memory 19. The timestamps are still present. If the images are dewarped and seamlessly stitched in the case of a digital rearview mirror (where there are typically at least three images), the steganographic information could be lost. It is therefore advantageous to extract the steganographically embedded information beforehand and store it in a few additional bytes of each image. When the stitched image is then serialized and transmitted to the monitor 16 (i.e., the "electronic rearview mirror") via the SDI protocol, the timestamp of all three images can be checked at that point in time; if the timestamp is too far back compared to the user's own clock, a message is sent to the monitor 16, e.g.,A black image is displayed, making it immediately clear that no current image is available. The identification code can also be used to verify, for example, whether the image content shown on the left actually originates from the camera located on the far left.

[0060] Furthermore, a controller 14 is provided, which can be used to control the FPGA. This controller 14 can, for example, be used to specify that only the image from the camera corresponding to the interior mirror or one of the exterior mirrors is displayed on the monitor as a full screen, which is sometimes useful when parking.

[0061] Although there is hardly any delay when displaying an SDI signal on monitor 16, the composite image can of course also be timestamped, which can then be checked by monitor 16. If the image from one of the cameras is passed through as a full frame, the timestamp is automatically present anyway due to the steganographic embedding.

[0062] Although there is hardly any delay when displaying an SDI signal on monitor 16, the composite image can of course also be timestamped, which can then be checked by monitor 16. If the image from one of the cameras is passed through as a full frame, the timestamp is automatically present anyway due to the steganographic embedding.

Claims

1. A method of reading and editing images in real time with an electronic rear-view mirror, in which rear-view mirror at least three video cameras (11) supply images as a serial digital signal in the form of an SDI signal, which video cameras correspond to a left exterior mirror, a right exterior mirror and an interior mirror, wherein the images are saved in a memory (19), wherein the images are assembled and delivered to at least one monitor (16), wherein all images in the video cameras (11) are provided with a timestamp in the least significant bit of the digital signal, wherein the timestamps are deserialised together with the images by an image processing unit realised by an FPGA (18) and the timestamps are saved in the memory (19) together with the images, wherein a control is provided, by which the FPGA (18) can be controlled, wherein the images are straightened out and seamlessly assembled and then the assembled image is serialised, wherein the timestamp of all rendered images is then read and compared to the current time and a signal for transmitting the images to the at least one monitor is only delivered to the at least one monitor (16) if all timestamps do not date back further then a predefined maximum timespan.

2. The method of claim 1, characterised in that the saved images are edited after being provided with the timestamp, in that the timestamp is read after editing and in that the edited images are delivered to the at least one monitor (16).

3. The method of any one of claims 1 and 2, characterised in that the signals are provided with an identification code (26) and in that said identification code (26) is verified to make sure that the correct image is shown in full-screen display and / or that the individual images are displayed in their correct positions when the images are shown together on one monitor.