Computer-implemented method for compressing a video stream and vehicle

Abstract

The invention relates to a computer-implemented method for compressing a video stream (1) by means of sparse coding, wherein individual frames (2) of the video stream (1) are represented by matrix multiplication of a basis function matrix (3) with a sparse coefficient matrix (4). The computer-implemented method according to the invention is characterized by the following process steps: - Feeding the video stream (1) to a convolutional neural network (CNN) trained to generate a set of sparse coefficients (5) for a single frame (2), and generating initial sparse coefficients (5.0) by the convolutional neural network (CNN) for the first single frame (2.0) of the video stream (1) in the form of an initial sparse coefficient matrix (4.0); - Calculating a sparse coefficient difference matrix (6) for the next frame of the video stream (1) for all frames (2) contained in the video stream (1), wherein the sparse coefficient difference matrix (6) of a frame (2) corresponds to the difference of the sparse coefficient matrices (4) of the currently viewed frame and the previous frame in the video stream (1), wherein the sparse coefficient matrix (4) of the currently viewed frame is determined by a sparse coding optimization based on the current frame (2), the previous frame (2), and the sparse coefficient matrix (4) of the previous frame (2); and - Storing or outputting the sparse coefficient difference matrices (6) calculated for all individual frames (2) contained in the video stream (1) in the form of a sparse coefficient stream (7).

Description

[0001] The invention relates to a computer-implemented method for compressing a video stream by means of sparse coding according to the type defined in more detail in the preamble of claim 1, and to a vehicle for carrying out the method.

[0002] Cameras are now standard equipment in modern vehicles. Mono or stereo cameras can capture the vehicle's surroundings, allowing for the detection and classification of static and dynamic objects. The relative distance of these objects to the vehicle can also be calculated. Suitable cameras can also capture the vehicle's interior, for example, to record the driver's eyes. This allows tracking of the driver's gaze and / or determination of their blink rate. This information can be used for fatigue detection or to assess the driver's level of attention.

[0003] To provide appropriate driver assistance functions, relevant objects must be recognizable in the images captured by the respective cameras. This is facilitated by the highest possible image resolution. Accordingly, high-resolution cameras are preferably installed in vehicles. However, the storage space occupied by such camera images on a computer-readable storage medium increases compared to lower-resolution camera images. Processing and storing the camera images, as well as transmitting them via a data line within the vehicle, is therefore more complex. Consequently, more storage space is occupied, and the corresponding data lines are more heavily utilized, leaving less bandwidth available for transmitting other data.Typically, due to space and cost constraints, the hardware components installed in vehicles are designed to be as simple as possible, meaning the available potential is quickly exhausted. Particularly in automated, autonomous, or connected vehicles, large amounts of data are continuously generated, making their processing and transmission in real time a challenge.

[0004] Various data compression algorithms are known, for example, dictionary-based compression algorithms. Such compression algorithms can be inefficient and may have an insufficient compression rate or too slow an execution speed.

[0005] CN 103595414 A discloses the sampling and compression of signals as well as a corresponding signal reconstruction using sparse coding. A disadvantage is the high computational complexity.

[0006] Furthermore, Wright, John, et al. “Robust face recognition via sparse representation.” Pattern Analysis and Machine Intelligence, IEEE Transactions on 31.2 (2009): 210-227, reveals the application of sparse coding for video compression in the context of face recognition.

[0007] From: Mallat, Stéphane. “A Wavelet Tour of Signal Processing.” Academic Press, 1999, it is further noted that sparse coding is known to be used in signal processing. It is also shown how data can be compressed using sparse coding. However, the methods revealed in the publication lead to a comparatively significant impairment of image quality.

[0008] The use of sparse coding for image compression is also known from: Emmanuel Candes, Justin Romberg, and Terence Tao: “Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information”, Applied and Computational Mathematics, Caltech, Pasadena, CA 91125, Department of Mathematics, University of California, Los Angeles, CA 90095, June 10, 2004, arXiv:math / 0409186v1 [math.NA] 10 Sep 2004.

[0009] Furthermore, US 2019 / 132586 A1 discloses a data processing device, a data processing method, and a program that improve a compression rate. It includes a first compression stage that compresses data using compressed sensing, and a second compression stage that compresses observation coefficients from the first compression stage using a different method than the one used in the first compression stage.

[0010] The present invention is based on the objective of providing an improved computer-implemented method for compressing a video stream, with the help of which video streams can be highly compressed, while preserving the respective image content as well as possible and which can be carried out quickly and efficiently.

[0011] According to the invention, this problem is solved by a computer-implemented method for compressing a video stream using sparse coding with the features of claim 1. Advantageous embodiments and further developments, as well as a vehicle for carrying out the method, are described in the dependent claims.

[0012] A generic computer-implemented method for compressing a video stream by means of sparse coding, wherein individual frames of the video stream are represented by a matrix multiplication of a basis function matrix with a sparse coefficient matrix, is further developed according to the invention by the following method steps: - Feeding the video stream into a convolutional neural network trained to generate a set of sparse coefficients for a single frame, and generating initial sparse coefficients for the first frame of the video stream by the convolutional neural network in the form of an initial sparse coefficient matrix; - Calculating a sparse coefficient difference matrix for the next frame of the video stream for all frames contained in the video stream, wherein the sparse coefficient difference matrix of a frame corresponds to the difference between the sparse coefficient matrices of the currently viewed frame and the previous frame in the video stream, wherein the sparse coefficient matrix of the currently viewed frame is determined by a sparse coding optimization based on the current frame, the previous frame, and the sparse coefficient matrix of the previous frame; and - Storing or exporting the sparse coefficient difference matrices calculated for all individual frames contained in the video stream in the form of a sparse coefficient stream.

[0013] The sparse coefficient stream has a significantly smaller storage size or file size compared to the video stream, so the video stream is compressed accordingly. Therefore, the sparse coefficient stream occupies less storage space on suitable computer-readable storage media and can be transmitted more quickly and completely over a data line for a given bandwidth. Alternatively, for a fixed data transmission duration, the bandwidth required to transmit the sparse coefficient stream can be reduced. This allows the underlying data line to be used for transmitting other data with a higher bandwidth. Due to the comparatively simple mathematical operations, the computer-implemented method according to the invention can be implemented efficiently.The initial sparse coefficient matrix is ​​determined using a convolutional neural network, which allows for the generation of particularly suitable initial sparse coefficients with minimal technical effort. This ensures that the image quality of each individual frame is preserved as much as possible. Furthermore, from iteration to iteration, the changes in the respective sparse coefficient difference matrices of successive frames in the video stream are minimized. This further reduces the resulting storage size of the sparse coefficient stream, as most entries are zero.

[0014] The computer-implemented method can be executed using common computing devices such as PCs, laptops, smartphones, tablet computers, servers, embedded systems like a "system-on-a-chip" (SoC), and the like. The video stream can be obtained from a computer-readable file or captured live from a camera.

[0015] The convolutional neural network (CNN) is also known as a "convolutional neural network." Convolutional neural networks have proven effective in the machine processing of image content. Through appropriate training with suitable training data, the convolutional neural network is enabled to derive appropriate sparse coefficients from individual frames, with the resulting set of sparse coefficients being provided in the form of a sparse coefficient matrix. The first frame of the video stream is fed to the convolutional neural network to determine these sparse coefficients, while the sparse coefficients for subsequent frames are determined through sparse coding optimization based on the current frame and the sparse coefficient matrix of the previous frame.Sparse coding optimization is a purely mathematical operation common in the context of sparse coding. It assumes that the differences between successive camera images are minimal. This allows the sparse coefficients of the current image to be calculated from the sparse coefficients of the previous image by comparing the current and previous frames.

[0016] Sparse coding aims to represent a signal or data set as a linear combination of a few basis functions, also called "kernels" or "atoms." These basis functions form an overdetermined basis set chosen to ensure the data representation is as economical, or "sparse," as possible. Mathematically, this can be expressed as follows, where bold letters represent tensors and matrices, respectively: X≈DA where: - X corresponds to the data tensor, i.e., the individual frames of the video stream, - D corresponds to the basis function matrix, which can also be called a "dictionary". The content of the dictionary is predefined by a developer; and - A represents the sparse coefficient matrix.

[0017] This approach uses the I0 norm to maximize the parsimony of the representation. This means minimizing the number of non-zero coefficients in matrix A to achieve the most efficient and compact representation possible. The I0 norm is defined as the number of non-zero elements in a vector or matrix. Therefore, sparse coding aims to maximize the number of zero coefficients, which contributes to a more economical representation of the data.

[0018] Mathematically, determining the initial sparse coefficients can be represented as: A0=CNN(X0) where X0 represents the first frame in the video stream and A0 is the first set of initial sparse coefficients estimated by the CNN. These sparse coefficients represent relevant and significant features of each frame. The CNN's estimation results in solutions that, while maintaining the same quality, are more sparsely populated and therefore more efficient in terms of storage space and processing speed.

[0019] Assuming that the current frame differs only marginally from the previous frame, it is possible, as already described, to determine the sparse coefficients A through a relatively small number of iterations. t to update the individual frame. The robust estimation of the initial sparse coefficients A0 by the CNN thus results in fewer sparse coefficients A. tin correlated individual images. This means that fewer differences need to be encoded for the same quality, making compression more efficient.

[0020] Then the difference ΔAt = A t - A t-1 The difference is calculated for the previous single image. This difference is then further processed using the same basis function matrix, i.e., a constant dictionary D, resulting in a new sparse coefficient matrix. In this context, this is referred to as the sparse coefficient difference matrix ΔAt. The sparse coefficients contained in the sparse coefficient difference matrix then represent the actual data to be processed, stored, and / or transmitted.

[0021] Due to the aforementioned difference calculation, the respective matrices consist primarily of zeros. Therefore, fewer entries in the respective matrices need to be updated from frame to frame. This facilitates the aforementioned extremely high data compression. Only the non-zero elements of the respective sparse matrices need to be stored. This method is particularly efficient for applications requiring real-time processing and high data compression rates, such as the transmission of video data in vehicle camera systems.

[0022] An advantageous further development of the method according to the invention provides that the convolutional neural network generates the initial sparse coefficient matrix through an iterative optimization process, in particular based on the Iterative Soft Thresholding Algorithm (ISTA). This allows even more suitable initial sparse coefficients to be determined quickly and with minimal technical effort.

[0023] According to a further advantageous embodiment of the method according to the invention, it is further provided that, starting from the initial sparse coefficient difference matrices, sparse coefficient matrices are calculated for all individual frames, respectively, and the individual frames of the video stream are reconstructed using the basis function matrix and the sparse coefficient matrices. By performing the mathematical operations in reverse order, the individual frames of the video stream can be reconstructed from the compressed data. This allows, for example, the video stream to be compressed on a first processing unit, sent via a data line to a second processing unit, and the original video stream to be reconstructed from the compressed data on the second processing unit.The reconstructed frames of the video stream are characterized by minimal loss of quality. This is particularly relevant for applications in vehicles, especially in the context of machine learning.

[0024] A further advantageous embodiment of the method according to the invention provides that the method is executed on a first computing unit in a vehicle. This allows for efficient use of the hardware installed in the vehicle. As already mentioned at the outset, the performance of the hardware installed in vehicles is usually more limited in comparison, which particularly favors the suitability of the method according to the invention for data processing in vehicles.

[0025] The video stream is preferably provided by a vehicle-integrated camera. This could be, for example, a mono or stereo camera, such as a so-called "multi-purpose camera" (MPC). Such cameras can capture the vehicle's surroundings. Generally, it could also be a camera that captures the vehicle's interior.

[0026] A further advantageous embodiment of the method according to the invention provides that the sparse coefficient current is transmitted via a fieldbus of the vehicle from the first computing unit to a second computing unit inside or outside the vehicle and / or is stored on a computer-readable storage medium of the first and / or second computing unit. The vehicle can have various fieldbus systems, such as a CAN bus, an Ethernet data line, or data lines based on Byteflight or LIN. Due to the reduced required transmission bandwidth, the corresponding data lines of the respective fieldbus system can be used efficiently. The sparse coefficient current can thus be transmitted quickly to the aforementioned second computing unit. The second computing unit can also be integrated into the vehicle.For example, it could be a central on-board computer, the control unit of a vehicle subsystem, a telematics unit, or similar. The sparse coefficient stream, and thus the video stream, can also be transmitted wirelessly to a server, such as a cloud server. For this purpose, the vehicle can be connected to the internet via mobile network using the aforementioned telecommunications unit.

[0027] According to a further advantageous embodiment of the method according to the invention, it is further provided that the individual frames of the reconstructed video stream are supplied to a driver assistance system in the form of input data. This can include all common driver assistance systems that rely on processing corresponding camera images. For example, environmental objects such as lane markings, other road users, traffic signs, traffic signals, and the like can be recognized in camera images. By applying established image recognition algorithms, also known as machine vision or "computer vision" (CV), corresponding environmental objects can be recognized and classified. Taking into account characteristic dimensions and / or stereoscopic vision, the relative distances of the environmental objects to the vehicle can also be determined.This can be used, for example, to derive automated control commands for the vehicle, enabling automated control of the vehicle's longitudinal and / or lateral acceleration. For instance, this can facilitate automatic lane changes, evasive maneuvers, or emergency braking. Automated or even autonomous vehicle control requires near real-time processing of the relevant data. Thanks to the video stream compression according to the invention, it can be transmitted even faster between processing units via the vehicle's fieldbus systems, ensuring time-critical processing by the respective driver assistance systems, even if these are provided by different processing units.

[0028] According to the invention, a vehicle comprising a first computing unit is further developed by configuring the first computing unit to execute a method described above. Thus, a vehicle employing the compression method according to the invention also falls within the scope of protection of the invention. The vehicle can be any road vehicle, such as a car, truck, van, bus, or the like. Generally, it could also be any rail vehicle, watercraft, or aircraft. In general, the application of the computer-implemented method according to the invention for compressing video streams is also possible in contexts unrelated to vehicles.

[0029] Further advantageous embodiments of the computer-implemented method according to the invention for compressing a video stream by means of sparse coding also result from the exemplary embodiments which are described in more detail below with reference to the figures.

[0030] This shows: Fig. 1 a schematic representation of the process of a method according to the invention for compressing a video stream by means of sparse coding; and Fig. 2 a schematic representation of the process from Fig. 1 for a specific application example.

[0031] Fig. Figure 1 shows a highly schematic representation of a video stream 1 generated by a camera or loaded from a computer-readable storage medium. The video stream 1 consists of a multitude of individual frames 2. These individual frames 2 can be described in the form of a tensor or a matrix, with the indices H, W, and T assigned to the individual matrix elements. For example, H can stand for height, W for width, and T for time. Thus, the individual entries of the matrix can correspond to pixels of a two-dimensional image. Each of these pixels can be assigned a color value, a gray value, and / or a brightness value. Furthermore, each individual frame 2 is provided with a timestamp in the form of the time "T" or "t", which defines the position of the respective individual frame 2 within the video stream 1. The first individual frame of the video stream 1 is labeled with the reference symbol 2.0.

[0032] The entirety of the individual images 2, and thus the video stream 1, is denoted here in tensor notation by X. X therefore corresponds to a data tensor that encompasses the individual images 2.

[0033] The first image 2.0 is now fed in step 101 to a suitably trained artificial neural network (CNN) in the form of a convolutional neural network (CNN). Due to its training, the CNN is able to process the first image 2.0 and generate initial sparse coefficients 5.0 from it. The initial sparse coefficients 5.0 are output as an initial sparse coefficient matrix 4.0. The sparse coefficient matrices 4 are represented in tensor notation by A.

[0034] The equation X ≈ DA applies, where D corresponds to a so-called basis function matrix 3. The basis function matrix 3, i.e., its contents, can be filled with suitable parameters before executing the method according to the invention. The basis function matrix 3 remains unchanged during the execution of the method. Depending on different data rates and memory requirements, suitable coefficients for the basis function matrix 3 can be determined manually by a developer or automatically by suitable algorithms.

[0035] Starting with the initial sparse coefficient matrix 4.0, the basis function matrix 3, and the known entries of the tensor X (i.e., the individual frames 2), a sparse coefficient difference matrix 6 is progressively generated for each individual frame 2 of the video stream 1. The sparse coefficient difference matrix 6 of an individual frame 2 represents the difference between the sparse coefficient matrices 4 of the currently considered frame and the previous frame in the video stream 1. The concatenated sparse coefficient difference matrices 6 then form a sparse coefficient stream 7, which corresponds to the compressed video stream 1.

[0036] First, the respective sparse coefficient matrices 4 for the individual images 2 must be determined. The sparse coefficients 5, which serve to populate the respective sparse coefficient matrices 4, are determined in step 102 by means of a sparse coding optimization based on the content of the current individual image 2, the previous individual image 2, and the sparse coefficients 5 of the previous individual image 2. The procedure starts with the first individual image 2.0, where the initial sparse coefficients 5.0 are known through the convolutional neural network CNN. Assuming an empty matrix, the initial sparse coefficients 5.0 can directly form the content of the sparse coefficient difference matrix 6 of the first individual image 2.0. The sparse coefficient difference matrices 6 are then calculated by taking the difference in step 103.

[0037] To reconstruct video stream 1 from sparse coefficient stream 7, starting with the first frame 2.0 or the initial sparse coefficients 5.0 and thus the tensor A0, the respective differences contained in the sparse coefficient difference matrices 6 are successively added to the sparse coefficient matrices 4. This makes it possible to calculate the underlying sparse coefficient matrix 4 for each frame 2. Once the sparse coefficient matrices 4 are known, video stream 1 can be reconstructed, since the basis function matrix 3 remains unchanged.

[0038] Fig.Figure 2 illustrates the procedure once again using a concrete example. The content of the individual images 2, or rather the tensors X representing the individual images 2, corresponds, for example, to a coded color value. Thanks to the arithmetic operations performed during sparse coding and the corresponding difference calculations, matrices of reduced size and reduced content are generated. In particular, the sparse coefficient difference matrices 6 are very small in their storage size due to the high number of zero entries.

[0039] The method according to the invention enables real-time video compression using sparse coding for video streams 1 generated by vehicle cameras. Besides its use in automotive applications, the described method is also suitable for data compression in mobile surveillance systems or for other video stream processing. It is characterized by high storage efficiency. By using sparse coding, video data is transformed in its form, thereby reducing the required storage space. By storing only significant features of the individual frames 2, the storage space requirement is considerably reduced. Additionally, differential coding enables a further reduction in data size, as only the changes between successive individual frames 2 are recorded.

[0040] Furthermore, this allows for bandwidth reduction in data transmission. The efficiency of video data compression enables the use of lower bandwidth for data transmission, or, with the same bandwidth, allows for data transmission in a shorter time, which is particularly relevant when limited resources are available in the underlying information technology system.

[0041] The method according to the invention allows real-time processing of the underlying data. Thus, corresponding individual frames 2 from video streams 1 can be compressed directly during their recording and, if required, immediately transmitted via appropriate data lines or otherwise processed. This is particularly critical for applications that require a fast response time, such as in the context of automated or autonomous driving.

[0042] The method according to the invention is also characterized by a high degree of flexibility. As already described, the entries of the basis function matrix 3 can be adapted to the specific requirements depending on different external boundary conditions. However, during the execution of the method, i.e., during the transformation and inverse transformation of a specific video stream 1, the contents of the basis function matrix 3 remain constant.

[0043] The method according to the invention also allows for compliance with high quality standards. Despite the high compression rate, the method enables high video playback quality, which is essential for the visual analysis and evaluation of camera images generated by vehicle camera systems. The reconstruction of the original data from the compressed data results in minimal loss of quality, thus ensuring the reliability and accuracy of the visual information.

Claims

[1] Computer-implemented method for compressing a video stream (1) using sparse coding, wherein respective frames (2) of the video stream (1) are mapped by matrix multiplication of a basis function matrix (3) with a sparse coefficient matrix (4), characterized by the following procedural steps: - Feeding the video stream (1) to a convolutional neural network (CNN) trained to generate a set of sparse coefficients (5) for a single frame (2), and generating initial sparse coefficients (5.0) by the convolutional neural network (CNN) for the first single frame (2.0) of the video stream (1) in the form of an initial sparse coefficient matrix (4.0); - Calculating a sparse coefficient difference matrix (6) for the next frame of the video stream (1) for all frames (2) contained in the video stream (1), wherein the sparse coefficient difference matrix (6) of a frame (2) corresponds to the difference of the sparse coefficient matrices (4) of the currently viewed frame and the previous frame in the video stream (1), wherein the sparse coefficient matrix (4) of the currently viewed frame is determined by a sparse coding optimization based on the current frame (2), the previous frame (2), and the sparse coefficient matrix (4) of the previous frame (2); and - Storing or outputting the sparse coefficient difference matrices (6) calculated for all individual frames (2) contained in the video stream (1) in the form of a sparse coefficient stream (7). [2] Method according to claim 1, characterized by, that the convolutional neural network (CNN) generates the initial sparse coefficient matrix (4.0) through an iterative optimization process, in particular based on the “Iterative Soft Thresholding Algorithm”. [3] Method according to claim 1 or 2, characterized by , that from the sparse coefficient difference matrices (6) and starting from the initial sparse coefficient matrix (4.0) , sparse coefficient matrices (4) are calculated for all individual frames (2) and, respectively, sparse coefficient matrices (4) are reconstructed using the basis function matrix (3) and the sparse coefficient matrices (4). [4] Method according to any one of claims 1 to 3, characterized by a version on a first computing unit in a vehicle. [5] Method according to claim 4, characterized by , that the video stream (1) is provided by a vehicle-integrated camera. [6] Method according to claim 4 or 5, characterized by, that the sparse coefficient stream (7) is transmitted via a fieldbus of the vehicle from the first computing unit to a second computing unit inside or outside the vehicle and / or is stored on a computer-readable storage medium of the first and / or second computing unit. [7] Method according to any one of claims 4 to 6 and claim 3, characterized by , that the individual frames (2) of the reconstructed video stream (1) are supplied to a driver assistance system in the form of input data. [8] Vehicle comprising a first computing unit, characterized by that the first computing unit is configured to execute a method according to one of claims 4 to 7.

Images