SYSTEMS AND METHODS FOR EFFICIENT AND LOSSILESS COMPPRESSION OF CAPTURED RAW IMAGE DATA

Abstract

A procedure encompassing: Receiving raw image data from an image acquisition device (110), wherein the raw image data includes pixel values ​​expressed in a first color space; Splitting the pixel values ​​into a base layer part (791, 793) and an extension layer part (792, 794), where the extension layer part contains the least significant bits of the pixel values; Reconfiguring the pixel values ​​of the base layer part (791, 793) as a pseudo-color space expression, wherein the pseudo-color space expression is compatible with a second color space compression format and differs from the first color space; Compressing the pseudo-color space expression to create compressed data, and Storing the compressed data and the extension layer portion as pixel values.

Description

FIELD OF THE INVENTION

[0001] The various embodiments of the present invention relate to the field of data storage. Systems and methods for efficient data compression are presented. BACKGROUND

[0002] Today, numerous technologies related to electronic data processing, such as computers, video equipment, and telephone systems, are used in various fields of business, science, and entertainment to increase productivity and reduce costs. Several of these applications require the capture and storage of large amounts of video or image data (e.g., for historical purposes, for subsequent analysis, etc.). Reliable and fast data storage is generally a key requirement for the proper processing and use of this information. However, several factors can affect data storage, such as limitations in memory access speed or storage capacity.

[0003] Storing large amounts of data can be problematic and costly, often requiring significant storage resources. However, storage resources are often limited by various factors such as cost, device dimensions, space, power supply, etc. Storage operations for large amounts of data can also be relatively time-consuming. To address these limitations, data compression techniques are sometimes used. However, data compression can also be complex and costly. The development and implementation of systems and methods for compression encoding and decoding can be associated with significant costs. Conventional data compression approaches often attempt to reduce these problems and costs by using lossy compression techniques.Unfortunately, lossy compression results in some of the original raw data being lost or distorted during processing. Conventional lossless compression approaches usually have limitations (e.g., the data must be appended by an extra bit due to color space conversion, which negatively impacts the compression ratio).

[0004] Large datasets are required in a variety of applications or disciplines, such as medical imaging, national security, autonomous vehicles, interactive video analytics, etc. Due to the nature of these applications, highly accurate, lossless, and fast data storage is required. Lossless compression must ensure that the data retrieved from storage can be decoded and restored to match the original raw data before compression, without mismatches or loss of information. Compression techniques that are only considered lossless for normal perception may not be sufficient. Therefore, in many cases, a user is interested in lossless compression coding even if it does not deliver such high compression ratios.US 2017 / 0 028 935 A1 describes a vehicle with a video system, and US 2003 / 0 179 938 A1 discloses a method for signal compression with bit division according to significance. US 2017 / 0 104 974 A1 discloses a method for processing video signals in which some low-order bits are deleted to reduce the bit layers. One object of the invention is to improve data compression without losing too much information. SUMMARY

[0005] This object is achieved by the features of the independent claims. A system according to claim 18 and methods according to claims 1 and 9 for efficient and lossless compression of acquired raw image data are presented.

[0006] The image information received from a camera or image sensor is divided into two parts or sections, among other things. One part of the image data is rearranged, compressed, and stored in memory. The other part is stored without rearranging or compression. In an example implementation, a part of the original image values ​​captured by a camera or sensor is compressed and stored as a base part, while another part of the image values ​​is uncompressed and stored as an extension part. The values ​​in the base part can be rearranged or reconfigured to be compatible with efficient compression techniques. Storing an uncompressed extension part and a compressed base part of the original image values ​​enables efficient and effective, lossless, and accurate storage of the data with a smaller footprint than conventional systems.Through the targeted rearrangement of the data, which is compatible with powerful compression methods, the information can also be compressed efficiently and faster than with conventional methods.

[0007] In one embodiment, the original data of an image is collected and arranged in a first configuration or format. This configuration may include color value types (e.g., red (R), blue (B), green (G), clear (C)). This data is then converted into a second configuration. In the second configuration, the image data is divided into a base part that can be easily compressed (e.g., repeating or redundant values) and an extension part that cannot be compressed so easily (e.g., unique or less repeated values). The color values ​​in the extension part are stored in memory. The color values ​​(e.g., R, B, G, C) in the base part are then specifically fitted into a third configuration. The third configuration can then be compatible with or tailored to a specific compression type (e.g., H.264, H.265, Av1).In one variant, the third configuration can, unlike conventional configurations or formats, contain slots or positions for other color value types (e.g., YUV, luminance, chrominance). The slots or positions of the third configuration are then filled with the original color value types (e.g., red (R), blue (B), green (G), clear (C)) instead of the other color value types (e.g., YUV, luminance, chrominance). The color values ​​of this third configuration from the base part are then compressed and stored in memory.

[0008] Another embodiment of the invention receives the raw image data from an image acquisition device in the form of pixels with the original color space values. The pixel values ​​are divided into a base layer portion and an extension layer portion, with the pixel values ​​in the base layer portion being reconfigured as pseudo-color value expressions. These pseudo-color values ​​are compatible with a specific compression format into which the reconfigured color values ​​of the base layer portion are compressed. The division may be based on various factors, including an analysis of the compression benefit threshold between the base layer portion and the extension layer portion. Reconfiguring the base layer portion may also involve dividing the base layer portion based on repeated components within the raw data and forming video frames from the repeated components in the base layer.The raw image data may be in a Red, Clear, Clear, Blue (RCCB) format, and the reconfigured input for compression of the base layer portion then contains red, clear, and blue components compatible with a YUV luminance-chrominance format. The threshold between the enhancement layer and the base layer may be based on a balance between the benefits of compressing the data versus the cost of compressing the data. Thus, the threshold may be a point where the benefits of compressing the data outweigh the cost of compressing the data. A base layer portion may contain more bits than the enhancement layer portion. The base layer portion may contain the high-order bits, with the lower-order bits then contained in the enhancement layer portion. The base layer video images may be the input to an H.264-compliant encoder.In one embodiment, the method further comprises training a deep learning network or deep neural network (TNN) for autonomous vehicle control systems, wherein the training uses data from the base layer part and the extension layer part.

[0009] In one embodiment, the reconfiguration includes splitting the base layer portion based on multiple elements within the raw data, forming a base layer video frame from the multiple elements, and arranging the data associated with the base layer video frames in a YUV color scheme compatible format. The multiple elements may include a red element, a blue element, a first clear element, and a second clear element, which together correspond to the pixel values. In one embodiment, the data of the base layer video frames is arranged in a compression-compatible format for the YUV color space by expressing the pixel values ​​corresponding to the RCCB color space as values ​​in a pseudo-YUV color space, instead of derived YUV color space values ​​converted from the RCCB color space values.The base layer video images may include a clear blue (CB) video image (formed by the blue element and the first clear element) and a clear red (CR) video image (formed by the red element and the second clear element). The base layer video images may correspond to various pseudo-YUV formats (e.g., a pseudo-YUV444 format, a pseudo-YUV420 format, etc.). Part of the base layer video images may correspond to a pseudo-Y component, and zero-plane frames are used as pseudo-UV components.

[0010] In one embodiment, a first portion of the blue element may correspond to a chrominance U input and a first portion of the red element may correspond to a chrominance V input, while a second portion of the red element, a second portion of the blue element, a first clear element, and a second clear element may correspond to a luminance Y input. The base layer video images may be compatible with a pseudo-YUV420 format, where: a pseudo-YUV width is equal to an input width, and a pseudo-YUV height is equal to (input height + 2) / 3 * 2, where input height is a value indicating the number of pixels in the height of the pseudo-YUV component.In an exemplary implementation, a 420 compatible height minus base layer part compression input divided by 2 is a first number of lines from the red element and the blue element contained in a luminance Y format section, and a second number of remaining lines from the red element and the blue element are contained in the chrominance U and V format sections.

[0011] In one embodiment, a system comprises: - An image capture device that captures raw image data, wherein the raw image data includes pixel values ​​expressed in a first color space. - A processing unit that manages storage of the raw image data, wherein a portion of the pixel values ​​is arranged as a pseudo-color expression compatible with a color compression format. - A data store. The data store stores a first portion of the pixel values ​​as a compressed version of the pseudo-color expression and a second portion of the pixel values ​​as raw image data in an uncompressed format. In one embodiment, the first portion is an extension portion and the second portion is a base portion. In one embodiment, the image capture device, the processing unit, and the memory are located in a vehicle, and the raw image data corresponds to scenes encountered during vehicle operation.The storage can be portable and selectively removable from the vehicle. SUMMARY

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification and in which like numerals represent like elements, illustrate the embodiments of the present publication and, together with the description, serve to explain the principles of the specification. The drawings are not to scale unless otherwise indicated. Fig. is a block diagram of an exemplary vehicle system according to one embodiment. Fig. is an exemplary block diagram of information flow and processing associated with TNN training according to one embodiment. Fig. is an exemplary block diagram with captured raw image data according to one embodiment. Fig. is an exemplary block diagram of raw image acquisition data having green luminance, first type red chrominance, and second type blue chrominance according to one embodiment. Fig. is a block diagram of an exemplary input and output data flow of a compression component according to one embodiment. Fig. is an exemplary block diagram of raw image capture data having clear luminance, first type red chrominance, and second type blue chrominance according to one embodiment. Fig. is a block diagram of an exemplary base / expansion configuration according to one embodiment. Fig. is a block diagram of another exemplary base / extension configuration according to one embodiment. Fig. is a block diagram of another exemplary base / extension configuration according to one embodiment. Fig. is an exemplary block diagram with captured raw image data according to one embodiment. Fig. are traditionally captured RCCB raw image values ​​with conventional YUV conversion / compression. Fig. is a block diagram in the conventional YUV420 format. Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base part into a pseudo-YUV444 format according to one embodiment. Fig. is a block diagram of an exemplary compression input stream in a pseudo-YUV444 format according to one embodiment. Fig. is a block diagram of another exemplary compression input stream in a pseudo-YUV444 format according to one embodiment. Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base part into a pseudo-YUV420 format according to one embodiment. Fig. is a block diagram of an exemplary compression input stream in a pseudo-YUV444 format according to one embodiment. Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base part into another pseudo-YUV420 format according to one embodiment. Fig. is a block diagram of an exemplary compression input stream in a pseudo-YUV444 format according to one embodiment. Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base part into a pseudo-YUV422 format according to one embodiment. Fig. is a block diagram of an exemplary compression input stream in a pseudo-YUV444 format according to one embodiment. Fig. is a block diagram of the compression process for data storage according to one embodiment. Fig. is a flowchart of an exemplary method according to one embodiment. Fig. is a block diagram of a computer system according to one embodiment. DETAILED PRESENTATION

[0013] Reference will now be made in detail to the preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Although the invention will be described with reference to these preferred embodiments, it is to be understood that the invention is not limited to these embodiments. On the contrary, the invention also includes alternatives, variations, and equivalents which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure the current invention with unnecessary aspects.

[0014] The embodiments of the presented systems and methods can provide efficient and effective data storage. In one embodiment, the captured raw image data is divided into a base layer portion and an enhancement layer portion, wherein the captured values ​​of the base layer portion are reconfigured from the raw data format into a compression encoder input format. The compression encoder input format contains the captured raw image data of the original color space values ​​configured in a pseudo-color space encoder input format. In an exemplary implementation, the division into a base layer portion and an enhancement layer portion can be based on various factors (e.g., a compression benefit analysis of the base layer portion, the type of compression scheme, the time required to perform the compression operations, etc.).Compression can reduce the amount of data written and stored on the storage medium compared to the amount of raw image data captured. In one embodiment, the compression is lossless.

[0015] In one embodiment, the captured raw image data is compressed in real time before being written to memory. Lossless compression of the raw image data can reduce the amount of data written to and stored on the storage medium. Reducing the amount of data written to memory minimizes bottlenecks associated with the storage medium's write speed and enables fast, real-time data acquisition.

[0016] In one embodiment, lossless compression is used in the development of autonomous vehicle control systems. Autonomous vehicle control systems require the collection of large amounts of high-quality datasets gathered during vehicle operation, especially large amounts of raw video data. These datasets are typically used for training and tuning deep neural networks developed for autonomous driving. The data can be acquired by installing multiple camera sensors on the vehicle. In an exemplary implementation, multiple sensors on a vehicle can be used to capture the raw image data of a full 360-degree view of the environment. These sensors can generate large amounts of raw image data every second, and this image information must be written to attached storage systems.

[0017] Fig. is a block diagram of an exemplary vehicle system (100) according to one embodiment. The vehicle system (100) includes the sensor (110), the computer system (150), and a portable removable storage device (140). In one embodiment, the sensor (110) is an image capture device (e.g., camera, infrared sensor) that captures the raw image data and forwards it to the computer system (150). The computer system (150) performs the operations of the capture app (130). In one embodiment, the operations of the capture app (130) include compressing at least a portion of the captured raw image data. Both the compressed capture data (145) and the uncompressed capture data (147) are stored in a portable removable storage device (140) (e.g., SSD, removable hard drive).

[0018] In one embodiment, other information (e.g., speed, distance, time) associated with vehicle operation may also be processed by the computer system (150) and stored in the portable removable storage device (140) (not shown). This other information may be associated with or projected onto the captured raw image data.

[0019] The portable storage device can be removed from the vehicle, and the data can be uploaded to a separate storage server. Various types of data analyses can be performed using the information stored on the portable storage device. The data can be used by various applications (e.g., artificial intelligence applications, machine learning applications). These applications can, for example, be machine learning methods (e.g., neural networks) with multiple layers (deep neural networks). The collected data can therefore be used to train a deep neural network (TNN) for autonomous vehicle applications.

[0020] Fig. is an exemplary block diagram of information flow and processing associated with TNN training according to one embodiment.

[0021] In one embodiment, a portable data storage device (210), a storage server (230), and a processing server (240) are located at a central location. Vehicles with systems similar to system (100) that capture raw image data locally are typically located far away from the central location. The portable data storage device (210) can be removed from a vehicle, similar to the removable storage device (140) from the vehicle (100), and can contain compressed raw image acquisition data. The compressed raw image acquisition data (221) is uploaded to the storage server (230). The compressed raw image acquisition data in the storage server (230) is then retrieved by the processing server (240). The uncompressed raw image acquisition data (222) can also be uploaded to the storage server (230) and then retrieved by the processing server (240).In an exemplary implementation, the compressed raw image acquisition data is decompressed or decoded by the processing server (240) and combined with the uncompressed raw image acquisition data. The processing server (240) may perform various processing operations, including TNN training. In an exemplary implementation, video acquisition data associated with the image of an object may be tagged with an identifying label (e.g., tree, pedestrian, car, truck) in a tagging process (241). Both tagged information from the tagging process (241) and untagged data may be passed from the storage server (230) to the TNN training process (242). The results of the TNN training process (242) may be fed into the prototxt process (243). In one embodiment, other information (e.g.,Speed, distance, time) coupled with vehicle operation are passed to the processing server (240) (not shown) and used in various processing operations, including TNN training.

[0022] The portable data storage (210) and the storage server (230) can, of course, also contain other data, and the processing server (240) can perform other processing besides TNN training. In one embodiment, a single server performs both the functions of the storage server (230) and the processing server (240). In one embodiment, multiple distributed servers perform operations of the storage server (230) and multiple distributed servers perform operations of the processing server (240).

[0023] It is recognized that TNN training is not limited to autonomous vehicle control. In one embodiment, TNN training is also used in automated product testing and analysis when the observer is a machine. In an exemplary implementation, object details can be captured as raw images and stored for analysis. The captured raw image data is input to a deep learning network. Deep learning networks can be compatible with a variety of sensor raw data acquisition formats (e.g., Bayer format, BCCR, BGGR, etc.). In one embodiment, camera sensors capture raw image data in Bayer format, which is used as input to the processing components of a deep learning network. However, the captured raw image data is typically incompatible with the input formats for compression.

[0024] In one embodiment, a color image sensor array is used to capture the raw image data. The sensor array may include a plurality of sensor elements. A sensor element may be used to capture the raw image data. In an exemplary implementation, a sensor element, or sensel, is a combination of a selectively transmissive filter with sensitivity to specific light properties and a charge-coupled device. The sensitivity to specific light properties may relate to a specific color (e.g., blue, red, green, white, clear).

[0025] The elements can be arranged in an array of luminance sensor elements and chrominance sensor elements. In one embodiment, the detected luminance is the color vector that is the main portion of the luminance information from the array, and the detected chrominance refers to other color vectors that form a basis for constructing an image. The luminance sensor element is the dominant type of sensor element in the array, while a chrominance element is the non-dominant sensor element in the array. The number of luminance sensor elements is greater than the number of a specific type of chrominance sensor elements. In one embodiment, the total number of different types of chrominance elements can be greater than or equal to the number of luminance sensor elements, as long as the number of each type of chrominance sensor element is less than that of luminance sensor elements.In one embodiment, luminance is considered as a brightness component and chrominance is considered as a color component.

[0026] In one embodiment, a mosaic of individually narrow sensor elements can have a relatively wide common range of light sensitivity. In one embodiment, luminance elements detect a first color, the first type of chrominance elements detect a second color, and the second type of chrominance elements detect a third color. In one embodiment, a raw image data detection array includes green luminance detection elements, red chrominance detection elements, and blue chrominance detection elements. The distribution of filter types in the mosaic depends on the array pattern of the luminance sensor elements and chrominance sensor elements. In one embodiment, the luminance elements at each element position are arranged to create a dominant, uniform pattern in two perpendicular directions (e.g., horizontal and vertical), with the chrominance elements filling the remaining element positions.

[0027] In one embodiment, a group of sensor elements corresponds to a pixel. In an exemplary implementation, the sensor elements for a pixel consist of two luminance sensors, a chrominance sensor of the first type, and a chrominance sensor element of the second type. Fig. is a block diagram of exemplary sensels (300) with the corresponding raw image data according to one embodiment. The sensels consist of the first-type chrominance sensor elements 311, 313, 331, and 333, the second-type chrominance sensor elements 322, 324, 342, and 344, and the luminance sensor elements 312, 314, 321, 323, 332, 334, 341, and 343. It can be seen that the elements for detecting luminance are more dominant (i.e., more frequently represented) than the first-type chrominance elements. The elements for detecting luminance are also more dominant (i.e., more frequently represented) than the second-type chrominance elements.

[0028] In one embodiment, a luminance sensor element corresponds to a green light characteristic filter, a chrominance sensor element of the first type corresponds to a red light characteristic filter, and a chrominance sensor element of the second type corresponds to a blue light characteristic filter. In one embodiment, the Y, C1, and C2 values ​​in Fig. the green luminance values, the red chrominance values ​​of the first type and the blue chrominance values ​​of the second type. Fig. is an exemplary block diagram of raw image acquisition data (400) with sensels for green luminance, first-type red chrominance, and second-type blue chrominance, according to one embodiment. The sensels consist of red chrominance sensor elements 411, 413, 431, and 433, blue chrominance sensor elements 422, 424, 442, and 444, and green sensor elements 412, 414, 421, 423, 432, 434, 441, and 443. In an exemplary implementation, the mosaic of green luminance, first-type red chrominance, and second-type blue chrominance sensors corresponds to a red / green / green / blue (RGGB) pixel array. In one embodiment, for each RGB pixel, there are two green luminance elements, one red chrominance element of the first type, and one blue chrominance element of the second type. In an exemplary implementation, the red chrominance element 411, the blue chrominance element 422, and the green elements 412 and 421 form a pixel (490).

[0029] Typically, sensor elements generate a very large amount of raw image data. In an exemplary use case, there are multiple camera sensors (3, 6, 8, etc.) that must capture data in 12-bit Bayer format (RCCB / RGGB) in real time and store it in a lossless format. In one embodiment, six camera sensors capture a 1080-pixel input signal (1920x1080, 16-bit Bayer format) at a rate of 30 frames per second. In an exemplary implementation, the amount of data generated per second corresponds to 746 MB (1920x1080 x 2 x 30 x 6). Thus, one hour of continuous acquisition generates approximately 2.5 TB of data. In one embodiment, data compression is used to reduce the amount of data. A video codec with a compression factor of 2.5 reduces the size from 2.5 TB to 1 TB.This compression can help reduce storage space, increase recording time, reduce bandwidth costs, or even increase the number of sensors to capture more detail.

[0030] Fig. is a block diagram of an exemplary input data flow of a compression pipeline according to one embodiment. The input stream (510) is fed to the compression encoder (520), which performs the operations for compressing the data of the input stream (510) and outputs compressed data (530). The compressed data (530) is stored in memory (540) in a write operation. The compressed data (530) can be retrieved from the memory (540) and during a read operation by input to the decoder (550). The decoder (550) performs the decompression operations on the compressed data (530) and outputs the output stream (570). In an embodiment with lossless encoding / decoding operations, the input stream (510) and the output stream (570) contain the same data. In one embodiment, the captured raw image values ​​of the sensor element form the input to the compression encoder (520).

[0031] Encoder performance is particularly important for use cases where real-time capture is required. There are also use cases where multiple encoder captures are running simultaneously and very fast encoding is required. Encoding is typically more computationally intensive than decoding, so it's important to carefully select the encoder's characteristics. There are several factors to consider when selecting a compression and encoding scheme and features.

[0032] In one embodiment, the redundancy of the acquired raw image data is utilized for compression operations to reduce the data volume. There are different types of redundancy that can be used, such as spatial redundancy (intra-frame compression), temporal redundancy (inter-frame redundancy), and so on. Inter-frame compression generally provides better compression efficiency because an encoder can utilize the temporal redundancy within the frames. Intra-frame compression encodes each frame independently, without references to other frames, allowing intra-frames to be accessed individually without having to decode a sequence of frames.

[0033] In one embodiment, the number of source input symbols encoded or mapped to the number of output symbols or bits affects the loss / distortion of the data retrieved from memory. Fixed-length encoding encodes a fixed number of source symbols to a fixed number of output symbols. Fixed-length encoding can be susceptible to information loss. Variable-length encoding maps a source symbol to a variable number of bits. In an example implementation, variable-length encoding enables symbol-faithful reading of sources from memory without information loss.

[0034] In conventional approaches, variable-length coding can be a bottleneck in lossless coding. In one embodiment, context-adaptive variable-length coding (CAVLC) is faster than context-based adaptive binary arithmetic coding (CABAC). In one embodiment, better compression rates can be achieved with CABAC. In an example implementation, CABAC is used when compression efficiency is more important than processing speed, and CAVLC is used when processing speed is more important than compression rate. In one embodiment, CAVLC is used to keep pace with the rapid processing demands associated with large amounts of data that must be collected from multiple sensors and processed in real time.

[0035] In one embodiment, some codec functions are not required. A lossless storage system and method eliminates the need for quantization functions that would otherwise cause problems with lossy compression. In one embodiment, operations that do not contribute much to compression rates can also be bypassed.

[0036] It is generally accepted that different numbers of bits can be used to express the raw image data of the captured element. In various applications (e.g., autonomous vehicles, medical procedures, security systems), it is often important that the image acquisition data be very accurate and granular. Using a larger number of bits to represent the captured raw image values ​​of the captured element can ensure greater accuracy and granularity. However, using a larger number of bits to express the captured raw image values ​​of the captured element can significantly increase the coding processing and complexity, resulting in slower coding operations compared to a smaller number of input bits for compression.In one embodiment, larger bit compression or large codec inputs require improved encoder profiles, and it may be difficult to achieve real-time performance with a larger bit encoder.

[0037] In one embodiment, a measured element raw image value is expressed in 12 bits. The 12-bit values ​​are assigned to the sensels in a mosaic corresponding to a red / clear / clear / blue (RCCB) pixel array. In one embodiment, for each RCCB pixel, there are two clear luminance elements, one first-type red chrominance element, and one second-type blue chrominance element. Fig. is an exemplary block diagram of raw image capture data (600) having clear luminance, first type red chrominance, and second type blue chrominance according to one embodiment. The raw image acquisition data consists of the red chrominance sensor elements 611, 613, 631, and 633, the blue chrominance sensor elements 622, 624, 642, and 644, and the clear sensor elements 612, 614, 621, 623, 632, 634, 641, and 643. In one embodiment, 12 bits are used to express the raw image data of each acquired element (e.g., 12 bits are used to express the red chrominance element 611, another 12 bits are used to express the red chrominance element 613, another 12 bits are used to express the blue chrominance element 624, etc.). In one embodiment, the 12 bits are sufficient for accuracy, but problematic for encoders that are required to perform compression processing in a real-time environment.

[0038] It is recognized that for image acquisition applications, the lower-significance bits of each sensel can be very different, and there may be little or no similarity in these bits. Therefore, low-significance bits typically do not contribute much to redundancy compression or coding. In an exemplary implementation, the last remaining bits effectively have a negative impact on the compression ratio results. In one embodiment, the low-significance bits are treated differently than other bits in the raw image acquisition data.

[0039] In one embodiment, the acquired element values ​​are divided into two groups of values. This can be thought of as splitting an input frame into two layers. The first layer is an enhancement layer, which contains the low-order bits of a sensel. The second layer is a base layer, which contains the remaining bits of a sensel. In one embodiment, a base layer contains the most significant 8 bits and an enhancement layer contains the least significant 4 bits of each sensel. The enhancement layer can be stored directly in memory as uncompressed bits of the raw image acquisition data, without the need for processing through a compression encoder, since there is little or no similarity between the values. In one embodiment, the base layer can be used to provide the values ​​for the input stream of a compression component.In one embodiment, the base layer may be encoded using a lossless compression technique. In an exemplary implementation, splitting the acquired raw image data into a base layer portion and an enhancement layer portion enables increased speed and encoding performance without compromising encoding efficiency.

[0040] Fig. is a block diagram of an exemplary base / extension layer configuration according to one embodiment. In one embodiment, a pixel includes a red sensor element value 711, a first clear sensor element value 721, a second clear sensor element value 712, and a blue sensor element value 722. The red sensor element value 711, the first clear sensor element value 721, the second clear sensor element value 721, and the blue sensor element value 722 may correspond to the red sensor element value 611, the first clear sensor element value 612, the second clear sensor element value 621, and the blue sensor element value 622 in Fig. Each sensor element value 711 to 722 comprises 12 bits, which are labeled as bit 0 to bit 11, respectively. The respective bits 0 to 3 of the sensor element values ​​711 to 722 are divided into the extension layer 792, and the remaining bits 4 to 11 are divided into the base layer part 791. The data in the base layer 791 can be fed into a compression pipeline as an input stream, and the data in the extension layer 792 is stored directly in memory.

[0041] In one embodiment, threshold 790 defines the division or separation between the extension portion or layer 792 and the base portion or layer 791. In an example implementation, the configuration or location of threshold 790 may be based on a variety of factors. In one embodiment, the respective likely compression ratios and storage benefits that can be achieved by compressing the respective sizes of the extension portion and the base portion may be used as a factor in determining the position of threshold 790. As indicated above, due to the low probability of redundancy in the low-order bits, there is a relatively low probability of achieving a favorable compression ratio that saves storage space.In one embodiment, the threshold 790 is variable and can move right or left to increase or decrease the number of bits in the base layer.

[0042] Fig. is a block diagram of an exemplary base / extension layer configuration with the threshold shift to the right, according to one embodiment. Since the threshold 790 moves to the right to include fewer low-order bits in the extension layer portion 794 compared to the extension layer portion 792 in Fig. the amount of uncompressed data stored in memory is reduced. However, as the threshold 790 moves to the right, more low-order bits are moved into the base layer part 793 compared to the base layer part 791 in Fig. included, the probability of redundancies is lower, and the compression ratio in the base layer 793 will decrease compared to the base layer 791. In an exemplary implementation, the threshold 790 moves slightly to the right to the position between the respective bits 3 and 2, so that bits 11 to 3 are contained in the base layer and bits 0 to 2 are contained in the extension layer 794. Since the bit 3 position is now in the base layer part 793 and there is likely to be a significant variance of the values ​​at the bit 3 position of the respective different sensor elements (e.g. 711, 712, 721, 722, etc.), the probability of duplicates is lower, which improves compression.

[0043] Fig. is a block diagram of an exemplary base / extension layer configuration with the threshold shifted left, according to one embodiment. When the threshold 790 is shifted left to include higher-order bits in the extension layer portion 798, more data is added to the uncompressed memory sections, thus consuming more storage space. When the threshold 790 is shifted left to remove more lower-order bits from the base layer portion 797, the redundancy between the bits is likely to increase, thereby facilitating compression.

[0044] The storage time delay introduced by compression processing is also a factor to consider. If threshold 790 is shifted to the left to decrease the number of bits in the base layer portion 797 that are fed into the compression pipeline, the faster and less complex the compression operations will be (i.e., also relatively low delays). However, if threshold 790 is shifted to the right to increase the number of bits in the base layer portion that are fed into the compression pipeline, the slower and more complex the compression operations will be (i.e., also relatively larger delays). The type of compression employed can also be a factor in selecting the position of threshold 790.

[0045] Fig. is a block diagram of an exemplary raw data storage flow according to one embodiment. After capture, the raw image data is divided into the base layer 810 and the enhancement layer 850. The data in the enhancement layer (850) is stored in memory (840) in a write operation. The base layer input stream (810) is fed into the compression pipeline (820), which performs the operations to compress the data of the input stream (810) and output compressed data (830). The compressed data (830) is stored in memory (840) in write operations. In one embodiment, lossless encoding / decoding operations are used, and the compressed data (830) can be retrieved from memory (840) during the read operation and decoded such that the input stream and the output stream contain the same data.

[0046] In one embodiment, a compression pipeline is compatible with various compression standards (e.g., H.264, H2.65, AV1). Many of the standards operate on data in a specific color space format, which may differ from the respective color space format of the raw image data. In one embodiment, an image sensor captures the raw image data in RCCB format, while the compression of the data is performed in a YUV format. In an exemplary implementation, Y is a derived or calculated luminance value, U is a derived chrominance value of the first type, and V is a derived chrominance value of the second type. It is recognized that the measured luminance values ​​and chrominance values ​​corresponding to those in the Fig. similar, can differ significantly from the derived or calculated luminance and chrominance values.

[0047] In one embodiment, a distinction is made between the acquired luminance and chrominance values ​​and the derived luminance and chrominance values. Many conventional systems use a derived YUV luminance / chrominance color space, or Y'UV luma / chroma color space, in the pipeline processing of color image data. The term YUV is often used to describe file formats encoded using the YCbCr format. The UV representation of chrominance in conventional systems is typically different from linear R and B signals / values ​​because U and V are color difference signals / values. Conventional YUV / Y'UV / YCbCr color spaces typically result in some loss when converting to and from an RGB color space. Fig. are traditionally acquired RCCB raw image values ​​with conventional YUV conversion / compression. The acquired RCCB raw values ​​(910) are passed through an interpolation process / algorithm (920) to generate the RGB values ​​(930). In one embodiment, the RGB values ​​are linear, and in another embodiment, the RGB values ​​are gamma-adjusted. The RGB values ​​(930) are passed through a conversion process / algorithm (940) to generate YUV values ​​(350). In one example, the weighted values ​​of R, G, and B are summed to generate Y', while U and V are calculated as the scaled difference between Y' and B and R values. In standard definition television (SDTV) and the BT.601 standard, the conversion is defined by: a. Y'=0.2899R+0.587G+0.114B b. U≅0.492(B-Y') c. V≅0.877(R-Y').

[0048] While conventional lossless conversions to overcome data loss are possible in some image compression applications, they typically increase the dynamic range of the input by at least one additional bit. These additional bits tend to negate or reduce compression rates.

[0049] Fig. is a block diagram in conventional YUV420 format. A pixel array (1001) contains pixels 1011 to 1016, 1021 to 1026, 1031 to 1036, and 1041 to 1046 with corresponding RGB pixel values. The RGB pixel values ​​are converted to derived YUV values. There is a Y value for each pixel in the pixel array (1001), a U value for all four pixels in the pixel array (1001), and a V value for all four pixels in the pixel array (1001). In an exemplary implementation, the values ​​Y1, Y2, Y7 and Y8 are assigned to pixels 1011, 1012, 1021 and 1022, the value U1 is divided among or corresponds to pixels 1011, 1012, 1021 and 1021, the value V1 is divided among or corresponds to pixels 1011, 1012, 1021 and 1021.Similarly, the values ​​Y3, Y4, Y9 and Y10 are assigned to pixels 1013, 1014, 1023 and 1024, the value U2 is divided among or equal to pixels 1013, 1014, 1023 and 1024 and the value V2 is divided among or equal to pixels 1013, 1014, 1023 and 1024.

[0050] The Y, U, and V values ​​can be divided into corresponding planes. The Y values ​​are divided into the Y plane (1002). The Y plane (1002) includes Y1 to Y24, i.e., values ​​derived or calculated based on the corresponding RBG values ​​from the pixel array (1001). The U values ​​are divided into the U plane (1003). The U plane (1003) includes U1 to U6, i.e., values ​​derived or calculated based on the corresponding Y values ​​in plane (1002) and R values ​​from the pixel array (1001). The V values ​​are divided into the V plane (1004). The V plane (1003) includes V1 to V6, i.e., values ​​derived or calculated based on the corresponding Y values ​​in plane (1002) and B values ​​from the pixel array (1001).

[0051] The YUV values ​​are placed in YUV420 format into the input stream (1005), which covers sequential slots 1051 to 1076 (not all are shown). Sequential slots 1051 to 1064 contain the respective Y1 to Y24 values. Sequential slots 1065 to 1070 contain the respective U1 to U6 values. Sequential slots 1071 to 1076 contain the respective V1 to V6 values.

[0052] Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base portion according to one embodiment. The configuration (710) comprises a base portion of the raw image acquisition data in a raw data format. The configuration (1110) comprises the base portion of the raw image acquisition data for red elements (1111, 1113, 1131, and 1133), blue elements (1122, 1124, 1142, and 1144), and clear elements (1112, 1114, 1121, 1123, 1132, 1134, 1141, and 1143). In an exemplary implementation, the base portion includes bits 4-11 of the respective RCCB elements 1111 to 1144. The configuration (1120) comprises the base portion of the raw image acquisition data divided into corresponding color planes. The configuration (1130) includes the base part of the raw image acquisition data rearranged in a codec-compatible configuration.

[0053] In one embodiment, the configuration (1130) is considered a pseudo-YUV444 configuration or surface. In one embodiment, a pseudo-YUV444 configuration is similar to a conventional YUV444 configuration or surface, except that instead of lossy converted YUV values, the configuration is filled with losslessly acquired raw image data. In one embodiment, four different RCCB elements of the raw image acquisition data are rearranged into two video frames (1131) and (1132). As in Fig. As shown, the video frame (1131) may include the clear / red (CR) component and the video frame (1132) may include the clear / blue (CB) component. In one embodiment, the pseudo-YUV444 frames do not include converted YUV values, but rather raw image capture data (e.g., RCCB, RGGB). These frames may be considered pseudo-YUV444 frames, with the last component added as a zero frame. In one embodiment, a zero or "empty" value placeholder that is outside the normal range for a U or V value may be used to generate the last element or component of a frame (e.g., pseudo-V components 1151 and 1152).

[0054] Fig. is a block diagram of an exemplary compression input stream (1180) in a first configuration in pseudo-YUV444 format according to one embodiment. In one embodiment, the input stream (1180) is considered a pseudo-color printout. The compression input stream (1180) includes the video frame (1131) and the video frame (1132). The video frame (1131) is assigned to input stream slots 1151 to 1162, and the video frame (1132) is assigned to input stream slots 1163 to 1174. A pseudo-Y format part (1181) contains the clear sense values ​​CL1, CL2, CL3, and CL4 inserted into input stream slots 1151, 1152, 1153, and 1154. A pseudo-U format part (1182) contains the clear sense values ​​R1, R2, R3, and R4, which are inserted into input stream slots 1155, 1156, 1157, and 1158. A pseudo-V format part (1183) contains empty values, which are inserted into input stream slots 1159, 1160, 1161, and 1162.These padding values ​​are selected to maximize compression. In one embodiment, the same value is repeatedly used as a padding duplicate to increase the likelihood of significant compression. In an exemplary implementation, a zero value is used as the padding value. A pseudo-Y format part (1187) contains the clear sensel values ​​CL5, CL6, CL7, and CL8 inserted into input stream slots 1163, 1164, 1165, and 1166. A pseudo-U format part (1188) contains the clear sensel values ​​B1, B2, B3, and B4 inserted into input stream slots 1167, 1168, 1169, and 1170. A pseudo-V-format part (1189) contains empty values ​​inserted into the input stream slots 1171, 1172, 1173 and 1174.

[0055] The compression input stream (1191) in a derived YUV444 format is shown to illustrate an exemplary difference from the compression input stream (1192) in a pseudo-YUV444 format. The compression input stream (1191) contains the video frame (1192) and the video frame (1193). The video frame (1192) is assigned to the input stream slots 1151' to 1162', and the video frame (1193) is assigned to the input stream slots 1163' to 1174'. The input stream slots 1151', 1152', 1153', and 1154' contain the derived Y1, Y2, Y3, and Y4 values, respectively. Input stream slots 1155', 1156', 1157', and 1158' contain the derived U1, U2, U3, and U4 values, respectively. Input stream slots 1159', 1160', 1161', and 1162' contain the derived V1, V2, V3, and V4 values, respectively. Input stream slots 1163', 1164', 1165', and 1166' contain the derived Y5, Y6, Y7, and Y8 values, respectively.Input stream slots 1167', 1168', 1169', and 1170' contain the derived U5, U6, U7, and U8 values, respectively. Input stream slots 1171', 1172', 1173', and 1174' contain the derived V5, V6, V7, and V8 values, respectively.

[0056] In one embodiment, the compression input stream (1180) is in pseudo-YUV444 format and is fed to the YUV-compatible compression encoder. In an example implementation, the YUV-compatible compression encoder operates on the CL1 value in slot 1151 as if it were the Y1 value in slot 1151', the CL2 value in slot 1152 as if it were the Y2 value in slot 1152', the CL3 value in slot 1153 as if it were the Y3 value in slot 1153', and so on. In one embodiment, due to the similarities in the input streams, the encoder attempts to compress CL1, CL2, CL3, and CL4 as if they were Y1, Y2, Y3, and Y4 values.

[0057] Fig. is a block diagram of an exemplary compression input stream (2180) in a first configuration in pseudo-YUV444 format according to one embodiment. In one embodiment, the input stream (2180) is considered a pseudo-color expression. The compression input stream (2180) includes the video frame (2131) and the video frame (2132). In one embodiment, the video frames (2131) and (2132) are mapped to the video frames (1131) and (1132) of Fig. similar. Portions of video frames 2131 and 2132 are interleaved in compression input stream 2180. Video frame (2131) is assigned to input stream slots 2151 to 2154, 2159 to 2162, and 2167 to 2170. Video frame (2132) is assigned to input stream slots 2155 to 2158, 2163 to 2166, and 2171 to 2174. A pseudo-Y format portion (2182) contains the clear sense values ​​CL1, CL2, CL3, and CL4, which are inserted into input stream slots 2151, 2152, 2153, and 2154. A pseudo-U format part (1182) contains the clear sense values ​​R1, R2, R3, and R4, inserted into input stream slots 2155, 2156, 2157, and 2158. A pseudo-V format part (2183) contains blank values ​​inserted into input stream slots 2159, 2160, 2161, and 2162. These padding values ​​are selected to maximize compression.In one embodiment, the same value is repeatedly used as a padding or padding duplicate to increase the likelihood of significant compression. In an exemplary implementation, a zero value is used as the padding or padding value. A pseudo-Y format part (2187) contains the clear sensel values ​​CL5, CL6, CL7, and CL8 inserted into input stream slots 2163, 2164, 2165, and 2166. A pseudo-U format part (2188) contains the red sensel values ​​B1, B2, B3, and B4 inserted into input stream slots 2167, 2168, 2169, and 2170. A pseudo-V format part (2189) contains blank values ​​inserted into input stream slots 2171, 2172, 2173, and 2174.

[0058] The compression input stream (2190) in a derived YUV444 format is shown to illustrate an example difference from the compression input stream (2180) in a pseudo-YUV444 format. The compression input stream (2190) contains the video frame (2191) and the video frame (2192). The video frame (2191) is assigned to the input stream slots 2151' to 2154', 2159' to 2162', and 2167' to 2170'. The input stream slots 2151', 2152', 2153', and 2154' contain the derived Y1, Y2, Y3, and Y4 values, respectively. Input stream slots 2159', 2160', 2161', and 2162' contain the derived U1, U2, U3, and U4 values, respectively. Input stream slots 2167', 2168', 2169', and 2170' contain the derived V1, V2, V3, and V4 values, respectively. The video frame (2192) is assigned to input stream slots 2155' to 2158', 2163' to 2166', and 2171' to 2192'. Input stream slots 2155', 2156', 2157', and 2158' contain the derived Y5, Y6, Y7, and Y8 values, respectively.Input stream slots 2163', 2164', 2165', and 2166' contain the derived U5, U6, U7, and U8 values, respectively. Input stream slots 2171', 2172', 2173', and 2174' contain the derived V5, V6, V7, and V8 values, respectively.

[0059] Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base portion according to one embodiment. The configuration (1210) comprises a base portion of the raw image acquisition data in a raw data format. The configuration (1210) comprises the base portion of the raw image acquisition data for red elements 1211, 1213, 1231, and 1233, blue elements 1222, 1224, 1242, and 1244, and clear elements 1212, 1214, 1221, 1223, 1232, 1234, 1241, and 1243. In an exemplary implementation, the base portion includes bits 4-11 of the respective RCCB elements 1211 to 1244. The configuration (1220) comprises the base portion of the raw image acquisition data divided into corresponding color planes. The configuration (1230) contains the base part of the raw image acquisition data rearranged in a codec-compatible configuration.In one embodiment, as part of the surface rearrangement, the base layer is considered a pseudo-Y component, and the pseudo-U and pseudo-V components are added as a pseudo-blank or pseudo-zero frame. The newly established frame is considered a pseudo-YUV420 frame and is further processed.

[0060] Fig. is a block diagram of an exemplary compression input stream (1292) in a pseudo-YUV420 format according to one embodiment. In one embodiment, the input stream (1292) is considered a pseudo-color expression. The compression input stream (1292) is assigned to input stream slots 1251 to 1274. A pseudo-Y format part (1281) contains the clear sense values ​​CL1 to CL8, and the values ​​R1 to R4 and B1 to B4 are inserted into input stream slots 1251 to 1266. A pseudo-U format part (1282) and a pseudo-V format part (1283) both contain dummy values ​​inserted into pseudo-input stream slots 1267 to 1270 and 1271 to 1274, respectively. These fill or padding values ​​are selected to maximize compression.

[0061] The compression input stream (1291) in a derived YUV444 format is shown to illustrate an example difference from the compression input stream (1292) in a pseudo-YUV444 format. The compression input stream (1291) contains the input stream slots 1251' to 1274'. The input stream slots 1251' to 1266' contain the derived Y1 to Y16 values. The input stream slots 1267 to 1270 contain the derived U1 to U4 values. The input stream slots 1271 to 1274 contain the derived V1 to V4 values.

[0062] In one embodiment, the compression input stream (1292) is in pseudo-YUV420 format and is fed to the YUV-compatible compression encoder. In an example implementation, the YUV-compatible compression encoder operates on the CL1 value in slot 1251 as if it were the Y1 value in slot 1251', the CL2 value in slot 1252 as if it were the Y2 value in slot 1252', the CL3 value in slot 1253 as if it were the Y3 value in slot 1253', and so on. In one embodiment, due to the similarities in the input streams, the encoder attempts to compress CL1, CL2, CL3, and CL4 as if they were Y1, Y2, Y3, and Y4 values.

[0063] In one embodiment, the base layer is rearranged into a pseudo-YUV420 video frame, where portions of the R and B components are considered U and V components. The remaining R, B, and C components are considered pseudo-Y components. Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base portion according to one embodiment. In one embodiment, the input image is rearranged as YUV420, and no zero plane is added. The configuration (1310) includes a base portion of the raw image acquisition data in a raw data format. The configuration (1310) includes the base portion of the raw image acquisition data for red elements 1311, 1313, 1331, and 1333, blue elements 1322, 1324, 1342, and 1344, and clear elements 1312, 1314, 1321, 1323, 1332, 1334, 1341, and 1343. In an exemplary implementation, the base portion includes bits 4-11 of the respective RCCB elements 1311 to 1344. The configuration (1320) includes the base portion of the raw image acquisition data divided into corresponding color planes. The configuration (1330) includes the base portion of the raw image acquisition data rearranged in a codec-compatible configuration.In one embodiment, as part of the surface rearrangement, the pseudo-Y part includes the clear values ​​CL1 to CL8, the red values ​​R1a and R2a, the blue values ​​B1a and B2a, the pseudo-U part includes the red values ​​R1b, R2b, R3, and R4, and the pseudo-V part includes the blue values ​​B1b, B2b, B3, and B4. In an exemplary implementation, R1a, R2a, B1a, and B2a include bits 4-7 of elements R1a, R2a, B1a, and B2a, and elements R1b, R2b, B1b, and B2b include bits 8-11 of elements R1a, R2a, B1a, and B2a. The newly established frame is considered a pseudo-YUV420 frame and is further processed.

[0064] Fig. is a block diagram of an exemplary compression input stream (1392) in a pseudo-YUV444 format according to one embodiment. In one embodiment, the input stream (1392) is considered a pseudo-color expression. The compression input stream (1392) is assigned to input stream slots 1351 to 1366. A pseudo-Y format part (1381) contains the clear sense values ​​CL1 to CL8, and the values ​​R1a, R2a, B1a, and B2a are inserted into input stream slots 1351 to 1360. A pseudo-U format part (1382) contains the values ​​R1b, R2b, R3 and R4, inserted into the input stream slots 1361 to 1363. A pseudo-V format part (1383) contains the values ​​B1b, B2b, B3 and B4, inserted into the input stream slots 1361 to 1366.

[0065] The compression input stream (1391) in a derived YUV444 format is shown to illustrate an example difference from the compression input stream (1392) in a pseudo-YUV444 format. The compression input stream (1391) contains the input stream slots 1351' to 1360'. The input stream slots 1351' to 1360' contain the derived Y1 to Y10 values. The input stream slots 1361 to 1363 contain the derived U1 to U4 values. The input stream slots 1364 to 1366 contain the derived V1 to V4 values.

[0066] In one embodiment, the compression input stream (1392) is in pseudo-YUV420 format and is fed to the YUV-compatible compression encoder. In an example implementation, the YUV-compatible compression encoder operates on the CL1 value in slot 1351 as if it were the Y1 value in slot 1351', the CL2 value in slot 1352 as if it were the Y2 value in slot 1352', the CL3 value in slot 1353 as if it were the Y3 value in slot 1353', and so on. In one embodiment, due to the similarities in the input streams, the encoder attempts to compress CL1, CL2, CL3, and CL4 as if they were Y1, Y2, Y3, and Y4 values.

[0067] Conventional YUV420 is a common video interface format and is widely supported. Although not obvious, a conventional YUV420 encoder can easily be adapted to perform encoding for a pseudo-YUV420 input. In one embodiment, similar to the configuration in Fig. , no pseudo-YUV420 null planes are added, and the base layer can be processed efficiently. In one embodiment, the resolution of the YUV420 surface is calculated as follows: a. YUV420_Width = input width b. YUV420_Height = (Input height + 2) / 3 * 2.

[0068] The C components of the sensor frame form the pseudo-Y surface. Part of the R and B components can also be used to form the pseudo-Y surface. In one embodiment (YUV420_Height - Input_Height / 2), rows of the R and B components are included in the Y surface, and the remaining rows are considered U and V surfaces.

[0069] In one embodiment, the resolution of a sensor frame is 1920x1240 and the resolution of a YUV420 video frame is 1920x828. In one embodiment, the resolution is based on setting the YUV420_width to a value (YUV420_width = 1920) and calculating the YUV420_height. In an example implementation, the YUV420_height = (1240 + 2) / 3 * 2 = 828 and the UV surface resolution is 960x414. The last two lines of the UV surface can be padded with content from the last row or zero.

[0070] It is generally accepted that various arrangements can be used for base layer levels. Another option for the base layer arrangement is to pack and encode the base layer portion in the single 4:2:2 format. Fig. is a block diagram of an exemplary partitioning and reconfiguration of raw image data of a base portion according to one embodiment. In one embodiment, the input image is rearranged as a pseudo-YUV422 configuration. The configuration (1410) includes a base portion of the raw image acquisition data in a raw data format. Configuration (1410) includes the base portion of the raw image acquisition data for red elements 1411, 1413, 1431, and 1433, blue elements 1422, 1424, 1442, and 1444, and clear elements 1412, 1414, 1421, 1423, 1432, 1434, 1441, and 1443. In an exemplary implementation, the base portion includes bits 4-11 of the respective RCCB elements 1411 to 1444. Configuration (1420) includes the base portion of the raw image acquisition data divided into corresponding color planes. Configuration (1430) includes the base portion of the raw image acquisition data rearranged in a codec-compatible configuration.In one embodiment, the base layer shall include a pseudo-Y component, a pseudo-V component, and a pseudo-U component as part of the surface rearrangement. The newly established frame is considered a pseudo-YUV422 frame and further processed.

[0071] Fig. is a block diagram of an exemplary compression input stream (1492) in a pseudo-YUV422 format according to one embodiment. In one embodiment, the input stream (1492) is considered a pseudo-color expression. The compression input stream (1492) is associated with input stream slots 1451 through 1466. A pseudo-Y format portion (1481) contains the clear sense values ​​CL1 through CL8 inserted into input stream slots 1451 through 1458. A pseudo-U format portion (1482) contains the values ​​R1 through R4 inserted into input stream slots 1459 through 1462. A pseudo-V format portion (1483) contains the values ​​B1 through B4 inserted into input stream slots 1463 through 1466.

[0072] The compression input stream (1491) in a derived YUV422 format is shown to illustrate an example difference from the compression input stream (1492) in a pseudo-YUV422 format. The compression input stream (1491) contains the input stream slots 1451' to 1466'. The input stream slots 1451' to 1458' contain the derived Y1 to Y8 values. The input stream slots 1459 to 1462 contain the derived U1 to U4 values. The input stream slots 1463 to 1466 contain the derived V1 to V4 values.

[0073] In one embodiment, the compression input stream (1492) is in pseudo-YUV422 format and is fed to the YUV-compatible compression encoder. In an example implementation, the YUV-compatible compression encoder operates on the CL1 value in slot 1451 as if it were the Y1 value in slot 1451', the CL2 value in slot 1452 as if it were the Y2 value in slot 1452', the CL3 value in slot 1453 as if it were the Y3 value in slot 1453', and so on. In one embodiment, due to the similarities in the input streams, the encoder attempts to compress CL1, CL2, CL3, and CL4 as if they were Y1, Y2, Y3, and Y4 values.

[0074] In one embodiment, data allocated to the base layer video images in a compression-compatible pseudo-YUV format is arranged using the RCCB color space values ​​as values ​​in the compression-compatible YUV format, rather than as YUV color space values ​​derived from the RCCB or RBG color space values. In an exemplary implementation, traditional YUV conversion values ​​are not used as compression input configured in a pseudo-YUV compression format. The traditional YUV conversion values ​​may include U calculated as if it were (BY) and V calculated as if it were proportional to (RY) using various scale factors and offset values; the weighted RGB values ​​are used to determine Y, and then U and V are calculated as scaled differences between Y and R and B values, and so on.

[0075] As already mentioned, it is important that the recorded values ​​are retained in many applications. Fig. is a block diagram for the compression process (1500) for data storage according to one embodiment. In an example implementation, the acquired values ​​of the color components are preserved during our compression processes for writing to the speaker and the later decompression processes when reading from memory. In one embodiment, the acquired values ​​of the color components are not converted to or from a different color space as part of the compression processes for writing to memory and the decompression processes for reading. In an example implementation, the acquired value compression process is lossless, in contrast to conventional conversion methods in which the acquired raw image values ​​(e.g., R, G, B) are converted to other color space values ​​(e.g., Y, U, V), thereby causing information loss or distortion.

[0076] In block (1510), the raw image values ​​are received in an image capture format (e.g., RCCB, RGGB, Bayer format). In one embodiment, the captured raw image values ​​are color values ​​output by an array of color sensors. The color sensor elements can be integrated into a camera. The captured raw image values ​​originate from sensors in a vehicle. In an exemplary implementation, the vehicle is similar to the vehicles described above.

[0077] In block (1520), the raw image values ​​are split into a base portion (1530) and an enhancement portion (1540). The boundary for the split may be based on a variety of factors (e.g., a compression benefit analysis of the base layer portion, the type of compression scheme, the time required to perform the compression operations, etc.). In one embodiment, the split is performed similarly to the Fig. .

[0078] In block (1540), the raw image values ​​(1530) of the base part are reordered from a raw image format (e.g., RCCB, RGGB, Bayer format) into a color plane format. In one embodiment, the different color components (e.g., red (R), clear (C), and blue (B)) are separated into different planes. The planes can be edited individually. A more detailed explanation of how the exemplary raw image RCCB values ​​are recorded in a raw image recording format can be found in the description of the Fig. .

[0079] In block (1550), the raw image data is reconfigured from the color plane format into a codec-compatible format. In one embodiment, the codec-compatible format is considered a pseudo-color expression. The codec-compatible format may be a pseudo-codec-compatible format in which the captured raw image data expressed in original color space values ​​is reconfigured into a second color space-encoded format. In one embodiment, the codec resembles various standards (e.g., an H.264-compatible codec, an H.265-compatible codec, a VP9-compatible codec, an AV1-compatible codec, and other video encoders). In one embodiment, a pseudo-YUV format is used. In an exemplary implementation, the pseudo-YUV resembles a YUV format, but the captured raw image values ​​are used instead of YUV values.A more detailed explanation of how the example raw image RCCB values ​​are recorded in a raw image recording format can be found in the description of the . Fig. In one embodiment, the input format for compressing the base layer portion is a pseudo-YUV format. In one embodiment, the pseudo-YUV compression format is similar to or compatible with YUV compression.

[0080] In block (1570), the compression process is performed. In one embodiment, the codec process is performed on the base portion of the raw image capture values ​​that have been arranged in a special color codec format. In an exemplary implementation, the encoding is performed on the base portion of the raw image values ​​configured in a pseudo-YUV format. In one embodiment, a YUV-compatible compression (e.g., H.264, H.265, AV1) of the captured raw image values ​​of the base portion is performed. In an exemplary implementation, the base portion of RCCB values ​​is output in a compressed pseudo-YUV format.

[0081] In block (1590) the output of the compression process is stored.

[0082] In one embodiment, to utilize intra-frame redundancies in the RCCB base layer, all four components must first be separated. These four different components are then arranged into new video frames.

[0083] Fig. is a flowchart of an exemplary method (1600) according to one embodiment.

[0084] In block (1610), the raw image data is captured by an image capture device, wherein the image data corresponds to a sensor element or sense values. The sense values ​​can be used to form pixel data. In one embodiment, the image capture device is a camera.

[0085] In block (1620), the pixel data is divided into a base layer portion and an extension layer portion, with the division based on various factors. The threshold between the base layer portion and the extension layer portion can be based on various factors.

[0086] In block (1630), the base layer portion is reconfigured into an input format for a compression encoder. In one embodiment, the reconfiguration includes converting the base layer portion from a raw image format to an input format for compression. In one embodiment, the reconfigured base layer is considered a pseudo-color expression.

[0087] In block (1640), the reconfigured base layer portion is compressed. In one embodiment, a pseudo-YUV encoder is used to compress the data. It is recognized that the reconfigured base layer portion may be compatible with various types of compression standards (e.g., H.264, H.265, VP9, ​​AV1).

[0088] In block (1650), the raw image data is stored. In one embodiment, an enhancement layer portion is stored in an uncompressed format and a base layer portion is stored in a compressed format.

[0089] It is generally accepted that significant data compression can be achieved by reconfiguring the original color space values ​​into a pseudo-encoder color space format. The following table contains exemplary compression results achieved with a pseudo-YUV420 format conversion. Nr.: Stream-Name Input-Größe Bit roh Dateigröße Gesamtgröße Kompressionsrate (in GB) Dateigröße H.264 H.264 H.264 1 video_0.raw 53,5 13,4 8,1 21,5 2,488372093 2 video_1.raw 53,5 13,4 9,2 22,6 2,367256637 3 video_2.raw 15,2 3,8 2,89 6,69 2,272047833 5 video_3.raw 15,2 3,8 3,14 6,94 2,190201729 5 video_4.raw 20,5 5,1 4,8 9,9 2,070707071 6 video_5.raw 20,5 5,1 5,1 10,2 2,009803922 7 video_6.raw 23,5 5,8 4,3 10,1 2,326732673 8 video_7.raw 23,5 5,8 3,4 9,2 2,554347826

[0090] In the use cases discussed above, encoding performance is more important than decoding performance, as encoding must be performed in real time. In an example implementation, a video frame can be encoded in approximately 3 ms. This allows approximately 330 sensor images to be processed per second. In one embodiment, this scheme supports multi-sensor acquisition on a T186 platform.

[0091] In one embodiment, video frames can be encoded with an H.264 CAVLC 4:4:4 Intra profile. Reasons for choosing an H.264 encoder can include better compression tools and higher encoding speed. Symmetric encoding and decoding operations and very good decoding performance can be achieved. H.264 YUV420 decoding is supported in NVDEC. The use of intra-only frames enables random access to each individual frame.

[0092] It is recognized that a pseudo-YUV format enables a YUV-compatible encoder to improve performance. The pseudo-YUV format allows a YUV-compatible encoder to effectively store more data in less memory than a conventional YUV input format. In one embodiment, the raw image data in pseudo-YUV format enables a YUV encoder to achieve lossless compression unattainable with conventional YUV encoders. Dividing the pseudo-YUV format into a base layer portion and an enhancement layer portion further improves the performance of a YUV encoder by eliminating those portions of the raw image data that are unlikely to be highly compressible. This reduces the computational overhead for compression processing.Pseudo-YUV splitting allows a YUV-compatible encoder to encode the base layer portion faster than a traditional approach that attempts to encode the entire derived YUV data.

[0093] Fig. is a block diagram of a computer system (2000) according to one embodiment. In one embodiment, the computer system (2000) is similar to the computer system (150). In one embodiment, the computer system (2000) is similar to a remote computer system (e.g., storage server (230), processing server (240)). In an example implementation, an optional display may be used to display the captured image or data associated with the captured image.

[0094] With reference to Fig. Shown here is a block diagram of an exemplary computer system (2000), one embodiment of a computer system on which embodiments of the present invention may be practiced. The computer system (2000) includes the central processing unit (CPU) 2001, the main memory (2002) (e.g., random access memory), the chipset (2003) with northbridge (2009) and southbridge (2005), the removable storage device (2004), the input device (2007), the signal transmission port (2008), and the graphics subsystem (2010) coupled to the display (2020). The computer system (2000) has several buses for communicatively connecting the components of the computer system (2000). The communication bus (2001) (e.g., a front side bus) couples the northbridge (2009) of the chipset (2003) to the central processing unit (2001). The communication bus (2092) (e.g. a main memory bus) couples the north bridge (2009) of the chipset (2003) with the main memory (2002).The communication bus (2093) (e.g., Advanced Graphics Port interface, Peripheral Component Interconnect (PCI) Express bus) couples the northbridge of the chipset (2003) to the graphics subsystem (2010). The communication buses (2094), (2095), and (2097) (e.g., a PCIe bus) couple the southbridge (2005) of the chipset (2003) to the removable storage device (2004), the input device (2007), and the signal transmission port (2008). The graphics subsystem (2010) includes the graphics processing unit (GPU) (2011) and the graphics memory (2015).

[0095] The components of the computer system (2000) work together to provide versatile functionality and performance. In an exemplary implementation, the components of the computer system (2000) work cooperatively to provide predetermined types of functionality. The communication buses (2091, 2092, 2093, 2094, 2095, and 2097) transfer the data. The central processing unit (2001) processes the data. The main memory (2002) stores data and instructions for the central processing unit (2001). The removable storage device (2004) also stores data and instructions (e.g., as a large data storage device). The input device (2007) provides a mechanism for entering information and / or pointing to or highlighting information on the display (2020). The signal communication port (2008) provides a communication interface to external devices (e.g., an interface to a network).The display (2020) outputs the information stored in the graphics memory (2015) for output. The graphics processor (2011) processes graphics commands from the central processing unit (2001) and makes the resulting data available to the graphics memory (2015) for storage and retrieval by the display (2020). The graphics memory (2015) can provide memory for various memory operations (e.g., CPU cache operations, frame buffer operations, raster operations, rendering operations). Hybrid frustum trace processes can be used for various operations within the computer system (2000) (e.g., for execution operations on the CPU (2001), execution operations on the GPU (2011)).

[0096] The image information received by a camera or image sensor is divided into two parts or sections (e.g. see Fig. ). A portion of the image data is rearranged, compressed, and stored in memory. The other portion is stored without rearranging or compression. In an exemplary implementation, a portion of the original image values ​​captured by a camera or sensor is compressed and stored as a base portion, while another portion of the image values ​​is not compressed and stored as an extension portion. The values ​​of the base portion can be rearranged or reconfigured (e.g. Fig.) to be compatible with efficient compression techniques. Storing an uncompressed extension portion and a compressed base portion of the original image values ​​enables efficient and effective, lossless, and accurate data storage with a smaller footprint than conventional systems. Targeted data reordering, compatible with powerful compression techniques, also allows information to be compressed efficiently and faster than conventional methods.

[0097] Some portions of the detailed descriptions represent procedures, logic blocks, processing, and other symbolic representations of operations on data bits in computer memory. These descriptions and representations are commonly used by professionals in the data processing industry to effectively communicate the content of their work to other professionals. A procedure, logic block, process, etc., is conceived here and generally as a self-consistent sequence of steps or instructions that leads to a desired result. The steps involve physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, optical, or quantum signals that can be stored, transferred, combined, compared, and otherwise manipulated within a computer system.It has proven useful to refer to these signals as bits, values, elements, symbols, characters, terms, numbers or the like, primarily for reasons of common usage.

[0098] It should be noted, however, that all these and similar terms are associated with the corresponding physical quantities and are merely convenient labels applied to these quantities. Unless the following explanations expressly state otherwise, it should be noted that throughout this application, explanations using terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like refer to the action and processes of a computer system or similar processing device (e.g., an electrical, optical, or quantum computing device) that manipulates and transforms data represented as physical (e.g., electronic) quantities. The terms refer to operations and processes of the processing devices that manipulate physical quantities within the component of a computer system (e.g.,Registers, memories, other such information storage, transmission, or display devices) into other data, which are also represented as physical quantities within other components.

[0099] References to "embodiment" or "an embodiment" mean that one or more features, structures, characteristics, etc. described in connection with the embodiment are included in at least one implementation or realization covered by the present specification. Therefore, references to the terms "embodiment" or "an embodiment" do not necessarily refer to the same embodiment. Moreover, the features, structures, characteristics, etc. may be included in other suitable forms than the illustrated embodiment.

[0100] The use of definite or indefinite articles is not intended to indicate cardinality. In particular, a reference to "the" object or "an" object is also intended to denote one of a possible plurality of such objects. It should also be understood that the phraseology and terminology used herein is descriptive and should not be considered limiting. The use of the disjunctive mood is intended to include the subjunctive mood.

[0101] The terms "first," "second," "third," "fourth," and the like in the description and, where applicable, the claims are used to distinguish between similar elements and not necessarily to describe a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances, so that, for example, the embodiments described herein may be employed in sequences other than those illustrated or otherwise described herein. Likewise, where a method is described herein as comprising a series of steps, the order of the steps illustrated herein is not necessarily the only order in which such steps may be performed, and some of the recited steps may be omitted and / or certain other steps not described herein may be added to the method.

Claims

[1] A method comprising: Receiving raw image data from an image capture device (110), the raw image data including pixel values ​​expressed in a first color space; Dividing the pixel values ​​into a base layer part (791, 793) and an extension layer part (792, 794), the extension layer part having the least significant bits of the pixel values; Reconfiguring the pixel values ​​of the base layer portion (791, 793) as a pseudo-color space expression, wherein the pseudo-color space expression is compatible with a second color space compression format and is different from the first color space; Compressing the pseudo-color space expression to create compressed data, and Saving the compressed data and the enhancement layer part as pixel values. [2] The method of claim 1, wherein said splitting comprises splitting the pixel data into a base layer portion (791, 793) and an enhancement layer portion (792, 794) at a boundary location between the base layer portion (791, 793) and the enhancement layer portion, wherein the boundary location is set based on a compression benefit analysis of the pixel data. [3] The method of claim 1 or 2, wherein the raw image data is in a Red, Clear, Clear, Blue (RCCB) format, and the reconfigured base layer portion includes red, clear, and blue components configured in the pseudo color space expression, wherein the pseudo color space expression is a pseudo YUV format including RCCB values ​​in the YUV slots of an image frame bitstream. [4] A method according to any one of the preceding claims, wherein a boundary location separating the bits in the base layer part (791, 793) from the bits in the enhancement layer part (792, 794) is based on a balance of the benefits of compressing the information against the costs of compressing the information. [5] A method according to any preceding claim, wherein reconfiguring includes forming a base layer video frame. [6] Method according to one of the preceding claims, wherein the base layer part (791, 793) comprises more bits than the extension layer part (792, 794). [7] A method according to any one of the preceding claims, wherein the base layer portion (791, 793) contains the most significant bits in the pixel data. [8] A method according to any one of the preceding claims, wherein the reconfigured base layer data is input to a video encoder selected from a group of video encoders comprising: an H.264-compatible encoder, an H.265-compatible encoder, a VP9-compatible encoder and an AV1-compatible encoder. [9] A method comprising: Receiving raw image data from an image capture device (110), the raw image data including pixel values ​​expressed in a red, clear, clear, blue (RCCB) color space; Dividing the pixel values ​​into a base layer part (791, 793) and an extension layer part (792, 794); and Reconfiguring the pixel values ​​of the base layer part (791, 793) as a pseudo-color space expression, wherein the pseudo-color space expression is compatible with a compression-compatible format for the YUV color space, wherein the reconfiguring comprises: Dividing the base layer portion (791, 793) based on a plurality of elements within the raw data, wherein the plurality of elements comprises a red element, a blue element, a first clear element, and a second clear element corresponding to the pixel values; Forming one or more base layer video images from the plurality of elements; and arranging information associated with one or more base layer video images in the compression-compatible format for the YUV color space using the pixel values ​​expressed in the RCCB color space as values ​​in the compression-compatible format for the YUV color space; and Compressing the one or more of the base layer video images, wherein the one or more base layer video images have the pseudo color space expression. [10] The method of claim 9, wherein the one or more base layer video images comprise: One or more Clear, Blue (CB) video images formed from the blue element and the first clear element; and One or more Clear, Red (CB) video images formed from the red element and the second clear element. [11] The method of claim 9 or 10, wherein the one or more base layer video images correspond to a pseudo-YUV444 format. [12] A method according to any one of claims 9 to 11, wherein zero-plane frames are used as pseudo-UV components and a portion of the one or more base layer video images corresponds to a pseudo-Y component. [13] A method according to any one of claims 9 to 12, wherein the one or more base layer video images correspond to a pseudo-YUV420 format. [14] The method of any one of claims 9 to 13, wherein the one or more base layer video images are input to a video encoder selected from a group comprising an H.264-compatible encoder, an H.265-compatible encoder, a VP9-compatible encoder, and an AV1-compatible encoder. [15] A method according to any one of claims 9 to 14, wherein a first part of the red element corresponds to a chrominance U input and a first part of the blue element corresponds to a chrominance V input, while a second part of the red element corresponds to a second part of the blue element, a first clear element and a second clear element corresponds to a luminance Y input. [16] The method of claim 15, wherein the base layer video images are compatible with a pseudo-YUV420 format, wherein: A pseudo-YUV width is equal to an input width; and A pseudo-YUV height is equal to (input_height + 2) / 3 * 2, where input_height is a value indicating the number of pixels in the input height of the pseudo-YUV component. [17] The method of claim 15 or 16, further comprising training a deep learning network or deep neural network (TNN) for autonomous vehicle control systems, wherein the training uses data from the base layer part and the extension layer part. [18] A system comprising: an image capture device (110) that captures raw image data, the raw image data including pixel values ​​expressed in a first color space; a processing unit that stores captured raw image data, wherein a first portion of the pixel values ​​is arranged as a pseudo-color expression compatible with a color compression format, divides the pixel values ​​into a base layer portion (791, 793) and an extension layer portion (792, 794), wherein the extension layer portion has the least significant bits of the pixel values; reconfigures the pixel values ​​of the base layer portion (791, 793) as a pseudo-color space expression, wherein the pseudo-color space expression is compatible with a second color space compression format and different from the first color space, and compresses the pseudo-color space expression to create compressed data; and a storage unit that stores the compressed data and the enhancement layer part as pixel values. [19] The system of claim 18, wherein the first part is an extension part and the second part is a base part. [20] The system of claim 18, wherein the image capture device, the processing unit and the memory are located in a vehicle, and the raw image data corresponds to scenes that occurred during vehicle operation. [21] The system of claim 20, wherein the data storage device is portable and selectively removable from the vehicle. [22] A system according to any one of claims 18 to 21, configured to perform a method according to any one of claims 1 to 17.

Images