Method and apparatus for dynamic codec configuration

Abstract

The invention relates to a system, apparatus and method for dynamic encoder configuration. In one exemplary embodiment, a machine learning model uses pixel features and encoding features from previous stages of an image processing pipeline (IPP) to dynamically adjust bit rates. The machine learning model is trained to select bit rate adjustments for an encoder such that the expected image quality of the video stream remains at a selected quality level (e.g., SSIM, VMAF, VIF, HVS-PSNR, etc. Conventional dynamic coding solutions focus on one-time coding over-delivery (best effort) applications, the exemplary IPP being designed for real-time applications that may not benefit from actual subsequent coding quality analysis; instead, proxy data (pixel features and encoding features), which are representative approximations of image complexity, are used.

Description

[0001] Related Applications

[0002] This application claims the benefit of PCT Application No. PCT / US23 / 62157, filed on Feb. 7, 2023, and entitled “Methods and Apparatus for Real-Time Guided Encoding,” which is incorporated herein by reference in its entirety.

[0003] Copyright

[0004] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyrights. Technical Field

[0005] This disclosure relates to encoding video content. More specifically, this disclosure relates to dynamically configuring a codec according to a desired image quality. Background Art

[0006] Existing video encoding techniques utilize so-called intra-coded frames (I-frames), predicted frames (P-frames), and bi-directional frames (B-frames). Three different frame types can be used in certain cases to improve video compression efficiency. Most codecs encode video based on image analysis and metrics. Image analysis is computationally complex and typically requires look-ahead / look-back comparisons between frames.

[0007] An embedded device is a computing device that contains a dedicated computing system. In many cases, an embedded device must operate within strict processing and / or memory constraints to ensure that a real-time budget is met. For example, an action camera (e.g., a GoPro HERO TM series device) must capture each video frame at a specific capture rate (e.g., 30 frames per second (fps)). In practice, the video compression quality in an embedded device is significantly limited. Summary of the Invention

[0008] On the one hand, the present disclosure provides a method for dynamically configuring an encoder in a pipeline, which includes: obtaining a model that correlates proxy data with image complexity; obtaining first proxy data of a first set of images; determining first encoding parameters for a consistent optimization objective based on the first proxy data and the model; configuring the encoder to encode a first video segment based on the first encoding parameters; obtaining second proxy data of a second set of images; determining second encoding parameters for the consistent optimization objective based on the second proxy data and the model; configuring the encoder to encode a second video segment based on the second encoding parameters; and wherein the first video segment and the second video segment are within a threshold tolerance of the consistent optimization objective.

[0009] On the other hand, the present disclosure further provides an apparatus, which includes: a camera configured to capture at least one first image; an image processing pipeline including an encoding element; machine learning logic trained to select a bitrate based on proxy data; a processor; and a non-transitory computer-readable medium including a set of instructions that, when executed by the processor, cause the processor to: provide first proxy data associated with at least the first image to the machine learning logic; obtain a first bitrate from the machine learning logic based on the first proxy data; and configure the encoding element to encode at least the first image based on the first bitrate.

[0010] In yet another aspect, the present disclosure further provides an encoding apparatus, which includes: an encoding element configured to encode according to a first mode and a second mode; machine learning logic trained to select a bitrate according to the first mode and the second mode; a processor; and a non-transitory computer-readable medium including a set of instructions that, when executed by the processor, cause the processor to: provide first proxy data associated with a first image to the machine learning logic according to the first mode; obtain a first bitrate from the machine learning logic based on the first proxy data; and switch to the second mode based on the first bitrate. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A logic flow diagram depicting an exemplary image processing pipeline (IPP) for explaining various aspects of the present disclosure.

[0012] Figure 2 A graphical representation depicting the variation of image quality with the bitrate used for encoding fixed-resolution images and an ideal convex hull for explaining various aspects of the present disclosure.

[0013] Figure 3 A graphical representation depicting the possible bitrate range of a target quality for explaining various aspects of the present disclosure.

[0014] Figure 4An illustrative diagram depicting single-pass constant bit rate (CBR) encoding for explaining various aspects of the present disclosure.

[0015] Figure 5 An illustrative diagram depicting "equal quality" bit rate encoding in accordance with various aspects of the present disclosure.

[0016] Figure 6 A graphical representation depicting an exemplary single-pass equal quality encoding process in accordance with various aspects of the present disclosure.

[0017] Figure 7 A graphical representation depicting an exemplary dual-resolution equal quality encoding process in accordance with various aspects of the present disclosure.

[0018] Figure 8 A graphical representation depicting a first lower-upper bound variant of an equal quality encoding process in accordance with various aspects of the present disclosure.

[0019] Figure 9 A graphical representation depicting a second lower-upper bound variant of an equal quality encoding process in accordance with various aspects of the present disclosure.

[0020] Figure 10 A logical block diagram depicting an exemplary system in accordance with various aspects of the present disclosure.

[0021] Figure 11 A logical block diagram depicting an exemplary encoding device in accordance with various aspects of the present disclosure. Detailed Description

[0022] In the following detailed description, reference is made to the accompanying drawings. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Accordingly, the following detailed description is not to be considered limiting, and the scope of embodiments is defined by the appended claims and their equivalents.

[0023] Aspects of the present disclosure are disclosed in the accompanying description. Alternative embodiments and their equivalents of the present disclosure may be designed without departing from the spirit or scope of the present disclosure. It should be noted that any discussion regarding "one embodiment", "an embodiment", and "exemplary embodiment" etc. indicates that the described embodiment may include specific features, structures, or characteristics, and such features, structures, or characteristics may not necessarily be included in every embodiment. Additionally, references above do not necessarily include references to the same embodiment. Finally, whether explicitly described or not, those of ordinary skill in the art will readily understand that each of the features, structures, or characteristics of a given embodiment may be combined or utilized in combination with the features, structures, or characteristics of any other embodiment discussed herein.

[0024] The various operations may be described sequentially as a number of discrete actions or operations in a manner most helpful in understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations are necessarily order dependent. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed and / or the described operations may be omitted.

[0025] Image processing pipeline

[0026] Figure 1 is a logical flow diagram of an exemplary Image Processing Pipeline (IPP) 100 for illustrating various aspects of the present disclosure. As shown, the exemplary IPP has three (3) stages: a first stage 102 that captures raw data and converts the raw data into a color space (e.g., YUV); a second stage 104 that performs in-camera preprocessing; and a third stage 106 that is used for encoding video. Transitions between stages of the pipeline are facilitated by DDR buffers 108A, 108B.

[0027] Briefly, the first stage 102 converts the “raw” light information captured by the sensor into “pixel” data suitable for display and / or post-capture image manipulation (post-processing). Typically, ISP operations include demosaicking, white balancing, denoising, color correction, tone mapping, sharpening, etc. Although the following discussion is presented in terms of a specific set of operations and order, other operations and / or orders may be equally successful alternatives.

[0028] In one exemplary embodiment, the first stage 102 is implemented within an Image Signal Processor (ISP). As shown, the ISP controls the light capture of the camera sensor and may also perform color space conversion. The camera captures light information by “exposing” its photoelectric sensor for a short period of time. The “exposure” can be characterized by three parameters: aperture, ISO (sensor gain), and shutter speed (exposure time). The exposure determines how bright or dark the image will appear when captured by the camera. During normal operation, a digital camera can automatically adjust the aperture, ISO, and shutter speed to control the amount of light received; this functionality is commonly referred to as “auto exposure”. Due to form factor limitations and its most common use cases (different lighting conditions), most action cameras are fixed aperture cameras that only adjust ISO and shutter speed.

[0029] After each exposure, the ISP reads the raw luminance data from the photoelectric camera sensor; the luminance data is associated with the positions of a color filter array (CFA) to create a “mosaic” of chrominance values. The ISP demosaics the luminance and chrominance data to produce the standard color space of the image; for example, in the illustrated embodiment, the raw data is converted into the YUV (or YCrCb) color space.

[0030] The ISP performs white balance and color correction to compensate for lighting differences. White balance attempts to mimic human perception of "white" under different lighting conditions. Simply put, the way a camera captures chromaticity information is different from that of the human eye. The human visual system perceives light through three different types of "cone" cells, with spectral sensitivity peaks at short wavelengths ("blue", 420nm to 440nm), medium wavelengths ("green", 530nm to 540nm), and long wavelengths ("red", 560nm to 580nm). Human sensitivity to red, blue, and green varies with different lighting conditions; under low light conditions, the human eye's sensitivity to red light decreases, but the blue / green sensitivity is maintained, and under bright conditions, the human eye has full-color vision. Without proper white balance, the ambient color temperature appears unnatural. For example, an image taken indoors under fluorescent lighting will appear "greenish", an indoor tungsten lamp will appear "yellowish", and shadows can be "bluish". White balance can correct the "white point", but additional color correction is needed to balance the rest of the color spectrum. Color correction can mimic natural light or add artistic effects (e.g., making blues and oranges "pop", etc.).

[0031] After color space conversion, the output image of the first stage 102 of the IPP can be written to the DDR buffer 108A. In a particular implementation, the DDR buffer 108A can be a first-in-first-out (FIFO) buffer sized to achieve maximum IPP throughput; for example, 10-bit image data of 5.3K (15.8 megapixels) at 60 frames per second (fps) with a 1-second buffer would require approximately 10 Gbit (or 1.2 GByte) of working memory. In some cases, the storage buffer can be allocated from system memory; for example, a 10 Gbit region from a 32 Gbit DRAM can be used to provide the DDR buffer 108A. In the illustrated embodiment, for peak data rates, the storage buffer can be accessed at double data rate (DDR), but whenever possible, single data rate (SDR) should be used to minimize power consumption and improve battery life. Although for clarity the illustrated embodiment depicts two storage buffers, any number of physical storage buffers can be virtually subdivided or combined with equal success.

[0032] In an exemplary embodiment, auto exposure and color space conversion statistics can be written as metadata associated with the output image. As just one example, the auto exposure settings (ISO and shutter speed) for each image can be stored within a metadata track. Similarly, white balance and color correction adjustments can be stored within the metadata track. In some cases, additional statistics can be provided; for example, color correction can indicate "characteristic" spectra (e.g., skin color for face detection, spectral distributions associated with common landscapes (leaves, snow, water, cement), and / or specific regions of interest. In fact, some ISPs explicitly provide, for example, face detection, scene classification, and / or region of interest (ROI) detection.

[0033] One of ordinary skill in the relevant art will readily understand that the first stage 102 of the IPP may include other functionality, which is for illustrative purposes only above. As an example only, some ISPs may additionally denoise each image space before writing to the DDR buffer 108A. As used herein, "spatial denoising" refers to a noise reduction technique applied to an image area. Spatial denoising generally corrects chromatic noise (color fluctuations) and luminance noise (bright / dark fluctuations). Other examples of ISP functionality may include (but are not limited to) autofocus, image sharpening, contrast enhancement, tone mapping, and any other sensor management / image enhancement techniques.

[0034] Return reference Figure 1 , and the second stage 104 is implemented within a central processing unit (CPU) and / or a graphics processing unit (GPU). The second stage 104 retrieves the set of images from the DDR buffer 108A and incorporates sensor data to perform image stabilization and other temporal denoising. As used herein, "temporal denoising" refers to a noise reduction technique applied across multiple images.

[0035] Briefly, action photography is captured under difficult conditions that are often beyond the control of the photographer. For example, action cameras are typically used while in motion. Thus, when the footage is subsequently viewed in a stable reference frame, the relative motion between the camera motion and the subject motion can create a sense of apparent motion. In-camera image stabilization can have significant benefits for downstream processing; for example, video codecs use motion estimation between frames to compress similar video frames, and stabilizing the video results in much better compression (e.g., smaller file size, smaller quantization error, etc.).

[0036] There are various different stabilization techniques to eliminate unwanted camera motion. For example, so-called electronic image stabilization (EIS) relies on image manipulation techniques to compensate for camera motion. As used herein, "captured view" refers to the total image data that can be used for electronic image stabilization (EIS) manipulation. The "designated view" of an image is the visual portion of the image that can be presented on a display and / or used to generate frames of video content. The EIS algorithm generates a designated view to create an illusion of stability; the designated view corresponds to the "stable" portion of the captured view. In some cases, the designated view may also be referred to as a "crop" of the image, a "cropped portion" of the image, or a "punch-out" of the image.

[0037] Images captured using a sensor with an electronic rolling shutter (ERS) can also introduce unwanted rolling shutter artifacts in cases where the camera or the subject moves significantly. The ERS exposes pixel rows to light at slightly different times during image capture. Put simply, a CMOS image sensor uses two pointers to clear and write to each pixel value. The erase pointer discharges the photosensitive elements (or element rows / columns / arrays) of the sensor to erase them; the read pointer then follows the erase pointer to read the contents of the photosensitive elements / pixels. The capture time is the time delay between the erase and read pointers. Each photosensitive element / pixel accumulates light during the same exposure time, but since the pointers scan through the rows, they are not erased / read simultaneously. If the image capture device (or the subject) moves, this slight time shift between the start of each row can cause image distortion.

[0038] ERS compensation can be performed to correct rolling shutter artifacts from camera motion. In one particular implementation, the capture device determines the orientation change of the sensor during pixel acquisition to correct the input image distortion associated with the motion of the image capture device. Specifically, the orientation change between different captured pixels can be compensated for by warping, shifting, shrinking, stretching, etc. to compensate for the motion of the camera.

[0039] In some cases, temporal denoising can be used to further improve the overall quality of video content by reducing visual artifacts or distortions caused by temporal variations. This can include random noise, flicker, or other irregularities that may be present in consecutive frames of a video or image. Temporal denoising techniques eliminate the differences in pixel movement between consecutive images. This technique can be parameterized according to a temporal filter radius and a temporal filter threshold. The temporal filter radius determines the number of consecutive frames used for temporal filtering. A higher value for this setting results in more aggressive (and slower) temporal filtering, while a lower value results in less aggressive (and faster) filtering. The temporal filter threshold setting determines how sensitive the filter is to pixel changes in consecutive frames. A higher value for this setting results in more aggressive filtering and less attention to temporal variations (lower motion sensitivity). A lower value results in less aggressive filtering and more attention to temporal variations and better retention of moving details (higher motion sensitivity). Temporal denoising can include calculating pixel motion vectors between images for elimination; these calculations are, in effect, similar to the motion vector calculations performed by codecs and can predict the subsequent codec workload.

[0040] Return reference Figure 1 , the image processing pipeline includes a third stage 106. The third stage 106 encodes the image sequence into a video format for transmission, storage, and / or rendering. Complementary decoding operations can be used to decode the image sequence from the video format. Depending on the encoding technique, the encoding can be lossy or lossless (the decoded image may or may not be an exact reproduction of the original input image).

[0041] In one exemplary embodiment, the encoder is implemented within a codec configured via an application programming interface (API) call from a CPU. The codec operations can be streamlined into the following steps: opening an encoding session, determining encoder properties, determining an encoding configuration, initializing a hardware pipeline, allocating input / output (I / O) resources, encoding one or more video frames, writing an output bitstream, and closing the encoding session. More specifically, the encoding session is "opened" by calling the codec (physical hardware or virtualized software) via the API. The API allows the codec to determine its properties (e.g., encoder globally unique identifier (GUID), profile GUID, and hardware support capabilities) and its encoding configuration. In some implementations, the encoding configuration is based on real-time guidance (e.g., quantization, compression, bitrate adjustment, and / or group of pictures (GOP) size can be based on parameters provided by upstream IPP operations). Thereafter, the codec can initialize its parameters and allocate appropriate I / O resources based on its properties and encoding configuration, at which point the codec is ready to encode data. Subsequent codec operations retrieve input frames, encode the frames into an output bitstream, and write the output bitstream to a data structure for storage / transmission. After encoding is terminated, the encoding session can be "closed" to return the codec resources to the system.

[0042] In some embodiments, the real-time guidance can update and / or correct the encoding configuration during real-time capture and (in some variations) throughout the real-time capture. Specifically, the third stage 106 of the IPP can configure the encoding parameters using capture and conversion statistics (from the first stage 102) and sensed motion data (from the second stage 104) before processing. For example, the CPU can determine quantization parameters based on the auto-exposure and color space conversion statistics of the output image discussed above. In some cases, the quantization parameters can be based on pixel motion vectors obtained from the temporal denoising discussed above. Where available, face recognition, scene classification, and / or region of interest (ROI) metadata can also be used. Additionally, flag images and / or orientation information can be used to determine the GOP size. Similar adjustments can be made to compression and bitrate adjustment. Advantageously, the real-time guidance information from the previous stage can be retrieved before encoding, which is a function of the IPP pipeline. More directly, rather than buffering 1 second of images within the codec so that the codec can perform forward / backward prediction, the CPU can configure the encoding parameters of the codec based on the 1 second of real-time guidance provided by the earlier stage of the IPP.

[0043] A comprehensive list of various encoding parameters and API calls can be found at the following link (last retrieved on February 3, 2022), the entire text of which is incorporated herein by reference:

[0044] https: / / ffmpeg.org / doxygen / 3.3 / group__ENCODER__STRUCTURE.html ;

[0045] https: / / ffmpeg.org / doxygen / 3.3 / structNV__ENC__PIC__PARAMS.html ; and

[0046] https: / / ffmpeg.org / doxygen / 3.3 / structNV__ENC__CONFIG.html 。

[0047] During the encoding operation, the codec performs splitting, prediction, transformation, quantization, and entropy coding according to the above encoding parameters. Although the following discussion is presented in terms of a particular set of operations and order, other operations and / or orders may be equally successful alternatives.

[0048] Splitting divides the input image in time into temporal blocks called groups of pictures (GOPs). Each GOP contains a number of frames. Depending on the parameterization of the encoder, each frame can be assigned a frame type: intra-coded (I) frame, predicted (P) frame, bi-predicted (B) frame. The GOP structure defines the way the frame types are distributed; each GOP starts with an I frame, followed by a number of P frames and / or B frames. Each frame can be further divided spatially into blocks and in some cases into sub-blocks.

[0049] Prediction is performed based on the frame type. Intra prediction uses only blocks from the same frame, and inter prediction uses blocks from other frames. I frames use only intra prediction. P frames can use only inter prediction from previous frames; B frames can use inter prediction from both previous and future frames. P frames and B frames can also use intra prediction.

[0050] Typically, intra prediction occurs in raster scan order (e.g., from right to left, top to bottom); a block is predicted based on the horizontal line of pixels directly above it or the vertical line immediately to its left. The encoder selects the best option from a list of possible predictions, such as taking the average of these pixels or horizontally copying the row above it or vertically copying the row to its left, etc. In contrast, inter prediction uses motion estimation (ME) between two frames (e.g., previous and current, current and future, etc.) to estimate the motion vector (MV) for motion-compensated prediction (MCP). The MV can initially be approximated roughly and then refined further with local search.

[0051] Prediction is rarely perfect; thus, the error between the prediction and the actual pixel value is commonly referred to as the "residual". Different prediction modes can be selected based on the amount of residual and the motion vector between frames. In the "inter" mode, the pixel residual and the MV approximation refinement are transmitted. The inter mode is efficient when there is overall movement in many directions everywhere (e.g., no common motion vector). The "merge" mode transmits the pixel residual and only the rough MV approximation (not refined); the merge mode is efficient when there is stable and constant movement between frames. The "skip" mode does not transmit the residual or the MV approximation; the skip mode is efficient when there is little motion between frames.

[0052] Ideally, the prediction residuals are expected to mainly contain zeros and low values. The transform step converts the prediction residuals to the frequency domain, which groups the low spatial frequency information in only a few coefficients. In most video coding implementations, the transform is applied to the predicted residuals rather than the image values themselves (image coding transforms are typically applied to the image values). The most common transforms are the Discrete Cosine Transform (DCT) and the Discrete Sine Transform (DST); some encoders can actively select between different transforms.

[0053] The above steps can be implemented losslessly, but in some cases, a certain amount of loss can be tolerated. In such implementations, the quantization step can be used to quantize the residuals, etc., to reduce the data size. The quantization parameter (QP) is used to affect the amount of quantization; a larger QP results in higher quantization and loss, and a lower QP results in lower quantization and higher fidelity. The QP is a key parameter for encoder operation and strongly affects the quality and size of the encoded image.

[0054] Entropy coding translates the encoder's data and quantized transform residuals into a bitstream encapsulated in a container file. In some cases, entropy coding can additionally incorporate lossless coding to improve compression.

[0055] Additional discussion regarding the image processing pipeline, various modifications, and improvements can be found in the PCT application with serial number PCT / US23 / 62157, filed on February 7, 2023, and titled "Methods and Apparatus for Real-Time Guided Encoding", which is hereby incorporated by reference in its entirety.

[0056] Conventional dynamic coding techniques

[0057] Dynamic coding techniques seek to optimize the coding process for an optimization objective. Optimization techniques typically seek to balance multiple constraints; for example, dynamic coding techniques may attempt to trade off bitrate, quality, and computational complexity, etc.

[0058] Most trade-offs have some form of diminishing returns. For example, Figure 2 The first bitrate-quality curve 200 illustrates how the image quality varies with the bitrate used to encode a fixed-resolution image. The bitrate-quality curve 200 is characterized by a steep increase region 202, an inflection point 204, and a plateau region 206. As shown, a large increase in bitrate is required to achieve a small increase in image quality in the plateau region 206.

[0059] Figure 2Also compare multiple bitrate quality curves 250 (scaled comparison) at different resolutions. At lower bitrates, low-resolution encoding has an edge because fewer pixels are encoded. However, as the bitrate increases, the low-resolution curve quickly reaches the natural limit of fidelity. Higher resolutions are more difficult to encode at low bitrates, but the number of bits eventually becomes sufficient to encode a higher number of pixels, resulting in higher image quality (compared to upscaling low resolution).

[0060] Others have experimentally explored the observed relationships between bitrate, image quality, and resolution. For example, Aaron et al. in "Encoding Optimization by Title" published on December 14, 2015 (hereinafter referred to as "Aaron") with a last retrieval date of December 29, 2023 (the full text of which is incorporated herein by reference) observed that "each resolution has a bitrate region where it outperforms other resolutions". Combining these points describes a "convex hull", which identifies the point of diminishing returns (inflection point) for all bitrate-resolution combinations. Aaron used the convex hull to search for practical bitrate-resolution pairs that provide acceptable performance. Aaron stated that most conventional dynamic encoding works. Table 1 provides a brief overview of other notable works in this field. https: / / netflixtechblog.com / per-title-encode- optimization-7e99442b62a2

[0061]

[0062]

[0063] Table 1: Notable Dynamic Encoding Implementations

[0064] For bandwidth and / or playback considerations, most research on encoding techniques has focused on dynamically encoding video. For example, VOD services are typically limited to a few common supported resolutions (1080p, 4K, etc.) and must adapt their services within the available network capacity (bitrate). Similarly, live streaming is constrained. Practical solutions for these applications typically focus on fixed "ladders" of bitrate-resolution pairs. Table 2 provides an example of the bitrate-resolution ladder described by Aaron.

[0065] Bitrate (kbps) Resolution 235 320x240 375 384x288 560 512x384 750 512x384 1050 640x480 1750 720x480 2350 1280x720 3000 1270x720 4300 1920x1080 5800 1920x1080

[0066]

[0067] Figure 3 Table 2: Notable Dynamic Encoding ImplementationsImportantly, these ladders apply to all videos, regardless of their content. Although they are chosen to produce good encodings for most general content, they are not individually selected for specific content (titles). Interesting examples are live-action content and animated content. Due to the simplicity of cartoons, sufficient quality can be achieved at lower bitrates even for high resolutions. On the other hand, complex textures and live-action movement require more bits in their encoding. This effect is in Figure 3Rate-quality curve illustration for videos encoded with different complexities but the same encoder configuration. Conceptually, this is because the actual resolution of a physical display ("display resolution") is different from the resolution required to present perceptually acceptable content. VOD and live streaming applications employ practical methods to address known constraints such as available network bandwidth and display resolution, which is easier than attempting to determine the "perceptual acceptability" of each title.

[0068] GoPro is a camera manufacturer specializing in action camera photography. Action cameras have unique application requirements different from those of other cameras and / or media services. Different from VOD and live streaming services that provide ready-to-watch content, action cameras record videos that are usually post-processed. Since post-processing can only edit the captured signal and noise (i.e., the signal cannot be recovered from the noise later), action cameras usually record videos in the highest available quality.

[0069] GoPro's current camera models (HERO 11 ) use HEVC to encode videos in single-pass constant bitrate (CBR) mode. The GOP duration is fixed (1 second), corresponding to the hardware capabilities of the image processing pipeline; for similar reasons, the GOP structure is limited to 1 I-frame followed by several P-frames. Currently, users can only start recording in one of two encoding modes: standard and high. These bitrate values are selected to produce encoded videos with a subjective good image quality for average video content. In other words, the bitrate of the encoded video is independent of the content complexity, which means that the image quality of complex scenes may be insufficient while simple scenes may waste data. Table 3 provides the frame rates and resolutions of the standard bitrate mode of the current model camera (HERO 11 ), and Table 4 provides the frame rates and resolutions of the high bitrate mode of the current model camera.

[0070]

[0071] Table 3: Standard Bitrate Mode

[0072]

[0073] Table 4: High Bitrate Mode

[0074] Even more complex is that contemporary cameras manufactured by GoPro use third-party video codecs with proprietary hardware and / or software. Proprietary codecs provide some customized codec operations but also obscure rate control and rate-distortion optimization. In other words, codec operations are mainly controlled via API calls and parameters; the rate control / distortion behavior cannot be directly changed.

[0075] To address the specific application requirements of action photography, new solutions for dynamic encoding are needed. Ideally, new solutions should adapt the target bitrate to the complexity of the content. Real-time in-camera implementations can also benefit from low additional complexity and work within the constraints of existing codecs.

[0076] Exemplary dynamic encoding technology

[0077] Various embodiments of the present disclosure dynamically configure a video encoder to achieve consistent image quality across content complexity. Specifically, in one exemplary embodiment, the codec's target bitrate is dynamically configured to maintain consistent content complexity. In other words, the exemplary solution avoids wasting bitrate on marginal quality gains for low-complexity content and increases the bitrate to handle high-complexity content.

[0078] As a related improvement, the embedded real-time variant uses a staged implementation of the image processing pipeline to provide a lightweight solution. In other words, encoding guidance from earlier pipeline stages enables predictive single-pass encoding (rather than iterative processing or brute-force parallel encoding and selection).

[0079] First, consider an illustration of single-pass constant bit rate (CBR) encoding, such as Figure 4 4. Here, a reference video is divided into 1-second increments and encoded at a constant bitrate (20 Mbps) to produce the encoded video (e.g., 1080p resolution, 30 frames per second). CBR encoding prioritizes a consistent bitrate and is also referred to herein as "equal bitrate" encoding. As shown in the first plot 400, image complexity varies over time; in other words, some portions of the reference video will have high texture and / or high motion, and other portions of the reference video will have low texture and / or low motion. However, in constant bitrate encoding, the resulting bitrate of the encoded video remains nominally constant within a threshold tolerance (as shown in the second plot 410, the bitrate is nominally 20 Mbps).

[0080] Simply put, bitrate selection has a certain tolerance threshold for nominal values because rate control is based on imperfect predictions. Typically, this variance is caused by fluctuations in video complexity. For example, rapidly switching from low-complexity video to high-complexity video will result in a significant shift in bitrate. This is because rate control will initially use a small quantization parameter (QP) to achieve low complexity, resulting in high bitrate peaks at high complexity.

[0081] Figure 4 The third plot 420 plots the image quality of each 1-second segment of the encoded video as a function of the bit rate. Here, the reference video (I ) is measured using the peak signal-to-noise ratio (PSNR) defined in Equation 1 according to the mean square error (MSE) defined in Equation 2. ref ) and reconstructed coded video (I rec) to calculate the image quality based on the difference between them.

[0082] Equation 1

[0083] Equation 2 As shown, the isometric rate distribution 422 varies greatly in image quality. In other words, low-complexity (low texture, low motion) regions are presented with high quality and relatively few artifacts; in contrast, high-complexity (high texture and / or high motion) regions are presented with low quality and multiple artifacts.

[0084] In contrast, consider Figure 5 the illustrative diagram of exemplary single-pass image quality-priority encoding (also referred to herein as "equal quality" encoding) depicted in. Similar to Figure 4 very much, the original sample video is divided into 1-second increments and encoded to produce an encoded video (e.g., 1080p resolution, 30 frames per second). However, instead of a constant bit rate, the bit rate of the encoder is dynamically configured based on pixel features identified in the previous stage and / or encoded features identified in the previous encoded frames. As discussed in more detail below, the image quality is not known until after encoding, so pixel features and encoded features are used as proxies for predicting image complexity.

[0085] As shown in the first plot 500, the image complexity varies over time. The change in complexity is predicted by pixel features and / or encoded features, which are then used to dynamically configure the bit rate of the encoder (see the second plot 510).

[0086] Figure 5 The third plot 520 of plots the image quality of each 1-second video segment as a function of the bit rate. As mentioned before, the image quality is compared by measuring the difference between the reference video and its corresponding encoded video. However, in the exemplary embodiment, the equal quality distribution 522 provides relatively consistent image quality over a wide range of bit rates (the isometric rate distribution 422 is also shown for comparison). It is worth noting that the adjustment range is limited, so certain configurations may result in irrelevant behavior. For example, when the bit rate drops to the "lower limit" bit rate (as represented by the lower limit region 524), the image quality may improve, and when the bit rate rises to the "upper limit" (as represented by the upper limit region 526), the image quality may decrease.

[0087] Figure 6It is a graphical representation of an exemplary single-pass equal-quality encoding process. As shown, an image sequence (Image 1602A, Image 2 602B, Image 3 602C) is encoded into a frame sequence (Frame 0 604A, Frame 1 604B, Frame 2604C) in a single pass. For example, pixel features from Image 1 602A and encoded features from Frame 0 604A are used to select a first bitrate 1 (BR1), and the encoder is configured according to the first bitrate 1 to generate Frame 1 604B. Similarly, pixel features from Image 2 602B and encoded features from Frame 1 604B are used to select a second bitrate 2 (BR2), and the encoder is configured according to the second bitrate 2 to generate Frame 2604C, and so on.

[0088] In one exemplary embodiment, "pixel features" refer to metadata extracted from raw pixel values and / or image processing (e.g., during Figure 1 the first stage 102). Examples of pixel features can include (but are not limited to) the average value, standard deviation, skewness, kurtosis, etc. of pixel values. Pixel features can be further subdivided into spatial features and temporal features. Examples of spatial pixel features include gray-level co-occurrence matrix (GLCM), color measurement, noise measurement, and / or spatial information (as defined in accordance with ITU-T P.910, "Subjective Video Quality Assessment Methods for Multimedia Applications," Recommendation ITU, 2022, the full text of which is incorporated herein by reference). Temporal features can be calculated for each pair of temporally adjacent images; temporal features can include normalized cross-correlation, average value of temporal coherence, standard deviation of temporal coherence, and / or temporal information (as defined in accordance with the above ITU-T P.910). More generally, in view of the content of the present disclosure, one of ordinary skill in the relevant art can equally successfully substitute any metadata derived, identified, or generated in the image signal processing from sensed optical information to pixel data.

[0089] In one exemplary embodiment, "encoded features" refer to metadata extracted from previously encoded video frames (e.g., during Figure 1During the third stage 106). Examples of encoding characteristics may include (but are not limited to) quantization parameters (QP) of the encoded frames (aggregated by median, minimum, maximum, amplitude between maximum and minimum, etc.), number of bits of the encoded frames (aggregated by average, minimum, maximum, P-frame maximum, P-frame amplitude, etc.), YUV-PSNR between the encoded frames and the reference frames (aggregated by average, standard deviation, minimum, maximum,, P-frame maximum, P-frame amplitude, etc.), percentage of intra blocks in the encoded frames (aggregated by median, minimum, P-frame maximum, P-frame amplitude, etc.), percentage of skipped blocks in the encoded frames (aggregated by median, P-frame minimum, maximum, P-frame amplitude, etc.) and / or percentage of merged blocks in the encoded frames (aggregated by median, minimum, maximum, P-frame minimum, P-frame amplitude, etc.). More generally, in view of the present disclosure, one of ordinary skill in the relevant art may equally successfully substitute any metadata derived, identified or generated from the encoding of previous frames.

[0090] Although Figure 6 depicts a single-pass encoding, the technique is more generally applicable to any time-constrained encoding process. Here, time-constrained refers to an encoding process with limited information from the future and / or the past (look-ahead / look-back buffers, etc.). Figure 7 is a graphical representation of an exemplary dual-resolution constant-quality encoding process. Dual encoding provides a low-resolution video for reducing post-playback or post-capture processing and a main-resolution video for high-quality post-processing.

[0091] As shown, an image sequence (Image 1 702A, Image 2 702B, Image 3 702C) is encoded into a first sequence of low-resolution frames (LRV Frame 1 704A, LRV Frame 2 704B, LRV Frame 3 704C) and a second sequence of main-resolution frames (MRV Frame 0 706A, MRV Frame 1 706B, MRV Frame 2 706C). Note that in this embodiment, the LRV frames are encoded before their MRV corresponding frames (e.g., LRV Frame 1 704A is successfully encoded before MRV Frame 1 706B starts encoding). In this example, the pixel characteristics from Image 1 702A, the encoding characteristics from LRV Frame 1 704A, and the encoding characteristics from MRV Frame 0 706A are used to select a first bitrate 1 (BR1), and the encoder is configured according to the first bitrate 1 to generate MRV Frame 1 706B. Similarly, the pixel characteristics from Image 2 702B, the encoding characteristics from LRV Frame 2 704B, and the encoding characteristics from MRV Frame 1 706B are used to select a second bitrate 1 (BR2), and the encoder is configured according to the second bitrate 2 to generate MRV Frame 2 706C, and so on.

[0092] More generally, the techniques described herein can be widely applied to the dynamic encoding of any optimization objective based on surrogate information obtained from a previous stage of pipeline processing. Herein, image quality / bitrate can be generalized to a subjective level of image quality, such as "fair quality", "high quality", and "visually lossless quality", etc. Although the above discussion is presented in the context of bitrate, other embodiments can be optimized for computational complexity, memory usage, bandwidth, image quality, and / or any combination of the above. Additionally, although the above discussion is presented in the context of pixel features and encoding features, other embodiments can consider various other forms of metadata, including (but not limited to): stabilization data, stitching data, user configuration, filtering data, and / or other capture information.

[0093] Surrogate data and in-camera machine learning variants

[0094] Although various concepts of the present disclosure are illustrated with reference to graphical representations of image quality as a function of bitrate, most practical applications do not have a reference image (i.e., no reference image for real-world settings). Herein, machine learning embodiments implement "open-loop" operation, which is particularly useful when optimizing unmeasurable image quality using surrogate data.

[0095] Exemplary embodiments of the present disclosure utilize pixel features and / or encoding features as surrogates for optimization objectives. Herein, the term "surrogate" and its linguistic derivatives refer to information that represents target data. Surrogate data can be useful in situations where the target data cannot be directly measured or observed. For example, pixel features and encoding features predict the image quality of an encoding process that has not yet occurred. Similarly, the captured image data may not have a reference image to compare with (i.e., the image quality is also unknown). More generally, surrogate data can be useful in any application where the target data is unknown, inaccessible, or otherwise unavailable.

[0096] Surrogate data typically represents the target data only to a certain degree of accuracy and / or precision. Herein, precision refers to the magnitude of the error between the surrogate data and the target data, while accuracy refers to the consistency of the error over multiple iterations. The degree of representation can be qualitatively and / or quantitatively characterized according to any numerical and / or statistical analysis; typically, it is expressed in terms of error margin, standard deviation, or other similar metrics.

[0097] Conceptually, the error of each individual surrogate data can be uncorrelated with other surrogates, so the "diversity" of using multiple surrogates can be used to improve the overall degree of representation. There are various statistical and / or numerical techniques for combining different surrogate data.

[0098] As independent but related tangents, the progress of machine learning has led to the widespread adoption of machine learning chips; these chips are particularly useful for implementing adaptive behavior in real-time applications. Adaptive behavior is typically described implicitly (by "learning" from training instances) rather than defined explicitly (formalized according to rules).

[0099] Machine learning models are "trained" based on a training database ("offline" or "closed-loop" mode). In closed-loop operation, the model can use feedback to improve and correct unsatisfactory results. For example, within the context of the present disclosure, a machine learning model can use pixel features and encoded features to train the objective image quality (PSNR) between an encoded image and a reference image. Once the model produces acceptable results for the training library, it can switch to real-world data ("online" or "open-loop" mode). During online operation, the model uses its trained behavior to produce results without feedback based on errors. In other words, the machine learning model does not need to compare the resulting encoded image with the reference image to determine the image quality.

[0100] In one exemplary embodiment, machine learning logic is used to select an optimization objective based on pixel features and / or encoded features. In a particular implementation, "random forest" machine learning logic provides bitrate selection within the embedded constraints of a camera device. "Random forest" refers to an ensemble learning method for classification or regression tasks. A random forest typically consists of a collection of decision trees, each of which is a node that makes a decision based on its input values. Each decision tree is independently trained on a random subset of the training data and makes its own prediction. In addition to using random subsets of data, random forest training also introduces additional randomness by considering only a random subset of features at each split in the decision tree. This helps to decorrelate the individual trees and improve the robustness of the overall model. During a classification task, the random forest combines the predictions of the individual trees through a majority voting mechanism. For a regression task, it averages the predictions of the individual trees.

[0101] Random forests are well-suited to noisy data and can generalize well to new unseen instances. The combination of multiple trees helps to reduce overfitting, making the model more robust. Random forests also provide a measure of feature importance, indicating the contribution of each feature to the model's prediction. This helps to understand the most influential factors in a given problem.

[0102] In another embodiment, the machine learning logic may use Gradient Boosting Trees (GBT) to provide bitrate selection. Gradient boosting is a technique for building additive models in a forward stagewise manner, where each new tree corrects the errors of the previously trained trees. GBTs typically consist of shallow decision trees, often referred to as "base learners" or "weak learners" because their predictions are only slightly better than random guessing (50 / 50). Trees are added sequentially, and each tree is trained to correct the residuals (the difference between the actual and predicted values) of the set of trees built so far; in other words, the weak learners are arranged in sequence such that each subsequent tree focuses on correcting the errors made by its previous tree. The training algorithm uses the gradient boosting technique to add trees to minimize the loss function of the ensemble; the algorithm identifies the direction (gradient) in which the loss function is steepest and updates the model in this direction. In some variants, a learning rate parameter is used to control the contribution of each tree to the overall ensemble. A lower learning rate makes the model more robust, but requires more trees to achieve similar performance. Similarly, GBTs may incorporate regularization techniques to prevent overfitting, such as limiting the depth of each tree or adding a penalty for complex models.

[0103] In one exemplary embodiment, the machine learning logic is trained to perform image classification using surrogate data. Classification refers to the general area of machine learning techniques for labeling input data. In this case, the machine learning logic is trained to classify input images according to complexity based on pixel features and / or encoded features; the resulting classification is used to select an appropriate bitrate. In one particular instance, training samples may be extracted from a video library and segmented into 1-second chunks for each previous encoded bitrate (and in some cases, the current LRV encoded bitrate) and each PSNR threshold. The machine learning logic is trained according to "ground truth labels", which are based on the actual PSNR of the encoded video compared to a reference video. Some training techniques may additionally weight the samples and / or provide ordered classification (ordinal classes) according to the impact of misclassification.

[0104] More generally, the techniques described herein may be widely applied to any scenario for mapping surrogate data to optimized target data. While the above discussion is presented in the context of machine learning, other embodiments may use explicit rules and logic that govern the optimization behavior. Explicit rules are relatively less complex and can mishandle unknown scenarios; however, they are simpler to implement and can be executed with lower resource utilization (processing cycles, memory space, power, etc.).

[0105] Lower and upper bound variations

[0106] As described above, exemplary constant-quality encoding implementations may have "lower bound" and "upper bound" regions where the control parameter (bit rate) is outside the adjustment range of the optimization objective (image quality). Some implementations may assume that lower bound and upper bound behavior is rare, or otherwise acceptable "as is". Other implementations may use special handling for these regions.

[0107] One exemplary implementation dynamically adjusts the optimization objective based on bit rate selection within the lower bound and upper bound regions. Consider Figure 8 an example where the optimization objective is initially set to a first image quality 802A; this optimization objective has a constant-quality distribution region 822 where the bit rate can be dynamically adjusted, a lower bound region 824 where the bit rate cannot be further reduced, and an upper bound region 826 where the bit rate cannot be further increased. During normal (constant-quality) operation, the bit rate selection logic selects the bit rate based on surrogate data within the constant-quality distribution region 822. However, if the surrogate data predicts a very high complexity (high texture, high motion) where the bit rate will be selected in the upper bound region 826, then the optimization objective (image quality) can be reduced to 802B (with corresponding adjustments to its constant-quality region, lower bound region, and upper bound region). Conversely, if the surrogate data predicts a very low complexity (low texture, low motion) where the bit rate will be selected in the lower bound region 824, then the optimization objective can be increased to 802C (with corresponding adjustments to the constant-quality region, lower bound region, and upper bound region).

[0108] Another exemplary implementation dynamically adjusts the video resolution based on bit rate selection within the lower bound and upper bound regions. Consider Figure 9 an example where the video resolution is initially set to a first resolution 902A; this resolution has a constant-quality distribution region 922 where the bit rate can be dynamically adjusted, a lower bound region 924 where the bit rate cannot be further reduced, and an upper bound region 926 where the bit rate cannot be further increased. During normal (constant-quality) operation, the bit rate selection logic selects the bit rate based on surrogate data within the constant-quality distribution region 922. However, if the surrogate data predicts a very high complexity (high texture, high motion) where the bit rate will be selected in the upper bound region 926, then the resolution can be reduced to 902B. Conversely, if the surrogate data predicts a very low complexity (low texture, low motion) where the bit rate will be selected in the lower bound region 924, then the resolution can be increased to 902C.

[0109] The above examples demonstrate various modalities of dynamic codec configuration. Here, a first operating mode (modality) can adjust codec parameters based on an optimization goal. For example, the bitrate can be adjusted to maintain a consistent image quality (equal quality). This modality can provide "fine tuning" that occurs periodically, e.g., every frame interval and / or GOP interval (once per second, etc.). A second mode can adjust the optimization goal itself (e.g., change the target image quality); this modality can provide "coarse tuning" on an aperiodic basis, but not more frequently than the frame interval or GOP interval granularity. A third mode can affect codec parameters (e.g., resolution, frame rate, GOP structure, etc.); these can be "major adjustments" that can change the codec pipeline state (e.g., flush stale data, reset the codec state, etc.) and / or the codec timing / delay. More generally, the various techniques described herein can be widely extended to any dynamic coding system with multiple modalities, where each modality encapsulates a specific set of conditions, behaviors, or characteristics of the encoding process.

[0110] Various embodiments can additionally incorporate hysteresis and / or differential information to switch between modalities. For example, a hysteresis system can consider the time-averaged bitrate to avoid sudden changes in bitrate (avoid "churn"). Similarly, a differential system can consider the difference between the current, previous, and / or next bitrates to quickly compensate for large shifts in bitrate. Given the content of this disclosure, those of ordinary skill in the relevant art will readily appreciate various other schemes for addressing "leading" and "lagging" shifts.

[0111] Technical Improvements and Other Considerations

[0112] The above systems and methods address the technical problems associated with encoder configuration for real-time capture in industrial practice. Conventional video coding techniques are optimized for content delivery networks where the content is typically encoded once for delivery. In practice, conventional encoders have an unconstrained ability to look ahead or look back in the video to maximize compression and video quality. In many cases, such encoders improve compression performance by increasing the search space both in terms of the number of frames saved in memory and the intra-pixel search for motion estimation. These techniques typically "do their best" given unconstrained processing power and memory. Action photography often must capture the shot in real time as it occurs. Additionally, the form factor requirements for action cameras can impose severe embedded constraints (processing power, storage space). More directly, the above techniques overcome the problems introduced by and rooted in the unconventional nature of action photography.

[0113] As noted by relevant parties, conventional video encoding assumes a division of tasks among dedicated devices. For example, studio-quality footage is typically captured with dedicated cameras, and the encoding is optimized for computationally intensive environments such as server farms and cloud computing. Embedded devices create efficiency opportunities that are not otherwise available in disparate devices. For example, action cameras can have shared memory among various processing units, allowing in-place data processing rather than moving data across a data bus between processing units. As a specific optimization, in-camera image signal processing (ISP) and / or dual encoding (e.g., LRV / MRV) can provide pixel features and / or encoding feature information that can be used as inputs to ongoing encoding operations. More directly, the techniques described herein are capable of making specific improvements to the operation of computers, particularly those of a mobile / embedded nature.

[0114] In addition, the various techniques described throughout utilize supplementary data to improve the real-time encoding of the primary data stream. Just as an example of this, image signal processing (ISP) data and / or low-resolution video (LRV) are supplementary data and are not widely available on general-purpose cameras or computing devices. Additionally, conventional encoded media do not contain supplementary data because it is not displayed during normal playback. Thus, the improvements described throughout relate to specific components that play an important role in real-time encoding.

[0115] System Architecture

[0116] Figure 10 is a logical block diagram of an exemplary system 1000 that includes: an encoding device 1100, a decoding device 1200, and a communication network 1002. The encoding device 1100 can capture data in real time (or near real time) and encode the captured data for transmission directly or via the communication network 1002 to the decoding device 1200. In some cases, video can be streamed in real time over the communication network 1002.

[0117] Although the following discussion is presented in the context of the encoding device 1100 and the decoding device 1200, those of ordinary skill in the relevant art will readily appreciate that the techniques can be extended widely to other topologies and / or systems. For example, an encoding device can transmit an equal-quality encoded video to a streaming media server for real-time streaming to a streaming media client (e.g., this would require re-encoding to an equal-bitrate format). As another example, a capture device can capture media at a constant bitrate (possibly with metadata), and the media is provided to an encoding device for equal-quality encoding.

[0118] The following discussion provides a functional description of various logical entities of exemplary system 1000. Those of ordinary skill in the relevant art will readily appreciate that other logical entities that perform the same work in substantially the same manner to achieve the same result are equivalent and can be freely interchanged. A specific discussion of the structural implementation, internal operation, design considerations, and / or alternatives of each of the logical entities of exemplary system 1000 is provided separately below.

[0119] Functional Overview of the Encoding Device

[0120] Functionally, encoding device 1100 encodes a continuous image stream into video based on proxy data. On the one hand, encoding device 1100 collects and / or generates proxy data to predict image quality using machine learning logic for encoding. On the other hand, encoding device 1100 performs real-time (or near-real-time) encoding with a nominally consistent image quality (equal quality). On yet another hand, encoding device 1100 can shift between different encoding control modes.

[0121] The techniques described throughout can be widely applied to encoding devices such as: cameras including action cameras, digital cameras, digital video cameras; cellular phones; laptop computers; smartwatches; and / or IoT devices. For example, a smartphone or a laptop computer is capable of capturing and processing video. Given the content of this disclosure, those of ordinary skill in the art can equally successfully substitute various other applications.

[0122] Figure 11 is a logical block diagram of exemplary encoding device 1100. Encoding device 1100 includes: a sensor subsystem, a user interface subsystem, a communication subsystem, a control and data subsystem, and a bus for enabling data transfer. The following discussion provides a specific discussion of the internal operation, design considerations, and / or alternatives of each subsystem of exemplary encoding device 1100.

[0123] As used herein, the term "real-time" refers to a task that must be performed within explicit constraints; for example, a camera must capture each frame of video at a specific capture rate (e.g., 30 frames per second (fps)). As used herein, the term "near-real-time" refers to a task that must be performed within explicit time constraints once it has started; for example, a smartphone can use near-real-time rendering for each frame of video at its specific display rate, but some queuing time can be allocated before display.

[0124] Unlike real-time tasks, so-called "best-effort" refers to tasks that can be handled with variable bitrates and / or latencies. Best-effort tasks are typically not time-sensitive and can run as low-priority background tasks (even for very high-complexity tasks) or be queued for cloud-based processing, etc.

[0125] Functional Overview of the Sensor Subsystem

[0126] Functionally, the sensor subsystem senses the physical environment and captures and / or records the sensed environment as data. In some embodiments, the sensor data may be stored as a function of the capture time (so-called "tracks"). The tracks may be synchronized (aligned) or asynchronous (unaligned) with each other. In some embodiments, the sensor data may be compressed, encoded, and / or encrypted into a data structure (such as MPEG, WAV, etc.).

[0127] The illustrated sensor subsystem includes: a camera sensor 1110, a microphone 1112, an accelerometer (ACCL 1114), a gyroscope (GYRO 1116), and a magnetometer (MAGN 1118).

[0128] Other sensor subsystem implementations may multiply, combine, further subdivide, enhance, and / or incorporate the above functionality within these or other subsystems. For example, two or more cameras may be used to capture panoramic (e.g., wide-angle or 360°) or stereoscopic content. Similarly, two or more microphones may be used to record stereo sound.

[0129] In some embodiments, the sensor subsystem is an integral part of the encoding device 1100. In other embodiments, the sensor subsystem may be enhanced by external devices and / or detachable components (e.g., hot shoe / cold shoe attachments, etc.). The following sections provide a detailed description of the individual components of the sensor subsystem.

[0130] Camera Implementation and Design Considerations

[0131] In one exemplary embodiment, the camera lens bends (distorts) light to focus on the camera sensor 1110. In one particular implementation, the optical properties of the camera lens are mathematically described by a lens polynomial. However, more generally, any characterization of the camera lens optical properties may be equally successful as a substitute; such characterizations may include (but are not limited to): polynomials, trigonometric functions, logarithms, look-up tables, and / or their piecewise or mixed functions. In one variant, the camera lens provides a wide field of view greater than 90°; examples of such lenses may include, for example, a panoramic lens of 120° and / or a super-hemispherical lens of 180°.

[0132] In one particular implementation, the camera sensor 1110 senses light (brightness) via a photosensor (such as a CMOS sensor). The color filter array (CFA) values provide the color (chromaticity) associated with each sensor. The combination of each brightness and chromaticity value provides a mosaic of discrete red, green, and blue values / positions, which can be "demosaicked" to recover the digital tuple (RGB, CMYK, YUV, YCrCb, etc.) of each pixel of the image.

[0133] However, more generally, the various techniques described herein can be widely applied to any camera assembly; including, for example, narrow fields of view (30° to 90°) and / or stitching variants (e.g., 360° panoramas). While the above techniques are described in the context of visible light, the techniques can be applied to other EM radiation capture and focusing devices, including (but not limited to): infrared, ultraviolet, and / or X-ray, etc.

[0134] Put simply, "exposure" is based on three parameters: aperture, ISO (sensor gain), and shutter speed (exposure time). Exposure determines how bright or dark an image will appear when the camera captures it. During normal operation, a digital camera can automatically adjust one or more settings including aperture, ISO, and shutter speed to control the amount of light received. Due to form factor limitations and its most common use cases (different lighting conditions), most action cameras are fixed-aperture cameras, which only adjust ISO and shutter speed. Traditional digital photography allows users to set fixed values and / or ranges to achieve desired aesthetic effects (e.g., shooting location, blur, depth of field, noise, etc.).

[0135] The term "shutter speed" refers to the amount of time light is captured. Historically, a mechanical "shutter" was used to expose film; even in digital cameras lacking such a mechanism, the term "shutter" is still used. For example, some digital cameras use an electronic rolling shutter (ERS), which exposes pixel rows at slightly different times during image capture. Specifically, a CMOS image sensor uses two pointers to clear and write to each pixel value. The erase pointer discharges the photosensitive cells (or cell rows / columns / arrays) of the sensor to erase them; the read pointer then follows the erase pointer to read the contents of the photosensitive cells / pixels. The capture time is the time delay between the erase and read pointers. Each photosensitive cell / pixel accumulates light for the same exposure time, but since the pointers scan through the rows, they are not erased / read simultaneously. Faster shutter speeds have shorter capture times, and slower shutter speeds have longer capture times.

[0136] The related term "shutter angle" describes the shutter speed relative to the frame rate of a video. A shutter angle of 360° means capturing all the motion from one video frame to the next. For example, a video at 24 frames per second (FPS) with a 360° shutter angle will expose the photosensitive sensor for 1 / 24 of a second. Similarly, 120 FPS with a 360° shutter angle will expose the photosensitive sensor for 1 / 120 of a second. In low light, the camera will typically expose for a longer time, thus increasing the shutter angle to cause more motion blur. A larger shutter angle results in a more flexible and smoother motion because the end of the blur in one frame extends into the next frame closer to the start of the blur. A smaller shutter angle results in jerkiness and disjointedness because the blur gap between the discrete frames of the video increases. In some cases, a smaller shutter angle is desired for capturing clear details in each frame. For example, the most common setting in a cinema is a shutter angle of nearly 180°, which is equal to a shutter speed of nearly 1 / 48 of a second at 24 FPS. Some users may use other shutter angles (shorter than 180°) that mimic old newsreels from the 1950s.

[0137] In some embodiments, the camera resolution directly corresponds to light information. In other words, a Bayer sensor can match one pixel to color and light intensity (each pixel corresponding to a picture element). However, in some embodiments, the camera resolution does not directly correspond to light information. Some high-resolution cameras use an N-Bayer sensor that groups four or even nine pixels per picture element. During image signal processing, the color information is redistributed among the pixels through a technique called "pixel binning". Pixel binning provides better results and versatility than just interpolation / enlargement. For example, a camera can capture a high-resolution image (e.g., 108M pixels) in full light; but in low light conditions, the camera can simulate a much larger picture element with the same sensor (e.g., grouping pixels into 9 groups to get a 12M pixel "unbinned" resolution). Unfortunately, piecing together picture elements can cause light "leakage" (i.e., sensor noise) between adjacent pixels. In other words, smaller sensors and smaller picture elements increase noise and reduce the dynamic range.

[0138] Microphone Implementation and Design Considerations

[0139] In a particular implementation, the microphone 1112 senses sound vibrations and converts the vibrations into an electrical signal (via a transducer, condenser, etc.). The electrical signal can be further transformed into frequency domain information. The electrical signal is provided to an audio codec that samples the electrical signal and converts the time-domain waveform into its frequency-domain representation. Typically, additional filtering and noise reduction can be performed to compensate for microphone characteristics. The resulting audio waveform can be compressed for delivery via any number of audio data formats.

[0140] Commercial audio codecs are generally divided into speech codecs and full-spectrum codecs. Full-spectrum codecs use modified discrete cosine transform (mDCT) and / or Mel-frequency cepstral coefficients (MFCC) to represent the full audible spectrum. Speech codecs mimic voice communication by taking advantage of the characteristics of the human auditory / voice system to reduce coding complexity. Speech codecs typically make significant trade-offs to maintain intelligibility, pleasantness, and / or data transmission considerations (robustness, latency, bandwidth, etc.).

[0141] However, more generally, the various techniques described herein can be widely applied to any integrated or handheld microphone or microphone array, including, for example, boom and / or shotgun microphones. Although the above techniques are described in the context of a single microphone, multiple microphones can be used to collect stereo and / or implement audio processing. For example, any number of individual microphones can be used to constructively and / or destructively combine sound waves (also known as beamforming).

[0142] IMU Implementation and Design Considerations

[0143] An inertial measurement unit (IMU) includes one or more accelerometers, gyroscopes, and / or magnetometers. In one particular implementation, an accelerometer (ACCL 1114) measures acceleration, and a gyroscope (GYRO 1116) measures rotation in one or more dimensions. These measurements can be mathematically transformed into a four-dimensional (4D) quaternion to describe the device motion, and electronic image stabilization (EIS) can be used to offset the image orientation to counteract the device motion (e.g., CORI / IORI 1120). In one particular implementation, a magnetometer (MAGN 1118) can provide a magnetic north vector (which can be used for "north locking" video and / or enhancing location services such as GPS). Similarly, an accelerometer (ACCL 1114) can also be used to calculate a gravity vector (GRAV 1122).

[0144] Generally, accelerometers use a damped mass and spring assembly to measure proper acceleration (i.e., acceleration in its own instantaneous rest frame). In many cases, accelerometers can have a variable frequency response. Most gyroscopes use a rotating mass to measure angular velocity; MEMS (microelectromechanical) gyroscopes can use a pendulum mass to achieve a similar effect by measuring the perturbation of the pendulum. Most magnetometers use ferromagnetic elements to measure the vector and intensity of a magnetic field; other magnetometers can rely on induced current and / or pickup coils. An IMU uses acceleration, angular velocity, and / or magnetic information to calculate a quaternion that defines the relative motion of an object in four-dimensional (4D) space. The quaternion can be effectively calculated to determine velocity (both device orientation and velocity).

[0145] However, more generally, any scheme for detecting device velocity (direction and speed) can equally successfully replace any of the above tasks. Although the above techniques are described in the context of an inertial measurement unit (IMU) that provides quaternion vectors, those of ordinary skill in the relevant art will readily appreciate that the raw data (acceleration, rotation, magnetic field) and any of its derivatives can equally successfully be replaced.

[0146] Agent data based on general sensors

[0147] As previously mentioned, exemplary embodiments of the present disclosure use sensed data to create sensor-based agent data in real time (or near real time) for control and data subsystems. For example, some image sensors can identify characteristics of pixel data (average pixel value, standard deviation, skewness, kurtosis, etc.) related to the image texture that can be used to infer the likelihood of high-texture content. This gives rise to various different coding problems (e.g., aliasing when sampling repeated patterns, quantization when quantifying gradients, etc.). Other examples include inferring the likelihood of motion blur from camera motion and / or subject motion. Additionally, sensors in an image processing pipeline can collect information for, e.g., autofocus, color correction, white balance, and / or other automatic image enhancements.

[0148] Conceptually, motion and texture are generally closely related to image complexity; these two factors can be sensed by sensors within the device. For example, an IMU can be used to physically sense device motion; although device motion is not a perfect simulation of the apparent motion of a video, the two tend to be closely related. Similarly, many camera sensors incorporate logic for automatically detecting certain types of textures and patterns, and this information can be used for object detection, face detection, etc. This information can also be used as an agent for image texture, etc. More generally, sensor-based agent data can be broadly extended to any sensed data that can affect the quality of sampled data (and by extension, coding quality).

[0149] A variety of different factors can affect sensor operation (and by extension, the quality of the resulting sampled data). Some image sensors can infer operation outside of their expected or calibrated range, which can introduce noise (e.g., the received light can be below the minimum ISO setting, which may result in image graininess). As another example, very large or very small magnitude gradients are affected by sensor tolerances. More generally, sensor-based operating condition agent data can be used as agent data for sample quality. Examples can include conditions that affect sensor sensitivity, analog-to-digital conversion (bit depth, quantization, etc.), accuracy / precision (signal-to-noise ratio), resolution, range, linearity, response time, stability, environmental conditions, durability / lifetime, power consumption, etc.

[0150] Although the foregoing discussion is presented in the context of photography, the techniques are widely applicable to any medium (e.g., audio, visual, tactile, etc.). For example, audio encoding can benefit from certain types of sensor-based proxy data. For example, directional or stereo microphones can capture audio waveforms and use inertial measurements to infer the likelihood of motion-based sound artifacts. Similarly, certain types of noise (e.g., wind noise, etc.) can greatly affect the resulting dynamic audio quality (e.g., whispering, shouting, etc.). Additionally, although the discussion presented throughout is in the context of media suitable for human consumption, the techniques can be equally successfully applied to other types of environmental data (e.g., temperature, lidar, radar, sonar, etc.). This data can be useful in applications including, but not limited to: computer vision, industrial automation, autonomous vehicles, Internet of Things (IoT), etc.

[0151] Functional Overview of the User Interface Subsystem

[0152] Functionally, the user interface subsystem 1124 can be used to present media to a human user and / or receive input from a human user. The media can include any form of auditory, visual, and / or tactile content suitable for human consumption. Examples include images, videos, sounds, and / or vibrations. The input can include any data input directly (via user input) or indirectly (e.g., by reference to a configuration file or other source) by the user.

[0153] The illustrated user interface subsystem 1124 can include: a touchscreen, physical buttons, and a microphone. In some embodiments, input can be interpreted from touchscreen gestures, button presses, device motion, and / or commands (spoken aloud). The user interface subsystem can include physical components (e.g., buttons, keyboards, switches, rollers, etc.) or virtualized components (via the touchscreen).

[0154] Other user interface subsystem 1124 implementations can multiply, combine, further subdivide, enhance, and / or incorporate the foregoing functionality within these or other subsystems. For example, audio input can incorporate elements of a microphone (discussed above with respect to the sensor subsystem). Similarly, IMU-based input can incorporate the foregoing IMU to measure "jitter," "bump," and other gestures.

[0155] In some embodiments, the user interface subsystem 1124 is an integral part of the encoding device 1100. In other embodiments, the user interface subsystem can be enhanced by an external device (e.g., the decoding device 1200 discussed below) and / or a removably attached component (e.g., a hot shoe / cold shoe attachment, etc.). The following sections provide a detailed description of the individual components of the sensor subsystem.

[0156] Touchscreen and Button Implementation and Design Considerations

[0157] In some embodiments, the user interface subsystem 1124 may include a touch screen panel. A touch screen is a combination of a touch-sensitive panel superimposed on a visual display. Typical displays are liquid crystal displays (LCDs), organic light emitting diodes (OLEDs), and / or active matrix OLEDs (AMOLEDs). Touch screens are commonly used to enable a user to interact with a dynamic display, which provides both flexibility and an intuitive user interface. In the context of an action camera, touch screen displays are particularly useful because they can be sealed (waterproof, dustproof, shockproof, etc.).

[0158] Most commercial touch screen displays are resistive or capacitive. Generally, these systems use changes in resistance and / or capacitance to sense the position of a human finger or other touch input. Other touch screen technologies may include, for example, surface acoustic wave, surface capacitance, projected capacitance, mutual capacitance, and / or self-capacitance. Other similar technologies may include, for example, projection screens with optical imaging and / or computer vision.

[0159] In some embodiments, the user interface subsystem 1124 may also include mechanical buttons, a keyboard, switches, a scroll wheel, and / or other mechanical input devices. Mechanical user interfaces are commonly used to open or close a mechanical switch, which results in a distinguishable electrical signal. Although physical buttons may be more difficult to seal components, they can still be used in low-power applications because they do not require active current consumption. For example, many BLE applications can be triggered by pressing a physical button to further reduce GUI power requirements.

[0160] However, more generally, any scheme for detecting user input may equally successfully replace any of the above tasks. Although the above technologies are described in the context of touch screens and physical buttons for implementing user data input, those of ordinary skill in the relevant art will readily understand that any derivatives thereof may equally successfully be replaced.

[0161] Microphone / Speaker Implementations and Design Considerations

[0162] Audio input may incorporate a microphone and codec (discussed above) and a speaker. As previously mentioned, the microphone can capture and convert the audio of voice commands. For auditory feedback, the audio codec can obtain the audio data and decode the data into an electrical signal. The electrical signal can be amplified and used to drive the speaker to produce sound waves.

[0163] As previously mentioned, the microphone and speaker can have any number of microphones and / or speakers for beamforming. For example, two speakers can be used to provide stereo. Multiple microphones can be used to collect both the user's voice commands and ambient sounds.

[0164] Functional Overview of the Communication Subsystem

[0165] Functionally, the communication subsystem can be used to transfer data to and / or receive data from external entities. The communication subsystem is typically divided into a network interface and a removable media (data) interface. The network interface is configured to communicate with other nodes of a communication network according to a communication protocol. Data can be received / transmitted as transient signals (e.g., electrical signals on a transmission medium). The data interface is configured to read / write data to a removable non-transitory computer-readable medium (e.g., a flash drive or similar storage medium).

[0166] The illustrated network / data interface 1126 can include network interfaces, including (but not limited to): Wi-Fi, Bluetooth, Global Positioning System (GPS), USB, and / or Ethernet interfaces. Additionally, the network / data interface 1126 can include data interfaces such as Secure Digital (SD) cards (and their derivatives) and / or any other optical / electrical / magnetic media (e.g., MMC cards, CDs, DVDs, magnetic tapes, etc.).

[0167] Network Interface Implementation and Design Considerations

[0168] The communication subsystem of the encoding device 1100 including the network / data interface 1126 can include one or more radios and / or modems. As used herein, the term "modem" refers to a modulator-demodulator for converting computer data (digital) into waveforms (baseband analog). The term "radio" refers to the front-end part of the modem that up-converts baseband analog waveforms to RF carrier frequencies and / or down-converts from RF carrier frequencies.

[0169] As previously mentioned, the communication subsystem with the network / data interface 1126 can include wireless subsystems (e.g., 5th / 6th generation (5G / 6G) cellular networks, Wi-Fi, Bluetooth (including Bluetooth Low Energy (BLE) communication networks), etc.). Additionally, the techniques described throughout can be equally successfully applied to wired network devices. Examples of wired communications include (but are not limited to) Ethernet, USB, PCI-e. Additionally, some applications can operate within a hybrid environment and / or task. In such cases, multiple different connections can be provided via multiple different communication protocols. Other network connection solutions can be equally successfully substituted.

[0170] More generally, any scheme for transferring data via transient media can be equally successfully substituted for any of the above tasks.

[0171] Data Interface Implementation and Design Considerations

[0172] The communication subsystem of the encoding device 1100 can include one or more data interfaces for removable media. In one exemplary embodiment, the encoding device 1100 can read from and write to Secure Digital (SD) cards or similar card memories.

[0173] Although the foregoing discussion has been presented in the context of SD, those of ordinary skill in the relevant art will readily appreciate that other removable media may be equally successfully substituted (e.g., flash drives, MMC cards, etc.). Additionally, the techniques described throughout may be equally successfully applied to optical media (e.g., DVDs, CD-ROMs, etc.).

[0174] More generally, any scheme for storing data onto non-transitory media may equally successfully substitute for any of the foregoing tasks.

[0175] Functional Overview of the Control and Data Processing Subsystem

[0176] Functionally, the control and data processing subsystem is used to read / write and store data to enable the computation and / or actuation of the sensor subsystem, user interface subsystem, and / or communication subsystem. Although the following discussion is presented in the context of a processing unit executing instructions stored in a non-transitory computer-readable medium (memory), other forms of control and / or data may be equally successfully substituted, including, for example, neural network processors, dedicated logic (field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs)), and / or other software, firmware, and / or hardware implementations.

[0177] As Figure 11 shown, the control and data subsystem may include one or more of the following: a central processing unit (CPU1106), an image signal processor (ISP 1102), a graphics processing unit (GPU 1104), a codec 1108, and a non-transitory computer-readable medium 1128 storing program instructions and / or data. In some embodiments, a neural network processing unit (NPU 1107) for machine learning applications may be additionally included.

[0178] Processor-Memory Implementations and Design Considerations

[0179] In fact, different processor architectures attempt to optimize their designs for their most likely uses. More specialized logic generally leads to higher performance (e.g., by avoiding unnecessary operations, memory accesses, and / or conditional branches). For example, a general-purpose CPU (e.g., Figure 6 shown) may be primarily used to control device operations and / or execute tasks of arbitrary complexity / best effort. CPU operations may include (but are not limited to): general-purpose operating system (OS) functionality (power management, UX), memory management, etc. Typically, such CPUs are selected to have a relatively short pipeline, a long word (e.g., 32-bit, 64-bit, and / or superscalar word), and / or an addressable space that can access both local cache memory and / or system virtual memory pages. More directly, a CPU typically switches between tasks and must account for branch interrupts and / or any memory accesses.

[0180] In contrast, an Image Signal Processor (ISP) repetitively performs many identical tasks on well - defined data structures. Specifically, the ISP maps the captured camera sensor data into a color space. ISP operations typically include (but are not limited to): demosaicking, color correction, white balance, and / or auto - exposure. Most of these operations can be accomplished through scalar - vector - matrix multiplications. The raw image data has a defined size and capture rate (for video) and the same ISP operations are performed on each pixel; thus, ISP designs are highly pipelined (and have few branches), can incorporate specialized vector - matrix logic, and typically rely on reducing the addressable space and other task - specific optimizations. The ISP design only needs to understand the camera sensor output to stay within the real - time budget; thus, ISPs typically benefit from larger registers / data structures and do not require parallelization. In many cases, the ISP can execute its own Real - Time Operating System (RTOS) locally to schedule tasks according to real - time constraints.

[0181] Very similar to the ISP, the GPU is mainly used to modify image data and can be highly pipelined (with few branches), and can incorporate specialized vector - matrix logic. However, different from the ISP, the GPU usually accelerates image processing for the CPU, so the GPU needs to operate on multiple images at once and / or perform other image processing tasks of arbitrary complexity. In many cases, GPU tasks can be parallelized and / or are constrained by the real - time budget. GPU operations can include (but are not limited to): stabilization, lens correction (stitching, warping, stretching), image correction (coloring, blending), noise reduction (filtering, etc.). The GPU can have a much larger addressable space, which can access both local cache memory and / or system virtual memory pages. Additionally, the GPU can include multiple parallel cores and load - balancing logic to manage, for example, power consumption and / or performance. In some cases, the GPU can execute its own operating system locally to schedule tasks according to its own scheduling constraints (pipelines, etc.).

[0182] The hardware codec converts image data into encoded data for transmission and / or converts encoded data into image data for playback. Very similar to the ISP, hardware codecs are typically designed according to specific use cases and are highly commoditized. Typical hardware codecs are highly pipelined, can incorporate Discrete Cosine Transform (DCT) logic (which is used by most compression standards), and usually have large internal memories to store multiple video frames for motion estimation (spatial and / or temporal). Like the ISP, codecs typically encounter network connection and / or processor bandwidth bottlenecks, so codecs are rarely parallelized and can have specialized data structures (such as registers that are multiples of the image row width, etc.). In some cases, the codec can execute its own operating system locally to schedule tasks according to its own scheduling constraints (bandwidth, real - time frame rate, etc.).

[0183] Other processor subsystems may implement the above functionality multiplied, combined, further subdivided, enhanced, and / or incorporated within these or other processing elements. For example, multiple ISPs may be used to serve multiple camera sensors. Similarly, codec functionality may be included within GPU or CPU operations via software emulation.

[0184] In one embodiment, the memory subsystem may be used to locally store data at the encoding device 1100. In an exemplary embodiment, the data may be stored as non-transitory symbols (e.g., bits read from a non-transitory computer-readable medium). In a particular implementation, the memory subsystem including the non-transitory computer-readable medium 1128 is physically implemented as one or more physical memory chips (e.g., NAND / NOR flash) that are logically separated into memory data structures. The memory subsystem may be divided into program code 1130 and / or program data 1132. In some variations, the program code and / or program data may be further organized for dedicated and / or collaborative use. For example, the GPU and CPU may share a common storage buffer to facilitate large amounts of data transfer therebetween. Similarly, the codec may have a dedicated storage buffer to avoid resource contention.

[0185] In some embodiments, the program code may be statically stored as firmware within the encoding device 1100. In other embodiments, the program code may be dynamically stored (and variable) via a software update. In some such variations, the software may then be updated by external parties and / or users based on various access rights and procedures.

[0186] Neural network and machine learning implementations

[0187] Historically, machine learning logic has typically been implemented as large vector-matrix operations that can be executed on dedicated vector-matrix logic (e.g., found in GPUs). However, more recently, machine learning logic can be implemented as a completely separate logic dedicated to accelerating neural network computations. Generally, an NPU includes hardware acceleration for highly parallel matrix multiplication and non-linear processing (for activation functions).

[0188] Unlike traditional “Turing”-based processor architectures (discussed above), neural network processing emulates a network of connected nodes (also called “neurons”) that loosely models the neurobiological functionality found in the human brain. Although neural network computations are still in their infancy, such technologies have great promise in, for example, computationally intensive, low-power, and / or continuous processing applications.

[0189] Each processor node of the neural network is a computational unit that can have any number of weighted input connections and any number of weighted output connections. The inputs are combined according to a transfer function to produce an output. In one particular embodiment, each processor node of the neural network combines its inputs with a set of coefficients (weights) that amplify or attenuate the constituent components of its input data. The input-weight products are summed and then the sum is passed through the activation function of the node to determine the magnitude and value of the output data. "Activating" the neuron (processor node) produces the output data. The output data can be fed to another neuron (processor node) or result in an action on the environment. The coefficients can be updated iteratively through feedback to amplify beneficial inputs while attenuating unbeneficial inputs.

[0190] Many neural network processors model individual neural network nodes as software threads and a large number of vector-matrix multiply accumulations. A "thread" is the smallest discrete unit of processor utilization that can be scheduled for execution on a core. A thread is characterized by: (i) a set of instructions to be executed by the processor, (ii) a program counter that identifies the current execution point of the thread, (iii) a stack data structure for temporarily storing thread data, and (iv) registers for storing the arguments of opcode executions. Other embodiments may use hardware or dedicated logic to implement the processor node logic.

[0191] As used herein, the term "emulate" and its linguistic derivatives refer to a software process that reproduces the functionality of an entity based on a processing description. For example, a processor node of a machine learning algorithm can be emulated with a "state input" and a "transfer function" that produces an "action".

[0192] Unlike Turing-based processor architectures, machine learning algorithms learn tasks that are not explicitly described by instructions. In other words, machine learning algorithms attempt to create inferences from patterns in data using, for example, statistical models and / or analysis. The inferences can then be used to formulate a predicted output that can be compared to the actual output to produce feedback. Each iteration of inference and feedback is used to improve the underlying statistical model. Since the task is accomplished through dynamic coefficient weighting rather than explicit instructions, machine learning algorithms can change their behavior over time, such as improving performance, changing tasks, etc.

[0193] Typically, a machine learning algorithm is "trained" until its predicted output matches the desired output (within a threshold similarity). Training can be done "offline" using batch-prepared data preprocessed by the system or can be done "online" using real-time data. Many embodiments combine offline and online training to, for example, provide accurate initial performance that is adjusted over time based on system-specific considerations. Once the NPU has "learned" the appropriate behavior, the NPU can be used in real-world scenarios. NPU-based solutions are generally more resilient to environmental changes and can behave reasonably (e.g., similar to humans) even in unexpected situations.

[0194] Processor-based proxy data

[0195] Exemplary embodiments of the present disclosure use metadata from a previous processing stage to create processor-based proxy data in real time (or near real time) for control and data subsystems. For example, an encoder may use information from a previous frame to predict encoding complexity. However, more generally, an encoder is just one processing element in a pipeline of processing elements. Each pipeline stage processes data in sequence, and many issues that may affect encoding can also affect upstream processing. Thus, the concept can be broadly extended to any information from a previous processing stage that can be used as a proxy for encoding complexity.

[0196] Conceptually, encoding quality is affected by various factors. Examples can include (but are not limited to) data volume, bit width, granularity, temporal variance, ambiguity / redundancy, noise, resource constraints, compression, etc. Many of these factors also affect and / or can be introduced by previous pipeline stages. For example, some pipeline stages may reduce the amount of data being processed due to, for example, bandwidth or resource limitations. In some such variants, this information can be used to predict the resulting encoding bit rate; for example, a bandwidth limitation can also directly result in a reduced encoding bit rate downstream, etc. Similarly, some stages may increase the data volume, for example, to improve the robustness and resilience of downstream encoding. Common examples can include, for example, line coding, forward error correction, etc. In such variants, this information can be roughly analogous to the image content itself; for example, image data with relatively minimal error correction can also be of low complexity, and vice versa. More generally, any processing metadata that can be used to predict the resulting encoding quality can be used as processor-based proxy data.

[0197] While the above discussion is presented in the context of an image processing pipeline within a device having a particular set of stages (such as an image signal processing stage, a stabilization and denoising stage, and an encoding stage), the concept can be widely applied to various other processing pipelines. For example, some devices may incorporate communication stages (such as transmit / receive, encryption / decryption, etc.) for transmission over a signaling medium. Other devices may incorporate machine learning, graphics processing, and / or other media operation stages. However, other devices may include data analysis, filtering, and / or other monitoring stages.

[0198] It is worth noting that processing pipelines are typically used in a variety of server-side and client-side applications. For example, a processing pipeline can enable traffic conditioning on the server side; other examples can allow for error recovery on the client side. Various user applications can also use pipeline processing to schedule processing loads for the resource limitations of the platform.

[0199] As a related matter, some embodiments of the present disclosure use metadata from concurrent processing stages to create processor-based proxy data for control and data subsystems in real time (or near real time). For example, an encoder may use information from corresponding frames of different encoders (e.g., LRV) to predict the encoding complexity of the current frame (e.g., MRV). In other words, the encoder may also be one of several concurrent tasks. In some cases, the concurrent tasks may affect and / or otherwise predict the current encoding quality. Thus, the concept can be broadly extended to any information from concurrent tasks that can be used as a proxy for encoding complexity.

[0200] In fact, the above discussion is in the context of a real-time operating system (RTOS) that schedules task completion within well-defined constraints. However, more generally, the concept can be broadly extended to high-level operating systems (OSs) that are typically best-effort. Simply put, many high-level OSs must dynamically manage different task loads; in some cases, this can affect encoding quality. For example, a laptop running a low-priority variable rate encoder may also be running other applications in the background. Under some conditions, the laptop may consider background memory usage as a proxy for the quality of the encoded image, since encoding quality can be strongly affected by available memory resources. Other examples of processor-based proxy data can include, for example, task load, memory load, memory availability, network bandwidth, power consumption, interrupt frequency, etc. More generally, processor-based proxy data can be broadly extended to any processing resource that can affect encoding quality.

[0201] General Dynamic Encoder Configuration Based on Proxy Data

[0202] Although the above discussion is presented in the context of an image processing pipeline within a device having a set of specific stages (e.g., an image signal processing stage, a stabilization and denoising stage, and an encoding stage), the concept can be broadly applied to a variety of other processing pipelines, and the techniques can be broadly extended to any media processing pipeline. As used herein, the term "pipeline" refers to a set of processing elements that process data in sequence such that each processing element may also operate in parallel with other processing elements. For example, a 3-stage pipeline may have first, second, and third processing elements that operate in parallel. During operation, the input to the second processing element includes at least the output of the first processing element, and the output of the second processing unit is at least one input to the third processing element. Although the above discussion is presented in the context of a pipeline having physical processing elements, one of ordinary skill in the relevant art will readily appreciate that virtualized and / or software-based pipelines may be equally successfully substituted.

[0203] In one embodiment, a non-transitory computer-readable medium includes routines that implement dynamic encoder configuration. When executed by a control and data subsystem, the routines cause an encoding device to: obtain a model that identifies an optimization target from proxy data; obtain proxy data; and adjust encoding based on the proxy data. Under certain conditions, the routines may further switch between modes of the dynamic codec configuration.

[0204] Instruction 1142 causes the encoding device to obtain a model that identifies an optimization target from proxy data. In one exemplary embodiment, the model includes a machine learning model that has been trained to classify images based on their complexity using proxy data. In a particular variant, the proxy data includes sensor-based proxy data (e.g., pixel features) and processor-based proxy data (e.g., encoding features). Here, the classification is based on the peak signal-to-noise ratio (PSNR) of YUV image data for a set of frames (e.g., a GOP of length 1 second). The complexity classification is used to identify a bitrate corresponding to the optimization target; for example, the pixel features and encoding features correspond to a predicted PSNR - the predicted PSNR is then used to select the corresponding bitrate. Here, the predicted PSNR is "learned" from a training video library. For example, machine learning logic may have been previously trained on a set of training video segments, where the actual PSNR of the encoded video is compared to a reference video.

[0205] In one embodiment, the machine learning model can be based on "random forest", gradient boosting trees, and / or any other machine learning logic configured to classify input data based on labels. Examples of such machine logic include logistic regression, decision trees, random forests, support vector machines (SVMs), k-nearest neighbors (KNN), and neural networks. The classification can be binary (complex, not complex) or multi-class (high motion high texture, high motion low texture, low motion low texture, low motion high texture, etc.). There are other classification techniques that can be labeled based on image PSNR (e.g., 5dB, 10dB, 15dB, etc.) or other optimization targets.

[0206] Although the above discussion is presented with reference to a machine learning model, other forms of logic can be equally successfully substituted. For example, a look-up table (LUT) can be used for very simple relationships between proxy data and optimization targets. More complex relationships can be described using univariate or multivariate equations. There is other logic that can incorporate conditional rules, triggers, and / or other heuristic-based logic.

[0207] The above examples are described in the context of a machine learning model that is trained offline and then stored or otherwise programmed into a device for online operation. Almost any other technique for obtaining or providing the model can be equally successfully substituted. In some cases, the model can be downloaded in an untrained format and then trained on real-time data during use. For example, multiple encoders operating in parallel can be used to provide both the encoded image and the reference image. Other embodiments can be downloaded via software updates, firmware updates, or similar programming.

[0208] Instruction 1144 causes the encoding device to obtain proxy data. In one embodiment, the proxy data is generated according to the real-time (or near real-time) constraints of the encoding device; for example, an embedded device may have a fixed buffer size that limits the amount of data that can be captured (e.g., a camera may only have a 1-second buffer for storing image data). In other cases, the encoding device may have a real-time operating system that imposes scheduling constraints according to its tasks.

[0209] In one embodiment, the proxy data may include real-time (or near real-time) information generated by a sensor. Examples of such information may include optical information, acoustic information, and / or inertial measurement data. In other embodiments, the real-time (or near real-time) proxy data may be determined from on-board processing. For example, an image stabilization algorithm may generate motion vectors based on sensed inertial measurements. In other embodiments, automatic exposure, white balance, and color correction algorithms may be based on the captured image data.

[0210] Although the above discussion is presented in the context of the "previous stage" of the pipeline, those of ordinary skill in the relevant art will readily understand that some embodiments may obtain proxy data information from subsequent stages of the pipeline. For example, a real-time streaming embodiment may encode video for transmission over a network; in some cases, a modem may provide network capacity information that identifies bottlenecks in the data transfer capacity (and by extension, the encoding complexity). As another example, a computer vision application (such as an autonomous vehicle) may adjust encoding according to application requirements; for example, a neural network processor may provide proxy data based on object recognition from image data, etc. As yet another example, the CPU may provide information from the OS on behalf of user input received from a user interface.

[0211] More generally, the proxy data may include any real-time (or near real-time) information captured or generated by any subsystem of the encoding device. Although the above discussion is presented in the context of ISP image correction data and codec encoding data, those of ordinary skill in the relevant art will readily understand that other proxy data may come from the CPU, modem, neural network processor, and / or any other entity of the device.

[0212] In some embodiments, the proxy data may be included as part of the metadata accompanying the image data. In other embodiments, the proxy data may be provided out-of-band via a dedicated storage buffer and / or signaling. For example, the image processing pipeline (discussed above in Figure 1 contains two DDR storage buffers, which may store the image data and any corresponding proxy data. Although the above discussion is presented in the context of a FIFO (first-in, first-out) circular buffer, various other memory organizations may be equally successfully substituted. Examples may include, for example, last-in, first-out (LIFO), ping-pong buffers, stacks (thread-specific), heaps (thread-independent), and / or other memory organizations commonly used in the computing field. However, more generally, any scheme for obtaining, providing, or otherwise transmitting data between stages of the pipeline may be equally successfully substituted. Examples may include shared mailboxes, packet-based delivery, bus signaling, interrupt-based signaling, and / or any other communication mode.

[0213] As previously mentioned, real-time (and near real-time) processing is typically subject to time-related constraints. In some embodiments, the proxy data may include an explicit timestamp or other message, which directly associates it with the corresponding image and / or video frame. This may be particularly useful for proxy data of arbitrary or unknown timing (e.g., user input provided via a stack or heap type data structure, neural network classification, etc.).

[0214] Instruction 1146 causes the encoding device to adjust control parameters based on the proxy data. In one exemplary embodiment, the encoding device uses a trained machine learning model that classifies image complexity based on image-based pixel features and encoding features; the marked image complexity is used to select a bitrate corresponding to the expected image quality. In some embodiments, the resulting encoded image may be compared with a reference image to calculate the actual image quality to improve the machine learning model; in other embodiments, the resulting encoded image may not have a reference image for comparison. Here, the term "expected" refers to the result based on model prediction, while "actual" refers to the result verified by a process external to the model. More generally, the various principles described herein utilize the representative nature of the proxy data to achieve the expected image quality to inform dynamic codec configuration without subsequent verification of the actual image quality.

[0215] Although the disclosed examples use peak signal-to-noise ratio (PSNR) of pixel values to measure image quality, other metrics may be substituted for image quality. Examples may include subjective metrics, an agreed-upon image library, user-specified models, and / or other metrics. Similarly, although the above examples use bitrate selection as a configurable control parameter, other control parameters may be equally successfully substituted. Examples may include minimum quantization parameter (QP), resolution, frame rate, bit depth, and / or other codec parameters may be equally successfully substituted.

[0216] Exemplary embodiments are optimized for image quality, and other optimization goals may include, for example, local storage space, power consumption, network storage space, transmission size, and / or other resources. Different optimization goals may use different control parameters; generally, control parameters may affect, for example, complexity, latency, throughput, bit rate, media quality, data format, resolution, size, and / or any number of other media characteristics. More generally, any parameter that modifies the way the encoding is performed and / or the output of the encoding process may be equally successfully substituted.

[0217] In one embodiment, the control parameters may be pre-generated and retrieved from a lookup table or similar reference data structure. In other embodiments, the control parameters may be calculated according to heuristics or algorithms. In other embodiments, the control parameters may be selected from a history of acceptable parameters under similar conditions. Other embodiments may use, for example, machine learning algorithms or artificial intelligence logic to select a suitable configuration. In some embodiments, an external entity (e.g., a network or a decoding device) may provide additional guidance, a selection of acceptable parameters from which the encoding device may choose, or even the control parameters themselves. However, more generally, any scheme for adjusting control parameters based on surrogate data from previous stages of the pipeline and / or other concurrent processing may be equally successfully substituted.

[0218] Instruction 1148 causes the encoding device to switch between modes of a dynamic codec configuration. In one exemplary embodiment, the model may have a first operating range and one or more boundary ranges. In a particular implementation, a machine learning model may have an equal-quality zone, a lower-bound zone, and an upper-bound zone. Here, the "lower-bound" and "upper-bound" zones correspond to zones where the control parameter (bit rate) is outside the adjustment range of the optimization goal (image quality). More generally, the various techniques described herein may be widely extended to any dynamic coding system having multiple modes, where each mode encapsulates a specific set of conditions, behaviors, or characteristics of the encoding process.

[0219] In one embodiment, different modes are characterized by different behaviors of the control parameters and / or optimization goals. For example, the encoding device may adjust its image quality goal based on the bit rate in the lower-bound or upper-bound zone. As another example, the encoding device may adjust its encoding configuration (e.g., resolution, frame rate, etc.) to accommodate a bit rate below the lower-bound or above the upper-bound zone.

[0220] In other embodiments, different modes may be characterized by adjusting the behavior of previous or subsequent stages. For example, the encoding device may adjust its image signal processing front end based on the bit rate in the lower-bound or upper-bound zone. As another example, the encoding device may adjust its network configuration (e.g., resolution, frame rate, etc.) and / or dynamic memory allocation to accommodate a bit rate below the lower-bound or above the upper-bound zone.

[0221] In some embodiments, shifting between modalities requires changing the processing pipeline, which invalidates previously processed data. As previously mentioned, the pipeline divides data processing into several stages; data propagates synchronously through the pipeline. If an error or an exceptional condition occurs, then the pipeline needs to be flushed to maintain consistent data stream handling. Pipeline flushing significantly affects the latency and throughput of the pipeline and should be used sparingly. Thus, some implementations may additionally incorporate hysteresis and / or differential information before switching between modalities.

[0222] In some variants, the encoding device may consider the time-averaged behavior to avoid resistance to sudden changes (avoid "churning"). In other variants, the encoding device may prepare preemptively for upcoming changes by, for example, adjusting the encoder configuration in advance. In some cases, the preemptive adjustment may additionally propagate to downstream processing. For example, a change made to the encoder modality to accommodate an expected increase / decrease in bitrate may propagate to downstream memory allocation and / or processing, etc.

[0223] Additional configuration considerations

[0224] In this specification, some embodiments have used the expressions "comprising", "including", "having", or any other variant thereof, all of which are intended to cover non-exclusive inclusion. For example, a process, method, article, or device that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to this process, method, article, or device.

[0225] Additionally, the article "a" is used to describe elements and components of the embodiments herein. This is merely for convenience and to give a general sense of the invention. This description should be construed to include one or at least one and the singular also includes the plural unless it is obvious that it has a different meaning.

[0226] As used herein, any reference to "an embodiment" or "embodiments", "a variant" or "variants", and "an implementation" or "implementations" means that a particular element, feature, structure, or characteristic described in connection with the embodiment, variant, or implementation is included in at least one embodiment, variant, or implementation. Such phrases appearing in various places in the specification do not necessarily refer to the same embodiment, variant, or implementation.

[0227] As used herein, the term "computer program" or "software" means any sequence of human- or machine-perceivable steps that performs a function. This program may be presented in almost any programming language or environment, including, for example, Python, JavaScript, Java, C# / C++, C, Go / Golang, R, Swift, PHP, Dart, Kotlin, MATLAB, Perl, Ruby, Rust, Scala, etc.

[0228] As used herein, the term "integrated circuit" means an electronic circuit fabricated by patterning and diffusing trace elements onto the surface of a thin substrate of semiconductor material. By way of non-limiting example, an integrated circuit may include a field programmable gate array (e.g., FPGA), a programmable logic device (PLD), a reconfigurable computing fabric (RCF), a system on a chip (SoC), an application specific integrated circuit (ASIC), and / or other types of integrated circuits.

[0229] As used herein, the term "memory" includes any type of integrated circuit or other storage device suitable for storing digital data, including (but not limited to) ROM, PROM, EEPROM, DRAM, mobile DRAM, SDRAM, DDR / 2 SDRAM, EDO / FPMS, RLDRAM, SRAM, "flash" memory (e.g., NAND / NOR), memristor memory, and PSRAM.

[0230] As used herein, the term "processing unit" generally means a digital processing device. By way of non-limiting example, a digital processing device may include one or more of a digital signal processor (DSP), a reduced instruction set computer (RISC), a general purpose (CISC) processor, a microprocessor, a gate array (e.g., a field programmable gate array (FPGA)), a PLD, a reconfigurable computing fabric (RCF), an array processor, a secure microprocessor, an application specific integrated circuit (ASIC), and / or other digital processing devices. Such digital processors may be contained on a single monolithic IC die or distributed across multiple components.

[0231] As used herein, the term "camera" or "image capture device" may be used to refer to (but not limited to) any imaging device or sensor configured to capture, record, and / or transmit still and / or video images, which may be sensitive to the visible portion of the electromagnetic spectrum and / or the invisible portion of the electromagnetic spectrum (e.g., infrared, ultraviolet) and / or other energy (e.g., pressure waves).

[0232] After reading this disclosure, those skilled in the art will appreciate additional alternative structural and functional designs for the principles disclosed herein. Accordingly, while specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes, and variations apparent to those skilled in the art may be made to the arrangement, operation, and details of the methods and apparatuses disclosed herein without departing from the spirit and scope defined in the appended claims.

[0233] It should be recognized that while certain aspects of the present technology are described in terms of a particular sequence of method steps, these descriptions merely illustrate the broader method of the disclosure and may be modified according to the needs of a particular application. In some cases, certain steps may become unnecessary or optional. Additionally, certain steps or functionality may be added to the disclosed embodiments, or the order of execution of two or more steps may be arranged. All such variations are considered to be covered by the disclosure as disclosed and claimed herein.

[0234] While the foregoing detailed description has shown, described, and pointed out the novel features of the disclosure as applied to various embodiments, it should be understood that various omissions, substitutions, and changes in the form and details of the devices or processes illustrated may be made by those skilled in the art without departing from the disclosure. The foregoing description is the best mode presently contemplated for carrying out the principles of the disclosure. This description is in no way limiting, but rather should be regarded as illustrative of the general principles of the technology. The scope of the disclosure should be determined with reference to the claims.

[0235] It should be understood that each of the foregoing aspects of the disclosure, or any part or function thereof, may be implemented using hardware, software, firmware, tangible and non-transitory computer-readable or computer-usable storage media having instructions stored thereon, or a combination thereof, and may be implemented in one or more computer systems.

[0236] Those skilled in the art will appreciate that various modifications and variations can be made to the disclosed embodiments of the disclosed devices and associated methods without departing from the spirit or scope of the disclosure. Accordingly, the disclosure is intended to cover modifications and variations of the embodiments disclosed above, provided they are within the scope of any claim and its equivalents.

Claims

1. A method for dynamically configuring an encoder in a pipeline, comprising: Obtaining a model that correlates proxy data with image complexity; Obtaining first proxy data for a first set of images; Determining first encoding parameters for a consistent optimization objective based on the first proxy data and the model; Configuring the encoder to encode a first video segment based on the first encoding parameters; Obtaining second proxy data for a second set of images; Determining second encoding parameters for the consistent optimization objective based on the second proxy data and the model; Configuring the encoder to encode a second video segment based on the second encoding parameters; And Wherein the first video segment and the second video segment are within a threshold tolerance of the consistent optimization objective.

2. The method according to claim 1, wherein the proxy data includes pixel features from a previous stage of the pipeline, and wherein configuring the encoder occurs in the current stage of the pipeline.

3. The method according to claim 1, wherein the proxy data includes encoded features from a previous video segment.

4. The method according to claim 1, wherein the model includes a machine learning model configured to predict the image quality of an encoded video segment of a reference video segment based on pixel features of the reference video segment.

5. The method according to claim 1, wherein the model includes a machine learning model configured to predict the image quality of an encoded video segment of a reference video segment based on encoded features of a previous reference video segment.

6. The method according to claim 1, wherein the model includes a machine learning model configured to predict the image quality of an encoded video segment of a reference video segment based on encoded features of a lower resolution reference video segment.

7. The method according to claim 1, wherein the first encoding parameter is a first bitrate, the second encoding parameter is a second bitrate, and the consistent optimization objective is peak signal-to-noise ratio.

8. An apparatus, comprising: A camera configured to capture at least one first image; An image processing pipeline including an encoding element; Machine learning logic trained to select a bitrate based on proxy data; A processor; And A non-transitory computer-readable medium including a set of instructions that, when executed by the processor, cause the processor to: Provide first proxy data associated with at least the first image to the machine learning logic; Obtain a first bitrate from the machine learning logic based on the first proxy data; And Configure the encoding element to encode at least the first image based on the first bitrate.

9. The apparatus according to claim 8, wherein the image processing pipeline further includes an image signal processor, and the first proxy data includes pixel features calculated by the image signal processor from at least the first image.

10. The apparatus according to claim 8, wherein the proxy data includes encoded features corresponding to at least one previous image calculated by the encoding element.

11. The apparatus according to claim 8, wherein the proxy data includes encoding features corresponding to at least the low-resolution encoding of the first image calculated by other encoding elements.

12. The apparatus according to claim 8, wherein the set of instructions, when executed by the processor, further causes the processor to: Cause the camera to capture at least a second image and at least the first image according to a real-time frame rate; Provide second proxy data associated with the second image to the machine learning logic; Obtain a second bit rate from the machine learning logic based on the second proxy data; And Configure the encoding element to encode at least the second image based on the second bit rate according to the real-time frame rate.

13. The apparatus according to claim 12, wherein the first bit rate is equal to the second bit rate.

14. The apparatus according to claim 12, wherein the first bit rate is different from the second bit rate.

15. An encoding apparatus, comprising: An encoding element configured to encode according to a first mode and a second mode; Machine learning logic trained to select a bit rate according to the first mode and the second mode; A processor; And A non-transitory computer-readable medium including a set of instructions that, when executed by the processor, cause the processor to: Provide first proxy data associated with a first image to the machine learning logic according to the first mode; Obtain a first bit rate from the machine learning logic based on the first proxy data; And Switch to the second mode based on the first bit rate.

16. The encoding apparatus according to claim 15, wherein the encoding element encodes the first image according to the first mode based on the first bit rate.

17. The encoding apparatus according to claim 15, wherein the first mode is based on a first image quality and the second mode is based on a second image quality.

18. The encoding apparatus according to claim 15, wherein the first mode is based on a first image resolution and the second mode is based on a second image resolution.

19. The encoding apparatus according to claim 15, wherein the first mode is based on a first video frame rate and the second mode is based on a second video frame rate.

20. The encoding apparatus according to claim 15, wherein the first mode is based on a first video encoding standard and the second mode is based on a second video encoding standard.

Images