Training and using machine learning models using a hierarchical data format

Abstract

There is described a method of providing data to a machine learning model using a hierarchical data format, the method comprising: identifying a data source with a hierarchical format; determining first data from the data source, the first data having a first level of quality; determining second data from the data source, the second data having a second level of quality; and providing the first data and the second data to the machine learning model.

Description

[0001] Training and using machine learning models using a hierarchical data format

[0002] Field of the Disclosure

[0003] The present disclosure relates to methods, systems, and apparatuses for training and / or using a machine learning model. In particular, the present disclosure relates to methods, systems, and apparatuses that use a hierarchical data format to train and / or use a machine learning model, such as a large multimodal model (LMM).

[0004] Background to the Disclosure

[0005] Large multimodal models are advanced artificial intelligence systems designed to process and interpret multiple forms of data, such as text, images, audio, and video. These models leverage deep learning techniques to understand and generate content across these various modalities, enabling them to perform tasks like image captioning, video summarization, or cross-modal translations. By integrating information from diverse sources, multimodal models offer a more holistic understanding of complex datasets compared to unimodal systems (such as large language models, LLM), which are limited to a single data type.

[0006] LMMs have mixed modal prompts and mixed output, e.g. Speech to text prompt, the prompt may further include video. LMMs may extract meaning from multiple inputs (e.g. sensors).

[0007] LMMs are known and information may be found at multiple sources, for example:

[0008] "Multimodal Machine Learning: A Survey and Taxonomy" by Baltrusaitis et aL, This comprehensive survey provides a taxonomy of multimodal machine learning, detailing different approaches for fusion and representation learning. [1705.09406] Multimodal Machine Learning: A Survey and Taxonomy (arxiv.org).

[0009] "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Chapter on Representation Learning covers foundational techniques for multimodal integration., "Multimodal Deep Learning" by Jiajun Chen and Aniket Anand Deshmukh.

[0010] Problematically, training and using existing large multimodal models can be a difficult, inefficient, and timeconsuming process. Therefore, improvements in this area are desired.

[0011] Summary of the Disclosure

[0012] According to various aspects of the present disclosure there are disclosed: methods of (and apparatuses and systems for) providing data to a machine learning model using a hierarchical data format; methods of (and apparatuses and systems for) generating and / or transmitting data in a hierarchical data format (for use with a machine learning mode); and methods of (and apparatuses and systems for) training and / or performing inference using a machine learning model, preferably using data obtained from a file with a hierarchical data format.

[0013] According to an aspect of the present disclosure, there is described a method of providing data to a machine learning model using a hierarchical data format, the method comprising: identifying a data source with a hierarchical format; determining first data from the data source, the first data having a first level of quality (e.g. a first resolution); determining second data from the data source, the second data having a second level of quality (e.g. a second resolution); and providing the first data and the second data to the machine learning model.

[0014] Preferably, the method comprises: training the machine learning model using the first data; and re-training and / or fine tuning the machine learning model using the second data.

[0015] Preferably, the second level of quality is higher than the first level of quality (e.g. the second data has a higher resolution than the first data). Preferably, the method comprises iteratively training the machine learning model using data of increasing data with levels of quality.

[0016] Preferably, the method comprises: providing the first data to the machine learning model; receiving an output from the machine learning model; comparing the output to the second data; and training the machine learning model based on the comparison of the output and the data.

[0017] Preferably, the output comprises generated data, the generated data being generated by the machine learning model based on the first data and the generated data having the second level of quality.

[0018] Preferably, the method comprises: determining a further output based on the second data, the output and the further output being the same type of output (e.g. the output and the further output relating, respectively, to a characteristic of the first data and the second data); comparing the output and the further output; and training the machine learning model based on the comparison of the output and the further output.

[0019] Preferably, determining the second data comprises combining the first data with one or more enhancement layers. Preferably, the method comprises extracting the one or more enhancement layers from the data source.

[0020] Preferably, the first data comprises a first version of an image and / or video and the second data comprises a second version of the same image and / or video.

[0021] Preferably, the first data is associated with a first element of the data source and the second data is associated with a second element of the data source.

[0022] Preferably, the first element and the second element are different elements of the data source, e.g. different portions or sections of an image and / or a video.

[0023] Preferably, determining the first data comprises determining the first data at a first time, and determining the second data comprises determining the second data at a second time.

[0024] Preferably, the method is performed by a first computer device, wherein the method comprises receiving the first data from a second computer device at a first time.

[0025] Preferably, the method comprises receiving the second data and / or one or more enhancement layers from a second computer at a second time. Preferably, the second time is substantially after the first time (e.g. at least 1 second, at least 10 seconds, and / or at least one minute after the first time).

[0026] Preferably, the method comprises: receiving the first data at the first time; transmitting a request for the second data and / or the enhancement layers; and receiving the second data at the second time.

[0027] Preferably, the method comprises transmitting the request for the second data in dependence on a feature of the first data and / or in dependence on an output from the machine learning model.

[0028] Preferably, the method comprises: providing the first data to the machine learning model; receiving an output from the machine learning model; and transmitting the request in dependence on the output. Preferably, the transmitting is based on a confidence level associated with the output being below a threshold value.

[0029] Preferably, the method comprises determining, at the second computer device, the second level of quality. Preferably, the method comprises determining the second level of quality in dependence on an input quality required by the machine learning model.

[0030] Preferably, the method comprises determining the second data in dependence on an output of the machine learning model. Preferably, the method comprises: providing the first data to the machine learning model; receiving an output from the machine learning model; and determining the second data in dependence on the output, preferably based on a confidence level associated with the output being below a threshold value. Preferably, the method comprises determining the second level of quality in dependence on a feature of the data source.

[0031] Preferably, the method comprises determining the second level of quality in dependence on an input quality required by the machine learning model.

[0032] Preferably, the method comprises determining the second level of quality as the lowest possible level of quality that meets a required input quality threshold.

[0033] According to another aspect of the present disclosure, there is described a method of providing data to a machine learning model using a hierarchical data format, the method comprising: identifying a data source with a hierarchical format, the data source encoding first data having a first level of quality (e.g. a first resolution) and second data having a second level of quality (e.g. a second resolution); and providing the first data and the second data to the machine learning model by transmitting the data source to a further computer device; the method comprising determining the second level of quality as a lowest possible level of quality that meets a required input threshold.

[0034] Preferably, the method comprises selecting the second level of quality from a plurality of possible levels of quality.

[0035] Preferably, the method is performed by a sending computer device and wherein providing the first data and the second data to the machine learning model comprises transmitting the first data and the second data to a receiving computer device on which the machine learning model is implemented.

[0036] Preferably, the method comprises encoding the first data and the second data prior to transmitting the data to the receiving computer device. Preferably, the method comprises encoding the first data and the second data in a hierarchical coding scheme. Preferably, the method comprises encoding the first data and the second data using a hierarchical coding scheme that allows dynamic resolution adjustments during model training.

[0037] Preferably, the method comprises selectively transfer only necessary LOQs of the encoded data between the sending computer device and the receiving computing device.

[0038] Preferably, the method comprises progressively transmitting data starting with a lower resolution and increasing resolution based on subsequent requirements, using a hierarchical encoding scheme.

[0039] Preferably, the method comprises initially sending low-resolution data to decrease response times, followed by higher resolution data as needed.

[0040] Preferably, the method comprises transmitting the first data at a first time and transmitting the second data at a second time. Preferably, the method comprises transmitting the second data and / or an enhancement layer relating to the second data in response to a request from the receiving device.

[0041] Preferably, determining the first data and the second data comprises selectively decoding only required Levels of Quality, LOQs, from a hierarchically compressed data file. Preferably, the method comprises comprising determining the required LOQs based on the machine learning model, preferably based on the training needs of the machine learning model.

[0042] Preferably, the method comprises a method for optimizing processor (e.g. GPU, Tensor Processing unit, and / or CPU) resource utilization during the training of large multimodal models.

[0043] Preferably, the method comprises processing only selectively decoded data segments of the data source.

[0044] Preferably, determining the first data and the second data comprises dynamically decoding only the necessary segments of the data source to match a resolution required by the machine learning model. Preferably, the method comprises dynamically decoding the segments to match a resolution required by one or more specific training tasks of the machine learning model. Preferably, the method comprises the data source comprises a first set of one or more elements that can be determined at the first level of quality and a second set one or more elements that can be determined at the second level of quality. Preferably, the second set is a subset of the first set.

[0045] Preferably, the method comprises: determining the first data using a first device, preferably a first GPU; and determining the second data using a second device, preferably a second GPU.

[0046] Preferably, the method comprises training a plurality of different aspects and / or functionalities of the machine learning model using data at different levels of quality.

[0047] Preferably, the method comprises: training a first aspect (or module or functionality) of the machine learning model using the first data; and training a second aspect (or module or functionality) of the machine learning model using the second data.

[0048] Preferably, the aspects of the machine learning model comprise one or more of: a movement analysis aspect, a semantic analysis aspect, and an object classification aspect.

[0049] Preferably, the method comprises determining the second level of quality based on an aspect of the machine learning model being used to process the second data and / or based on an aspect (or a module or a functionality) of the machine learning model being trained using the second data.

[0050] Preferably, the method comprises training a plurality of, e.g. various, aspects of the model (e.g. semantic analysis, movement analysis, and object classification) at different levels of quality optimized for each aspect using a single hierarchical-encoded data file.

[0051] Preferably, the method comprises adjusting the quality of data used for training specific aspects based on performance feedback and resource availability.

[0052] Preferably, the data source is encoded using a hierarchical codec that allows selective decoding of data at varying qualities necessary for specific training aspects.

[0053] Preferably, the method comprises receiving an output from the machine learning model. Preferably, the method comprises a method of inference using the machine learning model.

[0054] Preferably, the method comprises using a unified software layer on general-purpose GPUs (and or CPUs and or Tensor Processing units, TPUs) to decode multiple types of data modalities without the need for specialized hardware decoders.

[0055] Preferably, the method comprises implementing a multimodal decoding software that dynamically adapts to various types of data, including text, speech, and video.

[0056] Preferably, the machine learning model comprises a large multimodal model, LMM.

[0057] Preferably, the data source comprises an image and / or a video.

[0058] Preferably, the first data and / or the second data comprises an image and / or a video.

[0059] Preferably, the data source is encoded using a hierarchical codec.

[0060] Preferably, the data source is encoded using a SMPTE VC 6 or an LCEVC codec.

[0061] Preferably, the first level of quality and the second level of quality are associated with different resolutions.

[0062] Preferably, the method comprises determining the first data and the second data using a graphical processing unit, GPU, and / or a tensor processing unit, TPU.

[0063] Preferably, the method is (arranged to be) performed by a graphical processing unit, GPU, and / or a tensor processing unit, TPU. According to another aspect of the present disclosure, there is described a computer programme product comprising instructions that, when executed by a processor, cause the processor to perform the aforesaid method.

[0064] According to another aspect of the present disclosure, there is described a non-transitory computer- readable storage medium, the computer-readable storage medium including instructions that when executed by a processor cause the processor to perform the aforesaid method.

[0065] According to another aspect of the present disclosure, there is described an apparatus for providing data to a machine learning model using a hierarchical data format, the apparatus comprising: means for (e.g. a processor for) for identifying a data source with a hierarchical format; means for (e.g. a processor for) determining first data from the data source, the first data having a first level of quality; means for (e.g. a processor for) determining second data from the data source, the second data having a second level of quality; and means for (e.g. a processor for) providing the first data and the second data to the machine learning model.

[0066] Preferably, the apparatus comprises a memory for storing (e.g. parameters of) the machine learning model.

[0067] According to another aspect of the present disclosure, there is described an apparatus for providing data to a machine learning model using a hierarchical data format, the apparatus comprising: means for (e.g. a processor for) identifying a data source with a hierarchical format, the data source encoding first data having a first level of quality (e.g. a first resolution) and second data having a second level of quality (e.g. a second resolution); and means for (e.g. a processor for) providing the first data and the second data to the machine learning model by transmitting the data source to a further computer device; the apparatus comprising means for (e.g. a processor for) determining the second level of quality as a lowest possible level of quality that meets a required input threshold

[0068] According to another aspect of the present disclosure, there is described a system for providing data to a machine learning model using a hierarchical data format, the system comprising: means for (e.g. a processor) for identifying a data source with a hierarchical format; means for (e.g. a processor) determining first data from the data source, the first data having a first level of quality; means for (e.g. a processor) determining second data from the data source, the second data having a second level of quality; and means for (e.g. a processor) providing the first data and the second data to the machine learning model.

[0069] Preferably, the system comprises: a receiving device for transmitting one or more of: the data source, the first data, and the second data to a sending device; and the sending device, wherein the sending device hosts the machine learning model.

[0070] According to another aspect of the present disclosure, there is described a system for providing data to a machine learning model using a hierarchical data format, the system comprising: means for (e.g. a processor for) identifying a data source with a hierarchical format, the data source encoding first data having a first level of quality (e.g. a first resolution) and second data having a second level of quality (e.g. a second resolution); and means for (e.g. a processor for) providing the first data and the second data to the machine learning model by transmitting the data source to a further computer device; the apparatus comprising means for (e.g. a processor for) determining the second level of quality as a lowest possible level of quality that meets a required input threshold

[0071] Any feature in one aspect of the disclosure may be applied to other aspects of the invention, in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa.

[0072] Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly. Any apparatus feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0073] It should also be appreciated that particular combinations of the various features described and defined in any aspects of the disclosure can be implemented and / or supplied and / or used independently.

[0074] The disclosure also provides a computer program and a computer program product comprising software code adapted, when executed on a data processing apparatus, to perform any of the methods described herein, including any or all of their component steps.

[0075] The disclosure also provides a computer program and a computer program product comprising software code which, when executed on a data processing apparatus, comprises any of the apparatus features described herein.

[0076] The disclosure also provides a computer program and a computer program product having an operating system which supports a computer program for carrying out any of the methods described herein and / or for embodying any of the apparatus features described herein.

[0077] The disclosure also provides a computer readable medium having stored thereon the computer program as aforesaid.

[0078] The disclosure also provides a signal carrying the computer program as aforesaid, and a method of transmitting such a signal.

[0079] The disclosure extends to methods and / or apparatus substantially as herein described with reference to the accompanying drawings.

[0080] The disclosure will now be described, by way of example, with reference to the accompanying drawings.

[0081] Description of the Drawings

[0082] Figures 1 to 7 shows features of a hierarchical data format (and the formation of such a format).

[0083] Figure 8 shows a computer apparatus on which aspects of the present disclosure may be implemented.

[0084] Figure 9 shows an exemplary process for training a machine learning model.

[0085] Figure 10 shows a method of training a machine learning model, e.g. an LMM, using a plurality of levels of quality.

[0086] Figure 11 shows a method of providing elements of different LOQs to a machine learning model.

[0087] Figure 12 shows a method of transmitting a plurality of different LOQs from a first computer device to a further computer device.

[0088] Figure 13 shows a method of using a machine learning model to provide an output.

[0089] Detailed description

[0090] A general architecture of a Large Multimodal Model (LMM) is described below, which LMM is an artificial intelligence system designed to process and analyze data from multiple sources and modalities to perform various computational tasks. The LMM integrates inputs such as text, images, audio, and video to generate outputs based on the synthesis of this data.

[0091] General Architecture of LMM

[0092] The architecture of the LMM comprises typically (one or more of, e.g. all of) several key components designed to handle different data modalities and facilitate their integration: 1. Input Processing Units: These units are tailored to process specific types of input data. For instance, a Text Processing Unit processes textual data using natural language processing techniques, an Image Processing Unit utilizes convolutional neural networks to handle image data, an Audio Processing Unit employs signal processing methods for audio data, and a Video Processing Unit analyzes video streams using both convolutional neural networks and temporal processing techniques.

[0093] 2. Data Fusion Layer: This layer integrates processed data from the Input Processing Units. It aligns and combines features from different modalities, using techniques such as feature concatenation, attention mechanisms, or more complex fusion strategies like multimodal transformers, to create a comprehensive multimodal representation.

[0094] 3. Contextual Analysis Module: Utilizing the integrated data representations, this module performs deep semantic analysis to understand context and inter-modal relationships. It employs advanced neural networks, such as transformers or recurrent neural networks, optimized for multimodal data.

[0095] 4. Output Generation Unit: Based on the analysis conducted by the Contextual Analysis Module, this unit generates outputs relevant to the task at hand. The output could be textual (e.g., summaries), auditory (e.g., synthesized speech), visual (e.g., images, videos), or a combination of these.

[0096] 5. Learning Optimization System: This system continually improves model performance using feedback loops and reinforcement learning techniques to adjust weights and parameters in the model based on performance metrics.

[0097] The LMM described above may be combined with and / or form part of any other element, system and / or method describe in the rest of the document.

[0098] The Large Multimodal Model (LMM) represents a significant advancement in the field of artificial intelligence by efficiently integrating and processing multiple data types to enhance decision-making processes in various applications. The flexibility and scalability of the model's architecture allow it to be adapted for a wide range of industry-specific applications, demonstrating its utility and innovative capabilities.

[0099] While the present disclosure focuses on LMMs, it will be appreciated that the disclosures herein can be applied to other machine learning models, such as large language models (LLMs).

[0100] In brief, a Large Language Model (LLM) is a type of artificial intelligence that processes and generates human-like text based on patterns and information it has learned from a massive dataset of existing text. These models are typically trained using deep learning techniques and are capable of performing a variety of language-based tasks such as translating languages, answering questions, summarizing documents, and even generating creative writing. They operate primarily on textual data and are designed to understand and manipulate language at a sophisticated level.

[0101] This is in contrast to a Large Multimodal Model (LMM), which is an advanced type of Al that processes and understands multiple types of input data, such as text, images, audio, and video. These models integrate information from different sensory modalities to perform tasks that require a holistic understanding of various forms of data. For instance, an LMM might analyze a video by interpreting its visual content, spoken dialogue, and associated text to generate a comprehensive summary. The key capability of an LMM is its ability to bridge multiple data types, offering richer and more context-aware Al applications compared to models that handle only one type of data.

[0102] We also describe herein example hierarchical coding schemes that may be used to train and to apply to machine learning models (e.g. for use in training or inference processes).

[0103] A number of example hierarchical coding schemes can be found in the following patent applications, which are incorporated by reference in their entirety: US13 / 188226 (US8531321), PCT / IB2012 / 053725 (WO2013 / 011495), PCT / IB2012 / 053726

[0104] (W02013 / 011496), US17 / 122434 (US2021 / 0211752), US13 / 188237 (US2013 / 0297466),

[0105] PCT / GB2020 / 051366 (WO2020 / 245599) , PCT / GB2020 / 050312 (W02020 / 165575), PCT / GB2017 / 052142 (WO2018 / 015764), PCT / GB2017 / 050673 (WO2017 / 153784), PCT / GB2020 / 050695 (W02020 / 188273), PCT / GB2018 / 053552 (WQ2019 / 111010), PCT / GB2023 / 052755 (WO2024084248A1).

[0106] These documents often use video as the primary data to be encoded. However, it is described in the patents, and is to be appreciated here, that the coding schemes are also relevant for other data types (e.g. sensor data, point cloud data, machine data, and so forth)

[0107] Certain examples described herein relate to methods for encoding signals. Processing data may include, but is not limited to, obtaining, deriving, outputting, receiving and reconstructing data.

[0108] Certain examples described herein relate to distributed processing of signals. In particular, examples relate to encoding a signal using a multi-layer encoding (also referred to as a tier-based and / or hierarchical encoding), where different layers represent different levels of quality. These levels of quality may comprise different spatial resolutions, different bitrates for transmission, different levels of quantisation, different levels of sampling and / or different bit depths.

[0109] In tier-based coding formats such as ISO / IEC MPEG-5 Part 2 LCEVC (hereafter “LCEVC”), or SMPTE VC- 62117 (hereafter “VC-6”), a signal is decomposed in multiple “echelons” (also known as “hierarchical tiers”) of data, each corresponding to a “Level of Quality” (“LoQ” or “LOQ”) of the signal, from the highest echelon at the sampling rate of the original signal to a lowest echelon, which typically has a lower sampling rate than the original signal. In the non-limiting example when the signal is a frame of a video stream, the lowest echelon may be a thumbnail of the original frame, or even just a single picture element. Other echelons contain information on corrections to apply to a reconstructed rendition in order to produce the final output. Echelons may be based on residual information, e.g. a difference between a version of the original signal at a particular level of quality and a reconstructed version of the signal at the same level of quality. A lowest echelon may not comprise residual information but may comprise a lowest sampling of the original signal. The decoded signal at a given Level of Quality is reconstructed by first decoding the lowest echelon (thus reconstructing the signal at the first - lowest - Level of Quality), then predicting a rendition of the signal at the second - next higher - Level of Quality, then decoding the corresponding second echelon of reconstruction data (also known as “residual data” at the second Level of Quality), then combining the prediction with the reconstruction data so as to reconstruct the rendition of the signal at the second - higher - Level of Quality, and so on, up to reconstructing the given Level of Quality. Reconstructing the signal may comprise decoding residual data and using this to correct a version at a particular Level of Quality that is derived from a version of the signal from a lower Level of Quality. Different echelons of data may be coded using different coding formats, and different Levels of Quality may have different sampling rates (e.g., resolutions, for the case of image or video signals). Subsequent echelons may refer to a same signal resolution (i.e., sampling rate) of the signal, or to a progressively higher signal resolution.

[0110] In preferred examples, the multi-layer coding schemes includes the video coding standard VC-6: SMPTE VC-6 ST-2117, as described in PCT / GB2018 / 053552 (and the associated published standard document), all of which are incorporated by reference herein. Certain examples may also use other multi-layer coding schemes such as LCEVC: MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”) as described in PCT / GB2020 / 050695 (and the associated published standard document), these documents also being incorporated herein by reference. However, the concepts illustrated herein need not be limited to these specific multi-layer coding schemes. In other examples, the multi-layer coding scheme may comprise a (scalable) wavelet-based coding scheme.

[0111] As further background, examples described herein relate to signal processing. A signal may be considered as a sequence of samples (i.e., two-dimensional images, video frames, video fields, sound frames, etc.). In the description, the terms “image”, “picture” or “plane” (intended with the broadest meaning of “hyperplane”, i.e., array of elements with any number of dimensions and a given sampling grid) will be often used to identify the digital rendition of a sample of the signal along the sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y), and comprises a set of plane elements (or “element”, or “pel”, or display element for two-dimensional images often called “pixel”, for volumetric images often called “voxel”, etc.) characterized by one or more “values” or “settings” (e.g., by ways of non-limiting examples, colour settings in a suitable colour space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch, settings indicating amplitude, settings indicating depth, settings indicating alpha channel transparency level, etc.). Each plane element is identified by a suitable set of coordinates, indicating the integer positions of said element in the sampling grid of the image. Signal dimensions can include only spatial dimensions (e.g., in the case of an image) or also a time dimension (e.g., in the case of a signal evolving over time, such as a video signal).

[0112] As examples, a signal can be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a 3DoF / 6DoF video signal, a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even signals with more than four dimensions.

[0113] For simplicity, examples described herein often refer to signals that are displayed as 2D planes of settings (e.g., 2D images in a suitable colour space), such as for instance a video signal. The terms “frame” or “field” will be used interchangeably with the term “image”, so as to indicate a sample in time of the video signal: any concepts and methods illustrated for video signals made of frames (progressive video signals) can be easily applicable also to video signals made of fields (interlaced video signals), and vice versa. Despite the focus of embodiments illustrated herein on image and video signals, people skilled in the art can easily understand that the same concepts and methods are also applicable to any other types of multidimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.).

[0114] Certain tier-based hierarchical formats described herein use a varying amount of correction (e.g., in the form of also “residual data”, or simply “residuals”) in order to generate a reconstruction of the signal at the given level of quality that best resembles (or even losslessly reconstructs) the original. The amount of correction may be based on a fidelity of a predicted rendition of a given level of quality.

[0115] Examples of a Tier-Based Hierarchical Coding Scheme or Format

[0116] In preferred examples, the encoders or decoders are part of a tier-based hierarchical coding scheme or format. Examples of a tier-based hierarchical coding scheme include LCEVC: MPEG-5 Part 2 LCEVC (“Low Complexity Enhancement Video Coding”) and VC-6: SMPTE VC-6 ST-2117, the former being described in PCT / GB2020 / 050695 (and the associated standard document) and the latter being described in PCT / GB2018 / 053552 (and the associated standard document), all of which are incorporated by reference herein. However, the concepts illustrated herein need not be limited to these specific hierarchical coding schemes. Certain examples described herein work optimally with a VC-6 encoding.

[0117] Figures 1 to 7 provide an overview of different example tier-based hierarchical coding formats. These are provided as context for the addition of certain further signal processing operations, which are set out in the figures following Figure 7. Figures 1 to 5 provide examples similar to the implementation of SMPTE VC-6 ST-2117, whereas Figures 6 and 7 provide examples similar to the implementation of MPEG-5 Part 2 LCEVC. It may be seen that both sets of examples utilise common underlying operations (e.g., downsampling, upsampling and residual generation) and may share modular implementing technologies.

[0118] Figure 1 illustrates, very generally, a hierarchical coding scheme. Data to be encoded 101 is retrieved by a hierarchical encoder 102 which outputs encoded data 103. Subsequently, the encoded data 103 is received by a hierarchical decoder 104 which decodes the data and outputs decoded data 105. Typically, the hierarchical coding schemes used in examples herein create a base or core level, which is a representation of the original data at a lower level of quality and one or more levels of residuals which can be used to recreate the original data at a higher level of quality using a decoded version of the base level data. In general, the term “residuals” as used herein refers to a difference between a value of a reference array or reference frame and an actual array or frame of data. The array may be a one or two-dimensional array that represents a coding unit. For example, a coding unit may be a 2x2 or 4x4 set of residual values that correspond to similar sized areas of an input video frame.

[0119] It should be noted that the generalised examples are agnostic as to the nature of the input signal. Reference to “residual data” as used herein refers to data derived from a set of residuals, e.g. a set of residuals themselves or an output of a set of data processing operations that are performed on the set of residuals. Throughout the present description, generally a set of residuals includes a plurality of residuals or residual elements, each residual or residual element corresponding to a signal element, that is, an element of the signal or original data.

[0120] In specific examples, the data may be an image or video. In these examples, the set of residuals corresponds to an image or frame of the video, with each residual being associated with a pixel of the signal, the pixel being the signal element.

[0121] The methods described herein may be applied to so-called planes of data that reflect different colour components of a video signal. For example, the methods may be applied to different planes ofYUV or RGB data reflecting different colour channels. Different colour channels may be processed in parallel. The components of each stream may be collated in any logical order.

[0122] A hierarchical coding scheme will now be described in which the concepts of the invention may be deployed. The scheme is conceptually illustrated in Figures 2 to 5 and corresponds generally to VC-6 described above. In such encoding techniques, residuals data is used in progressively higher levels of quality. In this proposed technique, a core layer represents the image at a first resolution and subsequent layers in the tiered hierarchy are residual data or adjustment layers necessary for the decoding side to reconstruct the image at a higher resolution. Each layer or level may be referred to as an echelon index, such that the residuals data is data required to correct low quality information present in a lower echelon index. Each layer or echelon index in this hierarchical technique, particularly each residual layer, is often a comparatively sparse data set having many zero value elements. When reference is made to an echelon index, it refers collectively to all echelons or sets of components at that level, for example, all subsets arising from a transform step performed at that level of quality.

[0123] In this particular hierarchical manner, the described data structure removes any requirement for, or dependency on, the preceding or proceeding level of quality. A level of quality may be encoded and decoded separately, and without reference to any other layer. Thus, in contrast to many known other hierarchical encoding schemes, where there is a requirement to decode the lowest level of quality in order to decode any higher levels of quality, the described methodology does not require the decoding of any other layer. Nevertheless, the principles of exchanging information described below may also be applicable to other hierarchical coding schemes.

[0124] As shown in Figure 2, the encoded data represents a set of layers or levels, generally referred to here as echelon indices. The base or core level represents the original data frame 210, albeit at the lowest level of quality or resolution and the subsequent residuals data echelons can combine with the data at the core echelon index to recreate the original image at progressively higher resolutions.

[0125] To create the core-echelon index, an input data frame 210 may be down-sampled using a number of downsampling operations 201 corresponding to the number of levels or echelon indices to be used in the hierarchical coding operation. One fewer down-sampling operation 201 is required than the number of levels in the hierarchy. In all examples illustrated herein, there are 4 levels or echelon indices of output encoded data and accordingly 3 down-sampling operations, but it will of course be understood that these are merely for illustration. Where n indicates the number of levels, the number of down-samplers is n-1 . The core level Ri.nis the output of the third down-sampling operation. As indicated above, the core level Ri.ncorresponds to a representation of the input data frame at a lowest level of quality.

[0126] To distinguish between down-sampling operations 201 , each will be referred to in the order in which the operation is performed on the input data 210 or by the data which its output represents. For example, the third down-sampling operation 201 i-nin the example may also be referred to as the core down-sampler as its output generates the core-echelon index or echelo -n, that is, the index of all echelons at this level is 1- n. Thus, in this example, the first down-sampling operation 201 -1 corresponds to the R-i down-sampler, the second down-sampling operation 201-2 corresponds to the R-2 down-sampler and the third down-sampling operation 201 i-ncorresponds to the core or R-3 down-sampler.

[0127] As shown in Figure 2, the data representing the core level of quality Ri-nundergoes an up-sampling operation 202i-n, referred to here as the core up-sampler. A difference 203-2 between the output of the second down-sampling operation 201-2 (the output of the R-2 down-sampler, i.e. the input to the core downsampler) and the output of the core up-sampler 202i-nis output as the first residuals data R-2. This first residuals data R-2 is accordingly representative of the error between the core level R-3 and the signal that was used to create that level. Since that signal has itself undergone two down-sampling operations in this example, the first residuals data R-2 is an adjustment layerwhich can be used to recreate the original signal at a higher level of quality than the core level of quality but a lower level than the input data frame 210.

[0128] Variations in how to create residuals data representing higher levels of quality are conceptually illustrated in Figures 2 and 3.

[0129] In Figure 2, the output of the second down-sampling operation 201-2 (or R-2 down-sampler, i.e. the signal used to create the first residuals data R-2), is up-sampled 202-2 and the difference 203-1 between the input to the second down-sampling operation 201-2 (or R-2 down-sampler, i.e. the output of the R-1 down-sampler) is calculated in much the same way as the first residuals data R-2 is created. This difference is accordingly the second residuals data R-1 and represents an adjustment layerwhich can be used to recreate the original signal at a higher level of quality using the data from the lower layers.

[0130] In the variation of Figure 3, however, the output of the second down-sampling operation 201 -2 (or R-2 downsampler) is combined or summed 304-2 with the first residuals data R-2 to recreate the output of the core up-sampler 202i-n. In this variation it is this recreated data which is up-sampled 202-2 rather than the down- sampled data. The up-sampled data is similarly compared 203-1 to the input to the second down-sampling operation (or R-2 down-sampler, i.e. the output of the R-1 down-sampler) to create the second residuals data R-i.

[0131] The variation between the implementations of Figures 2 and 3 results in slight variations in the residuals data between the two implementations. Figure 2 benefits from greater potential for parallelisation.

[0132] The process or cycle repeats to create the third residuals Ro. In the examples of Figures 2 and 3, the output residuals data Ro (i.e. the third residuals data) corresponds to the highest level and is used at the decoder to recreate the input data frame. At this level the difference operation is based on the input data frame which is the same as the input to the first down-sampling operation.

[0133] Figure 4 illustrates an example encoding process 401 for encoding each of the levels or echelon indices of data to produce a set of encoded echelons of data having an echelon index. This encoding process is used merely for example of a suitable encoding process for encoding each of the levels, but it will be understood that any suitable encoding process may be used. The input to the process is a respective level of residuals data output from Figure 2 or 3 and the output is a set of echelons of encoded residuals data, the echelons of encoded residuals data together hierarchically represent the encoded data. In a first step, a transform 402 is performed. The transform may be directional decomposition transform as described in WO2013 / 171173 or a wavelet or discrete cosine transform. If a directional decomposition transform is used, there may be output a set of four components (also referred to as transformed coefficients). When reference is made to an echelon index, it refers collectively to all directions (A, H, V, D), i.e., 4 echelons. The component set is then quantized 403 before entropy encoding. In this example, the entropy encoding operation 404 is coupled to a sparsification step 405 which takes advantage of the sparseness of the residuals data to reduce the overall data size and involves mapping data elements to an ordered quadtree. Such coupling of entropy coding and sparsification is described further in WO2019 / 111004 but the precise details of such a process is not relevant to the understanding of the invention. Each array of residuals may be thought of as an echelon.

[0134] The process set out above corresponds to an encoding process suitable for encoding data for reconstruction according to SMPTE ST 2117, VC-6 Multiplanar Picture Format. VC-6 is a flexible, multiresolution, intra-only bitstream format, capable of compressing any ordered set of integer element grids, each of independent size but is also designed for picture compression. It employs data agnostic techniques for compression and is capable of compressing low or high bit-depth pictures. The bitstream’s headers can contain a variety of metadata about the picture.

[0135] As will be understood, each echelon or echelon index may be implemented using a separate encoder or encoding operation. Similarly, an encoding module may be divided into the steps of down-sampling and comparing, to produce the residuals data, and subsequently encoding the residuals or alternatively each of the steps of the echelon may be implemented in a combined encoding module. Thus, the process may be for example be implemented using 4 encoders, one for each echelon index, 1 encoder and a plurality of encoding modules operating in parallel or series, or one encoder operating on different data sets repeatedly.

[0136] The following sets out an example of reconstructing an original data frame, the data frame having been encoded using the above exemplary process. This reconstruction process may be referred to as pyramidal reconstruction. Advantageously, the method provides an efficient technique for reconstructing an image encoded in a received set of data, which may be received by way of a data stream, for example, by way of individually decoding different component sets corresponding to different image size or resolution levels, and combining the image detail from one decoded component set with the upscaled decoded image data from a lower-resolution component set. Thus by performing this process for two or more component sets, digital images at the structure or detail therein may be reconstructed for progressively higher resolutions or greater numbers of pixels, without requiring the full or complete image detail of the highest-resolution component set to be received. Rather, the method facilitates the progressive addition of increasingly higher- resolution details while reconstructing an image from a lower-resolution component set, in a staged manner.

[0137] Moreover, the decoding of each component set separately facilitates the parallel processing of received component sets, thus improving reconstruction speed and efficiency in implementations wherein a plurality of processes is available.

[0138] Each resolution level corresponds to a level of quality or echelon index. This is a collective term, associated with a plane (in this example a representation of a grid of integer value elements) that describes all new inputs or received component sets, and the output reconstructed image for a cycle of index-m. The reconstructed image in echelon index zero, for instance, is the output of the final cycle of pyramidal reconstruction.

[0139] Pyramidal reconstruction may be a process of reconstructing an inverted pyramid starting from the initial echelon index and using cycles by new residuals to derive higher echelon indices up to the maximum quality, quality zero, at echelon index zero. A cycle may be thought of as a step in such pyramidal reconstruction, the step being identified by an index-m. The step typically comprises up-sampling data output from a possible previous step, for instance, upscaling the decoded first component set, and takes new residual data as further inputs in order to obtain output data to be up-sampled in a possible following step. Where only first and second component sets are received, the number of echelon indices will be two, and no possible following step is present. However, in examples where the number of component sets, or echelon indices, is three or greater, then the output data may be progressively upsampled in the following steps.

[0140] The first component set typically corresponds to the initial echelon index, which may be denoted by echelon index 1-N, where N is the number of echelon indices in the plane.

[0141] Typically, the upscaling of the decoded first component set comprises applying an upsamplerto the output of the decoding procedure for the initial echelon index. In examples, this involves bringing the resolution of a reconstructed picture output from the decoding of the initial echelon index component set into conformity with the resolution of the second component set, corresponding to 2-N. Typically, the upscaled output from the lower echelon index component set corresponds to a predicted image at the higher echelon index resolution. Owing to the lower-resolution initial echelon index image and the up-sampling process, the predicted image typically corresponds to a smoothed or blurred picture.

[0142] Adding to this predicted picture higher-resolution details from the echelon index above provides a combined, reconstructed image set. Advantageously, where the received component sets for one or more higher-echelon index component sets comprise residual image data, or data indicating the pixel value differences between upscaled predicted pictures and original, uncompressed, or pre-encoding images, the amount of received data required in order to reconstruct an image or data set of a given resolution or quality may be considerably less than the amount or rate of data that would be required in order to receive the same quality image using other techniques. Thus, by combining low-detail image data received at lower resolutions with progressively greater-detail image data received at increasingly higher resolutions in accordance with the method, data rate requirements are reduced.

[0143] Typically, the set of encoded data comprises one or more further component sets, wherein each of the one or more further component sets corresponds to a higher image resolution than the second component set, and wherein each of the one or more further component sets corresponds to a progressively higher image resolution, the method comprising, for each of the one or more further component sets, decoding the component set so as to obtain a decoded set, the method further comprising, for each of the one or more further component sets, in ascending order of corresponding image resolution: upscaling the reconstructed set having the highest corresponding image resolution so as to increase the corresponding image resolution of the reconstructed set to be equal to the corresponding image resolution of the further component set, and combining the reconstructed set and the further component set together so as to produce a further reconstructed set.

[0144] In this way, the method may involve taking the reconstructed image output of a given component set level or echelon index, upscaling that reconstructed set, and combining it with the decoded output of the component set or echelon index above, to produce a new, higher resolution reconstructed picture. It will be understood that this may be performed repeatedly, for progressively higher echelon indices, depending on the total number of component sets in the received set.

[0145] In typical examples, each of the component sets corresponds to a progressively higher image resolution, wherein each progressively higher image resolution corresponds to a factor-of-four increase in the number of pixels in a corresponding image. Typically, therefore, the image size corresponding to a given component set is four times the size or number of pixels, or double the height and double the width, of the image corresponding to the component set below, that is the component set with the echelon index one less than the echelon index in question. A received set of component sets in which the linear size of each corresponding image is double with respect to the image size below may facilitate more simple upscaling operations, for example. In the illustrated example, the number of further component sets is two. Thus, the total number of component sets in the received set is four. This corresponds to the initial echelon index being echelon-3.

[0146] The first component set may correspond to image data, and the second and any further component sets correspond to residual image data. As noted above, the method provides particularly advantageous data rate requirement reductions for a given image size in cases where the lowest echelon index, that is the first component set, contains a low resolution, or down sampled, version of the image being transmitted. In this way, with each cycle of reconstruction, starting with a low resolution image, that image is upscaled so as to produce a high resolution albeit smoothed version, and that image is then improved by way of adding the differences between that upscaled predicted picture and the actual image to be transmitted at that resolution, and this additive improvement may be repeated for each cycle. Therefore, each component set above that of the initial echelon index needs only contain residual data in order to reintroduce the information that may have been lost in down sampling the original image to the lowest echelon index.

[0147] The method provides a way of obtaining image data, which may be residual data, upon receipt of a set containing data that has been compressed, for example, by way of decomposition, quantization, entropyencoding, and sparsification, for instance.

[0148] The sparsification step is particularly advantageous when used in connection with sets for which the original or pre-transmission data was sparse, which may typically correspond to residual image data. A residual may be a difference between elements of a first image and elements of a second image, typically colocated. Such residual image data may typically have a high degree of sparseness. This may be thought of as corresponding to an image wherein areas of detail are sparsely distributed amongst areas in which details are minimal, negligible, or absent. Such sparse data may be described as an array of data wherein the data are organised in at least a two-dimensional structure (e.g., a grid), and wherein a large portion of the data so organised are zero (logically or numerically) or are considered to be below a certain threshold. Residual data are just one example. Additionally, metadata may be sparse and so be reduced in size to a significant degree by this process. Sending data that has been sparsified allows a significant reduction in required data rate to be achieved by way of omitting to send such sparse areas, and instead reintroducing them at appropriate locations within a received byteset at a decoder.

[0149] Typically, the entropy-decoding, de-quantizing, and directional composition transform steps are performed in accordance with parameters defined by an encoder or a node from which the received set of encoded data is sent. For each echelon index, or component set, the steps serve to decode image data so as to arrive at a set which may be combined with different echelon indices as per the technique disclosed above, while allowing the set for each level to be transmitted in a data-efficient manner.

[0150] There may also be provided a method of reconstructing a set of encoded data according to the method disclosed above, wherein the decoding of each of the first and second component sets is performed according to the method disclosed above. Thus, the advantageous decoding method of the present disclosure may be utilised for each component set or echelon index in a received set of image data and reconstructed accordingly.

[0151] With reference to Figure 5, a decoding example is now described. A set of encoded data 501 is received, wherein the set comprises four echelon indices, each echelon index comprising four echelons: from echelono, the highest resolution or level of quality, to echelon-3, the initial echelon. The image data carried in the echelon-3 component set corresponds to image data, and the other component sets contain residual data for that transmitted image. While each of the levels may output data that can be considered as residuals, the residuals in the initial echelon level, that is echelon.3, effectively correspond to the actual reconstructed image. At stage 503, each of the component sets is processed in parallel so as to decode that encoded set. With reference to the initial echelon index, or the core-echelon index, the following decoding steps are carried out for each component set echelon-3 to echelono.

[0152] At step 507, the component set is de-sparsified. De-sparsification may be an optional step that is not performed in other tier-based hierarchical formats. In this example, the de-sparsification causes a sparse two-dimensional array to be recreated from the encoded byteset received at each echelon. Zero values grouped at locations within the two-dimensional array which were not received (owing to there being omitted from the transmitted byteset in order to reduce the quantity of data transmitted) are repopulated by this process. Non-zero values in the array retain their correct values and positions within the recreated two- dimensional array, with the de-sparsification step repopulating the transmitted zero values at the appropriate locations or groups of locations there between.

[0153] At step 509, a range decoder, the configured parameters of which correspond to those using which the transmitted data was encoded prior to transmission, is applied to the de-sparsified set at each echelon in order to substitute the encoded symbols within the array with pixel values. The encoded symbols in the received set are substituted for pixel values in accordance with an approximation of the pixel value distribution forthe image. The use of an approximation of the distribution, that is relative frequency of each value across all pixel values in the image, ratherthan the true distribution, permits a reduction in the amount of data required to decode the set, since the distribution information is required by the range decoder in order to carry out this step. As described in the present disclosure, the steps of de-sparsification and range decoding are interdependent, rather than sequential. This is indicated by the loop formed by the arrows in the flow diagram.

[0154] At step 511 , the array of values is de-quantized. This process is again carried out in accordance with the parameters with which the decomposed image was quantized prior to transmission.

[0155] Following de-quantization, the set is transformed at step 513 by a composition transform which comprises applying an inverse directional decomposition operation to the de-quantized array. This causes the directional filtering, according to an operator set comprising average, horizontal, vertical, and diagonal operators, to be reversed, such that the resultant array is image data for echelon-3 and residual data for echelon-2 to echelono.

[0156] Stage 505 illustrates the several cycles involved in the reconstruction utilising the output of the composition transform for each of the echelon component sets 501 . Stage 515 indicates the reconstructed image data output from the decoder 503 for the initial echelon. In an example, the reconstructed picture 515 has a resolution of 64x64. At 516, this reconstructed picture is up-sampled so as to increase its constituent number of pixels by a factor of four, thereby a predicted picture 517 having a resolution of 128x128 is produced. At stage 520, the predicted picture 517 is added to the decoded residuals 518 from the output of the decoder at echelon-2. The addition of these two 128x128-size images produces a 128x128-size reconstructed image, containing the smoothed image detail from the initial echelon enhanced by the higher- resolution detail of the residuals from echelon-2. This resultant reconstructed picture 519 may be output or displayed if the required output resolution is that corresponding to echelon-2. In the present example, the reconstructed picture 519 is used for a further cycle. At step 512, the reconstructed image 519 is up- sampled in the same manner as at step 516, so as to produce a 256x256-size predicted picture 524. This is then combined at step 528 with the decoded echelon-1 output 526, thereby producing a 256x256-size reconstructed picture 527 which is an upscaled version of prediction 519 enhanced with the higher- resolution details of residuals 526. At 530 this process is repeated a final time, and the reconstructed picture 527 is upscaled to a resolution of 512x512, for combination with the echelonO residual at stage 532. Thereby a 512x512 reconstructed picture 531 is obtained.

[0157] A further hierarchical coding technology with which the principles of the present invention may be utilised is illustrated in Figures 6 and 7. This technology is a flexible, adaptable, highly efficient and computationally inexpensive coding format which combines a different video coding format, a base codec, (e.g., AVC, HEVC, or any other present or future codec) with at least two enhancement levels of coded data.

[0158] The general structure of the encoding scheme uses a down-sampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to generate a corrected picture, and then adds a further level of enhancement data to an up-sampled version of the corrected picture. Thus, the streams are considered to be a base stream and an enhancement stream, which may be further multiplexed or otherwise combined to generate an encoded data stream. In certain cases, the base stream and the enhancement stream may be transmitted separately. References to an encoded data as described herein may refer to the enhancement stream or a combination of the base stream and the enhancement stream. The base stream may be decoded by a hardware decoder while the enhancement stream is may be suitable for software processing implementation with suitable power consumption. This general encoding structure creates a plurality of degrees of freedom that allow great flexibility and adaptability to many situations, thus making the coding format suitable for many use cases including OTT transmission, live streaming, live ultra-high-definition UHD broadcast, and so on. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and, where considered suitable, also usable as a lower resolution output.

[0159] In certain examples, each or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). The NALUs are meant to encapsulate the enhancement bitstream in order to apply the enhancement to the correct base reconstructed frame. The NALU may for example contain a reference index to the NALU containing the base decoder reconstructed frame bitstream to which the enhancement has to be applied. In this way, the enhancement can be synchronised to the base stream and the frames of each bitstream combined to produce the decoded output video (i.e. the residuals of each frame of enhancement level are combined with the frame of the base decoded stream). A group of pictures may represent multiple NALUs.

[0160] Returning to the initial process described above, where a base stream is provided along with two levels (or sub-levels) of enhancement within an enhancement stream, an example of a generalised encoding process is depicted in the block diagram of Figure 6. An input video 600 at an initial resolution is processed to generate various encoded streams 601 , 602, 603. A first encoded stream (encoded base stream) is produced by feeding a base codec (e.g., AVC, HEVC, or any other codec) with a down-sampled version of the input video. The encoded base stream may be referred to as the base layer or base level. A second encoded stream (encoded level 1 stream) is produced by processing the residuals obtained by taking the difference between a reconstructed base codec video and the down-sampled version of the input video. A third encoded stream (encoded level 2 stream) is produced by processing the residuals obtained by taking the difference between an up-sampled version of a corrected version of the reconstructed base coded video and the input video. In certain cases, the components of Figure 6 may provide a general low complexity encoder. In certain cases, the enhancement streams may be generated by encoding processes that form part of the low complexity encoder and the low complexity encoder may be configured to control an independent base encoder and decoder (e.g., as packaged as a base codec). In other cases, the base encoder and decoder may be supplied as part of the low complexity encoder. In one case, the low complexity encoder of Figure 6 may be seen as a form of wrapper for the base codec, where the functionality of the base codec may be hidden from an entity implementing the low complexity encoder.

[0161] A down-sampling operation illustrated by down-sampling component 105 may be applied to the input video to produce a down-sampled video to be encoded by a base encoder 613 of a base codec. The downsampling can be done either in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 613 and a base decoder 614 may be implemented by a base codec (e.g., as different functions of a common codec). The base codec, and / or one or more of the base encoder 613 and the base decoder 614 may comprise suitably configured electronic circuitry (e.g., a hardware encoder / decoder) and / or computer program code that is executed by a processor.

[0162] Each enhancement stream encoding process may not necessarily include an upsampling step. In Figure 6 for example, the first enhancement stream is conceptually a correction stream while the second enhancement stream is upsampled to provide a level of enhancement.

[0163] Looking at the process of generating the enhancement streams in more detail, to generate the encoded Level 1 stream, the encoded base stream is decoded by the base decoder 614 (i.e. a decoding operation is applied to the encoded base stream to generate a decoded base stream). Decoding may be performed by a decoding function or mode of a base codec. The difference between the decoded base stream and the down-sampled input video is then created at a level 1 comparator 610 (i.e. a subtraction operation is applied to the down-sampled input video and the decoded base stream to generate a first set of residuals). The output of the comparator 610 may be referred to as a first set of residuals, e.g. a surface or frame of residual data, where a residual value is determined for each picture element at the resolution of the base encoder 613, the base decoder 614 and the output of the down-sampling block 605.

[0164] The difference is then encoded by a first encoder 615 (i.e. a level 1 encoder) to generate the encoded Level

[0165] 1 stream 602 (i.e. an encoding operation is applied to the first set of residuals to generate a first enhancement stream).

[0166] As noted above, the enhancement stream may comprise a first level of enhancement 602 and a second level of enhancement 603. The first level of enhancement 602 may be considered to be a corrected stream, e.g. a stream that provides a level of correction to the base encoded / decoded video signal at a lower resolution than the input video 600. The second level of enhancement 603 may be considered to be a further level of enhancement that converts the corrected stream to the original input video 600, e.g. that applies a level of enhancement or correction to a signal that is reconstructed from the corrected stream.

[0167] In the example of Figure 6, the second level of enhancement 603 is created by encoding a further set of residuals. The further set of residuals are generated by a level 2 comparator 619. The level 2 comparator 619 determines a difference between an upsampled version of a decoded level 1 stream, e.g. the output of an upsampling component 617, and the input video 600. The input to the up-sampling component 617 is generated by applying a first decoder (i.e. a level 1 decoder) to the output of the first encoder 615. This generates a decoded set of level 1 residuals. These are then combined with the output of the base decoder 614 at summation component 620. This effectively applies the level 1 residuals to the output of the base decoder 614. It allows for losses in the level 1 encoding and decoding process to be corrected by the level

[0168] 2 residuals. The output of summation component 620 may be seen as a simulated signal that represents an output of applying level 1 processing to the encoded base stream 601 and the encoded level 1 stream 602 at a decoder.

[0169] As noted, an upsampled stream is compared to the input video which creates a further set of residuals (i.e. a difference operation is applied to the upsampled re-created stream to generate a further set of residuals). The further set of residuals are then encoded by a second encoder 621 (i.e. a level 2 encoder) as the encoded level 2 enhancement stream (i.e. an encoding operation is then applied to the further set of residuals to generate an encoded further enhancement stream).

[0170] Thus, as illustrated in Figure 6 and described above, the output of the encoding process is a base stream 601 and one or more enhancement streams 602, 603 which preferably comprise a first level of enhancement and a further level of enhancement. The three streams 601 , 602 and 603 may be combined, with or without additional information such as control headers, to generate a combined stream for the video encoding framework that represents the input video 600. It should be noted that the components shown in Figure 6 may operate on blocks or coding units of data, e.g. corresponding to 2x2 or 4x4 portions of a frame at a particular level of resolution. The components operate without any inter-block dependencies, hence they may be applied in parallel to multiple blocks or coding units within a frame. This differs from comparative video encoding schemes wherein there are dependencies between blocks (e.g., either spatial dependencies ortemporal dependencies). The dependencies of comparative video encoding schemes limit the level of parallelism and require a much higher complexity.

[0171] A corresponding generalised decoding process is depicted in the block diagram of Figure 7. Figure 7 may be said to show a low complexity decoder that corresponds to the low complexity encoder of Figure 6. The low complexity decoder receives the three streams 601 , 602, 603 generated by the low complexity encoder together with headers 704 containing further decoding information. The encoded base stream 601 is decoded by a base decoder 710 corresponding to the base codec used in the low complexity encoder. The encoded level 1 stream 602 is received by a first decoder 711 (i.e. a level 1 decoder), which decodes a first set of residuals as encoded by the first encoder 615 of Figure 1 . At a first summation component 712, the output of the base decoder 710 is combined with the decoded residuals obtained from the first decoder 711. The combined video, which may be said to be a level 1 reconstructed video signal, is upsampled by upsampling component 713. The encoded level 2 stream 103 is received by a second decoder 714 (i.e. a level 2 decoder). The second decoder 714 decodes a second set of residuals as encoded by the second encoder 621 of Figure 1 . Although the headers 704 are shown in Figure 7 as being used by the second decoder 714, they may also be used by the first decoder 711 as well as the base decoder 710. The output of the second decoder 714 is a second set of decoded residuals. These may be at a higher resolution to the first set of residuals and the input to the upsampling component 713. At a second summation component 715, the second set of residuals from the second decoder 714 are combined with the output of the up- sampling component 713, i.e. an up-sampled reconstructed level 1 signal, to reconstruct decoded video 750.

[0172] As per the low complexity encoder, the low complexity decoder of Figure 7 may operate in parallel on different blocks or coding units of a given frame of the video signal. Additionally, decoding by two or more of the base decoder 710, the first decoder 711 and the second decoder 714 may be performed in parallel. This is possible as there are no inter-block dependencies.

[0173] In the decoding process, the decoder may parse the headers 704 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low complexity decoder based on those headers. In order to re-create the input video, the low complexity decoder may decode each of the base stream, the first enhancement stream and the further or second enhancement stream. The frames of the stream may be synchronised and then combined to derive the decoded video 750. The decoded video 750 may be a lossy or lossless reconstruction of the original input video 100 depending on the configuration of the low complexity encoder and decoder. In many cases, the decoded video 750 may be a lossy reconstruction of the original input video 600 where the losses have a reduced or minimal effect on the perception of the decoded video 750.

[0174] In each of Figures 6 and 7, the level 2 and level 1 encoding operations may include the steps of transformation, quantization and entropy encoding (e.g., in that order). These steps may be implemented in a similar manner to the operations shown in Figures 4 and 5. The encoding operations may also include residual ranking, weighting and filtering. Similarly, at the decoding stage, the residuals may be passed through an entropy decoder, a de-quantizer and an inverse transform module (e.g., in that order). Any suitable encoding and corresponding decoding operation may be used. Preferably however, the level 2 and level 1 encoding steps may be performed in software (e.g., as executed by one or more central or graphical processing units in an encoding device).

[0175] The transform as described herein may use a directional decomposition transform such as a Hadamard- based transform. Both may comprise a small kernel or matrix that is applied to flattened coding units of residuals (i.e. 2x2 or 4x4 blocks of residuals). More details on the transform can be found for example in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder may select between different transforms to be used, for example between a size of kernel to be applied.

[0176] The transform may transform the residual information to four surfaces. For example, the transform may produce the following components or transformed coefficients: average, vertical, horizontal and diagonal. A particular surface may comprise all the values for a particular component, e.g. a first surface may comprise all the average values, a second all the vertical values and so on. As alluded to earlier in this disclosure, these components that are output by the transform may be taken in such embodiments as the coefficients to be quantized in accordance with the described methods. A quantization scheme may be useful to create the residual signals into quanta, so that certain variables can assume only certain discrete magnitudes. Entropy encoding in this example may comprise run length encoding (RLE), then processing the encoded output is processed using a Huffman encoder. In certain cases, only one of these schemes may be used when entropy encoding is desirable.

[0177] In summary, the methods and apparatuses herein are based on an overall approach which is built over an existing encoding and / or decoding algorithm (such as MPEG standards such as AVC / H.264, HEVC / H.265, etc. as well as non-standard algorithm such as VP9, AV1 , and others) which works as a baseline for an enhancement layer which works accordingly to a different encoding and / or decoding approach. The idea behind the overall approach of the examples is to hierarchically encode / decode the video frame as opposed to the use block-based approaches as used in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a decimated frame and so on.

[0178] As indicated above, the processes may be applied in parallel to coding units or blocks of a colour component of a frame as there are no inter-block dependencies. The encoding of each colour component within a set of colour components may also be performed in parallel (e.g., such that the operations are duplicated according to (number of frames) * (number of colour components) * (number of coding units per frame)). It should also be noted that different colour components may have a different number of coding units per frame, e.g. a luma (e.g., Y) component may be processed at a higher resolution than a set of chroma (e.g., U or V) components as human vision may detect lightness changes more than colour changes.

[0179] Thus, as illustrated and described above, the output of the decoding process is an (optional) base reconstruction, and an original signal reconstruction at a higher level. This example is particularly well- suited to creating encoded and decoded video at different frame resolutions. For example, the input signal 30 may be an HD video signal comprising frames at 1920 x 1080 resolution. In certain cases, the base reconstruction and the level 2 reconstruction may both be used by a display device. For example, in cases of network traffic, the level 2 stream may be disrupted more than the level 1 and base streams (as it may contain up to 4x the amount of data where down-sampling reduces the dimensionality in each direction by 2). In this case, when traffic occurs the display device may revert to displaying the base reconstruction while the level 2 stream is disrupted (e.g., while a level 2 reconstruction is unavailable), and then return to displaying the level 2 reconstruction when network conditions improve. A similar approach may be applied when a decoding device suffers from resource constraints, e.g. a set-top box performing a systems update may have an operation base decoder 220 to output the base reconstruction but may not have processing capacity to compute the level 2 reconstruction.

[0180] The encoding arrangement also enables video distributors to distribute video to a set of heterogeneous devices; those with just a base decoder 720 view the base reconstruction, whereas those with the enhancement level may view a higher-quality level 2 reconstruction. In comparative cases, two full video streams at separate resolutions were required to service both sets of devices. As the level 2 and level 1 enhancement streams encode residual data, the level 2 and level 1 enhancement streams may be more efficiently encoded, e.g. distributions of residual data typically have much of their mass around 0 (i.e. where there is no difference) and typically take on a small range of values about 0. This may be particularly the case following quantization. In contrast, full video streams at different resolutions will have different distributions with a non-zero mean or median that require a higher bit rate for transmission to the decoder.

[0181] In the examples described herein residuals are encoded by an encoding pipeline. This may include transformation, quantization and entropy encoding operations. It may also include residual ranking, weighting and filtering. Residuals are then transmitted to a decoder, e.g. as L-1 and L-2 enhancement streams, which may be combined with a base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for a hybrid data stream that comprises the base stream and both enhancements streams, and then different adaptive bit rates are applied to the individual streams based on the data being processed to meet the set bit rate (e.g., high-quality video that is perceived with low levels of artefacts may be constructed by adaptively assigning a bit rate to different individual streams, even at a frame by frame level, such that constrained data may be used by the most perceptually influential individual streams, which may change as the image data changes).

[0182] The sets of residuals as described herein may be seen as sparse data, e.g. in many cases there is no difference for a given pixel or area and the resultant residual value is zero. When looking at the distribution of residuals much of the probability mass is allocated to small residual values located near zero - e.g. for certain videos values of -2, -1 , 0, 1 , 2 etc. occur the most frequently. In certain cases, the distribution of residual values is symmetric or near symmetric about 0. In certain test video cases, the distribution of residual values was found to take a shape similar to logarithmic or exponential distributions (e.g., symmetrically or near symmetrically) about 0. The exact distribution of residual values may depend on the content of the input video stream.

[0183] Residuals may be treated as a two-dimensional image in themselves, e.g. a delta image of differences. Seen in this manner the sparsity of the data may be seen to relate features like “dots”, small “lines”, “edges”, “corners”, etc. that are visible in the residual images. It has been found that these features are typically not fully correlated (e.g., in space and / or in time). They have characteristics that differ from the characteristics of the image data they are derived from (e.g., pixel characteristics of the original video signal).

[0184] As the characteristics of residuals differ from the characteristics of the image data they are derived from it is generally not possible to apply standard encoding approaches, e.g. such as those found in traditional Moving Picture Experts Group (MPEG) encoding and decoding standards. For example, many comparative schemes use large transforms (e.g., transforms of large areas of pixels in a normal video frame). Due to the characteristics of residuals, e.g. as described above, it would be very inefficient to use these comparative large transforms on residual images. For example, it would be very hard to encode a small dot in a residual image using a large block designed for an area of a normal image.

[0185] Certain examples described herein address these issues by instead using small and simple transform kernels (e.g., 2x2 or 4x4 kernels - the Directional Decomposition and the Directional Decomposition Squared - as presented herein). The transform described herein may be applied using a Hadamard matrix (e.g., a 4x4 matrix for a flattened 2x2 coding block or a 16x16 matrix for a flattened 4x4 coding block). This moves in a different direction from comparative video encoding approaches. Applying these new approaches to blocks of residuals generates compression efficiency. For example, certain transforms generate uncorrelated transformed coefficients (e.g., in space) that may be efficiently compressed. While correlations between transformed coefficients may be exploited, e.g. for lines in residual images, these can lead to encoding complexity, which is difficult to implement on legacy and low-resource devices, and often generates other complex artefacts that need to be corrected. Pre-processing residuals by setting certain residual values to 0 (i.e. not forwarding these for processing) may provide a controllable and flexible way to manage bitrates and stream bandwidths, as well as resource use. Referring to Figure 8, the devices herein may be implemented using a computer apparatus. Examples of the computer apparatus 1200 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, base station, phone device, desktop computer, laptop, notebook, netbook computer, mainframe computer system, handheld computer, workstation, network computer, application server, storage device, a consumer electronics device such as a camera, camcorder, mobile device, video game console, handheld video game device, a peripheral device such as a switch, modem, router, a vehicle etc., or in general any type of computing or electronic device. In a preferred example, the apparatus comprises a server computing device.

[0186] In this example, the apparatus 1200 comprises one or more processors 1201 configured to process information and / or instructions. The one or more processors 1201 may comprise a central processing unit (CPU). The one or more processors 1201 are coupled with a bus 1202. Operations performed by the one or more processors 1201 may be carried out by hardware and / or software. The one or more processors 1201 may comprise multiple co-located processors or multiple disparately located processors.

[0187] In this example, the apparatus 1200 comprises computer-useable volatile memory 1203 configured to store information and / or instructions for the one or more processors 1201. The computer-useable volatile memory 1203 is coupled with the bus 1202. The computer-useable volatile memory 1203 may comprise random access memory (RAM).

[0188] In this example, the apparatus 1200 comprises computer-useable non-volatile memory 1204 configured to store information and / or instructions for the one or more processors 501 . The computer-useable nonvolatile memory 1204 is coupled with the bus 1202. The computer-useable non-volatile memory 1204 may comprise read-only memory (ROM).

[0189] In this example, the apparatus 1200 comprises one or more data-storage units 1205 configured to store information and / or instructions. The one or more data-storage units 1205 are coupled with the bus 1202. The one or more data-storage units 1205 may for example comprise a magnetic or optical disk and disk drive or a solid-state drive (SSD).

[0190] In this example, the apparatus 1200 comprises one or more input / output (I / O) devices 1206 configured to communicate information to and / or from the one or more processors 1201 . The one or more I / O devices 506 are coupled with the bus 1202. The one or more I / O devices 506 comprise at least one network interface. The at least one network interface enables the apparatus 1200 to communicate via one or more data communications networks. Examples of data communications networks include, but are not limited to, a Wide Area Network (WAN such as the Internet) and a Local Area Network (LAN). Many data communications links may comprise communications over a plurality of interconnected networks as is known in the art. The one or more I / O devices 1206 may enable a user to provide input to the apparatus 1200 via one or more input devices (not shown). The one or more I / O devices 1206 may enable information to be provided to a user via one or more output devices (not shown).

[0191] Various other entities are depicted for the apparatus 1200. For example, when present, an operating system 1207, data processing module 1208, one or more further modules 1209, and data 1210 are shown as residing in one, or a combination, of the computer-usable volatile memory 1203, computer-usable nonvolatile memory 1204 and the one or more data-storage units 1205. The signal processing module 1208 may be implemented by way of computer program code stored in memory locations within the computer- usable non-volatile memory 1204, computer-readable storage media within the one or more data-storage units 1205 and / or other tangible computer-readable storage media. Examples of tangible computer- readable storage media include, but are not limited to, an optical medium (e.g., CD-ROM, DVD-ROM or Blu-ray), flash memory card, floppy or hard disk or any other medium capable of storing computer-readable instructions such as firmware or microcode in at least one ROM or RAM or Programmable ROM (PROM) chips or as an Application Specific Integrated Circuit (ASIC). The apparatus 1200 may therefore comprise a data processing module 1208 which can be executed by the one or more processors 1201. The data processing module 1208 can be configured to include instructions to implement at least some of the operations described herein. During operation, the one or more processors 1201 launch, run, execute, interpret or otherwise perform the instructions in the data processing module 1208.

[0192] Although at least some aspects of the examples described herein with reference to the drawings comprise computer processes performed in processing systems or processors, examples described herein also extend to computer programs, for example computer programs on or in a carrier, adapted for putting the examples into practice. The carrier may be any entity or device capable of carrying the program.

[0193] It will be appreciated that the apparatus 1200 may comprise more, fewer and / or different components from those depicted in Figure 12. At each of the edge and cloud locations, the apparatus 1200 may be implemented as a single entity or as a distributed computing platform. For example, a remote processing device may comprise a plurality of processing devices that are interconnected with high-speed data communications links (e.g. as compared to a more limited link between the edge and the remote processing device). Similarly, processing at an edge location may be performed by a plurality of interconnected devices (e.g. coupled via a LAN).

[0194] The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. They may include configuring an apparatus to carry out and / or support any or all of techniques described herein.

[0195] The above embodiments are to be understood as illustrative examples. Further embodiments are envisaged. Different combinations of described options and features, including omission of certain features, are explicitly allowed and may be provided in different implementations. For example, it is to be understood that any feature described in relation to any one example may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention.

[0196] Training Large Multimodal Models

[0197] LMMs can be trained to process many different types of data. We describe below an example process for training a LMM on video data (and / or image data). Training a LLM on videos involves a more complex process than text-based training because it requires the model to interpret and generate responses based on both audio and visual cues, not just text. This type of model is often referred to as a multimodal model, as it processes multiple forms of input (text, audio, video). Referring to Figure 9, there is shown an exemplary process for performing such training of an LMM. This method is typically carried out by a computer device (such as the apparatus 1200) and / or a plurality of computer devices:

[0198] 1 . Data Collection. A first step 11 of data collection involves gathering a diverse set of video data that may include movies, TV shows, instructional videos, and other visual content. Typically, the data also includes associated transcriptions or subtitles to provide textual context for the audiovisual content. Non-video data (such as sensor data) could also be used.

[0199] 2. Preprocessing. In a second step 12, videos are processed to extract different types of data. For example, the second step may comprise extracting one or more of:

[0200] Visual Data: Frames are extracted from videos and may be resized or transformed to standardize inputs for the model.

[0201] Audio Data: Audio tracks are extracted and converted into a suitable format, such as spectrograms or mel-frequency cepstral coefficients (MFCCs), which are common ways of representing audio data. Text Data: Subtitles or transcriptions are synchronized with audio-visual data to create a cohesive dataset.

[0202] 3. Feature Extraction. In a third step 13, features are identified in the extracted data The model is then used to interpret the extracted features. For example:

[0203] Visual Features: Techniques such as convolutional neural networks (CNNs) might be used to analyze and understand visual patterns in video frames.

[0204] Audio Features: Audio data can be processed using models that are capable of understanding spoken language, like recurrent neural networks (RNNs) or Transformers specifically adapted for audio.

[0205] Textual Features: Text data is processed similarly to how traditional LLMs are trained, using tokenization and embedding.

[0206] Model Architecture. To enable the processing of various inputs, the architecture for a multimodal model typically involves integrating different specialized sub-models or layers that can process each type of input. These might include:

[0207] Vision Transformer: For processing visual data.

[0208] BERT-like Transformer: For textual data.

[0209] Audio Neural Network: For processing audio inputs.

[0210] These components are often connected through a central coordinating mechanism that allows the model to combine insights from each type of data to make decisions or generate outputs.

[0211] 4. Training. A fourth step 14 involves a training process that involves showing the model synchronized video, audio, and text data and teaching it to predict outcomes based on this combined input. This could be predicting the next frame, the next word in a subtitle, or even generating descriptive text based on video content.

[0212] 5. Fine-tuning and Evaluation. Similar to LLMs, a fifth step 15 of training multimodal models includes fine- tuning and evaluating the models to specialize in certain tasks, like video summarization, question answering based on video content, or interactive scenarios requiring understanding of video and audio context.

[0213] Challenges. Training LMMs (including in this example of video data) with videos introduces several challenges:

[0214] High computational costs: Video processing requires significant computational resources.

[0215] Data Collection and Curation: Gathering and curating a large-scale, diverse, and high-quality dataset that spans multiple modalities is a significant challenge for LMMs. The data must be accurately labeled and free from biases, which requires substantial human effort and domain expertise. The size (in terms of file size) may be large, indeed much larger than unimodal systems due to the variety of the data that LMM use to train.

[0216] Complex data synchronization: Aligning audio, text, and video accurately is complex.

[0217] Scalability: Handling large datasets of video is more challenging than text due to the sheer size of the data.

[0218] Training an LMM (e.g. with video data) is at the frontier of current Al research and involves significant challenges and innovations in model design and computational resources. The resulting multimodal models can perform tasks that integrate visual, auditory, and textual understanding in a cohesive manner.

[0219] The challenges of training LMMs can be further illustrated by way of comparison to training LLMs. Training Large Multimodal Models (LMMs) presents a unique set of technical challenges compared to training large language models (LLMs), which are primarily due to the complexities involved in handling multiple types of data inputs. Certain key challenges in training LMMs that differ or are more pronounced than those faced when training LLMs are considered below:

[0220] 1. Data Collection and Annotation

[0221] Multimodal Data Requirements: Unlike LLMs, which primarily require text data, LMMs need diverse datasets that include images, text, audio, video, and sometimes even sensory data. Collecting and curating such varied data to ensure high quality and diversity is significantly more complex and resource-intensive.

[0222] Annotation Complexity: Labeling multimodal data can be more challenging than text data. For instance, annotating images or videos requires detailed descriptions, object tagging, and sometimes bounding boxes, which is far more labor-intensive and requires domain expertise in image processing as well as the subject matter.

[0223] 2. Model Architecture and Integration

[0224] Integration of Modalities: LMMs must integrate multiple types of data effectively. Designing neural architectures that can handle such integration, such as choosing between early, late, or hybrid integration strategies for combining modalities, adds a layer of complexity absent in LLMs.

[0225] Synchronization and Alignment: Ensuring that different modal data types are properly synchronized and aligned during training is critical. This includes dealing with varying data resolutions, sampling rates, and formats, which is not a concern with purely text-based LLMs.

[0226] 3. Computational Demands

[0227] Increased Computational Load: LMMs often require more computational power than LLMs due to the need to process and learn from high-dimensional data like images and videos. This can increase the costs and logistical demands of model training.

[0228] Model Size and Scalability: Managing the size and scalability of LMMs can be more challenging. The complexity of multimodal data often necessitates larger and more complex neural network architectures, which can be difficult to scale efficiently compared to text-only models.

[0229] 4. Overfitting and Generalization

[0230] Risk of Overfitting: Training LMMs involves a higher risk of overfitting due to the complex and often noisy nature of multimodal data. Ensuring that these models generalize well across different types of unseen multimodal data is more challenging than with LLMs, which generally deal with more homogeneous data.

[0231] Bias and Fairness: Multimodal data can contain more varied and subtle biases across different data types. Addressing these biases to ensure fairness and ethical use of LMMs is a significant challenge, often more complex than handling biases in text-only data.

[0232] 5. Training Dynamics

[0233] Training Stability and Convergence: The dynamics of training LMMs, such as learning rate adjustments and optimization strategies, can be more complex due to the interactions between different data modalities. Achieving stable training and convergence in LMMs often requires more sophisticated techniques and fine- tuning compared to LLMs.

[0234] 6. Evaluation Metrics

[0235] Complex Evaluation Metrics: Evaluating the performance of LMMs often requires more complex and comprehensive metrics that can assess how well the model handles the interplay between different modalities, unlike LLMs where metrics like perplexity or BLEU score can be more straightforwardly applied.

[0236] Further challenges of training a model on multimodal content, e.g. video content, particularly for multimodal models that integrate text, audio, and video inputs, include: 1 . High Computational Requirements: Video data is inherently high-dimensional compared to text or even audio. Processing high-resolution video in real-time or near-real-time for training requires substantial computational power and memory. This includes handling the encoding and decoding of video streams, which can be computationally expensive.

[0237] 2. Large Data Storage Needs: Videos take up a significant amount of storage space. Managing and processing large datasets of videos fortraining purposes necessitates extensive data storage solutions and efficient data handling protocols, which can be costly and complex to maintain.

[0238] 3. Complex Data Preprocessing: Video data should be preprocessed before it can be used fortraining. This involves:

[0239] Extracting frames from videos.

[0240] Synchronizing audio tracks with corresponding video frames.

[0241] Aligning subtitles or transcriptions with the audio-visual content.

[0242] This preprocessing step is critical to ensure the data is usable for the model, and mistakes here can lead to poor training outcomes.

[0243] 4. Temporal Dependency Handling. Videos contain temporal dependencies, meaning the context or meaning of a scene can depend on preceding and succeeding frames. Capturing these dependencies accurately is crucial for the model to understand and generate coherent content based on video input. This often requires sophisticated model architectures that can handle long sequences of data, such as recurrent neural networks or specialized Transformers.

[0244] 5. Data Diversity and Bias. The diversity of video data in training sets can affect the model's performance and its ability to generalize across different scenarios. Furthermore, biases in the dataset (e.g., cultural, linguistic, demographic) can lead the model to develop skewed perceptions or inaccurate representations, which can perpetuate these biases in its outputs.

[0245] 6. Annotation and Labeling Challenges. Annotating video data fortraining machine learning models is more complex and labor-intensive than labeling images or text. For instance, annotating actions in video, understanding complex scenes, or providing detailed descriptions requires human annotators to understand context deeply, which can be subjective and time-consuming.

[0246] 7. Integration of Multimodal Features. Combining features from different modalities (text, audio, visual) into a cohesive model architecture is technically challenging. Each modality may require different neural network architectures and processing techniques, and their integration must allow the model to leverage each type of input effectively.

[0247] 8. Real-Time Processing Challenges. If the model is intended to operate in real-time (e.g., for live video captioning or interaction), it must process inputs and generate responses extremely quickly. Achieving this with high accuracy and low latency is a significant technical challenge.

[0248] These challenges necessitate novel solutions in data handling, model architecture, and computational efficiency. Addressing them is critical for the successful deployment of multimodal models trained on video content.

[0249] Therefore, some of the problems may be outlined as below:

[0250] 1 . When LMMs need to be trained on data - they may need to process data multiple times to extract meaning, this need a large memory bandwidth between model and training data..

[0251] 2. Time to train or update training on LMMs is longer than other Al systems (e.g. LLMs) 3. When training on multi modes (eg sensor output, text, video, speech), you can’t have hardware decoder for every different type of information mode. Thus any training, or usage, solution must should be flexible with respect to hardware.

[0252] 4. Inference today is typically sent over internet to somewhere with high computing power- e.g. cloud, edge rack etc. This is not generally a problem for LLMs transferring text requires only a low bandwidth. The time delay / limitation between prompt and response is the model rather than transmission of the prompt.

[0253] The present disclosure identifies that:

[0254] • With LMMs, transmitting the prompt may become the limiting factor because you are transmitting large files, e.g. videos, multi sensor outputs etc..

[0255] • LMMs compression / decompression should be software based - in this regard, it is undesirable to require hardware codecs for each data type and / or sensor input. So a codec that can be performed using general purpose (gpu / cpu) processors is typically desirable.

[0256] In summary, while large multimodal models hold tremendous potential for synthesizing information across various forms of data, they also present unique challenges in terms of their development, deployment, and ethical use. Addressing these challenges requires concerted efforts in model design, training techniques, and regulatory frameworks.

[0257] Large Multimodal Models (LMMs) come with a set of technical challenges that often relate to how they process and respond to prompts and perform inference. These challenges can significantly impact their effectiveness, scalability, and reliability in real-world applications. Here’s an overview of further key technical challenges:

[0258] 1. Prompt Engineering

[0259] Complexity of Prompts: LMMs require carefully designed prompts to function effectively. The complexity arises in crafting prompts that can guide the model to understand and integrate multiple data types (like text, images, and audio) simultaneously. This requires understanding not just the content but how different data types interact and influence each other.

[0260] Prompt Ambiguity and Misinterpretation: Ambiguity in prompts can lead to incorrect or irrelevant outputs. Multimodal prompts must be unambiguous and specific to direct the model accurately, particularly when the prompt must guide the model to synthesize or infer information across different modalities.

[0261] Dependence on High-Quality Prompting: The performance of LMMs can heavily depend on the quality of the input prompts. Inadequate or poorly structured prompts can lead to errors in model output, making prompt engineering a critical skill.

[0262] 2. Inference Challenges

[0263] Latency and Computational Cost: The inference process for LMMs, which involves generating predictions from new input data, can be computationally expensive and slow, especially when dealing with large inputs across multiple modalities. This is a significant challenge in deploying these models in environments where real-time processing is critical.

[0264] Scalability Issues: Scaling LMMs to handle varying volumes and velocities of data from different sources can be technically challenging. As data volume and input size increase, the resource requirements for maintaining performance and speed also grow, which can be a limiting factor for many applications.

[0265] Error Propagation: In multimodal systems, an error in interpreting one type of data can propagate and amplify across the system, leading to compounded errors in the final output. For example, a misinterpreted word in a text could lead to an incorrect image being generated or associated, affecting the overall system accuracy. Managing Data Synchronicity: Ensuring that data from different modalities is synchronized and timely processed is crucial. Issues in data alignment — where inputs from different sources are not correctly aligned in time — can lead to misinterpretations and errors in output.

[0266] 3. Generalization and Robustness

[0267] Handling Diverse Data Types: A technical challenge is ensuring that LMMs can handle a wide variety of data types and sub-types within modalities (e.g., different image formats, dialects in language). Models should be robust enough to generalize well across these variations to be practical in diverse real-world settings.

[0268] Computational Resources: Deploying large multimodal models often requires significant computational power, including specialized hardware like GPUs or TPUs. This can limit their accessibility for smaller computing systems and / or connection bandwidths under a threshold.

[0269] 4. Adaptability to New Contexts and Domains

[0270] LMMs trained on specific types of data or in specific contexts may struggle when applied to new scenarios or domains. This lack of flexibility can limit their usability across different applications without extensive retraining or fine-tuning.

[0271] The solutions described below generally address these technical challenges, focusing not just on improving the models' accuracy but also their usability, efficiency, and robustness in diverse applications. Addressing these issues is critical for the successful integration of LMMs into practical, everyday technology solutions.

[0272] Improvements in models and methods of training and operating models

[0273] In general, we describe the use of hierarchically coded data with machine learning models. We describe the use of hierarchically coded data to train and / or use (e.g. perform inference) with machine learning models. The machine learning models may be LMMs. The structure of the hierarchically coded data may be as described elsewhere in the document (e.g. the hierarchically coded data may comprise a plurality of levels of quality as described with reference to Figures 1 to 7 above.

[0274] Exemplary advantages provided by (aspects of) the disclosures herein include:

[0275] 1 . Reduce data transmission: use of a (e.g. video) codec that has a high compression efficiency to reduce amount of data to be transmitted between model and content

[0276] 2. Speed up training: use a compression algo that has a quick decoding time

[0277] 3. Work on CPU / GPU rather than dedicated hardware: use a compression algo that can run on general purpose processor (cpu or gpu) rather than a specific codec hardware block

[0278] 4. More flexible training: different parts of the data may be processed at different qualities.

[0279] 5. Power consumption reduction: use of efficient codecs to reduce power and computation..

[0280] In particular, decoding data (e.g. video, sensor data, audio,) more efficiently can be highly beneficial in the training of models on such data, particularly for multimodal models that need to process and integrate visual information along with text and audio. Efficient video decoding can improve the training process in several key ways, which we describe in relation to video data (merely as an illustrative example, but may apply to any other data (sensor data from multiple sensors, Virtual reality data, machine readable data, and so forth):

[0281] 1. Increased Training Speed

[0282] Efficient video decoding allows for faster preprocessing of video data, which is a major component of the overall training pipeline. Reducing the time it takes to decode and preprocess video frames can significantly decrease the total time required for training cycles. This is crucial in a domain where models often need to iterate over large datasets multiple times during training. 2. Reduced Computational Load

[0283] Video decoding is typically resource-intensive, often requiring significant GPU or CPU capacities. By optimizing decoding — such as through more effective software algorithms (such as those described in this document) or leveraging existing hardware more efficiently — the overall computational load can be reduced. This frees up resources that can then be used for other parts of the training process, such as forward and backward propagation through the neural network, which are also computationally expensive.

[0284] 3. Scalability

[0285] Efficient video decoding improves scalability in training multimodal models. As models become more complex and datasets grow larger, the ability to efficiently process data in a hierarchical manner becomes crucial. Efficient decoding helps manage larger volumes of data without linearly scaling hardware resources, making it more feasible to train on extensive video datasets.

[0286] 4. Energy Efficiency

[0287] Efficient decoding not only saves computational resources but also contributes to greater energy efficiency. Video processing is power-intensive, and improvements in decoding efficiency can lead to significant reductions in the energy consumption of training processes, which is important both from a cost and an environmental standpoint.

[0288] 5. Enabling More Complex Models

[0289] By reducing the overhead associated with video decoding, more computational resources can be allocated to experimenting with and training more sophisticated and potentially more accurate models. This could involve deeper neural networks or more complex temporal dynamics that can better understand the nuances of video data.

[0290] 6. Real-Time Applications

[0291] For models that are intended for real-time applications, such as video-based interactive systems, efficient video decoding is crucial. Faster decoding, including hierarchical decoding, allows for quicker response times, which is essential for applications that rely on low-latency interactions, such as real-time sign language interpretation or live event captioning.

[0292] In summary, improving the efficiency of decoding can significantly impact the training and functionality of language models on data by enhancing training speed, reducing computational demands, and enabling the handling of larger datasets. This leads to more scalable, efficient, and potentially more capable multimodal systems.

[0293] Hierarchical Encoding in LMMs

[0294] We describe the use of a multi-layer codec for training and / or inference of LMM models (the codec described herein is not limited to images and video, it may also include sensor data or text). Different aspects of this codec include and enable:

[0295] • Storage and data transmission reduction: in text learning, LLMs may use autocomplete processes. For video - LMM training can use multi-layer codecs to 'autocomplete' to a next level of quality (LOQ). This provides an advantage that is a computer device does not need to have access to multiple different resolutions of a video file (e.g. a 540 & 720 & 1080 version) - instead, the device can use a single file. a. We describe training an LLM by feeding a low LOQ of a data encoded in a hierarchical format, and then using a higher LOQ to provide feedback to the model. The present disclosure considers the use of a comparison of an output of the LMM with the higher LOQ to train the model. The present disclosure envisages providing a higher LOQ by providing an enhancement layer that, when combined with a low LOQ, reconstructs the higher LOQ. b. We describe a method of compressing a whole dataset (e.g. an image) so that when a computer device accesses this dataset it does not need to process each element (e.g. token) in an image at same level, it can utilise different tokens at different layers of quality.

[0296] • We describe a method of processing data (e.g. video, sensor data, and so forth) without having to decompress the entire dataset. In other words, selectively decoding the data. This is useful for training because the model can decode only the LOQ (or part of the LOQ) that it needs to train. Also, it may be that objects present in the same video may be decoded at different LOQ, for example a ball moving towards an object may require less LoQs to be decode than the object towards which it goes (for example, to distinguish whether the object is a wall, an animal or a person). Therefore, the present disclosure considers a codec is not just multi-layer in terms of data structure, but also in terms of objects in the video. Each object may have its own hierarchical structure. This is also useful as GPUs in racks may compress data before sending to one another, therefore this can be used for communication within training, (while you process you need to transfer this data - with the disclosed hierarchical codec you don’t need to transfer the entire file, you can selectively transfer only the LOQs that are needed), all within the gpu. This provides for training with data different locations, e.g. different server farms. Moreover, it is not a requirement to transfer all data at same time, e.g. some data will not be used at 4k.

[0297] We describe transferring a low LOQ representation of a data source fortraining and sending enhancement layers for further training after the training using the low LOQ has commenced.

[0298] We describe training (e.g. an LMM) using a ROI at a low LOQ for a first training task, and utilising a second ROI at a higher LOQ for a second task. The ROIs, at the different LOQs, are obtained from a single (hierarchically encoded) data source.

[0299] The described method may be performed on the gpu. The hierarchical structure and coding methods may comprise providing independently (en-Zde-)codable blocks within a layer, and / or independently decodable layers. This means that parallelisation is increased compared to other coding methods. This allows or GPUs to be used for coding.

[0300] • We describe training for different aspects (e.g. movement, semantics, object classification) by utilizing different LOQs / qualities. Further LOQs (level of qualities) can be used to re-train the model (e.g. the LMM) following training. For example, some aspects may need lower or higher quality of data to be trained for inference (e.g. movement training may not need as high quality of object recognition). By utilizing hierarchical data formats and codecs, it is possible to have a single data file that can serve each of these different training aspects - at different qualities

[0301] We further describe performing iterative training using a hierarchical structure (e.g., a video or a series of objects in a video, etc). This can speed up the learning but also can produce a more accurate predictive model. For example, if we train the network on the lowest layer first, and then once trained we re-train it on the layer immediately above to improve accuracy, we can get both a quicker model that can be improved over time, and a more precise one.

[0302] Not only, but for example, models could then be trained in a selective and bespoke way. For example, a model that is trained on a coarse set of data, and then only when specified it is trained on a more specific set of the same data. For example, a model designed to predict soccer / sport events: we describe training the model on a series of very low-res videos to make sure it is good enough to predict scores, etc. However, we may change it in practice to a model that predicts the actual players involved in a match. In that sense, we describe re-training the original model only focusing on the players, at a higher res. This can be done without having to re-train from scratch, but simply re-feeding the same videos at a different level or with objects at a different level.

[0303] Furthermore, we describe: • A proposed Solution for Scalability in LMM Training Using Hierarchical Coding Scheme that enhances the efficiency of storage and data transmission for video data during LMM training.

[0304] Implementing a hierarchical coding scheme, e.g. the 'SMPTE VC-6' standard, for video data encoding.

[0305] Referring to Figure 10, there is described a method of training a machine learning model, e.g. an LMM, using a plurality of levels of quality. This method may be performed by one or more computer devices being used to train a machine learning model.

[0306] In a first step 21 , the computer device identifies data in a hierarchical format. In particular, the computer device may identify (e.g. receive) a data source or data file that comprises a plurality of levels of quality (LOQs), where the data file may further comprise one or more enhancement layers. The computer device may then be able to extract a low LOQ version of the data from the data file and to combine this low LOQ version with an enhancement layer to generate a (relatively) high LOQ version of the data.

[0307] In a second step 22, the computer device feeds a low LOQ version of the data to a machine learning model, such as an LMM.

[0308] In a third step 23, the computer device determines an output of the machine learning model following application of the machine learning model to the low LOQ version of the data. For example, the output may comprise a classification of the data and / or of an object represented by the data. Equally, the output may comprise a modified version of the data. In particular, the machine learning model may be arranged to generate a predicted high LOQ version of the data based on the low LOQ version of the data.

[0309] In a fourth step 24, the computer device compares the output to a high LOQ version of the data (e.g. generated using an enhancement layer).

[0310] In a fifth step 25, the computer device trains the machine learning model based on a difference between the determined output and the high LOQ version of the data. For example, where the machine learning model is arranged to generate a predicted high LOQ version of the data, the difference may be a difference between the predicted high LOQ version and the actual high LOQ version (generated using the enhancement layer). Where the output is another type of output, such as an object classification, the method may comprise: determining a first output using the machine learning model and based on the low LOQ data; and determining a second output based on the high LOQ data (e.g. using the machine learning model and / or based on a user input). The machine learning model is then trained based on a difference between the first output and the second output (e.g. a parameter or a weighting of the machine learning model may be updated based on the difference).

[0311] This method of training a machine learning model enables the model to be trained so as to be useable on lower quality data. The training enables a single hierarchical data file to be used to perform this training offering substantial improvements in training efficiency as compared to conventional arrangements and reducing the amount of file transfer required to train the machine learning model.

[0312] Referring to Figure 11 , there is described a method of providing elements of different LOQs to a machine learning model. This method may be performed by one or more computer devices using a machine learning model and / or being used to train a machine learning model.

[0313] In a first step 31 , the computer device identifies data in a hierarchical format.

[0314] In a second step 32, the computer device determines a first element of the data with a first level of quality and, in a third step 33, the computer device determines a second element of the data with a second level of quality.

[0315] This typically comprises determining a first portion of the data (e.g. of an image or of a video) in a first LOQ and determining a second portion of the data in a second, higher, LOQ. This enables particularly important, of unclear, portions of the data to be determined in the higher LOQ. In a simple example, an image (or a video) may comprise a first portion that shows a wall or a door and a second portion that shows a dynamic scene with many small objects. The computer device may determine the first portion in a low LOQ, since it is simple to identify the wall or the door even in a low resolution image. The computer device may determine the second portion in a high LOQ (e.g. using an enhancement layer), since a higher resolution may be required to identify the images in this more complex portion of the scene.

[0316] In a fourth step 34, the first element and the second element are provided to the machine learning model. This may involve determining an output based on the first element and the second element, e.g. so as to classify the data or so as to generate a modified version of the data using the machine learning model.

[0317] Equally, the fourth step 34 may comprise providing the first element and the second element to the machine learning model in order to train the machine learning model.

[0318] In some embodiments, the output comprises predicted data, for example, the output may comprise a predicted frame of a video and / or a predicted part of a frame. For example, the output may comprise predicted data at the second level of quality, where this data is predicted based on the first element. Furthermore, the output may comprise predicted data at a third level of quality. The machine learning model may then be arranged to predict frames or portions of a video based on input data. In a specific use, the machine learning model may be arranged to more efficiently encode a video, where the machine learning model is used to predict (e.g. parts of) a second frame of the video based on a first frame ofthe video. This predicted frame may then be encoded alongside residual values that indicate any differences between the predicted frame and the (actual) second frame.

[0319] The determination of the first element and the second element may be based on an initial classification step. For example, the machine learning model may be used to classify one or more objects in a first, low LOQ, version of the data. The machine learning model may identify one or more elements in the low LOQ version of the data that are difficult to classify (e.g. the machine learning model may output a level of confidence for one or more elements in the data) and thereafter one or more of these elements may be determined in the second LOQ with these second-LOQ elements then being provided to the machine learning model.

[0320] This process may be repeated, where an input dataset for the machine learning model may be updated until each element within the input dataset can be classified with a threshold level of confidence. For example, the input dataset may include elements with a first, low, LOQ, a second, intermediate, LOQ, and a third, high, LOQ.

[0321] The determination of the first element and the second element may be based on an algorithmic or analytic approach. For example, a non-machine-learning object classification algorithm may be used to identify low complexity elements in the data and high complexity elements in the data, where only the high complexity elements are determined in the second LOQ.

[0322] As described above, this selective decoding of different elements in the data enables provides a more efficient method of providing data to a machine learning model that avoids unnecessary decompression of a data file.

[0323] In some embodiments, the method comprises selectively decoding a part of a data source at a single LOQ; for example, decoding only a single part of an image or a video. This selective decoding may provide benefits for training or inference even where only a single LOQ is used.

[0324] In some embodiments, providing the first element and the second element to the machine learning model comprises transmitting the first element and the second element to a further computer device. In particular, a first computer device may determine suitable LOQs for one or more elements in the data and transmit (to a further computer device) a compressed file that contains only these LOQs. This enables the efficient transfer of data between devices, e.g. GPUs. The suitable LOQs may, for example, be based on a complexity of the elements or a first-pass classification of the elements (as described above).

[0325] The first computer device may be arranged to determine the second LOQ and / or to determine a required enhancement layer for transmission. The first computer device may then determine a compressed file based on the second LOQ or the required enhancement layer. This may involve the first computer device determining the enhancement layer for the entirety of an image or a video so that the first computer device transmits a compressed file that enables the entirety of the image or video to be determined in the second LOQ. Equally, the first computer device may only determine (or compress, or transmit) a portion of an enhancement layer that relates to the second element. Therefore, the further computer device may be able to determine only the second element (and not the first element) in the second LOQ. In this way, the first computer device can avoid the transmission of unnecessary data.

[0326] In some embodiments, a first aspect of the machine learning model - e.g. a movement identifying module - may be trained based on a first LOQ and a second aspect of the machine learning model - e.g. an object classification module - may be trained on a second LOQ. By using a hierarchical format of data, these different aspects may be trained using the same data files.

[0327] In some embodiments, a machine learning model may initially be trained using data with a first LOQ before being fine-tuned and / or retrained using data with a second, e.g. higher, LOQ.

[0328] In this regard, the present disclosure considers an iterative training process in which a machine learning model is trained using a plurality of different LOQs of the same data, in particular where a model is trained using a low LOQ and is then fine-tuned using a relatively high LOQ. This provides a quick and efficient method of training an accurate model. This iterative process may comprise a plurality of, e.g. 2, 3, or 5, iterations that are performed on increasingly high LOQs.

[0329] Equally, the model may be re-trained using higher LOQs to alter a use case of the model and / or to expand the capabilities of the mode.

[0330] In some embodiments, the method includes training the machine learning models using a first set of elements with a first LOQ fine-tuning and / or the re-training involves fine tuning or re-training the model based on a second set of elements with a second LOQ. The first set and second set may comprise identical sets of elements, overlapping sets of elements, or separate sets of elements. Typically, the second set of elements is a subset of the first set of elements so that a model can be re-trained using a high LOQ for a set of elements that have not been accurately processed or classified (e.g. if the model incorrectly classifies a particular object in the data, then the elements associated with this object may be provided in a higher LOQ and used to re-train the model.

[0331] Referring to Figure 12, there is described a method of transmitting a plurality of different LOQs from a first computer device to a further computer device. This method may be performed by one or more computer devices using a machine learning model and / or being used to train a machine learning model.

[0332] In a first step 41 , the first computer device identifies data in a hierarchical format and in a second step 42, performed at a first time, the first computer device transmits a version of the data with a first LOQ to the further computer device.

[0333] Thereafter, in a third step 43 that is performed at a second time following the first time, the first computer device transmits a version of the data with a second LOQ to the further device.

[0334] In some embodiments, the third step 43 comprises transmitting one or more enhancement layers associated with the second LOQ. This enables the further computer device to generate the second LOQ by combining the first LOQ with the enhancement layers. The third step 43 may be performed in response to a request. For example, the further computer device may request an enhancement layer for one or more elements of the data (e.g. to obtain a portion of a video in higher quality). This may involve the further computer device performing a first-pass analysis of the data based on the first LOQ and the further computer device identifying, based on this first-pass analysis, one or more elements of the data for which a higher LOQ would be useful. The further computer device may then request this higher LOQ from the first computer device so as to trigger the transmission of the second LOQ.

[0335] Typically, the methods of Figures 9 - 12 are performed using one or more graphical processing units (GPUs). The methods may be performed using a plurality of GPUs, where different GPUs may perform operations on different LOQs and / or on different elements of the data. For example, referring to Figure 11 , a first GPU may determine and / or analyse the first element in the first LOQ and a second GPU may determine and / or analyse the second element in the second LOQ.

[0336] In some embodiments, the data includes video data. In these embodiments, the method may comprise providing a first frame of a video (or a set of elements of the first frame) in a first LOQ and providing a second frame of a video (or a set of elements of the second frame) in a second LOQ. The second LOQ may be determined based on the first LOQ, based on an output of the machine learning model, and / or based on a request from a further device. For example, the machine learning model may be used to classify the first frame and the second LOQ may then be determined based on a confidence of the classification (so that a quality of video can be increased if the model is not able to confidently classify the first frame). This process may occur in real time or near real time, where the first frame and the second frame may be adjacent frames but equally where the first frame and the second frame may be separated frames to account for processing time needed to apply the first frame to the machine learning model.

[0337] The machine learning model may comprise a plurality of different modules and / or may be arranged to perform a plurality of different functions (e.g. movement analysis, object classification, semantic analysis, etc.). The present disclosure considers training different modules or functions of a machine learning model using different LOQs (e.g. using elements of different LOQs or frames of different LOQs). The present disclosure also considers applying a machine learning model to data with different LOQs. For example, a first output may be obtained by applying a machine learning model to data with a first LOQ and a second output may be obtained by applying the machine learning model to (the same) data with a second LOQ. The first and the second output may be different types of outputs, for example the first output may comprise a movement classification and the second output may comprise an object classification. The first LOQ and the second LOQ are typically obtained using the same data from the same data file, e.g. the same hierarchically-encoded image file or video file.

[0338] The functions may include one or more of:

[0339] Semantic Analysis: Using the machine learning model to understand the meanings behind elements within the data, such as the context and thematic elements of text or scenes in videos.

[0340] Movement Analysis: Using the machine learning model to on understanding how objects move within data (e.g. within a video frame), which can be crucial for tasks like action recognition or tracking.

[0341] Object Classification: Using the machine learning model to recognize and categorize objects within data, which is fundamental for numerous Al applications like automated surveillance or retail analytics.

[0342] Referring to Figure 13, in use, the machine learning model is arranged to provide an output based on a given input. Specifically, referring to Figure 13, to use the machine learning model a computer device: in a first step 51 , identifies an input, in a second step 52, provides the input to the machine learning model, and in a third step 53, receives an output from the machine learning model.

[0343] The output may comprise, for example, a classification of the data, a classification of one or more objects in the data, an indication of a feature or a characteristic of the data (e.g. an indication of a movement of an object, and indication of a quality or authenticity of the data, etc.).

[0344] Typically, the machine learning model is an LMM and the input is a multi-modal input. As an example, the input may comprise a video and a question about the video, e.g. ‘what is in this video?’, ‘what do you think about this video?’, or ‘should I trust this video?’ In particular, the input may comprise one or more of image, video, or sensor data. Typically, a bottleneck for the use (or training) of a machine learning model is the provision of data to the machine learning model. The present disclosure envisages the use of hierarchical data formats in order to provide the input to the machine learning model. In this regard, identifying the input may involve a first computer device receiving input data from a second computer device and identifying the input based on this input data. For example, a user of a consumer PC may transmit a request to a company server that is hosting the machine learning model.

[0345] The present disclosure considers input data that comprises one or more hierarchical data files, one or more levels of quality, and / or one or more enhancement layers, as well as a computer device that hosts a machine learning model with the computer device being arranged to receive a hierarchical data file and to decode this data to obtain the input data.

[0346] In particular, the first computer device may be arranged to identify an input quality required by the machine learning model and to decode the data file to obtain an LOQ that corresponds to (e.g. is higher than) this required input quality. The first computer device may obtain the lowest available LOQ that provides an input with a quality that is greater than a required input quality.

[0347] Similarly, the second computer device may be arranged to determine the LOQ that corresponds to the required input quality (e.g. the required input quality may be received by the second computer device from the first computer device) and to encode data so as to provide this LOQ. For example, the second computer device may be arranged to identify an LOQ and / or a number of enhancement levels required to provide the required input quality and to encode a data file based on this LOQ or number of enhancement levels (e.g. to provide a data file that meets a quality threshold without providing unnecessary enhancement levels that would increase the size of this data file).

[0348] As described above in relation to Figure 12, the method of providing the data file may comprise transmitting a first level of quality at a first time. If this first level of quality does not meet a quality threshold, the first computer device may request from the second computer device that a second, higher, level of quality is transmitted to the first computer device.

[0349] The input data may be transmitted in real time or near real time (e.g. where the use of the hierarchical data enables the first device to decompress the input data and to provide as high an LOQ as is possible in order to maintain a certain throughput to the machine learning model given the resources of the first computing device).

[0350] In some embodiments, the input comprises one or more images or video frames and the output comprises a predicted image or video frame. In particular, the machine learning model may be arranged to predict a future video frame given one or more preceding video frames. The computer device may be arranged to further receive residuals that can be combined with the predicted video frame to obtain an actual video frame.

[0351] In this regard, both an encoder and a decoder may have access to the machine learning model so that the encoder is able to generate the predicted frame and to determine residuals based on this predicted frame and an actual frame. These residuals can then be transmitted to the decoder so that the residuals can be combined with the predicted frame at the decoder to obtain the actual video frame.

[0352] Similarly, the input may comprise an image or a video at a first LOQ (e.g. a first resolution) and the output may comprise a predicted image or video at a second LOQ. A similar arrangement of an encoder and decoder may be used in these embodiments, where the residuals provided to the decoder by the encoder indicate a difference between this predicted video / image and an actual video / image at the second LOQ. In this way, images / videos can be efficiently encoded / decoded using the machine learning model.

[0353] The present disclosure further envisages a method of generating a hierarchical data file for transmitting, the method being performed by a first compute device that is arranged to transmit the data file to a second computer device. The method may comprise: downsampling a source file at a first resolution, e.g. a 1080p source file, to create a base file at a second resolution, e.g. a 720p base file; encoding the base file using a base encoder; and then upsampling the base file to the resolution of the source file. The upsampler used to perform the upsampling may vary and may be specified in the transmission to the second computer device (alongside the data file). Typically, the upsampler uses a scale of 1 .5 (e.g. to upscale from 720p to 1080p.

[0354] Key Advantages of the present disclosure include:

[0355] Reduced Data Redundancy:

[0356] Unlike traditional methods that require multiple versions of the same video at different resolutions (e.g., 540p, 720p, 1080p), a hierarchical coding scheme allows for the storage of one encoded file. This file can then be decoded to the required resolution as needed, effectively reducing the amount of data to be stored and transmitted.

[0357] Efficiency in Data Usage:

[0358] By employing a method similar to 'autocomplete' for video data, the hierarchical coding scheme predicts and fills in the next Level of Quality (LOQ) based on the existing data stream. This predictive capability minimizes the need to process and store every frame at every quality level, thus optimizing the training data fed into the LMM.

[0359] Scalability and Resource Optimization:

[0360] This approach significantly cuts down the computational load and memory usage, as it circumvents the necessity of handling multiple discrete files for different resolutions. It ensures that the model training is more scalable and resource-efficient, catering effectively to the expansive data requirements of LMMs.

[0361] Thus, adopting a hierarchical coding scheme for video data in LMM training can provide a substantial reduction in storage requirements and data transmission overhead, making the training process more feasible and efficient.

[0362] • Data processing for artificial intelligence systems, particularly large multimodal models, focusing on improving data storage and transmission efficiency through hierarchical coding schemes.

[0363] Training large multimodal models requires handling extensive datasets, including high-resolution video files. Conventional approaches necessitate multiple files at different resolutions, which increases storage requirements and complicates data transmission, particularly across GPUs.

[0364] A method and system that utilize a hierarchical coding scheme, such as 'SMPTE VC-6', to encode and decode data reduces the need for multiple resolution-specific files by enabling a single encoded file that can be decoded to various Levels of Quality (LOQ) as required by the training process. Hierarchical Coding Implementation: The system encodes video data using a hierarchical coding scheme, which allows data to be stored in a compressed format that can be expanded to various resolutions based on specific training requirements, thus conserving storage space and reducing data redundancy.

[0365] Efficient Data Transmission: In multi-GPU environments, typically used for training large models, the invention facilitates efficient data communication by enabling the transmission of only the necessary LOQs between GPUs. This selective transmission prevents the need to transfer large amounts of redundant data, optimizing bandwidth usage and improving transmission speeds.

[0366] Adaptive Resolution Processing: The training system is configured to adaptively decode portions of the data file to the required resolution, analogous to an 'autocomplete' feature in text processing. This selective decoding adjusts dynamically to the model’s current training needs, enhancing computational efficiency.

[0367] Aspects of the present invention provide:

[0368] A method for optimizing data storage in the training of large multimodal models, comprising: encoding video data using a hierarchical coding scheme that allows dynamic resolution adjustments during model training.

[0369] A system for efficient data transmission in multimodal model training environments, characterized by: mechanisms for selectively transmitting encoded data at specific Levels of Quality, thus reducing the total volume of data transmitted between processors.

[0370] A method for adaptive resolution processing in large multimodal model training, involving: dynamically decoding only the necessary segments of a hierarchically compressed data file to match the resolution required by specific training tasks.

[0371] • Processing data (e.g. video) without having to decompress entire data / video. E.g. selectively decode a portion of a dataset, such as a video or an image. This is useful for training because the model can decode only the LOQ that it needs to train. This is also useful as GPUs in racks may compress data before sending to one another, therefore this can be used for communication within training. (While processing you need to transfer this data - with the disclosed hierarchical arrangement, a computer device does not need to transfer an entire file, it can instead selectively transfer only the LOQs that are needed by a recipient device / component), all within the GPU. The present disclosure describes training with data that is not in the same location, e.g. that is in different server farms. Using the disclosures herein, a computer device can perform a training progress without needing to transfer all data at same time, e.g. some data will not be used at 4k.

[0372] • A Proposed Solution for Efficient Data Handling in LMM Training Using Selective Decoding

[0373] Improved data processing efficiency by enabling selective decoding during the training of large multimodal models (LMMs).

[0374] We describe implementing a method that allows for processing data, such as video, without the need to decompress the entire file. This approach involves selectively decoding only the required Levels of Quality (LOQs).

[0375] Targeted Data Usage:

[0376] The model decodes only the specific LOQs necessary for its current training task. This selective decoding reduces computational overhead and speeds up the training process by avoiding unnecessary processing of high-resolution data when lower resolutions suffice.

[0377] Efficient Data Communication:

[0378] Within GPU racks, data often needs to be compressed before being sent to another GPU for processing. By using hierarchical encoding, only the necessary LOQs are transferred between GPUs, not the entire file. This selective transfer significantly reduces the volume of data being moved, which is crucial for maintaining high throughput and efficiency in GPU-intensive tasks.

[0379] Optimized Resource Utilization:

[0380] This approach maximizes the use of GPU resources by ensuring that only relevant data is decompressed and processed. By minimizing data transfer and decoding only what is necessary, the overall system's efficiency is enhanced, allowing for faster training cycles and lower energy consumption.

[0381] Selective decoding in the training of LMMs offers substantial improvements in the efficiency of data handling. By enabling the model to process only the necessary data, and facilitating efficient communication between GPUs, this strategy ensures more effective use of computational resources, paving the way for more scalable and agile machine learning operations.

[0382] The enhancement layers disclosed herein are typically determined, encoded, and decoded using parallelizable operations, such as convolutions and interpolations. Therefore, the hierarchical data format considered herein is particularly suitable for encoding / decoding using GPUs (in contrast to the conventional, codecs, e.g. AVC or HEVC, that may be used to encode / decode the base layer). Providing further examples, tasks like upsampling, residual addition, and filtering, that may be used to encode / decode the hierarchical data file are highly parallel and naturally map to GPU shader cores or compute kernels (e.g., CUDA or OpenCL).

[0383] Typically, the hierarchical data file and / or the enhancement layers is encoded without using intra prediction between blocks of a frame, where this enables the parallel decoding of data, e.g. frames, of the hierarchical data file. Furthermore, typically the hierarchical data file and / or the enhancement layers is encoded without, e.g. bitstream parsing, CABAC, variable-length decoding, etc., which are inherently serial and branchheavy and are hard to parallelize.

[0384] Typically, the hierarchical data file and / or the enhancement layers are decoded without using motion compensation.

[0385] In some embodiments, the hierarchical data file and / or the enhancement layers uses intra-only, blockless, hierarchical transforms for encoding frames of the data.

[0386] Typically, the hierarchical data format is encoded using an s-tree representation (e.g. as with VC-6). More generally, the hierarchical data format may be encoded using a spatial hierarchy that can be split across threads and / or GPU cores.

[0387] In specific embodiments, LCEVC and / or VC-6 based-codecs are used to provide these benefits of parallelization. The method of encoding / decoding / generating data may then comprise using a plurality of GPUs (or TPUs) to encode / decode / generate the data using parallel processing processes.

[0388] Various aspects of the present disclosure relate to:

[0389] A method for training large multimodal models, comprising: selectively decoding only required Levels of Quality (LOQs) from a hierarchically compressed data file based on the training needs of the model.

[0390] A system for efficient data handling in large multimodal model training, characterized by: a mechanism to selectively transfer only necessary LOQs between processing units, thereby reducing the bandwidth and storage requirements during model training.

[0391] A method for optimizing processor (e.g. GPU, Tensor Processing unit, and / or CPU) resource utilization during the training of large multimodal models, involving: processing only selectively decoded data segments, thereby minimizing unnecessary data processing and enhancing overall training efficiency.

[0392] • Flexible hardware - no need for specific decoding / encoding hardware. We describe data processing hardware, specifically to systems and methods that enable flexible video encoding and decoding directly within GPU environments, without the need for specialized hardware. Traditionally, video encoding and decoding for training large multimodal models require specialized hardware that supports specific codecs like HEVC, VVC, etc. This specialized hardware dependency limits the flexibility and scalability of training systems, increasing costs and complexity.

[0393] We describe a method and system that enables hierarchical encoding and decoding directly within GPUs, using standard hardware capabilities without reliance on specialized codec hardware. This approach allows for greater flexibility in training large multimodal models, reducing dependency on specific hardware and enhancing the adaptability of training systems.

[0394] Flexible GPU Implementation:

[0395] • The system is designed to leverage the existing computational capabilities of GPUs to perform video encoding and decoding tasks. This method utilizes the general-purpose processing power of GPUs to handle various codecs, including but not limited to SMPTE VC-6 and LCEVC.

[0396] Hardware Efficiency:

[0397] • By eliminating the need for specialized decoding / encoding hardware, the invention significantly reduces hardware overhead. This not only lowers the cost and complexity of the training setup but also enhances the efficiency of the training process by simplifying hardware requirements.

[0398] Scalability and Adaptability:

[0399] • The method allows for easy scalability within diverse training environments as it can adapt to different codecs and processing needs without additional hardware. This adaptability is crucial for organizations scaling their Al capabilities across varied hardware configurations.

[0400] An aspect of the present disclosure provides a method for encoding and decoding video data within a GPU, comprising utilizing the general-purpose processing capabilities of GPUs to perform encoding and decoding tasks traditionally handled by specialized hardware.

[0401] An aspect of the present disclosure provides a system for flexible video processing in multimodal model training, characterized by a GPU-configured to handle multiple video codecs without the need for additional specialized codec hardware.

[0402] An aspect of the present disclosure provides a method for enhancing hardware efficiency in Al model training, involving reducing hardware dependency by enabling standard GPUs to perform complex video encoding and decoding tasks.

[0403] Described examples revolutionize the hardware approach for training large multimodal models by leveraging the inherent capabilities of standard GPUs for video processing tasks. This eliminates the need for specific codec hardware, offering a flexible, cost-effective, and scalable solution that can adapt to various training needs and environments.

[0404] Training aspects of a model:

[0405] • Training is made of different steps - semantic, how objects move, etc. Training different aspects of a model (e.g. movement, semantics, object classification) at different qualities could be useful & possible with the disclosures herein. Can also retrain at different LOQs.

[0406] • With known solutions, users are forced to have all the parts of the training or the inference at the same level of inference. However, some aspects may need lower or higher quality of data to be trained or for inference (e.g. movement training may not need as high quality of object recognition). With a hierarchical codec, it is possible to have a single data file that can serve each of these different training aspects - at different qualities. We describe a Method and System for Multi-Aspect Training of Large Multimodal Models Using Hierarchical Encoding. We describe artificial intelligence training methodologies, particularly a system and method for training different aspects of large multimodal models using a single data file that can be decoded at various levels of quality.

[0407] In the training of artificial intelligence models, particularly large multimodal models, it is often desirable to process and interpret complex data such as video content, which involves understanding various attributes like movement, semantics, and object classification. Traditional training methods require the data for all these aspects to be processed at the same level of quality, which is not always efficient or necessary, leading to increased computational load and slower training processes.

[0408] Our described examples propose a novel training methodology that utilizes hierarchical encoding to allow different aspects of model training to be conducted at optimally suited levels of quality. This method enhances the efficiency and flexibility of training processes by allowing selective quality application tailored to the training needs of each aspect.

[0409] Training Aspects Defined:

[0410] Semantic Analysis: Involves understanding the meanings behind elements within the data, such as the context and thematic elements of text or scenes in videos.

[0411] Movement Analysis: Focuses on understanding how objects move within the video frame, which can be crucial for tasks like action recognition or tracking.

[0412] Object Classification: Entails recognizing and categorizing objects within data, which is fundamental for numerous Al applications like automated surveillance or retail analytics.

[0413] Hierarchical Encoding Application: the system encodes data using a hierarchical codec that allows it to be stored in a single file but decoded at different levels of quality (LOQs). Each training aspect can access the quality level most suited to its needs, enhancing training effectiveness and reducing unnecessary data processing.

[0414] Adaptive Quality Training: Training modules are designed to dynamically select the LOQ based on the specific requirements of the task at hand. For instance, movement analysis might not require high- resolution data as much as object classification does. This selective quality usage optimizes computational resources and accelerates the training process.

[0415] Retraining Capabilities: The system also supports retraining different aspects of the model at different LOQs as needed, allowing continuous improvement of the model based on adaptive quality feedback and performance assessment.

[0416] We describe a method for training large multimodal models, comprising: training various aspects of the model (e.g. semantic analysis, movement analysis, and object classification) at different levels of quality optimized for each aspect using a single hierarchical-encoded data file.

[0417] We describe a system for efficient data processing in Al model training, characterized by: a hierarchical codec that allows selective decoding of data at varying qualities necessary for specific training aspects.

[0418] We describe a method for dynamic retraining of Al models, involving: adjusting the quality of data used for training specific aspects based on performance feedback and resource availability.

[0419] These described examples significantly improve the flexibility and efficiency of training large multimodal models by allowing different aspects of the model to be trained or retrained at various levels of data quality. They optimize resource use and enhances the speed and adaptability of training processes, making it highly beneficial for advanced Al development environments. We describe using hierarchically encoded data to train Large Multimodal Models (LMMs). Hierarchical encoding involves structuring data in layers or levels that capture different granularities or aspects of the information. This approach can be particularly advantageous in multimodal contexts where different types of data need to be integrated effectively. Here are some potential benefits and the mechanisms through which they are achieved:

[0420] Benefits of Hierarchical Encoding in LMMs include:

[0421] 1. Improved Data Representation:

[0422] • Hierarchical encoding allows for more nuanced representations of data by capturing both low-level features (such as pixel intensity in images or phonemes in speech) and high-level features (like objects in images or semantics in text). This can lead to a more comprehensive understanding of the data across modalities.

[0423] • Mechanism: By training models on these layered representations, LMMs can learn to recognize patterns at multiple scales and abstract higher-level concepts from more granular details.

[0424] 2. Enhanced Model Generalization:

[0425] • Models trained on hierarchically encoded data can generalize better to new, unseen data. This is because they not only learn specific features but also how these features are structured and related in complex data.

[0426] • Mechanism: Hierarchical representations help the model to abstract commonalities across different instances of data, reducing overfitting and improving the model's ability to adapt to new scenarios without losing accuracy.

[0427] 3. Increased Efficiency in Learning:

[0428] • Hierarchical encoding can help reduce the dimensionality of the input data by focusing on relevant features at different levels, potentially speeding up the learning process.

[0429] • Mechanism: By focusing computational resources on significant features at each level of the hierarchy, the model can efficiently learn important patterns without being bogged down by less informative data.

[0430] 4. Better Handling of Complexity in Data:

[0431] • In multimodal data, different modalities may naturally align into hierarchical structures (e.g., frames in a video to scenes, words to sentences in text). Hierarchical encoding leverages this structure to better synchronize and align data from these modalities.

[0432] • Mechanism: By mapping each modality's hierarchical features onto a common framework, the model can more effectively integrate data, leading to improved performance on tasks that require synthesis of information across modalities.

[0433] 5. Improved Interpretability:

[0434] • Hierarchical models can offer clearer insights into how decisions are made, as each layer of the hierarchy can be designed to correspond to recognizable and interpretable features or concepts.

[0435] • Mechanism: Transparency is enhanced when each level of the model's hierarchy is understandable, allowing easier identification of how inputs are transformed into outputs.

[0436] An exemplary embodiment we describe an LMM designed for a purpose (e.g. medical diagnosis using patient data, including clinical notes, radiographs, and test results). By hierarchically encoding the data — clinical notes analyzed from words to sentences to patient history, radiographs processed from edges to textures to diagnostic features, and test results categorized from raw data to trends over time — the LMM can integrate these varied data sources more effectively. It could then provide a diagnosis by considering both the micro-level details (specific test results) and the macro-level context (overall patient health trends).

[0437] Thus, using hierarchically encoded data to train LMMs can significantly enhance their effectiveness across a range of dimensions, including learning efficiency, model generalization, and data integration, making it a valuable approach in the development of advanced multimodal systems.

[0438] Using hierarchically encoded data during the inference phase of Large Multimodal Models (LMMs) can provide several significant benefits. Hierarchical encoding, even during inference, allows the model to leverage structured representations that have been learned during training. This can enhance the model's ability to interpret new, unseen data, among other advantages. Here’s how these benefits manifest:

[0439] We describe sending a prompt in a hierarchical codec (where there are layers which give extra, layers are entropy coded). This can be used to compress info other than just video.

[0440] We describe multiple solutions / aspects to previously described problems:

[0441] • If you generate a hierarchical image, you are more efficient with the prompt because instead of sending a maximum / single quality video (or data), you can send an exact quality that matches with the models desired / train quality - whilst still only storing a single copy of the prompt.

[0442] I. Single layer videos are usually downsampled to the quality that the model uses, so it is not always beneficial to send the maximum available quality prompt to the model because the model will just downsample it to meet the model's input requirements.

[0443] • We describe sending a low resolution prompt first and follow up with high layers, if you use hierarchical codec, to decrease response time.

[0444] • Hierarchical codecs have been found to improve compression efficiency, so even without other features, advantages are found.

[0445] We describe a multi-layer software (de)coding, which means the prompt can be decoded using GPUs (regardless of how many different modes of data is contains) rather than needing a specialized hardware decoders for each type of mode.

[0446] We describe sending a prompt in a hierarchical codec (where there are layers which give extra information, and the layers are entropy coded). These can be used to compress info other than just video.

[0447] We describe a Method and System for Progressive Data Transmission in Large Multimodal Models Using Hierarchical Encoding.

[0448] We describe data processing, specifically to a method and system for improving data transmission efficiency during the operation of large multimodal models by using hierarchical encoding to enable progressive data transmission. Traditional data transmission methods for artificial intelligence systems typically require sending data at a single, fixed quality, which can lead to delays, particularly in bandwidthlimited scenarios.

[0449] We describe providing a progressive data transmission method that utilizes hierarchical encoding to initially send a low-resolution prompt and subsequently transmit higher resolution layers as needed. This approach significantly decreases initial response times and enhances overall system responsiveness.

[0450] We describe a Progressive Transmission Strategy. This may comprise utilizing a hierarchical codec, the system first sends a low-resolution version of data, ensuring rapid delivery and quick initial processing. This is followed by higher resolution layers based on the specific requirements of the task or the user's request, optimizing the balance between speed and quality. This provides for efficient resource utilization. This method allows for better management of bandwidth and computational resources, as the higher quality data layers are only transmitted when necessary, thereby reducing unnecessary data load.

[0451] We describe a method for optimizing data transmission in artificial intelligence systems, comprising: progressively transmitting data starting with a lower resolution and increasing resolution based on subsequent requirements, using a hierarchical encoding scheme.

[0452] We describe a system for efficient progressive data handling in large multimodal model operations, characterized by: a mechanism to initially send low-resolution data to decrease response times, followed by higher resolution data as needed.

[0453] We describe a Method and System for Enhanced Compression Efficiency in Large Multimodal Models Using Hierarchical Codecs

[0454] We describe data compression technologies, particularly improving compression efficiency in the processing and storage of data for large multimodal models using hierarchical codecs. High data compression efficiency is crucial for reducing storage and transmission costs in large-scale artificial intelligence applications. Traditional codecs often do not optimize for the specific needs of multimodal data, which can include varying types of media such as text, images, and video.

[0455] We describe employing hierarchical codecs, which have been found to improve compression efficiency significantly. This system enhances the capability to store and transmit multimodal data more effectively, even without the incorporation of other advanced features.

[0456] This provides for Improved Compression Mechanism: By implementing hierarchical codecs, the system achieves higher compression rates, which reduces the storage space required and minimizes the bandwidth needed for data transmission.

[0457] Application in Multimodal Systems: The enhanced compression is particularly beneficial in environments involving large multimodal models, where diverse data types are common and efficient handling is critical.

[0458] We describe a method for enhancing data compression in multimodal systems, comprising: utilizing hierarchical codecs to achieve improved compression rates for diverse data types.

[0459] We describe a system for reduced data storage and transmission costs in artificial intelligence applications, characterized by: a use of hierarchical encoding techniques to enhance overall compression efficiency.

[0460] We describe a System and Method for Efficient Data Handling and Inference in Large Multimodal Models Using Hierarchical Encoding

[0461] We describe artificial intelligence systems, particularly the use of hierarchical (en-Zde-)coding techniques during the inference phase of large multimodal models (LMMs) to optimize data handling and model efficiency. In the inference phase of LMMs, data is often required at varying levels of quality depending on the specific task or model capability. Traditionally, this necessitates the storage of multiple copies of the same data at different qualities or sending data at a single, often unnecessarily high quality, leading to inefficiencies in storage and data transmission.

[0462] We describe a method and system that utilizes hierarchical encoding, e.g. using the SMPTE VC-6 standard, to store and transmit data in a format that allows multiple quality levels from a single encoded source. This capability significantly enhances the efficiency of the inference process by enabling the delivery of data at the precise quality level needed by the model, while maintaining a single copy of the data.

[0463] Hierarchical Encoding for Inference: The system encodes data using a hierarchical method that allows the data to be stored in a compressed format. This format is capable of being decoded at multiple quality levels as required during inference, ensuring optimal data utilization. Efficient Data Transmission: During inference, data is transmitted at the quality level that matches the model’s trained or desired quality, rather than at a maximum or single quality level. This selective quality transmission ensures that data handling is more efficient and reduces bandwidth usage.

[0464] Storage Optimization: By storing only a single copy of the data in a hierarchical encoded format, the system minimizes storage requirements while maximizing the flexibility to access data at various quality levels. This approach is particularly beneficial in environments where storage space and data retrieval speed are critical.

[0465] We describe a method for optimizing data usage during the inference phase of large multimodal models, comprising: storing and transmitting data in a hierarchical encoded format that can be selectively decoded at desired quality levels to match the model's requirements.

[0466] We describe a system for efficient data handling in artificial intelligence inference operations, characterized by: the capability to transmit data at various quality levels from a single encoded source, tailored to the specific needs of the inference task.

[0467] We describe a method for storage optimization in large multimodal model inference, involving: utilizing hierarchical encoding to maintain a single copy of data that can be accessed at multiple quality levels as demanded by different inference tasks.

[0468] Embodiments provide a significant advancement in the field of artificial intelligence by enhancing the efficiency and effectiveness of data handling during the inference phase of LMMs. By allowing precise quality matching and reducing the overhead associated with traditional data storage and transmission methods, the system facilitates faster and more resource-efficient model operations.

[0469] This structured approach highlights the inventive steps and potential claims that focus on optimizing inference operations in LMMs through the use of advanced hierarchical encoding techniques. It ensures that the description is clear about the benefits and technical advancements provided by the proposed system

[0470] We describe a system and Method for Multimodal Software-Based Decoding Using GPUs.

[0471] We describe data processing technologies, specifically to a system and method for flexible, multimodal software-based decoding using general-purpose GPUs, eliminating the need for specialized hardware decoders.

[0472] Traditional approaches to handling diverse data modalities such as text, speech, and video typically require specialized hardware decoders for each type of data. This requirement increases the complexity and cost of the data processing systems and limits their flexibility and scalability.

[0473] We describe a flexible, multi-layer software coding and decoding approach that leverages the computational power and versatility of general-purpose GPUs (and or CPUs and or Tensor Processing units, TPUs). This method enables the decoding of various data modalities using a single, unified system architecture, which simplifies the hardware setup and enhances system adaptability.

[0474] Flexible Software-Based Decoding: The system utilizes a software layer designed to operate on general- purpose GPUs (and or CPUs and or Tensor Processing units, TPUs), capable of decoding multiple types of data modalities, including text, speech, and video. This software layer adapts dynamically to the data being processed, utilizing the inherent parallel processing capabilities of GPUs.

[0475] Elimination of Specialized Hardware: By deploying the multimodal decoding process on GPUs, the need for specialized hardware decoders for each data type is eliminated. This not only reduces hardware costs but also streamlines the data processing pipeline, making it more efficient and less prone to compatibility issues. Scalability and Versatility: The software-based approach allows for easy updates and scalability. As new data modalities or improved decoding algorithms become available, the system can be updated with minimal disruption, ensuring it remains at the cutting edge of technology without requiring hardware modifications.

[0476] Hierarchical coding is more amendable to software implementations than single layer coding (e.g. HEVC, VVC, AV1). Thus, utilising a multi layer approach allows for software based coding.

[0477] A method for decoding diverse data modalities in a data processing system, comprising: using a unified software layer on general-purpose GPUs (and or CPUs and or Tensor Processing units, TPUs) to decode multiple types of data modalities without the need for specialized hardware decoders.

[0478] A system for flexible multimodal decoding, characterized by: a general-purpose GPU (and or CPUs and or Tensor Processing units, TPUs) implementing a multimodal decoding software that dynamically adapts to various types of data, including text, speech, and video.

[0479] A method for enhancing the adaptability of a data processing system, involving: deploying a software- based decoding approach on GPUs (and or CPUs and or Tensor Processing units, TPUs) that can be easily updated to incorporate new decoding capabilities or data modalities as they become available.

[0480] Examples provide a significant advancement in the processing of multimodal data by leveraging general- purpose GPUs (and or CPUs and or Tensor Processing units, TPUs) for flexible, software-based decoding. It offers a scalable and cost-effective solution that enhances the adaptability and efficiency of data processing systems, suitable for a wide range of applications from Al to multimedia content analysis.

[0481] The described example provide for further advantages:

[0482] Enhanced Robustness to Variability in Data:

[0483] • Hierarchically encoded data during inference allows the model to better handle variations in input data by focusing on relevant features at different levels. This is particularly beneficial when dealing with real-world data, which can vary widely from the conditions seen during training.

[0484] • Mechanism: The model can dynamically adjust which level of the hierarchy to focus on based on the quality and type of input data. For instance, if high-level features are noisy or ambiguous, the model can rely on more granular features to make decisions.

[0485] 2. Improved Accuracy and Precision:

[0486] • By processing data at multiple hierarchical levels, the model can make more informed and precise decisions. This is because it considers both detailed and contextual information, allowing for a more nuanced understanding of complex situations.

[0487] • Mechanism: For example, in a medical imaging LMM, hierarchical encoding allows the model to simultaneously consider fine details visible in a specific image section (like a tumor's texture) and broader patterns across a sequence of images (like the growth rate of the tumor).

[0488] 3. Faster Processing and Reduced Computational Load:

[0489] • Hierarchical encoding can also lead to faster inference times by enabling the model to bypass unnecessary computations for certain tasks. If higher-level features provide sufficient information for decision-making, lower levels can be skipped.

[0490] • Mechanism: This selective processing reduces the computational burden, especially valuable in real-time applications where decision speed is critical.

[0491] 4. Scalability to Complex Tasks: • Hierarchically encoded data supports models in tackling more complex and layered tasks by structuring the inference process. This structured approach can be crucial when dealing with multifaceted problems that require synthesizing information across various domains and scales.

[0492] • Mechanism: In an autonomous driving LMM, hierarchical data encoding during inference helps integrate moment-to-moment sensor data (low-level) with broader navigational objectives (high-level), such as route planning and traffic pattern analysis.

[0493] 5. Adaptability to Different Contexts or Domains:

[0494] • Hierarchical encoding enhances the adaptability of LMMs to different contexts or domains without retraining. The model can adjust its focus across different levels of the hierarchy based on the domainspecific characteristics of the input data.

[0495] • Mechanism: For instance, an LMM used for both customer service and technical support might shift its focus within the hierarchy based on whether the query is about service feedback (requiring emotion recognition at a higher level) or technical details (focusing on specific product features at a lower level).

[0496] Example Scenario: Video Content Analysis

[0497] Consider an LMM designed for analyzing video content to identify context-specific advertising opportunities. During inference, the model processes hierarchical data encoded from video frames (low-level features like colors and motion), scenes (medium-level features like setting and activity), and entire episodes (high-level features like theme and narrative structure). This hierarchical approach allows the model to identify suitable moments for placing relevant ads based on both the immediate content and the broader context of the video, thereby enhancing the effectiveness and relevance of the advertising.

[0498] In conclusion, using hierarchically encoded data during the inference phase can substantially improve an LMM's performance across various dimensions. This approach not only makes the model more efficient and adaptable but also increases its capability to deal with complex, real-world datasets in a robust and scalable manner.

[0499] The above describe methods and systems may be used for various use cases, e.g. these methods and systems may be used in conjunction with the described use cases:

[0500] Healthcare: The LMMs are used in medical diagnosis by analyzing patient data that includes text (clinical notes), images (X-rays, MRIs), and structured data (lab results). They help in predicting diseases, personalizing treatment plans, and even in surgical robotics where they process visual, textual, and sensor data.

[0501] Autonomous Vehicles: In the automotive industry, LMMs process inputs from cameras, LiDAR, and radar to understand the vehicle's surroundings, aiding in navigation and decision-making for autonomous driving.

[0502] Customer Support: Multimodal Al can enhance customer service by interpreting requests that may come in the form of text, voice, or video, enabling more effective and responsive support systems that better understand user sentiments and needs.

[0503] E-Commerce: By analyzing customer reviews that include text, images, and ratings, LMMs provide insights into consumer preferences and product performance, enhancing recommendation systems and targeted marketing.

[0504] Education: In educational technologies, LMMs can analyze students’ written work, spoken responses, and even emotional expressions to provide personalized feedback and adaptive learning experiences.

[0505] Augmented and Virtual Reality: LMMs could be central to creating more immersive AR and VR experiences by integrating real-time environmental data with user-generated content like gestures and voice commands, thereby enhancing interaction within virtual environments. Smart Cities: In smart city applications, LMMs could integrate data from loT devices, traffic cameras, and environmental sensors to manage urban services such as traffic control, public safety, and resource management more efficiently.

[0506] Environmental Monitoring: LMMs analyze satellite imagery, sensor data from oceans or forests, and textual weather reports to monitor climate change, predict natural disasters, and manage natural resources more effectively.

[0507] Entertainment and Media: In the future, LMMs could be used to create dynamic media content that adapts to user interactions across different formats, such as combining video, text, and interactive elements for personalized storytelling or gaming.

[0508] Legal and Compliance: LMMs might be employed to analyze legal documents, client communications, and relevant case law to assist in legal research, compliance monitoring, and case preparation, providing a comprehensive view by integrating diverse data types.

[0509] In particular, we describe:

[0510] A Healthcare Diagnosis System: An LMM configured as a healthcare diagnostic tool processes patient data, including clinical notes (Text Processing Unit), X-ray images (Image Processing Unit), and recorded patient interviews (Audio Processing Unit). The Data Fusion Layer integrates these inputs, which are then analyzed to predict health conditions or suggest treatments.

[0511] An Autonomous Vehicle Navigation System: In this application, the LMM integrates data from vehicle cameras (Image Processing Unit), radar (Sensor Data Processing Unit), and audio commands from the driver (Audio Processing Unit). The Contextual Analysis Module interprets this data to make real-time driving decisions.

[0512] A Customer Support Chatbot: This system uses the Text Processing Unit to understand customer queries and the Contextual Analysis Module to interpret the emotional tone via text and audio inputs. The Output Generation Unit crafts and delivers appropriate responses in text or audio format.

[0513] Alternatives and modifications

[0514] It will be understood that the present invention has been described above purely by way of example, and modifications of detail can be made within the scope of the invention.

[0515] For example, it will be appreciated that the methods disclosed herein may be used with various machine learning techniques. For example, the method may be used with supervised or unsupervised training techniques. The methods may be used with various types of machine learning models. The methods may be used with, e.g. convolutional neural networks (CNNs), deep neural networks, long short term models (LSTMs), etc.

[0516] Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

Claims

Claims1 . A method of providing data to a machine learning model using a hierarchical data format, the method comprising: identifying a data source with a hierarchical format; determining first data from the data source, the first data having a first level of quality; determining second data from the data source, the second data having a second level of quality; and providing the first data and the second data to the machine learning model.

2. The method of claim 1 , comprising: training the machine learning model using the first data; and re-training and / or fine tuning the machine learning model using the second data.

3. The method of claim 2, wherein: the second level of quality is higher than the first level of quality and / or the method comprises iteratively training the machine learning model using data of increasing data with levels of quality.

4. The method of any preceding claim, comprising: providing the first data to the machine learning model; receiving an output from the machine learning model; comparing the output to the second data; and training the machine learning model based on the comparison of the output and the data; preferably, wherein the output comprises generated data, the generated data being generated by the machine learning model based on the first data and the generated data having the second level of quality.

5. The method of claim 4, comprising: determining a further output based on the second data, the output and the further output being the same type of output (e.g. the output and the further output relating, respectively, to a characteristic of the first data and the second data); comparing the output and the further output; and training the machine learning model based on the comparison of the output and the further output.

6. The method of any preceding claim, wherein determining the second data comprises combining the first data with one or more enhancement layers, preferably wherein the method comprises extracting the one or more enhancement layers from the data source.

7. The method of any preceding claim, wherein the first data comprises a first version of an image and / or video and the second data comprises a second version of the same image and / or video.

8. The method of any preceding claim, wherein the first data is associated with a first element of the data source and the second data is associated with a second element of the data source; preferably, wherein the first element and the second element are different elements of the data source, e.g. different portions or sections of an image and / or a video.

9. The method of any preceding claim, wherein determining the first data comprises determining the first data at a first time, and determining the second data comprises determining the second data at a second time.

10. The method of claim 9, being performed by a first computer device, wherein the method comprises receiving the first data from a second computer device at a first time.11 . The method of claim 10, comprising receiving the second data and / or one or more enhancement layers from a second computer at a second time, preferably wherein the second time is substantially after the first time (e.g. at least 1 second, at least 10 seconds, and / or at least one minute after the first time).

12. The method of claim 11 , comprising: receiving the first data at the first time; transmitting a request for the second data and / or the enhancement layers; and receiving the second data at the second time.

13. The method of claim 12, comprising transmitting the request for the second data in dependence on a feature of the first data and / or in dependence on an output from the machine learning model.

14. The method of claim 13, comprising: providing the first data to the machine learning model; receiving an output from the machine learning model; and transmitting the request in dependence on the output, preferably based on a confidence level associated with the output being below a threshold value.

15. The method of any of claims 10 to 14, comprising determining, at the second computer device, the second level of quality, preferably comprising determining the second level of quality in dependence on an input quality required by the machine learning model.

16. The method of any preceding claim, comprising determining the second data in dependence on an output of the machine learning model, preferably comprising: providing the first data to the machine learning model; receiving an output from the machine learning model; and determining the second data in dependence on the output, preferably based on a confidence level associated with the output being below a threshold value.

17. The method of any preceding claim, comprising determining the second level of quality in dependence on a feature of the data source.

18. The method of any preceding claim, comprising determining the second level of quality in dependence on an input quality required by the machine learning model; preferably comprising determining the second level of quality as the lowest possible level of quality that meets a required input quality threshold.

19. A method of providing data to a machine learning model using a hierarchical data format, the method comprising: identifying a data source with a hierarchical format, the data source encoding first data having a first level of quality (e.g. a first resolution) and second data having a second level of quality (e.g. a second resolution); and providing the first data and the second data to the machine learning model by transmitting the data source to a further computer device; the method comprising determining the second level of quality as a lowest possible level of quality that meets a required input threshold.

20. The method of claim 18 or 19, comprising selecting the second level of quality from a plurality of possible levels of quality.21 . The method of any preceding claim, wherein the method is performed by a sending computer device and wherein providing the first data and the second data to the machine learning model comprises transmitting the first data and the second data to a receiving computer device on which the machine learning model is implemented; preferably, comprising encoding the first data and the second data prior to transmitting the data to the receiving computer device, preferably encoding the first data and the second data in a hierarchical coding scheme, more preferably encoding the first data and the second data using a hierarchical coding scheme that allows dynamic resolution adjustments during model training.

22. The method of claim 21 , comprising selectively transfer only necessary LOQs of the encoded data between the sending computer device and the receiving computing device.

23. The method of claim 21 or 22, comprising progressively transmitting data starting with a lower resolution and increasing resolution based on subsequent requirements, using a hierarchical encoding scheme.

24. The method of any of claims 21 to 23, comprising initially sending low-resolution data to decrease response times, followed by higher resolution data as needed.

25. The method of any of claims 21 to 24, comprising transmitting the first data at a first time and transmitting the second data at a second time, preferably comprising transmitting the second data and / or an enhancement layer relating to the second data in response to a request from the receiving device.

26. The method of any preceding claim, wherein determining the first data and the second data comprises selectively decoding only required Levels of Quality, LOQs, from a hierarchically compressed data file, preferably comprising determining the required LOQs based on the machine learning model, preferably based on the training needs of the machine learning model.

27. The method of any preceding claim, comprising a method for optimizing processor (e.g. GPU, Tensor Processing unit, and / or CPU) resource utilization during the training of large multimodal models; preferably, comprising processing only selectively decoded data segments of the data source.

28. The method of any preceding claim, wherein determining the first data and the second data comprises dynamically decoding only the necessary segments of the data source to match a resolution required by the machine learning model, preferably to match a resolution required by one or more specific training tasks of the machine learning model.

29. The method of any preceding claim, wherein the data source comprises a first set of one or more elements that can be determined at the first level of quality and a second set one or more elements that can be determined at the second level of quality, preferably wherein the second set is a subset of the first set.

30. The method of any preceding claim, comprising: determining the first data using a first device, preferably a first GPU; and determining the second data using a second device, preferably a second GPU.31 . The method of any preceding claim, comprising training a plurality of different aspects and / or functionalities of the machine learning model using data at different levels of quality; preferably, comprising: training a first aspect of the machine learning model using the first data; and training a second aspect of the machine learning model using the second data.

32. The method of claim 31 , wherein the aspects of the machine learning model comprise one or more of: a movement analysis aspect, a semantic analysis aspect, and an object classification aspect.

33. The method of any preceding claim, comprising determining the second level of quality based on an aspect of the machine learning model being used to process the second data and / or based on an aspect (or a functionality) of the machine learning model being trained using the second data.

34. The method of any preceding claim, comprising training a plurality of, e.g. various, aspects of the model (preferably one or more of: a semantic analysis aspect; a movement analysis aspect; and an object classification aspect) at different levels of quality optimized for each aspect using a single hierarchical- encoded data file; preferably, comprising adjusting the quality of data used for training specific aspects based on performance feedback and resource availability.

35. The method of any preceding claim, wherein the data source is encoded using a hierarchical codec that allows selective decoding of data at varying qualities necessary for specific training aspects.

36. The method of any preceding claim, further comprising receiving an output from the machine learning model, preferably wherein the method comprises a method of inference using the machine learning model.

37. The method of any preceding claim, comprising using a unified software layer on general-purpose GPUs (and or CPUs and or Tensor Processing units, TPUs) to decode multiple types of data modalities without the need for specialized hardware decoders.

38. The method of any preceding claim, comprising implementing a multimodal decoding software that dynamically adapts to various types of data, including text, speech, and video.

39. The method of any preceding claim, wherein the machine learning model comprises a large multimodal model, LMM.

40. The method of any preceding claim, wherein the data source comprises an image and / or a video.41 . The method of any preceding claim, wherein the first data and / or the second data comprises an image and / or a video.

42. The method of any preceding claim, wherein the data source is encoded using a hierarchical codec, preferably wherein the data source is encoded using a SMPTE VC 6 or an LCEVC codec.

43. The method of any preceding claim, wherein the first level of quality and the second level of quality are associated with different resolutions.

44. The method of any preceding claim, comprising determining the first data and the second data using a graphical processing unit, GPU, and / or a tensor processing unit, TPU.

45. The method of any preceding claim, being performed by a graphical processing unit, GPU, and / or a tensor processing unit, TPU.

46. A computer programme product comprising instructions that, when executed by a processor, cause the processor to perform the method of any preceding claim.

47. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor cause the processor to perform the method of any preceding claim.

48. An apparatus for providing data to a machine learning model using a hierarchical data format, the apparatus comprising: means for (e.g. a processor) for identifying a data source with a hierarchical format; means for (e.g. a processor) determining first data from the data source, the first data having a first level of quality; means for (e.g. a processor) determining second data from the data source, the second data having a second level of quality; and means for (e.g. a processor) providing the first data and the second data to the machine learning model.

49. The apparatus of claim 48, comprising a memory for storing (e.g. parameters of) the machine learning model.

50. A system for providing data to a machine learning model using a hierarchical data format, the system comprising: means for (e.g. a processor) for identifying a data source with a hierarchical format; means for (e.g. a processor) determining first data from the data source, the first data having a first level of quality; means for (e.g. a processor) determining second data from the data source, the second data having a second level of quality; and means for (e.g. a processor) providing the first data and the second data to the machine learning model.51 . The system of claim 50, comprising: a receiving device for transmitting one or more of: the data source, the first data, and the second data to a sending device; and the sending device, wherein the sending device hosts the machine learning model.

Images