Updating another machine learning model using a machine learning model

Abstract

A device includes a memory configured to store a first model and a second model. The first model is configured to perform inferences based on a first set of parameters corresponding to a first context. The device includes one or more processors configured to process the first set of parameters and an input corresponding to a second context using the second model to generate an output of the second model. The one or more processors are further configured to update the first model based on the output of the second model to perform inferences using an updated set of parameters.

Description

[0001] I. Cross-references to related applications

[0002] This application claims the benefit of priority to jointly owned U.S. non-provisional patent application No. 18 / 545,871, filed on December 19, 2023, the entire contents of which are expressly incorporated herein by reference.

[0003] II. Technical Field

[0004] This disclosure pertains to updating machine learning models.

[0005] III. Related Technical Descriptions

[0006] Technological advancements have led to smaller and more powerful computing devices. For example, a wide variety of portable personal computing devices exist today, including small, lightweight, and easily portable cordless phones (such as mobile and smartphones, tablets, and laptops). These devices can transmit voice and data packets over wireless networks. Furthermore, many of these devices incorporate additional functionality, such as digital still cameras, digital camcorders, digital recorders, and audio file players. Moreover, such devices can process executable instructions, including software applications such as web browser applications that can be used to access the internet. Accordingly, these devices can include significant computing power.

[0007] Such computing devices can incorporate functionality based on machine learning (ML) models. ML models, such as deep neural networks (DNNs), include parameters (e.g., weights and / or biases) that enable training such models to perform inference corresponding to a specific context. For example, a neural radiation field (NeRF) model can be trained to encapsulate an environment (e.g., a 2D image, 3D object, or 3D scene) by implicitly learning the geometry of the environment given a set of observations, thereby enabling the generation of novel and consistent views of the environment. For example, a NeRF model can be trained based on a two-dimensional (2D) initial image I1. f This is used to generate pixel predictions for I1 using the parameter set θ1. During the inference operation, the pixel location (e.g., (x, y) coordinates) is input to... f ,and f Generate a predicted output pixel value for this pixel location. Iterate sequentially at all pixel locations to generate an output that reconstructs the initial image I1.

[0008] However, a NeRF model can only be trained for a single environment at a time, and another round of training is required to update the parameters θ1 in order to apply the NeRF model to another environment. This training is computationally expensive and time-consuming.

[0009] IV. Summary of the Invention

[0010] According to a specific implementation of the technology disclosed herein, an apparatus includes a memory configured to store a first model and a second model. The first model is configured to perform inference based on a first set of parameters corresponding to a first context. The apparatus includes one or more processors configured to use the second model to process the first set of parameters and inputs corresponding to a second context to generate an output of the second model. The one or more processors are further configured to update the first model based on the parameters of the output of the second model to perform inference using the updated set of parameters.

[0011] According to a specific implementation of the technology disclosed herein, a method includes: obtaining a first model and a second model. The first model is configured to perform inference based on a first set of parameters corresponding to a first context. The method includes: using the second model to process the first set of parameters and inputs corresponding to a second context to generate an output of the second model. The method further includes: updating the first model based on the output of the second model to perform inference using the updated set of parameters.

[0012] According to a specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to obtain a first model and a second model. The first model is configured to perform inference based on a first set of parameters corresponding to a first context. When executed by the one or more processors, the instructions cause the one or more processors to use the second model to process the first set of parameters and input corresponding to a second context to generate an output of the second model. When executed by the one or more processors, the instructions also cause the one or more processors to update the first model based on the output of the second model to perform inference using the updated set of parameters.

[0013] According to a specific implementation of the technology disclosed herein, an apparatus includes components for obtaining a first model and a second model. The first model is configured to perform inference based on a first set of parameters corresponding to a first context. The apparatus includes components for processing the first set of parameters and inputs corresponding to a second context using the second model to generate an output of the second model. The apparatus further includes components for updating the first model based on the output of the second model to perform inference using the updated set of parameters.

[0014] Other specific embodiments, advantages, and features of this disclosure will become apparent upon examination of the entire application, which includes the following parts: description of drawings, detailed description, and claims.

[0015] V. Illustrations

[0016] Figure 1 This is a block diagram illustrating examples of systems operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0017] Figure 2A and Figure 2B These are examples illustrating some aspects of this disclosure. Figure 1 The diagram shows examples of components and operations implemented in the system.

[0018] Figure 3 These are examples illustrating some aspects of this disclosure. Figure 1 A block diagram illustrating examples of components and operations implemented in the system.

[0019] Figure 4 These are examples illustrating some aspects of this disclosure. Figure 1 A block diagram illustrating examples of components implemented in the system.

[0020] Figure 5 These are examples illustrating some aspects of this disclosure. Figure 1 A graph illustrating the system's performance.

[0021] Figure 6 This is a block diagram illustrating examples of systems operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0022] Figure 7 This is a block diagram illustrating examples of components that can be implemented in a system operable to use an ML model to update another ML model, according to some examples of this disclosure.

[0023] Figure 8 This is a block diagram illustrating examples of components that can be implemented in a system operable to use an ML model to update another ML model, according to some examples of this disclosure.

[0024] Figure 9A and Figure 9B This is a block diagram illustrating examples of components that can be implemented in a system operable to use an ML model to update another ML model, according to some examples of this disclosure.

[0025] Figure 10 The diagram illustrates examples of components and operations that can be implemented in a system operable to use an ML model to update another ML model, according to some examples of this disclosure.

[0026] Figure 11 This is a diagram illustrating an example of an integrated circuit operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0027] Figure 12 This is an illustration of a mobile device that can operate by using an ML model to update another ML model, based on some examples of this disclosure.

[0028] Figure 13 This is an illustration of a wearable electronic device that is operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0029] Figure 14 This is an illustration of a mixed reality or augmented reality glasses device that is operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0030] Figure 15 This is an illustration of a head-mounted device (such as a virtual reality, mixed reality, or augmented reality head-mounted device) that is operable to use an ML model to update another ML model according to some examples of this disclosure.

[0031] Figure 16 This is an illustration of a camera that is operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0032] Figure 17 This is a diagram illustrating, based on some examples of this disclosure, an operable voice-controlled speaker system that uses an ML model to update another ML model.

[0033] Figure 18 This is a diagram illustrating a first example of a vehicle that is operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0034] Figure 19 This is a second example of a vehicle that is operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0035] Figure 20 This is an illustration of an example of a method for using an ML model to update another ML model, based on some examples of this disclosure.

[0036] Figure 21 This is a block diagram illustrating a particular exemplary example of a device operable to use an ML model to update another ML model, based on some examples of this disclosure.

[0037] VI. Detailed Implementation

[0038] Systems and methods for using ML models to update another ML model are disclosed. For example, although deep neural network models such as NeRF models can be trained to encapsulate the environment by implicitly learning the geometry of the environment given a set of observations, thereby enabling the generation of novel and consistent views of the environment, retraining such models for another environment is computationally expensive and time-consuming.

[0039] The disclosed systems and methods enable the retraining of a first model (e.g., a first DNN), such as a NeRF trained to encapsulate a first environment, using a second model (e.g., a second DNN). The second model is configured to tune the parameters of the first model, or generate tuned values ​​of the parameters of the first model used to update the first model to encapsulate the second environment. According to one aspect, the parameters of the first model and information corresponding to the second environment are input into the second model, and one or more inference operations of the second model are performed to generate tuned parameter values, which are then used to update the first model. According to one aspect, compared to performing regular iterative training to train the first model from scratch (which could take several minutes or longer to perform 1,000 training iterations to achieve a specific accuracy), a relatively small number of inference operations of the second model (e.g., 50 inferences) can be performed in seconds to generate tuned parameters for updating the first model, wherein the updated first model provides substantially the same accuracy as that provided by 1,000 regular training iterations.

[0040] Therefore, compared to conventional training, the computationally expensive and time-consuming nature of conventional ML model training is addressed by using a second ML model that predicts the updated parameters of the ML model much faster and employs operations with lower complexity. Consequently, ML models can be retrained by mobile devices or other resource-constrained systems for novel environments and can be used in situations where latency of several minutes or more is unsatisfactory (e.g., in virtual reality use cases) and / or intolerable (e.g., in driver assistance use cases).

[0041] Specific aspects of this disclosure are described below with reference to the accompanying drawings. In this description, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing particular embodiments and are not intended to limit the scope of the embodiments. For example, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some embodiments and plural in others. For example, Figure 1 It describes a system that includes one or more processors ( Figure 1The device 102 (of which the “processor” 190) indicates that in some embodiments, device 102 includes a single processor 190, while in other embodiments, device 102 includes multiple processors 190. For ease of reference herein, such features are generally introduced as “one or more” features and are subsequently referred to in the singular or optional plural form (as indicated by “(multiple)” in the feature name) unless an aspect relating to multiple features is described.

[0042] In some figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically different, the same reference numerals are used for each feature, and these different instances are distinguished by adding letters to the reference numerals. Reference numerals are used without distinguishing letters when a feature is referenced herein as a group or a type of feature (e.g., when a specific feature among these features is not referenced). However, reference numerals are used with distinguishing letters when a specific feature among multiple features of the same type is mentioned herein. For example, Figure 8 The figure depicts the parametric encoder 410 associated with reference numerals 410A-410N. When referring to a specific parametric encoder among these parametric encoders (such as parametric encoder 410A), the distinguishing letter "A" is used. However, when referring to any arbitrary parametric encoder among these parametric encoders or when referring to these parametric encoders as a group, reference numeral 410 is used without the distinguishing letter.

[0043] As used herein, the term “comprising” may be used interchangeably with “including”. Additionally, it should be understood that the term “wherein” may be used interchangeably with “where”. As used herein, “exemplary” may indicate an example, implementation, and / or aspect, and should not be construed as restrictive or indicating a preference or preferred implementation. As used herein, ordinal terms used to modify elements (such as structures, components, operations, etc.) (e.g., “first,” “second,” “third,” etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element with the same name (but using ordinal terms). As used herein, the term “set” refers to one or more specific elements among specific elements, while the term “multiple” refers to multiple (e.g., two or more) specific elements.

[0044] As used herein, “coupling” can include “communicationally coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combination thereof. Two devices (or components) may be coupled directly or indirectly (e.g., communicationally coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some specific implementations, two communicationally coupled (such as electrical communication) devices (or components) may transmit and receive signals (e.g., digital or analog signals) directly or indirectly via one or more wires, buses, networks, etc. As used herein, “direct coupling” can include two devices coupled without intermediate components (e.g., communicationally coupled, electrically coupled, or physically coupled).

[0045] In this disclosure, terms such as “obtain,” “determine,” “calculate,” “estimate,” “transfer,” and “adjust” are used to describe how one or more operations are performed. It should be noted that such terms should not be construed as restrictive, and similar operations can be performed using other techniques. Additionally, as mentioned herein, “obtain,” “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” are used interchangeably. For example, “obtain,” “generate,” “calculate,” “estimate,” or “determine” a parameter (or signal) can refer to actively generating, estimating, calculating, or determining the parameter (or signal), or it can refer to using, selecting, retrieving, receiving, or accessing a parameter (or signal) such as one already generated by another component or device.

[0046] As used herein, the term “machine learning” should be understood to have any of its usual and conventional meanings within the fields of computer science and data science. Such meanings include, for example, processes or techniques by which one or more computers can learn to perform certain operations or functions without being explicitly programmed to do so. As a typical example, machine learning can be used to enable one or more computers to analyze data to identify patterns in the data and generate results based on that analysis. For some types of machine learning, the generated results include data that indicates the underlying structure or patterns of the data itself. For example, such techniques include so-called “clustering” techniques, which identify clusters (e.g., groupings of data elements).

[0047] For some types of machine learning, the resulting output includes a data model (also known as a "machine learning model" or simply a "model"). Typically, a model is generated using a first dataset to facilitate analysis on a second dataset. For example, the first portion of a large dataset can be used to generate a model that can then be used to analyze the remaining portion of the large dataset. As another example, historical datasets can be used to generate models that can be used to analyze future data.

[0048] Because a model can be used to evaluate datasets different from those used to generate the model, it can be viewed as a type of software (e.g., instructions, parameters, or both) automatically generated by a computer during the machine learning process. Therefore, the model can be transferable (e.g., it can be generated at a first computer and subsequently moved to a second computer for further training, use, or both). Additionally, the model can be combined with one or more other models to perform a desired analysis. For example, first data can be provided as input to a first model to generate first model output data, and the first model output data (alone, with the first data, or with other data) can be provided as input to a second model to generate second model output data indicative of the results of the desired analysis. Depending on the analysis and data involved, different combinations of models can be used to generate such results. In some examples, multiple models can provide model outputs that are input to a single model. In some examples, a single model provides model outputs as input to multiple models.

[0049] Examples of machine learning models include, but are not limited to, perceptrons, neural networks, support vector machines, regression models, decision trees, Bayesian models, Boltzmann machines, adaptive neurofuzzy inference systems, and combinations, sets, and variations of these and other types of models. Variations of neural networks include, for example, but not limited to, prototype networks, autoencoders, transformers, self-focused networks, convolutional neural networks, deep neural networks, deep belief networks, etc. Variations of decision trees include, for example, but not limited to, random forests, boosted decision trees, etc.

[0050] Since machine learning models are generated by computers based on input data, they can be discussed within at least two distinct time windows: the creation / training phase and the runtime phase. During the creation / training phase, a model is created, trained, adapted, validated, or otherwise configured by a computer based on input data (often referred to as "training data" during the creation / training phase). It's important to note that a trained model corresponds to software that has been generated and / or refined during the creation / training phase to perform a specific operation (such as classification, prediction, encoding, or other data analysis or data synthesis operations). During the runtime phase (or "inference" phase), the model is used to analyze the input data to generate model outputs. The content of the model outputs depends on the type of model. For example, as a non-limiting example, a model can be trained to perform a classification task or a regression task. In some implementations, the model may be updated continuously, periodically, or occasionally, in which case training time and runtime may be interleaved, or one form of the model may be used for inference while a copy is updated, and subsequently, the updated copy may be deployed for inference.

[0051] In some implementations, machine learning techniques are used to train (or retrain) a previously generated model. In this context, "training" refers to adapting a model or its parameters to a specific dataset. Unless otherwise clearly understood from the specific context, the term "training" as used herein includes "retraining" or refining a model for a specific dataset. For example, training may include so-called "transfer learning." In transfer learning, a base model is trained using a general or typical dataset, and subsequently refined (e.g., retrained or further trained) the base model using a more specific dataset.

[0052] The dataset used during training is called the "training dataset" or simply "training data." The dataset can be labeled or unlabeled. "Labeled data" refers to data that has been assigned classification labels indicating the groups or categories associated with the data, and "unlabeled data" refers to unlabeled data. Typically, "supervised machine learning processes" use labeled data to train machine learning models, while "unsupervised machine learning processes" use unlabeled data; however, it should be understood that the labels associated with the data are simply another data element that can be used in any appropriate machine learning process. For example, many clustering operations can be performed using unlabeled data; however, such clustering operations can use labeled data by ignoring the labels assigned to the data or by treating the labels in the same way as other data elements.

[0053] Training a model on a training dataset typically involves modifying the model's parameters with the goal of making the model's output possess specific characteristics based on the data input to the model. To distinguish it from model generation operations, model training may be referred to as optimization or optimization training in this paper. In this context, "optimization" refers to improving a metric, not necessarily finding an ideal value for that metric (e.g., a global maximum or minimum). Examples of optimization trainers include, but are not limited to, backpropagation trainers, derivative-free optimizers (DFO), and extreme learning machines (ELM). As an example of training a model, during supervised training of a neural network, input data samples are associated with labels. When input data samples are fed to the model, the model generates output data, comparing that output data with the labels associated with the input data samples to generate error values. The model's parameters are modified to attempt to reduce (e.g., optimize) the error values. As another example of training a model, during unsupervised training of an autoencoder, data samples are fed as input to the autoencoder, and the autoencoder reduces the dimensionality of the data samples (a lossy operation) and attempts to reconstruct the data samples into output data. In this example, the output data is compared with the input data samples to generate the reconstruction loss, and the parameters of the autoencoder are modified to attempt to reduce (e.g., optimize) the reconstruction loss.

[0054] Figure 1 A block diagram of a system 100 including device 102 is shown, which is configured to update another ML model using an ML model. Device 102 includes memory 120 coupled to one or more processors 190 and configured to store a first model (f θ1 122. Second Model (g) γ 124, a first set (θ1) 128 of parameters, and context data 114, such as data associated with a first context 126 and a second context 130. In a particular embodiment, memory 120 corresponds to dynamic random access memory (DRAM), static random access memory (SRAM), or one or more other types of devices or combinations thereof configured to store data and / or instructions executable by processor 190 in a double data rate (DDR) memory subsystem.

[0055] Processor 190 is configured to perform operations associated with ML engine 140. In various embodiments, some or all of the functionalities associated with ML engine 140 are executed via processor 190 executing instructions, executed by the processing circuitry of processor 190 in a hardware implementation, or a combination thereof.

[0056] ML engine 140 is configured to perform contextual reasoning operations 142 using a first model 122. The first model 122 is configured to perform reasoning based on a first set 128 of parameters corresponding to a first context 126 (e.g., a 2D or 3D representation of a first scene or a first 3D object). In this example, the first model 122 corresponds to a NeRF model. In the following examples, for illustrative purposes, the first context 126 and the second context 130 refer to 2D images and / or image information, and coordinates (x, y) are used to index pixels in the image. However, in other embodiments, the first context 126 and the second context 130 can generally be any information or meta-information about the scene or environment (e.g., multi-view images, depth information, 3D scans, descriptors provided from a large language model (LLM), camera pose information, etc.).

[0057] Contextual inference operation 142 includes using the first model 122 to perform inference operations to generate output associated with the first context 126, such as a representation of a first scene from one or more viewpoints. For example, each inference operation using the first model 122 processes coordinate input 144, such as (x, y) coordinates, to generate a first contextual prediction 146, such as predicted pixel values ​​associated with the coordinate input 144.

[0058] The ML engine 140 is also configured to perform a model update operation 152 to update the first model 122 to generate output associated with the second context 130. For example, the processor 190 may receive input corresponding to the second context 130, such as a 2D or 3D representation of a second scene or a second 3D object. For illustration, the input corresponding to the second context 130 may include multiple images of a scene captured by one or more cameras and camera pose information associated with each of the multiple images.

[0059] During model update operation 152, ML engine 140 updates the first set 128 of parameters of the first model 122 to generate an updated first model (f) that can be used in the second context 130. θ2The updated set of parameters (θ2) 134 of the first model 148. For example, the ML engine 140 is configured to perform one or more iterations of a parametric inference operation 154, which includes using the second model 124 to process a first set 128 of parameters and inputs corresponding to a second context 130 to generate an output 132 of the second model 124. For illustration, generating the output 132 of the second model 124 may include performing multiple iterations of inference at the second model 124, wherein the updated parameters associated with the output 132 of each inference iteration provide an improved accuracy of the updated first model 148. The model update operation 152 also includes updating the first model 122 based on the output of the second model 124 to perform inference using the updated set of parameters 134. According to one aspect, each of the first set 128 of parameters, the updated set 134 of parameters, and the output 132 of the second model 124 is a vector of weights, biases, or any other parameters (or adjusted values ​​of these parameters) that can be input to the first model 122.

[0060] In certain implementations, such as references Figure 4 Further described, the second model 124 includes a parametric encoder configured to process a set of input parameters, a context encoder configured to process inputs corresponding to the second context 130, a joint encoder configured to process the outputs of the parametric encoder and the context encoder, and a parametric decoder configured to process the output of the joint encoder to generate output 132. (See reference...) Figure 8 , Figure 9A and Figure 9B Additional exemplary examples of components that can be implemented in the second model 124 are described.

[0061] Model update operation 152 also includes updating the first model 122 based on the output 132 of the second model 124 to perform inference using the updated set 134 of parameters. For illustration, the output 132 of the second model 124 may include the updated set 134 of parameters, or may include a set of adjusted values ​​to be applied to the first set 128 of parameters to generate the updated set 134 of parameters. (See reference) Figure 2A and Figure 2B Examples of operations associated with model update operation 152 are described in further detail.

[0062] In some implementations, after updating the first model 122 based on the output 132 of the second model 124 to generate an updated first model 148, as part of a model update operation 152, the ML engine 140 is further configured to perform one or more training operations on the updated first model 148 to enhance the inference accuracy of the updated first model 148 against the second context 130. In examples, training operations may be performed until the inference accuracy reaches an accuracy threshold. In some implementations, one or more processors are configured to alternate between parameter updates using training operations and parameter updates using the second model 124 until the inference accuracy reaches the accuracy threshold.

[0063] The updated first model 148 is configured to perform inference based on an updated set 134 of parameters corresponding to the second context 130. For example, the ML engine 140 is configured to perform context inference operation 162 using the updated first model 148. Context inference operation 162 includes processing coordinate input 164 at the updated first model 148 to generate a second context prediction 166 corresponding to the second context 130.

[0064] Device 102 (e.g., processor 190) is configured to output data based on a first context prediction 146, a second context prediction 166, or both. For example, device 102 may optionally be coupled to one or more output devices 112 configured to provide output associated with the operation of ML engine 140 to a user of device 102. For illustration, output device 112 may include a display device configured to display image data generated using a first model 122, image data generated using an updated first model 148, or both.

[0065] Device 102 may optionally be coupled to one or more context data sources 110, which are configured to provide at least a portion of context data 114 to processor 190. For example, context data source 110 may include one or more cameras configured to generate context data 114 associated with a second context 130 and integrated within or coupled to device 102. As another example, context data source 110 may include one or more remote devices 180. Figure 1 In the illustrated example, device 102 may optionally include modem 170, which is coupled to processor 190 and configured to receive a first model 122, a second model 124, a first set 128 of parameters, data corresponding to a first context 126, data corresponding to a second context 130, or a combination thereof, from remote device 180.

[0066] In some implementations, to improve the speed and / or accuracy of generating the updated set 134 of parameters, the processor 190 may perform a starting parameter set selection operation to find an existing set of parameters that corresponds to a context similar to the second context 130. In some implementations, such as references Figure 6 and Figure 7 Further described, processor 190 is configured to access a collection of stored parameter sets corresponding to multiple contexts of the first model 122, and to identify a specific context among the multiple contexts that has the closest similarity to the second context 130 based on a similarity metric. Processor 190 may select a collection of stored parameter sets corresponding to the identified specific context as a first set 128 of parameters to serve as a starting point for generating the updated first model 148. The collection of stored parameter sets may be stored at memory 120, at a remote device 180, or a combination of both.

[0067] Compared to performing conventional iterative training to train the first model 122 from scratch (which could take anywhere from several minutes to several hours), a relatively small number of inference operations on the second model 124 can be performed in seconds to generate updated parameters 134 for the updated first model 148, providing substantially the same accuracy as conventional iterative training. Therefore, the model update operation 152 enables the device 102 to rapidly generate updated models for new environments in resource-constrained systems such as mobile devices or head-mounted displays.

[0068] Although the second model 124 is exemplified as processing a first set 128 of parameters and inputs corresponding to the second context 130 to generate output 132, in some embodiments, the second model 124 is configured to further generate output 132 based on inputs corresponding to the first context 126 or based on a difference measurement between the first context 126 and the second context 130, such as reference... Figure 9A and Figure 9B Further detailed description.

[0069] Although one or more portions of context data 114 (e.g., data associated with second context 130) are described as being provided by context data source 110, such as from one or more cameras or from memory 120, in other embodiments, one or more portions of context data 114 may alternatively be generated by the processor 190 (e.g., a digital signal processor (DSP), such as audio including speech corresponding to the output of a game engine or other speech generation application), the output of another component of device 102, or received from another device (e.g., remote device 180).

[0070] In some specific implementations, device 102 corresponds to or is included in a device of a variety of types of devices. In an exemplary example, ML engine 140 (e.g., processor 190) is integrated into at least one of the following: as referenced Figure 12 The described mobile phone or tablet computer device, as shown in the reference Figure 13 The wearable electronic devices described, as in the reference Figure 14 The described mixed reality or augmented reality glasses device, such as the reference Figure 15 The virtual reality, mixed reality, or augmented reality head-mounted device described, as in the reference Figure 16 The described camera device or as in the reference Figure 17 The described voice-activated speaker system. In another exemplary example, processor 190 is integrated into a vehicle including a camera configured to generate context data associated with a second context 130, such as reference data. Figure 18 and Figure 19 Further description.

[0071] Figure 2A and Figure 2B Depicting what can be done Figure 1 Examples of components and operations implemented in the system. Figure 2A and Figure 2B In each of these, a sequence of operations that can be performed by the ML engine 140 during model update operation 152 is depicted. The operations include multiple iterations of inference performed at the second model 124 (e.g., multiple iterations of parameter inference operation 154) to generate an updated set of parameters θ2 134. The second model 124 is denoted as “g γ The instruction states that the second model 124 uses a parameter set "γ", which enables the second model 124 to generate inference outputs based on the context data input to the second model 124 to modify the parameters of the first model 122. (See reference...) Figure 3 An example of training the second model 124 to obtain the parameter set γ is described in further detail.

[0072] exist Figure 2A In the first iteration (inference 1) of the sequence, operation 200 includes: inputting a first set of parameters θ1128 and information (e.g., pixels) corresponding to a second context 130 (e.g., image I2) into a second model 124. The second model 124 generates a first adjustment value Δθ1 for the first set of parameters θ1128 in the first iteration, and adds the first adjustment value Δθ1 to the first set of parameters θ1128 at a combiner (e.g., adder) 208 to generate a first intermediate set of parameters θ. t1 (For example, θ1+Δθ1).

[0073] In the second iteration (Inference 2), the first intermediate set θ of the parameters is... t1 Together with the information corresponding to the second context I2 130, it is input into the second model 124 to generate the second adjustment value Δθ2. At the combiner 208, the second adjustment value Δθ2 is compared with the first intermediate set of parameters θ. t1 Add the parameters to generate a second intermediate set θ t2 (For example, θ1+Δθ1+Δθ2).

[0074] Optionally, in each iteration, the iteration number 206 (e.g., 1, 2, etc.) is also input into the second model 124. The second model 124 can be trained to adjust the generated adjustment value Δθ based on the iteration number 206, such as by performing a coarser “adjustment” on the parameter set in earlier iterations and a finer adjustment in later iterations. For example, the second model 124 can be trained such that the proportion of the adjustment value Δθ should become smaller as the iteration number 206 increases. To illustrate, during the training of the second model 124, one or more loss functions can be weighted in different ways based on the iteration number 206.

[0075] One or more additional iterations of the inference at point 124 of the second model are performed in a manner substantially similar to that described in the second iteration. The iterative process outputs the Mth adjustment value Δθ. M The process ends after the Mth iteration (where M is a positive integer), and the adjusted value is compared with the (M-1)th intermediate set θ of the parameters. t(M-1) Combined to generate an updated set of parameters θ2134.

[0076] exist Figure 2A In the implementation plan, Figure 1 The output 132 includes the set of adjustment values ​​Δθ1 … Δθ M These adjustment values ​​are applied to the first set of parameters θ1 128 to generate the updated set of parameters θ2 134 as θ2 = θ1 + Δθ1 + ... + Δθ M Therefore, generating output 132 involves multiple iterations of inference performed at the second model 124.

[0077] Operation 200 also depicts operating the first model 122 using an updated set of parameters θ2 134 (i.e., using an updated first model 148) to result in the generation of a second context prediction 166, such as predicted pixel values ​​for a second context I2 130. For illustration, coordinate input 164 can be configured to scan pixel coordinates and processed by the updated first model 148 to generate predicted pixel values, thereby producing a potentially novel view of the second context I2 130.

[0078] In some implementations, the number of iterations M is a hyperparameter. Alternatively, M can be determined based on the quality of the second context prediction 166 generated by the updated first model 148 after one or more inference iterations. For example, operation 200 may include: using the current intermediate set θ of the parameters t The updated first model 148 is periodically tested to determine a quality metric (e.g., PSNR) of the resulting second context prediction 166. Operation 200 may terminate when the quality metric is above a quality threshold or when the difference metric between the second context prediction 166 and the second context 130 is below a difference threshold.

[0079] exist Figure 2B In, with Figure 2A Operation 250 is performed in a similar manner to operation 200. However, with Figure 2A In contrast, combiner 208 is omitted, and the second model 124 is configured to directly generate the intermediate set θ of parameters. t1 θ t2 、…、θ t(M-1) The updated set of parameters 134, instead of the generated set. Figure 2A The adjustment values ​​Δθ1, Δθ2, ..., Δθ M Therefore, in Figure 2B In the implementation scheme, the output 132 of the second model 124 includes an updated set 134 of parameters.

[0080] Output Figure 2B The intermediate set of parameters θ t1 θ t2 、…、θ t(M-1) Instead Figure 2A The adjustment values ​​Δθ1, Δθ2, ..., Δθ M This allows the combiner 208 to be omitted and reduces complexity. However, compared to direct output... Figure 2B The intermediate set of parameters θ t1 θ t2 、…、θ t(M-1) Compared to, output Figure 2A The adjustment values ​​Δθ1, Δθ2, ..., Δθ M This allows the second model 124 to provide a larger dynamic range for parameter tuning and potentially higher accuracy.

[0081] According to one aspect, such as Figure 2A or Figure 2BAs described, after 50 iterations (M=50) of running low-complexity inference, the quality (e.g., peak SNR) of the second context prediction 166 of the second context I2 130 can approximately or substantially match the quality produced by performing 1,000 regular training cycles of the first model 122 using higher-complexity backpropagation, thus significantly saving processing resources and time compared to conventional techniques.

[0082] Figure 3 It describes what can be done during the training of the second model 124 (such as in Figure 1 Examples 300 of components and operations implemented in device 102 or in a remote system (such as remote device 180). As illustrated, training a second model 124 to learn how to adjust the parameters of a first model 122 can be performed using gradient-based optimization operations including forward and backward propagation. During forward propagation, the second model 124 receives the current parameters θ 304 of the first model 122 and context information I 322, such as the set of pixel values ​​corresponding to all (x, y) pixel coordinates associated with a group of two-dimensional images. The second model 124 outputs a set of adjusted values ​​Δθ, denoted as β 332, which are added at a combiner 308 (e.g., an adder) to the parameters θ 304 of the first model 122 to generate an updated set 334 of the parameters of the first model 122.

[0083] Similarly, during the forward propagation, the first model 122 receives coordinate input 320 and processes the coordinate input 320 using an updated set 334 of parameters to generate inference output 330 (e.g., pixel prediction), denoted as f. θ+Δθ (x, y). Inference output 330 is used to determine the loss function. 340, the loss function is a function of the context information 322 and the inference output 330, expressed as: .

[0084] During backpropagation, backpropagation is performed to adjust the parameters γ of the second model 124, which is graphically represented as data path 350. Since a fully differentiable computational graph is formed under the depicted formula, the loss function... The gradient of parameter γ of model 340 with respect to model 124 can be expressed using the chain rule. To calculate and optimize accordingly.

[0085] The forward and backward propagation sequences can be iteratively repeated to adjust the parameters γ of the second model 124, such that the adjusted value β 332 output by the second model 124 and the updated set 334 of parameters therefrom produce the inference output 330 of the first model 122, which reduces the loss function. 340 or minimize the loss function. The selection of context information 322 can train the second model 124 for domain-specific purposes (such as for facial images as an exemplary, non-restrictive example), resulting in improved performance of the second model 124 for the trained domain.

[0086] Figure 4 Depicting what can be done Figure 1 Examples of components implemented in the system. Figure 4 In the illustrated implementation, the second model 124 includes a parametric encoder 410, a context encoder (e.g., an image encoder) 420, a joint encoder 430, and a parametric decoder 440. The parametric encoder 410 is configured to receive and process an input set of parameters (illustrated as a first set 128 of parameters associated with the first context 126) and an iteration count 206. The context encoder 420 is configured to process input corresponding to the second context. For example, the context encoder 420 may receive input corresponding to the second context 130, such as a 2D image, and the context encoder 420 may act as an image encoder.

[0087] The joint encoder 430 is configured to process the outputs of the parametric encoder 410 and the context encoder 420. For example, the joint encoder 430 can modulate the weights of the first set 128 of parameters based on information from the second context 130, such as a multilayer perceptron (MLP), which receives a concatenation of image features from the context encoder 420 and parametric features from the parametric encoder 410, and performs processing similar to matrix multiplication.

[0088] The parameter decoder 440 is configured to process the output of the joint encoder 430 and the number of iterations 206 to generate the output 132 of the second model 124, which is exemplified as a set of adjustment values ​​450.

[0089] In a particular implementation, each of the parametric encoder 410, context encoder 420, joint encoder 430, and parametric decoder 440 is implemented as an MLP. However, in other implementations, one or more of the parametric encoder 410, context encoder 420, joint encoder 430, or parametric decoder 440 may be different. In an exemplary example, the context encoder 420 may include or be implemented as a transformer or convolutional neural network (CNN), and the parametric encoder 410, joint encoder 430, and parametric decoder 440 may include or be implemented as transformer-based models that benefit from the use of self-attention or cross-attention mechanisms. Skip connections, similar to those in ResNet networks, may also be used in one or more of the parametric encoder 410, context encoder 420, joint encoder 430, and parametric decoder 440.

[0090] During the training of the second model 124, the parameter encoder 410, context encoder 420, joint encoder 430, and parameter decoder 440, such as reference encoders, can be trained jointly. Figure 3 As described. However, in some implementations, the context encoder 420 may be partially or completely "frozen" so that the behavior of the context encoder 420 does not change during the training of the second model 124.

[0091] The example provided above enables the use of the second model 124 to generate the updated parameters 134 and eliminates the conventional training requirements associated with updating the first model 122 to correspond to the second context 130. However, in some implementations, after updating the first model 122 based on the output of the second model 124, the ML engine 140 also performs one or more conventional training operations on the updated first model 148 to enhance the inference accuracy of the updated first model 148 with respect to the second context 130. Such conventional training operations can be performed until the inference accuracy reaches an accuracy threshold. For example, in situations based on... Figure 2A or Figure 2B After generating the updated parameters 134 using the sequence of inference operations described, the ML engine 140 can perform regular training (e.g., using backpropagation) during the model update operation 152 to further improve the accuracy of the updated first model 148 in predicting the second context 130, as referenced. Figure 5 As described.

[0092] exist Figure 5 In Figure 500, the quality metric (illustrated as PSNR) is plotted on the vertical axis and the number of iterations of backpropagation training is plotted on the horizontal axis, and illustrative, non-limiting examples of improved performance that can be obtained in some specific implementations are shown. First curve 502 illustrates the PSNR associated with the number of iterations that have been performed in a conventional system to adapt the first model 122 to the conventional training corresponding to the second context 130. Second curve 504 illustrates the PSNR, according to this disclosure, associated with the number of iterations that have been performed after 5 iterations of parametric inference operation 154 first performed using the second model 124 to adapt the first model 122 to the conventional training.

[0093] As illustrated in Table 500, training the first model 122 for 5 iterations of the parameter inference operation 154, followed by 50 iterations of regular training, produces the same PSNR (0 dB) as regular training for 1,000 iterations. Specifically, the second curve 504 reaches PSNR=0 dB after approximately 50 iterations, while the first curve 502 requires approximately 1,000 iterations to reach PSNR=0 dB. Furthermore, at 1,000 iterations, the second curve 504 has a PSNR approximately 10 dB lower than the first curve 502 (approximately 0 dB). Therefore, training the first model 122 for 5 iterations of the parameter inference operation 154, followed by 1,000 iterations of regular training, produces a significantly higher quality (+10 dB) compared to regular training using the same number of iterations.

[0094] Therefore, a small number of training iterations can produce high-quality output. Consequently, this type of training can be performed with fewer resources (e.g., processing power, time) compared to full-fledged regular training, making it possible to perform model training in resource-constrained environments such as mobile devices.

[0095] In some implementations, the ML engine 140 can perform multiple iterations alternating between parameter tuning using the second model 124 and parameter tuning using regular training. This can lead to faster convergence to the updated set of updated parameters 134 of the updated first model 148 that satisfy the accuracy criterion compared to using the second model 124 alone. For illustration, the ML engine 140 can be configured to alternate (e.g., interleave) between parameter updates using training operations and parameter updates using the second model 124 until the inference accuracy reaches an accuracy threshold.

[0096] For example, upon encountering a new context 130 (e.g., a new image, scene, or environment), the ML engine 140 can execute a first loop (e.g., one or more iterations) of the parameter inference operation 154 to update the parameters of the first model 122. Using the updated parameters obtained in the parameter inference operation 154, the ML engine 140 can also execute several iterations of explicit (regular) training of the updated first model 148, followed by a second loop of the parameter inference operation 154 to generate further updated parameters. For the second loop of the parameter inference operation 154, the second model 124 receives the latest set of updated parameters of the first model 148 as input and further adjusts the parameters. This alternation between using the parameter inference operation 154 and regular training iterations can continue, while simultaneously using regular training (e.g., for...) Figure 5Compared to the first curve 502 (where PSNR increases with each iteration), convergence (e.g., an increase in PSNR with each iteration) occurs faster. Alternating between parametric inference operation 154 and regular training iterations can improve... Figure 5 The performance illustrated in the second curve 504, for example, causes the PSNR to increase at a faster rate per iteration than that shown in the second curve 504.

[0097] Figure 6 An example of system 600 is depicted, which includes device 102 coupled to remote device 602 via communication network 604. Device 102 includes processor 190 and memory 120, and is coupled to (or includes) one or more cameras 610. Cameras 610 may be coupled to device 102 or included in the device, and may correspond to... Figure 1 The context data source 110. The memory 120 includes a first model 122, a second model 124, and a library 620 of stored parameter sets corresponding to multiple contexts (e.g., different images, scenes, environments, etc.), as further described below. The remote device 602 also stores a library 650 of stored parameter sets corresponding to multiple contexts. In a particular embodiment, the remote device 602 corresponds to... Figure 1 The remote device 180 can be a remote server with a larger processing and storage capacity than device 102, such as a cloud server.

[0098] To improve the speed and / or accuracy of generating an updated set θ2 134 of parameters for a new environment (e.g., one or more images received from camera 610 as a second context I2 130), processor 190 may perform a starting parameter set selection operation 606 to find an existing set of parameters corresponding to a context similar to the second context I2 130. For example, processor 190 may access a group of stored parameter sets (e.g., gallery 620, gallery 650, or both) corresponding to multiple contexts of the first model 122 and identify the specific context among the multiple contexts that has the closest similarity to the second context I2 130 based on a similarity metric 612. In this way, the first model 122 may be initialized using parameters that are most similar to the new image, scene, environment, etc., of its corresponding image, scene, environment, etc. For illustration, the second model 124 may compute parameter adjustments more efficiently because the adjustments may be relatively small due to the similarity of the corresponding contexts. In some implementations, if no stored parameter set corresponds to a context that is sufficiently similar to the new context by a threshold amount, the processor 190 does not use the parameter inference operation 154, but instead trains the first model 122 from scratch using regular training 608.

[0099] Furthermore, the initial parameter set selection operation 606 can be performed in a centralized or distributed manner. For illustration, a centralized initial parameter set selection can be performed using cloud-based processing or split-based processing (e.g., both cloud-based processing and processing on device 102). A distributed initial parameter set selection can be performed on device 102. Determining whether to perform the initial parameter set selection in a centralized or distributed manner can be based on use cases, or on power, computation, or timing criteria, or a combination thereof.

[0100] As illustrated, memory 120 stores a library 620 of stored parameter sets associated with various contexts. For a specific context 630, each entry 622 of library 620 includes a parameter set (PS) 632, a feature descriptor (FD) 634, and may optionally include a second resolution parameter set (SR-PS) 636. Specifically, library 620 includes: entry 622A, which includes parameter set "a" (PSa) for context "a"; entry 622B, which includes parameter set "b" (PSb) for context "b"; and one or more additional entries, including entry 622F, which includes parameter set "f" (PSf) for context "f". Each of the parameter sets PSa-PSf may correspond to a weight set of the first model 122 to perform a prediction for a specific context. For illustration, one parameter set in parameter sets PSa-PSf corresponds to a first set of parameters θ1 128 to perform a prediction for a first context 126.

[0101] The similarity measure 612 used to identify the context most similar to the second context I2 130 can be based on a set of extracted feature descriptors 634 associated with multiple contexts 630 and the extracted feature descriptors associated with the second context I2 130. For example, entry 622A in the library 620 includes the feature descriptor FDa of context “a”, entry 622B includes the feature descriptor FDb of context “b”, and entry 622F includes the feature descriptor FDf of context “f”. As a non-limiting example, the feature descriptor can correspond to one or more of the following: scene type, object type, location, latent spatial representation of one or more features, pixel spatial representation of one or more features, features obtained via a large language model, or descriptors obtained via a large language model. In an exemplary embodiment, the similarity measure 612 can be based on the distance (e.g., Euclidean distance) between the extracted feature descriptors of I2 130 and each of the feature descriptors FDa-FDf.

[0102] In the example, processor 190 may determine that the feature descriptor FDb is most similar to (i.e., the closest to) the extracted feature descriptor of the second context 130 (e.g., the second context 130 is determined to be most similar to context "b"). Therefore, processor 190 selects the stored parameter set PSb corresponding to the identified specific context "b" as the first set of parameters θ1 128, as the starting point for generating the updated parameters θ2 134.

[0103] In some cases, instead of selecting one parameter set from the parameter set 632 at memory 120, processor 190 may retrieve parameter set 632 from a remote group of parameter sets at a remote device (such as remote device 602 storing library 650). Library 650 includes: entry 622G, which includes parameter set "g" (PSg) and feature descriptor "g" (FDg) for context "g"; entry 622H, which includes parameter set "h" (PSh) and feature descriptor "h" (FDh) for context "h"; and one or more additional entries, including entry 622X, which includes parameter set "x" (PSx) and feature descriptor "x" for context "x". For example, processor 190 may access a remote group of parameter sets at a remote server (e.g., library 650 at remote device 602) via communication network 604 to obtain a first set θ1 128 of parameters based on the failure of the closest similarity of parameter set 632 stored at local memory 120 to meet a threshold similarity with a second context 130.

[0104] In an exemplary implementation, if the library 620 in memory 120 does not include a set of parameters 632 for context 630 that satisfy (e.g., equal to or exceed) a threshold similarity metric 612, a remote library 650 may also be searched. However, in some time-sensitive applications (e.g., when context I2 130 corresponds to sensor data of an autonomous vehicle), the amount of latency that may result from the latency associated with searching the remote library 650 and retrieving the parameter set via the communication network 604 may be considered unacceptable. In such cases, device 102 (e.g., processor 190) may select whether to access the remote library 650 based at least in part on timing criteria associated with updating the first model 122.

[0105] In some specific implementations, the conditions for generating the updated model f are met. θ2 The timing criterion of 148 can have more than the model f that ensures updates. θ2 148 provides higher priority for high-resolution, high-accuracy predictions of the second context I2 130. In such specific implementations, the updated model f can be generated using the "silhouette" or the second resolution parameter set 636.θ2 148, this “simplified” or second resolution parameter set corresponds to a lower resolution than the full resolution parameter set 632. For example, the library 620 may include multiple contexts “a”-“m” in each specific entry 622A-622F; a first resolution parameter set (PS) configured to enable inference of the specific context at a first resolution; a second resolution parameter set (SR-PS) configured to enable inference of the specific context at a reduced resolution; and a feature descriptor (FD) for the specific context. Thus, the processor 190 may be configured to use the second resolution parameter set (SR-PS) for the specific context as a first set of parameters θ1 138, and after performing updated weight prediction using the second model 124, the updated model f θ2 148 is configured to perform inference corresponding to the second context I2 130 at a reduced resolution. For example, when processor 190 uses the full-resolution parameter set 632 to generate an updated model, a reduced resolution can be used, such as via performing parameter inference operation 154, performing parameter inference operation 154 followed by iterations of regular training 608, or performing a loop of parameter inference operation 154 alternating with iterations of regular training 608, as described above.

[0106] In some implementations, the updated model 148 can be generated according to a latency-reducing tuning process, where the inference of the second model 124 is restricted to tuning only specific parameters in the parameter set of the first model 122. For illustration, in the various examples above, the second model 124 considers all parameters of the first model 122. However, not all parameters within a neural network contribute to overall performance, as they typically operate under an overparameterized paradigm. By identifying the most salient parameters, the second model 124 can be restricted to tuning only those salient parameter values, which can improve the efficiency of the second model 124 in generating the updated set 134 of parameters. In some implementations, the salient parameters of the neural network can typically be determined via a parameter saliency map, which is based on determining the gradient with respect to the weights for a fixed input to determine the sensitivity of the weights to the target task, such as image classification.

[0107] As an illustrative, non-restrictive example, Figure 7 Depicting what can be done Figure 1 Device 102, System 100 or Figure 6 Examples of components and operations implemented in System 600.

[0108] exist Figure 7In this process, selection operation 710 is used to select a context from the library 702 based on its similarity to the second context I2 130, and inputs the parameter set corresponding to the selected context into the second model 124. In a particular embodiment, the library 702 corresponds to... Figure 6 The library 620 in the memory 120 of device 102, a remote library (such as library 650 in remote device 602), or a combination thereof.

[0109] The second model 124 includes a parameter encoder 410 that encodes the input parameter set, a context encoder 420 (e.g., exemplified as an image encoder 420) that encodes information corresponding to the context, a joint encoder 430 that encodes the outputs of the parameter encoder 410 and the image encoder 420, and a parameter decoder 440 that generates an adjustment value Δθ 450 based on the output of the joint encoder 430, as previously referenced. Figure 4 As described, information corresponding to the second context I2 130 (e.g., pixel data) is provided to the image encoder 420, and this information is also input to the selection operation 710.

[0110] Operation 710 searches the library 702 to identify a specific set of parameters that has the highest similarity (minimum distance) to the second context I2 130, denoted as θ. 720. For example, selecting operation 710 can be done in... Figure 6 The initial parameter set selection operation 606 is used, and the parameter set θ can be selected by calculating the similarity (e.g., similarity measure 612) between the second context I2 130 and the context associated with that parameter set for each parameter set in the library 702. 720. The parameter set that results in the highest similarity is chosen as parameter set θ. 720 is input to parameter encoder 410 for the first iteration step of parameter inference operation 154.

[0111] In a specific implementation, the parameter set θ is selected according to the following expression. 720:

[0112] ,

[0113] Where d(...) is a distance function that can be computed in pixel space, the latent space of an autoencoder (e.g., a feature extractor latent), LLM features and / or LLM descriptions, meta-information (e.g., what kind of scene it is, what object is being viewed, whether it is an interior space, what furniture is there, location, etc.), or any combination thereof. The set of parameters θ that minimizes the distance function d is chosen as the parameter set θ. 720, the minimum value of the distance function d indicates the minimum distance to the second context I2 130 or the highest similarity to that second context.

[0114] Figure 8 An example of a component that can be implemented in the second model 124 to achieve streamlined parameter tuning by separating the set of input parameters (exemplified as a first set 128 of parameters) through network layers based on the first model 122 is described. This makes it possible to limit the influence of unrelated parameters on each other. For example, since the first model 122 includes multiple network layers, the second model 124 can be configured to include corresponding instances of a parameter encoder 410, a joint encoder 430, and a parameter decoder 440 for each of the multiple network layers of the first model 122, the parameter encoder, the joint encoder, and the parameter decoder being configured to generate an output 132 associated with that network layer of the first model 122. Although this specification refers to a “layer” of the first model 122, it should be understood that each such “layer” may alternatively refer to a group of two or more layers of the first model 122, such as a function block.

[0115] exist Figure 8 In an exemplary embodiment, the second model 124 includes a first row 810A, which includes a first layer parameter encoder 410A, a first layer joint encoder 430A, and a first layer parameter decoder 440A associated with a first (e.g., lowest or initial) layer corresponding to the first model 122; a second row 810B, which includes a second layer parameter encoder 410B, a second layer joint encoder 430B, and a second layer parameter decoder 440B associated with a second layer of the first model 122; and a third row 810C, which... The third line includes a third-layer parametric encoder 410C, a third-layer joint encoder 430C, and a third-layer parametric decoder 440C associated with the third layer of the first model 122; and one or more additional lines 810, including an Nth line 810N, which includes an Nth-layer parametric encoder 410N, an Nth-layer joint encoder 430N, and an Nth-layer parametric decoder 440N associated with the Nth (e.g., the highest or final) layer of the first model 122, where N is a positive integer corresponding to the number of layers in the first model 122.

[0116] In the first row 810A, the first-layer parameter encoder 410A is configured to receive the first-layer parameter θ. L1 128A and 206 iterations. First layer parameters θ L1128A corresponds to a subset of the first parameter 128 and includes weights and / or biases associated with the first layer of the first model 122. The output of the first-layer parametric encoder 410A and the output of the context encoder 420 (e.g., image encoder 420) are input to the first-layer joint encoder 430A. The output of the first-layer joint encoder 430A and the iteration count 206 are input to the first-layer parametric decoder 440A, which outputs the first-layer adjustment value Δθ. L1 132A. First layer adjustment value Δθ L1 132A indicates that the first layer parameter θ needs to be adjusted. L1 Adjustments made to 128A.

[0117] Each of the remaining rows 810B-810N operates in a manner substantially similar to that described with reference to the first row 810A: the second row 810B receives the second-layer parameter θ. L2 128B and output the second layer adjustment value Δθ L2 132B, the third line 810C receives the third layer parameter θ. L3 128C and output the second layer adjustment value Δθ L3 132C, and the Nth row 810N receives the Nth layer parameter θ. LN 128N and output the Nth layer adjustment value Δθ LN 132N. However, each of the rows 810B-810N also receives information from one or more other rows 810 to simulate the information flow within the first model 122.

[0118] For example, in a particular implementation, by implementing a connection from the parametric and joint encoder from line 810 corresponding to the lower layer to the joint and parametric decoder from line 810 corresponding to the higher layer, without implementing such a connection in the other direction (e.g., from the parametric and joint encoder from line 810 corresponding to the higher layer to the joint and parametric decoder from line 810 corresponding to the lower layer), line 810 corresponding to the lower layer of the first model 122 (e.g., line 810A) affects line 810 corresponding to the higher layer (e.g., lines 810B-810N). This in Figure 8 The example shown is a connection from the output of layer 1 joint encoder 430A to the input of layer 2 joint encoder 430B, from the output of layer 2 joint encoder 430B to the input of layer 3 joint encoder 430C, etc.

[0119] By simulating the information flow of the first model 122 Figure 8 The second model 124 can reduce the influence of higher-level parameters on lower-level parameters, which enables more effective and / or efficient training of the second model 124 and can produce more accurate output 132.

[0120] As an illustrative, non-restrictive example, Figure 9A and Figure 9B Examples are shown that can be included Figure 1 Device 102, System 100 or Figure 6 Examples of components in System 600. Figure 9A and Figure 9B In the example, the second model 124 is configured to generate output 132 based at least in part on the difference between the first context 126 and the second context 130 associated with the first model 122.

[0121] exist Figure 9A In the second model 124, it includes Figure 4 The second model 124 omits the context encoder 420 and instead receives context-based input in the form of a difference measurement 920 received from the difference measurement unit 910.

[0122] The difference measurement unit 910 receives information associated with a first context 126 and information associated with a second context 130, and calculates a difference measurement 920. In some embodiments, the difference measurement unit 910 performs a direct comparison of the first context 126 and the second context 130 to determine the difference measurement 920. For example, in an embodiment where the first context 126 and the second context 130 correspond to images, the difference measurement unit 910 may subtract one image from another image, and the difference measurement 920 may correspond to the difference image. In some embodiments, the difference measurement unit 910 includes one or more encoders, and the difference measurement 920 may correspond to the difference in the latent space of one or more encoders. For example, in an embodiment where the first context 126 and the second context 130 correspond to images, one or more encoders may act as feature extractors that extract features from each image, and the difference measurement 920 may indicate the difference between the features of each image.

[0123] The difference measurement 920, along with the output of the parameter encoder 410, is input into the joint encoder 430. Therefore, the second model 124 generates output 132 based on the difference measurement 920 between the first context 126 and the second context 130. Because the difference measurement 920 provides information about where the differences between the first context 126 and the second context 130 lie, the second model 124 can generate output 132 more efficiently and / or more accurately than in a specific implementation that does not consider the differences between the first context 126 and the second context 130 (e.g., adjusted values ​​of the first set 128 of parameters).

[0124] Although the difference measurement unit 910 is described as performing a direct comparison of the received input, or as encoding the received input to compare differences in extracted features, in other embodiments, the difference measurement unit 910 may use one or more other techniques to generate the difference measurement 920. For example, the difference measurement unit 910 may include or be coupled to a large language model (LLM) describing the differences between two inputs. For illustration, the output of the LLM may include a textual description of the differences, which may be transformed into a latent space by the difference measurement unit 910, or otherwise processed via one or more intermediate operations at the difference measurement unit 910 to convert the textual description into an embedding that can be input to the joint encoder 430.

[0125] Figure 9B Another embodiment in which the second model 124 includes a first context encoder 420A (e.g., a first image encoder 420A) and a second context encoder 420B (e.g., a second image encoder 420B) is depicted. The first context encoder 420A is configured to encode input corresponding to a first context 126, and the second context encoder 420B is configured to encode input corresponding to a second context 130.

[0126] The outputs of each of the context encoders 420A and 420B are provided as inputs to the joint encoder 430 along with the output of the parametric encoder 410. Therefore, the second model 124 is configured to generate output 132 based on the input corresponding to the first context 126 and the input corresponding to the second context 130. The second model 124 (e.g., the joint encoder 430) can be configured to compare features of the first context 126 and the second context 130, and translate this comparison into generating output 132 to adjust parameters 128.

[0127] As discussed previously, it should be understood that although context 126 and context 130 are described as images in various examples, context 126 and context 130 can generally refer to any measurement, feature, or meta-information about the corresponding environment, and are not limited to images. Additionally, although Figure 9A and Figure 9B The implementation scheme can typically be used in conjunction with the various implementation schemes described above, but it is also suitable for use in conjunction with the previously described technique of interweaving the loop of parameter inference operation 154 with the loop of regular training iterations. Figure 9A and Figure 9BThe implementation scheme may have a negative impact on performance because whenever a subsequent loop of parameter inference operation 154 begins, the context 130 associated with the updated weights must be recalculated—that is, the environment encapsulated by the first model 122 after the loop of regular training iterations has been completed, which may be computationally expensive.

[0128] According to one aspect, an example is given. Figures 2A to 9B The various components shown aid in understanding the operations performed by processor 190 according to some implementation schemes. These components may be implemented in one or more processors or in processing circuitry, such as fixed-function circuitry, programmable circuitry, or a combination thereof. Fixed-function circuitry refers to circuitry that provides specific functionality and is pre-configured regarding the operations that can be performed. Programmable circuitry refers to circuitry that can be programmed to perform various tasks and provide flexible functionality in terms of the operations that can be performed. For example, programmable circuitry may execute software or firmware that causes the programmable circuitry to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuitry may execute software instructions (e.g., to receive or output parameters), but the type of operations performed by fixed-function circuitry is generally immutable. In some examples, one or more units in the cell may be different circuit blocks (fixed-function circuit blocks or programmable circuit blocks), and in some examples, the one or more units may be integrated circuits.

[0129] Figure 10 Examples 1000 of components and operations that can be used in conjunction with device 102 in an extended reality (XR) head-mounted device (HMD) 1004 worn by user 1002 are depicted. As used herein, by way of illustrative example, XR may include virtual reality (VR), mixed reality (MR), or augmented reality (AR).

[0130] In Example 1000, HMD 1004 has already “learned” the surfaces in a specific environment depicted as first scene 1010. For illustration, HMD 1004 includes a first model 122 configured to perform inference on coordinate inputs 144 (such as (x, y, z) coordinates) using a first set of parameters θ1 128 to generate a resulting prediction 146 corresponding to the view of first scene 1010. For example, first scene 1010 could correspond to a first room, such as a room in a museum. In this example, first scene 1010 corresponds to... Figure 1 The first context 126.

[0131] HMD 1004 may then encounter a new scene, depicted as a second scene 1012. For example, user 1002 may move from the first room of a museum into a second room, which may be similar to the first room but may have different artwork and a different arrangement of furniture. HMD 1004 captures multiple images of the second scene 1012 via one or more cameras and as user 1002 moves relative to the second scene 1012. HMD 1004 also stores pose information, such as 6-DOF pose information, associated with each of the captured images and reconstructs the second scene 1012 using the multi-view images and the associated pose information. In a particular embodiment, a first set θ1 128 of inputs corresponding to the second scene 1012 (e.g., second context 130) and parameters associated with the first model 122 is provided to the second model 124, which generates an updated set θ2 134A of parameters. As an illustrative and non-limiting example, the input provided to the second model 124 corresponding to the second scene 1012 may include multi-view images and pose information captured by the HMD 1004, or may include a 3D reconstruction created by the HMD 1004 based on the multi-view images and pose information (e.g., using a truncated symbolic distance function (TSDF)), or may correspond to semantic information extracted from the multi-view images and pose information (e.g., a description generated by an LLM).

[0132] The generated updated set 134A of parameters includes performing one or more inference operations at the second model 124 to generate an adjusted value set Δθ, and applying the adjusted value set Δθ to a first set of parameters θ1 128 to generate an updated set θ2 134A of parameters, such as reference. Figure 2A As described. In some implementations, one or more inference operations at the second model 124 may be followed by additional iterations of regular training, or interleaved with one or more loops of regular training iterations, until the inference accuracy of the updated first model 148A reaches an accuracy threshold, such as a reference. Figure 5 As described. After generating an updated first model 148A that uses an updated set θ2 134A of parameters for the second scene 1012, the HMD 1004 uses the updated first model 148A to perform inference based on coordinate input 164A to generate a prediction 166A that enables the HMD 1004 to represent the second scene 1012 of user 1002.

[0133] After generating an updated first model 148A corresponding to the second scene 1012, the HMD 1004 may then encounter another new scene, depicted as a third scene 1014. For example, user 1002 may walk from the second room of a museum into the third room, which may be similar to the first room but may have different artwork and furniture arrangement than the first or second room. The HMD 1004 captures multiple images of the third scene 1014 and corresponding pose information via one or more cameras of the HMD 1004 as user 1002 moves relative to the third scene 1014, and reconstructs the third scene 1014 using the multi-view images and associated pose information. In a particular embodiment, a first set θ1 128 of inputs corresponding to the third scene 1014 and parameters associated with the first model 122 is provided to the second model 124, which generates another updated set θ3 134B of parameters in a manner similar to that described for the updated set θ2 134A of generated parameters. After generating an updated first model 148B that uses an updated set θ3 134B of parameters for the third scene 1014, the HMD 1004 uses the updated first model 148B to perform inference based on coordinate input 164B to generate a prediction 166B that enables the HMD 1004 to represent the third scene 1014 for user 1002.

[0134] Figure 11 The device 102 is depicted as a specific implementation 1100 of an integrated circuit 1102 including one or more processors 190. The integrated circuit 1102 also includes a data input 1104 (such as one or more bus interfaces) to enable context data 114 (e.g., information associated with a second context 130) to be received for processing. The integrated circuit 1102 also includes a signal output 1106 (such as a bus interface) to enable the transmission of output signals, such as context prediction data 1108. As an example, context prediction data 1108 may correspond to or include... Figure 1 The first context prediction 146, the second context prediction 166, or both. The integrated circuit 1102, including the ML engine 140, enables the use of an ML model to update another ML model as a component of the system, such as... Figure 12 The mobile phone or tablet computer described, such as Figure 13 The wearable electronic devices described, such as those in the reference Figure 14 The described mixed reality or augmented reality glasses device, such as Figure 15 The virtual reality, mixed reality, or augmented reality headsets described, such as Figure 16 The camera described, such as Figure 17 The described voice-controlled speaker system or such Figure 18 or Figure 19 The vehicles depicted.

[0135] Figure 12 A specific implementation 1200 of a mobile device 1202, in which device 102 includes a telephone or tablet computer (as an illustrative, non-limiting example), is depicted. Mobile device 1202 includes one or more microphones 1206, one or more speakers 1208, one or more cameras 1210, and a display screen 1204. Components of processor 190, including ML engine 140, are integrated into mobile device 1202 and are illustrated using dashed lines to indicate internal components that are generally not visible to the user of mobile device 1202. In a particular example, ML engine 140 is operable to obtain input corresponding to a specific context via camera 1210 or from a remote device. ML engine 140 is operable to perform model update operations using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. ML engine 140 may generate image data based on predictions made using the updated first model and output the image data for display at display screen 1204. Using the second model to update the parameters of the first model enables the mobile device 1202 to efficiently (in terms of latency, computing resources, and power) generate an updated first model that can generate accurate predictions corresponding to a specific context.

[0136] Figure 13A specific implementation 1300 of the device 102, including a wearable electronic device 1302 (illustrated as a "smartwatch"), is depicted. The wearable electronic device 102 includes a display screen 1304, one or more microphones 1306, one or more speakers 1308, and one or more cameras 1310. Components of the processor 190, including an ML engine 140, are integrated into the wearable electronic device 1302. In a particular example, the ML engine 140 is operable to obtain input corresponding to a specific context via the camera 1310 or from a remote device. The ML engine 140 is operable to perform a model update operation using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. The ML engine 140 can generate image data based on predictions made using the updated first model and output the image data for display at the display screen 1304. Using a second model to update the parameters of the first model enables the wearable electronic device 1302 to efficiently generate an updated first model (in terms of latency, computational resources, and power), which generates accurate predictions corresponding to a specific context. In some embodiments, the wearable electronic device 1302 is configured to generate notifications based on a specific context. For example, the display 1304 may generate visual information associated with image data generated using the updated first model. As another example, the wearable electronic device 1302 may include a haptic device that provides haptic notifications (e.g., vibration) based on predictions made using the updated first model.

[0137] Figure 14A specific implementation 1400 of the device 102 is depicted, comprising a portable electronic device corresponding to augmented reality or mixed reality glasses 1402. Glasses 1402 include a holographic projection unit 1404 configured to project visual data onto the surface of a lens 1406, or to reflect the visual data from the surface of the lens 1406 onto the wearer's retina. Glasses 1402 also includes one or more microphones 1408, one or more speakers 1410, and one or more cameras 1412. Components of processor 190, including an ML engine 140, are integrated into glasses 1402. In a particular example, the ML engine 140 is operable to obtain input corresponding to a specific context via camera 1412 or from a remote device. The ML engine 140 is operable to perform a model update operation using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. The ML engine 140 can generate image data based on predictions made using an updated first model and output the image data to the holographic projection unit 1404 for display to the wearer of the glasses 1402. Updating the parameters of the first model using a second model enables the glasses 1402 to efficiently generate the updated first model (in terms of latency, computational resources, and power), which generates accurate predictions corresponding to a specific context.

[0138] Figure 15A specific implementation 1500 is depicted in which device 102 includes a portable electronic device corresponding to a virtual reality, mixed reality, or augmented reality head-mounted device 1502. A visual interface device 1504 is positioned in front of the user's eyes to enable the display of augmented reality, mixed reality, or virtual reality images or scenes to the user when wearing the head-mounted device 1502. The head-mounted device 1502 also includes one or more microphones 1506, one or more speakers 1508, and one or more cameras 1510. Components of processor 190, including an ML engine 140, are integrated into the head-mounted device 1502. In a particular example, the ML engine 140 is operable to obtain input corresponding to a specific context via camera 1510 or from a remote device. The ML engine 140 is operable to perform model update operations using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. The ML engine 140 can generate image data based on predictions made using an updated first model and output the image data to the visual interface device 1504 for display to the wearer of the head-mounted device 1502. Updating the parameters of the first model using a second model enables the head-mounted device 1502 to efficiently generate the updated first model (in terms of latency, computational resources, and power), which generates accurate predictions corresponding to a specific context.

[0139] Figure 16 A specific implementation 1600 is depicted in which device 102 includes a portable electronic device corresponding to camera device 1602. Camera device 1602 includes one or more microphones 1606, one or more speakers 1608, and one or more image sensors 1610. Components of processor 190 (including ML engine 140) are integrated into camera device 1602. In a particular example, ML engine 140 is operable to obtain input corresponding to a specific context via image sensor 1610 or from a remote device. ML engine 140 is operable to perform model update operations using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. ML engine 140 may generate image data based on predictions made using the updated first model and store the image data in memory or a remote device, output the image data to a display screen of camera device 1602 for display to a user of camera device 1602, or a combination thereof. Using the second model to update the parameters of the first model enables the camera device 1602 to efficiently (in terms of latency, computational resources, and power) generate an updated first model that can generate accurate predictions corresponding to a specific context.

[0140] Figure 17 This is a specific implementation 1700 of device 102, which includes a wireless speaker and a voice activation device 1702. The wireless speaker and voice activation device 1702 may have wireless network connectivity and is configured to perform auxiliary operations. The wireless speaker and voice activation device 1702 includes one or more microphones 1706, one or more speakers 1708, one or more cameras 1710, and a display screen 1704. Components of processor 190, including ML engine 140, are integrated into the wireless speaker and voice activation device 1702. In a particular example, ML engine 140 is operable to obtain input corresponding to a specific context via camera 1710 or from a remote device. ML engine 140 is operable to perform a model update operation using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. The ML engine 140 can generate image data based on predictions made using an updated first model and store the image data in memory or a remote device, output the image data to a display screen 1704 for display to a user of the wireless speaker and voice activation device 1702, or a combination thereof. Updating the parameters of the first model using a second model enables the wireless speaker and voice activation device 1702 to efficiently generate the updated first model (in terms of latency, computational resources, and power), which generates accurate predictions corresponding to a specific context.

[0141] Figure 18A specific implementation 1800 is depicted in which device 102 corresponds to or is integrated within vehicle 1802 (exemplified as a manned or unmanned aerial device, such as a package delivery drone). Vehicle 1802 includes one or more microphones 1806, one or more speakers 1808, and one or more cameras 1810. Components of processor 190, including ML engine 140, are integrated into vehicle 1802. In a particular example, ML engine 140 is operable to obtain input corresponding to a specific context via camera 1810 or from a remote device (e.g., a remote navigation system). ML engine 140 is operable to perform a model update operation using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. The ML engine 140 can generate image data based on predictions made using an updated first model, store the image data in memory, transmit the image data to a remote device, output the image data to a display screen for display to the user of vehicle 1802, or a combination thereof. Using a second model to update the parameters of the first model enables vehicle 1802 to efficiently (in terms of latency, computational resources, and power) generate the updated first model, which can generate accurate predictions corresponding to a specific context. For example, images captured by camera 1810 can be used to generate the updated model, which can produce a novel view of the environment surrounding vehicle 1802, which can be provided to the operator of vehicle 1802 for navigation assistance.

[0142] Figure 19 Another specific embodiment 1900 in which device 102 corresponds to or is integrated within a vehicle 1902 (illustrated as an automobile) is depicted. Vehicle 1902 includes a display screen 1904, one or more microphones 1906, one or more speakers 1908, and one or more cameras 1910 (e.g., front-facing cameras, rear-facing cameras, or cameras with any other orientation or field of view for navigation, assisted driving operations, or autonomous driving operations). Components of processor 190 (including ML engine 140) are integrated into vehicle 1902.

[0143] In a specific example, ML engine 140 is operable to obtain input corresponding to a specific context via camera 1910 or from a remote device (e.g., a remote navigation system). ML engine 140 is operable to perform model update operations using a second model, a first set of parameters associated with a first model, and the input corresponding to the specific context to generate an updated set of parameters, and to update the first model to perform inference using the updated set of parameters. ML engine 140 may generate image data based on predictions made using the updated first model and store the image data in memory, transmit the image data to a remote device, output the image data to a display screen for display to a user of vehicle 1902, or a combination thereof. Using the second model to update the parameters of the first model enables vehicle 1902 to efficiently (in terms of latency, computational resources, and power) generate the updated first model, which generates accurate predictions corresponding to the specific context. For example, images captured by camera 1910 can be used to generate an updated model that can produce a novel view of the environment surrounding vehicle 1902, which can be provided to the operator of vehicle 1902 for navigation assistance.

[0144] In some implementation schemes, Figure 18 Transportation vehicle 1802 Figure 19 Vehicles 1902, one or more other devices (such as robots), etc., can support operator-assisted or autonomous operation, where one or more technologies are used to generate updated models with minimal latency. For example, such as Figure 6As described, vehicles or robots can use a "simplified" or second-resolution parameter set 636, corresponding to a lower resolution than the full-resolution parameter set 632, to train a lower-resolution but functional model more quickly for more direct use, such as when generating full-resolution models in parallel. Alternatively, vehicles or robots can restrict parameter tuning to those determined to be the most prominent parameters to further reduce the latency associated with generating the updated model. In such an implementation, vehicles or robots can perform operations based on a hierarchical structure of permissible latency. For example, the time required to identify and update the local set of model parameters (e.g., from the library 650) can be estimated compared to the time required to access, identify, retrieve, and update the model parameters from a remote set (e.g., from the library 620). This estimation can be further hierarchically based on the latency associated with identifying, retrieving, and updating the simplified parameter set 636, and / or based on restricting training to the most prominent parameters, compared to the full-resolution parameter set 632. If the estimated delay for generating the updated model exceeds the timing criteria associated with a specific operation that will use the updated model, the vehicle or robot can be configured to make operational decisions (such as collision avoidance maneuvers) instead of generating the updated model or while the generation of the updated model is in progress, to ensure the safe operation of the vehicle or robot.

[0145] refer to Figure 20 This illustrates a specific implementation of a method 2000 for updating another ML model using an ML model. In this particular implementation, one or more operations of method 2000 are performed by... Figure 1 The ML engine 140, processor 190, device 102, system 100, or at least one combination thereof shall be executed.

[0146] In some implementations, method 2000 includes, at block 2002, obtaining a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context. For example, Figure 1 The ML engine 140 obtains a first model 122 and a second model 124, and the first model 122 is configured to perform inference based on a first set 128 of parameters corresponding to a first context 126. In some embodiments, the first context corresponds to a 2D or 3D representation of a first scene or a first 3D object, and the second context corresponds to a 2D or 3D representation of a second scene or a second 3D object.

[0147] Method 2000 includes: at block 2004, using a second model to process a first set of parameters and inputs corresponding to a second context to generate the output of the second model. For example, ML engine 140 uses second model 124 to process a first set 128 of parameters and inputs corresponding to a second context 130 to generate the output 132 of second model 124.

[0148] Method 2000 includes: at block 2006, updating the first model based on the output of the second model to perform inference using the updated set of parameters. For example, ML engine 140 updates the first model 122 to perform inference using the updated set of parameters 134, thereby producing an updated first model 148, and the updated set of parameters 134 is based on the output 132 of the second model 124. For example, the updated set of parameters 134 may be derived from, as referenced... Figure 2B The second model 124 output described herein, or may be based on, as referenced Figure 2A The set of adjusted values ​​output by the second model 124 described is used to generate the model.

[0149] Optionally, method 2000 includes accessing a group of stored parameter sets corresponding to multiple contexts of the first model. For example, processor 190 may access... Figure 6 The parameter set stored in library 620, library 650, or both. Method 2000 may include: identifying a specific context among multiple contexts that has the closest similarity to the second context based on a similarity measure (e.g., similarity measure 612), and selecting the stored parameter set corresponding to the identified specific context as the first set of parameters, such as references. Figure 6 The initial parameter set selection operation is described in 606.

[0150] Figure 20 Method 2000 can be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. As an example, Figure 20 Method 2000 can be executed by a processor that executes instructions, such as references Figure 21 As described.

[0151] refer to Figure 21 This diagram depicts a specific, exemplary embodiment of the device, and generally designates the device as 2100. In various embodiments, device 2100 may have the same... Figure 21 The illustrated components may be more or fewer than the number of components. In an exemplary embodiment, device 2100 may correspond to device 102. In an exemplary embodiment, device 2100 may perform reference... Figures 1 to 20One or more operations as described.

[0152] In a particular implementation, device 2100 includes a processor 2106 (e.g., a central processing unit (CPU)). Device 2100 may include one or more additional processors 2110 (e.g., one or more DSPs). In a particular aspect, Figure 1 The processor 190 corresponds to processor 2106, processor 2110, or a combination thereof. Processor 2110 may include a speech and music decoder-decoder (codec) 2108, which includes a voice decoder ("vocoder") encoder 2136, a vocoder decoder 2138, an ML engine 140, or a combination thereof.

[0153] In this context, the term "processor" refers to an integrated circuit composed of logic units, interconnects, input / output blocks, clock management components, memory, and optional other dedicated hardware components, designed to execute instructions and perform various computational tasks. Examples of processors include, but are not limited to, central processing units (CPUs), digital signal processors (DSPs), neural processing units (NPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), microcontrollers, quantum processors, coprocessors, vector processors, other similar circuits, and variations and combinations thereof. In some cases, a processor may be integrated with other components, such as communication components, input / output components, etc., to form a system-on-a-chip (SoC) device or a packaged electronic device.

[0154] Starting with the CPU, a CPU typically includes one or more processor cores. Each of these cores comprises a complex network of interconnected transistors and other circuitry defining logic gates, memory elements, and so on. The cores are responsible for executing instructions to perform, for example, arithmetic and logical operations. Typically, a CPU includes an arithmetic logic unit (ALU) that handles mathematical operations and a control unit that generates signals to coordinate the operations of other CPU components, such as managing fetch-decode-execute loops.

[0155] CPUs and / or individual processor cores typically include local memory circuitry, such as registers and caches, to temporarily store data during operation. Registers consist of high-speed, small-size memory cells tightly connected to logic units within the CPU. Typically, registers include transistors arranged as groups of flip-flops configured to store binary data. Caches consist of fast on-chip memory circuitry for storing frequently accessed data. For example, a cache can be implemented using static random access memory (SRAM) circuitry.

[0156] CPU operations (e.g., arithmetic, logical, and flow control operations) are guided by software and firmware. At the lowest level, the CPU includes an Instruction Set Architecture (ISA), which specifies how hardware resources (e.g., registers, arithmetic units, etc.) are used to perform individual operations. Higher-level software and firmware are then translated into various combinations of ISA operations to enable the CPU to perform specific higher-level operations. For example, an ISA typically specifies how the CPU's hardware components move and modify data to perform operations such as addition, multiplication, and subtraction, and higher-level software is translated into sets of such operations to accomplish larger tasks, such as adding two columns in a spreadsheet. Generally, the CPU operates on various levels of software, including the kernel, operating system, applications, etc., where each higher-level software is typically more abstract than the ISA and is generally easier for human users to understand.

[0157] GPUs, NPUs, DSPs, microcontrollers, coprocessors, FPGAs, ASICs, and vector processors include components similar to those described above for CPUs. The differences between these various types of processors are often related to the use of dedicated interconnect schemes and ISAs to enhance the processor's ability to perform specific types of operations. For example, the logic gates, local memory circuitry, and interconnects between them of a GPU are specifically designed to improve parallel processing, data sharing between processor cores, and vector operations, and the GPU's ISA can define operations that utilize these structures. As another example, an ASIC is a highly specialized processor that includes similar circuitry arranged and interconnected for specific tasks, such as encryption or signal processing. As yet another example, an FPGA is a programmable device that includes an array of configurable logic blocks (e.g., an interconnect set of transistors and memory elements) that can be configured (typically in flight) to perform customizable logic functions.

[0158] Device 2100 may include memory 2186 and codec 2134. Memory 2186 may include instructions 2156 that can be executed by one or more additional processors 2110 (or processor 2106) to implement the functionality described with reference to ML engine 140, or both. Memory 2186 may also include data corresponding to context data 114, first model 122, second model 124, parameter set 128, or other data associated with the operation of ML engine 140. In a particular embodiment, memory 2186 corresponds to... Figure 1 The memory 120. The device 2100 may include a modem 170 coupled to the antenna 2152 via a transceiver 2150.

[0159] Device 2100 may include a display 2128 coupled to display controller 2126. One or more speakers 2192 and microphones 2194 may be coupled to codec 2134. Codec 2134 may include a digital-to-analog converter (DAC) 2102, an analog-to-digital converter (ADC) 2104, or both. In a particular embodiment, codec 2134 may receive analog signals from microphone 2194, convert these analog signals to digital signals using ADC 2104, and provide these digital signals to speech and music codec 2108. Speech and music codec 2108 may process digital signals. In a particular embodiment, speech and music codec 2108 may provide digital signals to codec 2134. Codec 2134 may use ADC 2102 to convert digital signals to analog signals and may provide the analog signals to speaker 2192.

[0160] In a particular embodiment, device 2100 may be included in a system-in-package (SiP) or system-on-a-chip (SoC) 2122. In a particular embodiment, memory 2186, processor 2106, processor 2110, display controller 2126, codec 2134, and modem 170 are included in the SiP or SoC 2122. In a particular embodiment, input device 2130, power supply 2144, and one or more cameras 2196 are coupled to the SiP or SoC 2122. Furthermore, in a particular embodiment, such as... Figure 21 As illustrated, the display 2128, input device 2130, speaker 2192, microphone 2194, camera 2196, antenna 2152, and power supply 2144 are external to the system-in-package or system-on-chip device 2122. In a particular implementation, each of the display 2128, input device 2130, speaker 2192, microphone 2194, camera 2196, antenna 2152, and power supply 2144 may be coupled to components of the system-in-package or system-on-chip device 2122, such as an interface or controller.

[0161] Device 2100 may include smart speakers, speaker bars, mobile communication devices, smartphones, cellular phones, laptops, computers, tablets, personal digital assistants, display devices, televisions, game consoles, music players, radios, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, vehicles, head-mounted devices, augmented reality head-mounted devices, mixed reality head-mounted devices, virtual reality head-mounted devices, air vehicles, home automation systems, voice-activated devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof.

[0162] In conjunction with the described specific embodiments, an apparatus includes components for obtaining a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context. For example, the components for obtaining the first model and the second model may include device 102, modem 170, processor 190, ML engine 140, integrated circuit 1102, processor 2106, processor 2110, system-in-package or system-on-chip device 2122, device 2100, other circuitry configured to obtain the first model and the second model, or combinations thereof.

[0163] The apparatus also includes components for processing the first set of parameters and inputs corresponding to the second context using the second model to generate the output of the second model. For example, the components for processing the first set of parameters and inputs corresponding to the second context using the second model to generate the output of the second model may include device 102, processor 190, ML engine 140, integrated circuit 1102, processor 2106, processor 2110, system-in-package or system-on-chip device 2122, device 2100, other circuitry configured to process the first set of parameters and inputs corresponding to the second context using the second model to generate the output of the second model, or combinations thereof.

[0164] The apparatus further includes components for updating the first model based on the output of the second model to perform inference using the updated set of parameters. For example, components for updating the first model based on the output of the second model to perform inference using the updated set of parameters may include device 102, processor 190, ML engine 140, integrated circuit 1102, processor 2106, processor 2110, system-in-package or system-on-a-chip device 2122, device 2100, other circuitry or combinations thereof configured to update the first model based on the output of the second model to perform inference using the updated set of parameters.

[0165] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 2186) includes instructions (e.g., instructions 2156) that, when executed by one or more processors (e.g., one or more processors 2110 or 2106), cause the one or more processors to obtain a first model (e.g., first model 122) and a second model (e.g., second model 124), wherein the first model is configured to perform inference based on a first set of parameters (e.g., first set of parameters 128) corresponding to a first context (e.g., first context 126). When executed by the one or more processors, these instructions also cause the one or more processors to use the second model to process the first set of parameters and the input corresponding to the second context to generate an output (e.g., output 132) of the second model 124, and to update the first model based on the output of the second model to perform inference using the updated set of parameters (e.g., updated set of parameters 134).

[0166] Specific aspects of this disclosure are described below in a collection of related embodiments:

[0167] According to Embodiment 1, an apparatus includes: a memory configured to store a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; and one or more processors configured to use the second model to process the first set of parameters and inputs corresponding to a second context to generate an output of the second model; and to update the first model based on the output of the second model to perform inference using the updated set of parameters.

[0168] Example 2 includes the device according to Example 1, wherein the first context corresponds to a 2D or 3D representation of a first scene or a first 3D object, and wherein the second context corresponds to a 2D or 3D representation of a second scene or a second 3D object.

[0169] Example 3 includes the device according to Example 1 or Example 2, wherein the first model corresponds to the neural radiation field (NeRF) model.

[0170] Example 4 includes the device according to any one of Examples 1 to 3, wherein generating the output of the second model includes multiple iterations of inference performed at the second model.

[0171] Example 5 includes a device according to any one of Examples 1 to 4, wherein the output of the second model includes the updated set of parameters or a set of adjustment values ​​to be applied to the first set of parameters to generate the updated set of parameters.

[0172] Example 6 includes a device according to any one of Examples 1 to 5, wherein the one or more processors are configured to: access a group of stored parameter sets corresponding to a plurality of contexts of the first model; and identify a specific context among the plurality of contexts that has the closest similarity to the second context based on a similarity metric.

[0173] Example 7 includes the device according to Example 6, wherein the similarity measure is based on a set of extracted feature descriptors associated with the plurality of contexts and an extracted feature descriptor associated with the second context.

[0174] Example 8 includes the device according to Example 7, wherein the feature descriptor corresponds to one or more of the following: scene type, object type, location, features obtained via a large language model, or descriptors obtained via a large language model.

[0175] Example 9 includes a device according to any one of Examples 6 to 8, wherein one or more processors are configured to select the stored set of parameters corresponding to a specific identified context as the first set of parameters.

[0176] Example 10 includes a device according to any one of Examples 6 to 9, wherein the group of stored parameter sets is stored in the memory, and wherein one or more processors are configured to access a remote group of parameter sets via a communication network to obtain the first set of parameters based on the failure of the closest similarity to meet a threshold similarity.

[0177] Example 11 includes the device according to Example 10, wherein one or more processors are configured to select whether to access the remote group based at least in part on a timing criterion associated with updating the first model.

[0178] Example 12 includes the device according to any one of Examples 6 to 11, wherein:

[0179] For each specific context among the plurality of contexts, the group of stored parameter sets further includes: a first resolution parameter set configured to enable inference of the specific context at a first resolution; a second resolution parameter set configured to enable inference of the specific context at a reduced resolution; and a feature descriptor for the specific context; the one or more processors configured to use the second resolution parameter set of the specific context as the first set of parameters; and an updated first model configured to perform inference corresponding to the second context at the reduced resolution.

[0180] Example 13 includes a device according to any one of Examples 1 to 12, wherein after updating the first model based on the output of the second model, the one or more processors are further configured to perform one or more training operations on the updated first model to enhance the inference accuracy of the updated first model for the second context.

[0181] Example 14 includes the device according to Example 13, wherein the one or more training operations are performed until the inference accuracy reaches an accuracy threshold.

[0182] Example 15 includes the device according to Example 13, wherein one or more processors are configured to alternate between parameter updates using a training operation and parameter updates using the second model until the inference accuracy reaches an accuracy threshold.

[0183] Example 16 includes a device according to any one of Examples 1 to 15, wherein the second model is configured to further generate the output based on input corresponding to the first context.

[0184] Example 17 includes a device according to any one of Examples 1 to 16, wherein the second model is configured to generate the output based on a difference measurement between the first context and the second context.

[0185] Example 18 includes a device according to any one of Examples 1 to 17, wherein the second model includes: a parameter encoder configured to process a set of input parameters; a context encoder configured to process the inputs corresponding to a second context; a joint encoder configured to process the outputs of the parameter encoder and the context encoder; and a parameter decoder configured to process the outputs of the joint encoder to generate the outputs.

[0186] Example 19 includes a device according to any one of Examples 1 to 18, wherein: the first model includes a plurality of network layers; and for each of the plurality of network layers of the first model, the second model includes a corresponding instance of a parametric encoder, a joint encoder, and a parametric decoder, the parametric encoder, the joint encoder, and the parametric decoder being configured to generate an output associated with the network layer of the first model.

[0187] Example 20 includes the device according to any one of Examples 1 to 19 and also includes a camera configured to generate context data associated with the second context.

[0188] Example 21 includes the device according to any one of Examples 1 to 20 and further includes a modem coupled to the one or more processors and configured to receive the first model, the second model, the first set of parameters, or a combination thereof from a remote device.

[0189] Example 22 includes the device according to any one of Examples 1 to 21 and further includes a display device configured to display image data generated using the updated first model.

[0190] Example 23 includes a device according to any one of Examples 1 to 22, wherein the one or more processors are integrated in at least one of: a virtual reality headset, a mixed reality headset, or an augmented reality headset.

[0191] Example 24 includes a device according to any one of Examples 1 to 22, wherein one or more processors are integrated in a vehicle, the vehicle further including a camera configured to generate context data associated with the second context.

[0192] Example 25 includes a device according to any one of Examples 1 to 24, wherein one or more processors are included in an integrated circuit.

[0193] According to embodiment 26, a method includes: obtaining a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; using the second model to process the first set of parameters and inputs corresponding to a second context to generate an output of the second model; and updating the first model based on the output of the second model to perform inference using the updated set of parameters.

[0194] Example 27 includes the method according to Example 26, wherein the first context corresponds to a 2D or 3D representation of a first scene or a first 3D object, and wherein the second context corresponds to a 2D or 3D representation of a second scene or a second 3D object.

[0195] Example 28 includes the method according to Example 26 or Example 27, wherein the first model corresponds to the Neural Radiation Field (NeRF) model.

[0196] Example 29 includes the method according to any one of Examples 26 to 28, wherein generating the output of the second model includes performing multiple iterations of inference at the second model.

[0197] Example 30 includes the method according to any one of Examples 26 to 29, wherein the output of the second model includes the updated set of parameters or a set of adjusted values ​​to be applied to the first set of parameters to generate the updated set of parameters.

[0198] Example 31 includes the method according to any one of Examples 26 to 30, and further includes: accessing a group of stored parameter sets corresponding to a plurality of contexts of the first model; and identifying a specific context among the plurality of contexts that has the closest similarity to the second context based on a similarity metric.

[0199] Example 32 includes the method according to Example 31, wherein the similarity measure is based on a set of extracted feature descriptors associated with the plurality of contexts and an extracted feature descriptor associated with the second context.

[0200] Example 33 includes the method according to Example 32, wherein the feature descriptor corresponds to one or more of the following: scene type, object type, location, features obtained via a large language model, or descriptors obtained via a large language model.

[0201] Example 34 includes the method according to any one of Examples 31 to 33, and further includes: selecting the stored set of parameters corresponding to the identified specific context as the first set of parameters.

[0202] Example 35 includes the method according to any one of Examples 31 to 34, and further includes: obtaining the first set of parameters by accessing a remote group of the parameter set via a communication network based on the failure of the closest similarity to meet the threshold similarity.

[0203] Example 36 includes the method according to Example 35, and further includes: selecting whether to access the remote group based at least in part on a timing criterion associated with updating the first model.

[0204] Example 37 includes the method according to any one of Examples 31 to 36, wherein: for each specific context among the plurality of contexts, the group of stored parameter sets further includes: a first resolution parameter set configured to enable inference of the specific context at a first resolution; a second resolution parameter set configured to enable inference of the specific context at a reduced resolution; and a feature descriptor of the specific context; the first set of parameters using the second resolution parameter set of the specific context; and an updated first model performing inference corresponding to the second context at the reduced resolution.

[0205] Example 38 includes the method according to any one of Examples 26 to 37, and further includes: after updating the first model based on the output of the second model, performing one or more training operations on the updated first model to enhance the inference accuracy of the updated first model for the second context.

[0206] Example 39 includes the method according to Example 38, wherein the one or more training operations are performed until the inference accuracy reaches an accuracy threshold.

[0207] Example 40 includes the method according to Example 38, and further includes alternating between parameter updates using the training operation and parameter updates using the second model until the inference accuracy reaches an accuracy threshold.

[0208] Example 41 includes the method according to any one of Examples 26 to 40, wherein the second model is configured to further generate the output based on input corresponding to the first context.

[0209] Example 42 includes the method according to any one of Examples 26 to 41, wherein the second model is configured to generate the output based on a difference measurement between the first context and the second context.

[0210] Example 43 includes a method according to any one of Examples 26 to 42, wherein the second model comprises: a parameter encoder configured to process a set of input parameters; a context encoder configured to process the inputs corresponding to a second context; a joint encoder configured to process the outputs of the parameter encoder and the context encoder; and a parameter decoder configured to process the outputs of the joint encoder to generate the outputs.

[0211] Example 44 includes a method according to any one of Examples 26 to 43, wherein: the first model includes a plurality of network layers; and for each of the plurality of network layers of the first model, the second model includes a corresponding instance of a parametric encoder, a joint encoder, and a parametric decoder, the parametric encoder, the joint encoder, and the parametric decoder being configured to generate an output associated with the network layer of the first model.

[0212] According to embodiment 45, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to: obtain a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; process the first set of parameters and inputs corresponding to a second context using the second model to generate an output of the second model; and update the first model based on the output of the second model to perform inference using the updated set of parameters.

[0213] Example 46 includes a non-transitory computer-readable medium according to Example 45, wherein the first context corresponds to a 2D or 3D representation of a first scene or a first 2D object, and wherein the second context corresponds to a 2D or 3D representation of a second scene or a second 2D object.

[0214] Example 47 includes a non-transitory computer-readable medium according to Example 45 or Example 46, wherein the first model corresponds to a neural radiation field (NeRF) model.

[0215] Example 48 includes a non-transitory computer-readable medium according to any one of Examples 45 to 47, wherein the output of generating the second model includes multiple iterations of inference performed at the second model.

[0216] Example 49 includes a non-transitory computer-readable medium according to any one of Examples 45 to 48, wherein the output of the second model includes the updated set of parameters or a set of adjusted values ​​to be applied to the first set of parameters to generate the updated set of parameters.

[0217] Example 50 includes a non-transitory computer-readable medium according to any one of Examples 45 to 49, wherein the instructions, when executed by the one or more processors, further enable the one or more processors to access a set of stored parameter sets corresponding to a plurality of contexts of the first model; and to identify a specific context among the plurality of contexts that has the closest similarity to the second context based on a similarity metric.

[0218] Example 51 includes a non-transitory computer-readable medium according to Example 50, wherein the similarity measure is based on a set of extracted feature descriptors associated with the plurality of contexts and an extracted feature descriptor associated with the second context.

[0219] Example 52 includes a non-transitory computer-readable medium according to Example 51, wherein the feature descriptor corresponds to one or more of the following: scene type, object type, location, features obtained via a large language model, or descriptors obtained via a large language model.

[0220] Example 53 includes a non-transitory computer-readable medium according to any one of Examples 50 to 52, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to select the stored set of parameters corresponding to the identified specific context as the first set of parameters.

[0221] Example 54 includes a non-transitory computer-readable medium according to any one of Examples 50 to 53, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to access a remote group of parameter sets via a communication network to obtain the first set of parameters based on the failure of the closest similarity to meet a threshold similarity.

[0222] Example 55 includes a non-transitory computer-readable medium according to Example 54, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to select whether to access the remote group based at least in part on a timing criterion associated with updating the first model.

[0223] Example 56 includes a non-transitory computer-readable medium according to any one of Examples 50 to 55, wherein: for each particular context among the plurality of contexts, the group of stored parameter sets further includes: a first resolution parameter set configured to enable inference of the particular context at a first resolution; a second resolution parameter set configured to enable inference of the particular context at a reduced resolution; and a feature descriptor of the particular context; a first set of parameters for which the second resolution parameter set of the particular context is used; and an updated first model performs inference corresponding to the second context at the reduced resolution.

[0224] Example 57 includes a non-transitory computer-readable medium according to any one of Examples 45 to 56, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform one or more training operations on the updated first model after updating the first model based on the output of the second model, to enhance the inference accuracy of the updated first model for the second context.

[0225] Example 58 includes a non-transitory computer-readable medium according to Example 57, wherein the one or more training operations are performed until the inference accuracy reaches an accuracy threshold.

[0226] Example 59 includes a non-transitory computer-readable medium according to Example 57, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to alternate between parameter updates using a training operation and parameter updates using the second model until the inference accuracy reaches an accuracy threshold.

[0227] Example 60 includes a non-transitory computer-readable medium according to any one of Examples 45 to 59, wherein the second model is configured to further generate the output based on input corresponding to the first context.

[0228] Example 61 includes a non-transitory computer-readable medium according to any one of Examples 45 to 60, wherein the second model is configured to generate the output based on a difference measurement between the first context and the second context.

[0229] Example 62 includes a non-transitory computer-readable medium according to any one of Examples 45 to 61, wherein the second model comprises: a parameter encoder configured to process a set of input parameters; a context encoder configured to process the inputs corresponding to a second context; a joint encoder configured to process the outputs of the parameter encoder and the context encoder; and a parameter decoder configured to process the outputs of the joint encoder to generate the outputs.

[0230] Example 63 includes a non-transitory computer-readable medium according to any one of Examples 45 to 61, wherein: the first model includes a plurality of network layers; and for each of the plurality of network layers of the first model, the second model includes a corresponding instance of a parametric encoder, a joint encoder, and a parametric decoder, the parametric encoder, the joint encoder, and the parametric decoder being configured to generate an output associated with the network layer of the first model.

[0231] According to embodiment 64, an apparatus includes components for obtaining a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; components for using the second model to process the first set of parameters and inputs corresponding to a second context to generate an output of the second model; and components for updating the first model based on the output of the second model to perform inference using the updated set of parameters.

[0232] Example 65 includes the apparatus according to Example 64, wherein the first context corresponds to a 2D or 3D representation of a first scene or a first 3D object, and wherein the second context corresponds to a 2D or 3D representation of a second scene or a second 3D object.

[0233] Example 66 includes the apparatus according to Example 64 or Example 65, wherein the first model corresponds to the neural radiation field (NeRF) model.

[0234] Example 67 includes an apparatus according to any one of Examples 64 to 66, wherein generating the output of the second model includes multiple iterations of inference performed at the second model.

[0235] Example 68 includes an apparatus according to any one of Examples 64 to 67, wherein the output of the second model includes the updated set of parameters or a set of adjusted values ​​to be applied to the first set of parameters to generate the updated set of parameters.

[0236] Example 69 includes the apparatus according to any one of Examples 64 to 68, and further includes: a set of stored parameter sets corresponding to a plurality of contexts of the first model; and a set of components for identifying a specific context among the plurality of contexts that has the closest similarity to the second context based on a similarity metric.

[0237] Example 70 includes the apparatus according to Example 69, wherein the similarity measure is based on a set of extracted feature descriptors associated with the plurality of contexts and an extracted feature descriptor associated with the second context.

[0238] Example 71 includes the apparatus according to Example 70, wherein the feature descriptor corresponds to one or more of the following: scene type, object type, location, features obtained via a large language model, or descriptors obtained via a large language model.

[0239] Example 72 includes the apparatus according to any one of Examples 69 to 71, and further includes: a component for selecting the stored set of parameters corresponding to the identified specific context as the first set of parameters.

[0240] Example 73 includes the apparatus according to any one of Examples 69 to 72, and further includes a component for accessing a remote group of the parameter set via a communication network to obtain the first set of parameters based on the failure of the closest similarity to meet the threshold similarity.

[0241] Example 74 includes the apparatus according to Example 73, and further includes components for selecting whether to access the remote group based at least in part on a timing criterion associated with updating the first model.

[0242] Example 75 includes an apparatus according to any one of Examples 69 to 74, wherein: for each particular context among the plurality of contexts, the group of stored parameter sets further includes: a first resolution parameter set configured to enable inference of the particular context at a first resolution; a second resolution parameter set configured to enable inference of the particular context at a reduced resolution; and a feature descriptor of the particular context; the first set of parameters for which the second resolution parameter set of the particular context is used; and an updated first model performs inference corresponding to the second context at the reduced resolution.

[0243] Example 76 includes the apparatus according to any one of Examples 64 to 75, and further includes: a component for performing one or more training operations on the updated first model after updating the first model based on the output of the second model to enhance the inference accuracy of the updated first model for the second context.

[0244] Example 77 includes the apparatus according to Example 76, wherein the one or more training operations are performed until the inference accuracy reaches an accuracy threshold.

[0245] Example 78 includes the apparatus according to Example 76, and further includes a component for alternating between parameter updates using the training operation and parameter updates using the second model until the inference accuracy reaches an accuracy threshold.

[0246] Example 79 includes an apparatus according to any one of Examples 64 to 78, wherein the second model is configured to further generate the output based on input corresponding to the first context.

[0247] Example 80 includes an apparatus according to any one of Examples 64 to 79, wherein the second model is configured to generate the output based on a difference measurement between the first context and the second context.

[0248] Example 81 includes an apparatus according to any one of Examples 64 to 80, wherein the second model comprises: a parameter encoder configured to process a set of input parameters; a context encoder configured to process the inputs corresponding to a second context; a joint encoder configured to process the outputs of the parameter encoder and the context encoder; and a parameter decoder configured to process the outputs of the joint encoder to generate the outputs.

[0249] Example 82 includes an apparatus according to any one of Examples 64 to 81, wherein: the first model includes a plurality of network layers; and for each of the plurality of network layers of the first model, the second model includes a corresponding instance of a parametric encoder, a joint encoder, and a parametric decoder, the parametric encoder, the joint encoder, and the parametric decoder being configured to generate an output associated with the network layer of the first model.

[0250] Those skilled in the art will also understand that the various exemplary logic blocks, configurations, modules, circuits, and algorithm steps described in connection with the specific embodiments disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of both. The various exemplary components, blocks, configurations, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or processor-executable instructions depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, and such implementation decisions shall not be construed as departing from the scope of this disclosure.

[0251] The steps of the methods or algorithms described in conjunction with the specific embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, compressed optical disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. Alternatively, the processor and storage medium may reside as discrete components in a computing device or a user terminal.

[0252] The prior description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but should be granted the broadest scope that may be consistent with the principles and novel features as defined by the following claims.

Claims

1. An apparatus, the apparatus comprising: A memory configured to store a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; and One or more processors, said one or more processors being configured to: The second model is used to process the first set of parameters and the input corresponding to the second context to generate the output of the second model; as well as The first model is updated based on the output of the second model to perform inference using the updated set of parameters.

2. The device according to claim 1, wherein the first context corresponds to a 2D or 3D representation of a first scene or a first 3D object, and wherein the second context corresponds to a 2D or 3D representation of a second scene or a second 3D object.

3. The device according to claim 1, wherein the first model corresponds to the neural radiation field (NeRF) model.

4. The device of claim 1, wherein generating the output of the second model includes multiple iterations of inference performed at the second model.

5. The device of claim 1, wherein the output of the second model comprises the updated set of parameters or a set of adjustment values ​​to be applied to the first set of parameters to generate the updated set of parameters.

6. The device of claim 1, wherein the one or more processors are configured to: Access a group of stored parameter sets corresponding to multiple contexts of the first model; and The specific context that has the closest similarity to the second context among the plurality of contexts is identified based on a similarity measure.

7. The device of claim 6, wherein the similarity measure is based on a set of extracted feature descriptors associated with the plurality of contexts and an extracted feature descriptor associated with the second context.

8. The device of claim 7, wherein the feature descriptor corresponds to one or more of the following: scene type, object type, location, feature obtained via a large language model, or descriptor obtained via a large language model.

9. The device of claim 6, wherein the one or more processors are configured to select the stored set of parameters corresponding to the identified specific context as the first set of parameters.

10. The device of claim 6, wherein the group of stored parameter sets is stored in the memory, and wherein one or more processors are configured to access a remote group of parameter sets via a communication network to obtain the first set of parameters based on the failure of the closest similarity to meet a threshold similarity.

11. The device of claim 10, wherein the one or more processors are configured to select whether to access the remote group based at least in part on a timing criterion associated with updating the first model.

12. The device according to claim 6, wherein: For each specific context among the plurality of contexts, the group of stored parameter sets further includes: A first resolution parameter set, configured to enable reasoning of the specific context at a first resolution; A second resolution parameter set, configured to enable reasoning of the specific context at a reduced resolution; and The feature descriptor of the specific context; The one or more processors are configured to use the second resolution parameter set of the specific context as the first set of parameters; and The updated first model is configured to perform inference corresponding to the second context at the reduced resolution.

13. The device of claim 1, wherein after updating the first model based on the output of the second model, the one or more processors are further configured to perform one or more training operations on the updated first model to enhance the inference accuracy of the updated first model for the second context.

14. The device of claim 13, wherein the one or more training operations are performed until the inference accuracy reaches an accuracy threshold.

15. The apparatus of claim 13, wherein the one or more processors are configured to alternate between parameter updates using the training operation and parameter updates using the second model until the inference accuracy reaches an accuracy threshold.

16. The device of claim 1, wherein the second model is configured to generate the output based on a difference measurement between the first context and the second context.

17. The device according to claim 1, wherein the second model comprises: A parameter encoder, configured to process a set of input parameters; A context encoder, configured to process the input corresponding to a second context; A combined encoder configured to process the outputs of the parametric encoder and the context encoder; and A parameter decoder, configured to process the output of the joint encoder to generate the output.

18. The device according to claim 1, wherein: The first model includes multiple network layers; and For each of the plurality of network layers in the first model, the second model includes a corresponding instance of a parametric encoder, a joint encoder, and a parametric decoder, which are configured to generate an output associated with the network layer of the first model.

19. The device of claim 1, further comprising a camera configured to generate context data associated with the second context.

20. The device of claim 1, further comprising a modem coupled to the one or more processors and configured to receive from a remote device the first model, the second model, the first set of parameters, or a combination thereof.

21. The apparatus of claim 1, further comprising a display device configured to display image data generated using the updated first model.

22. A method, the method comprising: Obtain a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; The second model is used to process the first set of parameters and the input corresponding to the second context to generate the output of the second model; as well as The first model is updated based on the output of the second model to perform inference using the updated set of parameters.

23. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: Obtain a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; The second model is used to process the first set of parameters and the input corresponding to the second context to generate the output of the second model; as well as The first model is updated based on the output of the second model to perform inference using the updated set of parameters.

24. An apparatus comprising: Components for obtaining a first model and a second model, wherein the first model is configured to perform inference based on a first set of parameters corresponding to a first context; A component for using the second model to process the first set of parameters and the input corresponding to the second context to generate the output of the second model; as well as A component for updating the first model based on the output of the second model to perform inference using the updated set of parameters.

Images