Training a decision module model in a split-point range architecture

Abstract

The determination module determines a split point for the deep neural network model. One device implements the DNN model up to the split point. The first device sends intermediate data to a second device that implements the remainder of the DNN model from the split point onwards. The second device sends further data back to the first device. In one embodiment, the first and second devices are a user device and an edge server. The DNN model can be trained before execution or can be trained through execution. Asynchronous or synchronous metric collection is implemented.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of European Patent Application No. 23315025.9, filed February 10, 2023, which is incorporated herein by reference in its entirety.

[0002] Technical Field At least one of the present embodiments generally relates to a method or apparatus for implementing an efficient decision module in a split point range architecture. [Background technology]

[0003] background Partitioned computation or distributed inference aims to achieve several goals:

[0004] Computational load is distributed to mobile devices and edge / cloud servers. Energy consumption is shared between mobile devices and remote servers. Privacy is protected by transmitting partially processed data instead of raw data. User task specific parts can be processed on the mobile side, while generic parts can be done on the server side.

[0005] The DNN model is first partitioned into multiple parts according to current system environment factors such as network bandwidth, device resources (memory, power, processing units), and edge server workload.

[0006] The UE device runs the DNN model up to a certain layer and sends the intermediate data to the edge server, which runs the remaining layers and sends the prediction results back to the device. Conversely, the server can run the first layer and send the intermediate data to the mobile for processing the remaining layers. Summary of the Invention

[0007] overview At least one of the present embodiments generally relates to a method or apparatus in the context of a split point range architecture. Specifically, one objective of the described embodiments is to enable smart and automatic changing of split points, and therefore subparts executed on the UE and network parts.

[0008] According to a first aspect, a method is provided, the method including: executing at least a portion of a deep neural network (DNN) up to a particular layer on a first device, transmitting intermediate data from the first device to a second device, executing remaining layers of the DNN on the second device, and transmitting information from the second device to the first device.

[0009] According to another aspect, an apparatus is provided, the apparatus including a processor, the processor may be configured to implement the general aspects by performing any of the described methods.

[0010] According to another aspect, an apparatus is provided that is configured to execute at least a portion of a deep neural network (DNN) up to a particular layer on a first device, transmit intermediate data from the first device to a second device, execute remaining layers of the DNN on the second device, and transmit information from the second device to the first device.

[0011] According to another aspect, a determination module is provided for determining at least one split point indicative of a particular layer of a deep neural network model for execution across a plurality of devices.

[0012] According to another aspect, there is provided an apparatus configured to receive intermediate data from another apparatus and execute a portion of a deep neural network model, the portion being indicated by at least one split point input.

[0013] According to another general aspect of at least one embodiment, there is provided an apparatus including a device according to any of the embodiments and at least one of (i) an antenna configured to receive a signal, the signal including a video block, (ii) a band limiter configured to limit the received signal to a frequency band including the video block, or (iii) a display configured to display an output representing the video block.

[0014] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0015] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any of the described encoding embodiments or variations.

[0016] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0017] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0018] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings.

[0019] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content that includes instructions for performing either an encoding or decoding method. [Brief explanation of the drawings]

[0020] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] Shows AI / ML model split where the UE runs the first part of the model (left) and the edge / cloud server runs the second part of the model (right). [Figure 2] 1 shows a block diagram of a local decision module. [Figure 3] 1 shows an example of I / O of a decision module. [Figure 4] This provides an overview of reinforcement learning. [Figure 5] An example of a decision module based on reinforcement learning (evaluation on the UE side) is shown below. [Figure 6] 1 shows an example of a variant of the decision module based on reinforcement learning (evaluation on the UE side). [Figure 7] An example of a decision module based on reinforcement learning (server-side evaluation) is shown below. [Figure 8] 1 shows an example of model training for a decision module. [Figure 9] 1 illustrates one embodiment of context metric transmission: UE data source. [Figure 10] Illustrates one embodiment of context metric transmission: network data source. [Figure 11] Illustrates one embodiment of context metric transmission: UE data source with initial processing in the network. [Figure 12] 1 illustrates an embodiment of a 3GPP sequence flow diagram (synchronous exchange). [Figure 13] 1 illustrates an embodiment of a 3GPP sequence flow diagram (asynchronous exchange, machine learning solution). [Figure 14] 1 illustrates an embodiment of a 3GPP sequence flow diagram (asynchronous switching, reinforcement learning, UE-side reward solution). [Figure 15] 1 illustrates one embodiment of a 3GPP sequence flow diagram (asynchronous switching, reinforcement learning, network-side reward solution). [Figure 16] 1 illustrates one embodiment of a method for executing a DNN model using the described embodiments. [Figure 17] 10 illustrates another embodiment of a method for executing at least a portion of a DNN model in a device using the described embodiments. [Figure 18] 1 illustrates one embodiment of an apparatus for executing at least a portion of a DNN model using the described embodiments. [Figure 19] Denotes a standard general-purpose video compression method. [Figure 20] A standard general-purpose video decompression method is shown. [Figure 21] 1 illustrates a processor-based system for encoding / decoding under the described general aspects. DETAILED DESCRIPTION OF THE INVENTION

[0021] Detailed Description Distributed inference consists of splitting the DNN model into at least two subparts, which are distributed towards network infrastructure nodes: UEs, edge servers, and cloud servers, as shown in Figure 1.

[0022] If a system architecture is defined that allows for a smart and automatic change of the division point and therefore a smart and automatic change of the subparts performed on the UE side and on the network side.

[0023] Messages between the UE and the network that allow this change are also defined.

[0024] In the defined architecture, the decision to change the split point relies on a module called the "local decision module". However, this "local decision module" is described as a black box, i.e., its inputs and outputs are described but no implementation details are provided here, as there are several ways in which split point changes can be communicated between the UE side and the network side.

[0025] The following embodiment addresses the implementation of a local decision module, for example, for the architecture of FIG.

[0026] The technical problem to be solved to obtain smart decisions by the local decision module is as follows:

[0027] The number of parameters influencing the decision is potentially very large and time-varying, meaning that solutions based on static rules established by experts are complex to set up and give inefficient results as soon as the system encounters new environments related to UE resources, input data sources, or network resources.

[0028] The number of models to manage can become very large, making the possibility of expert rule definition too heavy a task.

[0029] There are multiple optimization options, and the goals can be optimization of delay, optimization of energy consumption on the UE side, optimization of energy consumption on the network side, optimization of energy consumption on both sides, etc.

[0030] The local decision module must consider the effective quality of its responses and be able to improve them over time.

[0031] The local decision module may encounter new models or new parameter values ​​that it has not seen before.

[0032] This solution relies on the following ideas:

[0033] Implementing an efficient decision module within a split point range architecture using live learning (reinforcement learning) or offline learning (machine learning).

[0034] Propose the creation of metrics that will enable the training of the above machine learning and reinforcement learning solutions.

[0035] We propose two ways to exchange these metrics: synchronous and asynchronous.

[0036] The decision module takes the following inputs: · The AI / ML models used by the application. A set of expected / possible split points for this AI / ML model. UE local information (device 1, e.g. UE) based on which the first part of the model is inferred: Available memory oCPU / GPU / TPU (processing unit) used Edge / network local information based on which the second part of the module is inferred (device 2, e.g. edge server): Available memory oCPU / GPU / TPU (processing unit) used Input data characteristics oResolution (width, height) o Dimension (8-bit, 16-bit, 32-bit, ...) Frame rate (25FPS, 30FPS, ...) Optimization options: Minimizing oUE energy costs o Minimizing network energy costs Minimizing total energy costs Minimizing delays o Maximize accuracy Best compromise between energy cost / delay / prediction accuracy

[0037] The output is the best split point and / or expected performance per split point, including: Expected delay Expected energy cost of the UE Expected energy costs of the network Total expected energy costs -Accuracy of AI / ML model prediction results

[0038] This is a problem that can be solved by machine learning algorithms, for which there are two important characteristics to consider: 1) when the learning occurs, live or offline, and 2) what type of learning is used, supervised or reinforcement learning.

[0039] Live or offline learning Offline learning or live learning is not opposed and may depend on the application provider or network operator.

[0040] In the former case, it is an AI / ML model that is pre-trained and then deployed, while in the latter case, it is an algorithm that learns by experiencing the environment through trial and error.

[0041] In the former case, an algorithm that immediately gives relevant results, in the latter, an algorithm that may take time to learn.

[0042] In the former case, the algorithm cannot adapt without being updated, while in the latter case, the algorithm continues to learn. Even in live learning, the initial model is often first trained offline, deployed, and then finished training live.

[0043] Reinforcement learning or supervised learning Reinforcement learning and supervised learning are two machine learning techniques considered for the purposes of this disclosure. Both can be trained live or offline.

[0044] Therefore, unsupervised learning is rejected or outside the scope of this disclosure, and indeed the goal of the decision module is not to cluster the data or to find any category of data, but rather to make a decision from a set of input variables as described below.

[0045] In either case, the learning algorithm can be deployed on the UE or on the network.

[0046] Supervised learning is based on ground truth. The learning phase consists of a mapping between input sample data and this ground truth. The result of this learning phase is a model. This model can be based on a neural network architecture or on classic machine learning algorithms such as decision trees, random forests, etc. Once deployed on a device, e.g., a UE, the model can be operated by an application. The application feeds this AI / ML model with sample data to be inferred, and the model gives a prediction result. A decision module embedding such a model can thus provide predicted segmentation points, i.e., those that best match the input sample data.

[0047] Additionally, deep reinforcement learning is well suited to personalizing the selection of segmentation points. Indeed, the selection of segmentation points may vary from one UE to another, from one user to another, from one location to another, from one time period to another, etc. For a given state, if the action of the segmentation point results in a good delay (e.g., a reward can be applied). This mechanism is reproducible for energy.

[0048] Rewards depend on the application: · Online gaming requires very low latency. Applications that rely on file transfers require high bandwidth. Some services require a certain amount of bandwidth, even if the bandwidth is low, i.e. there is no jitter.

[0049] Live Reinforcement Learning Variants The principle of reinforcement learning can be seen in Figure 4.

[0050] One difference compared to other systems is that reinforcement learning evaluates the state of the system at each action and calculates a reward that reflects the quality of the current situation with respect to the end goal. This reward is used by the agent's training algorithm, i.e., the agent tries to maximize future rewards. Therefore, calculating this reward requires a quantitative evaluation of the system's performance based on the agent's choices. How to calculate this reward from performance metrics is a design decision. The described embodiment describes several possibilities.

[0051] In the context of this disclosure, the agent corresponds to the main function of the local decision module. Furthermore, this reward should also be based on the user's preferences to reflect the user's intentions.

[0052] The reward is tied to the objective of the system because it is the function the agent strives to maximize. Therefore, it is logical to construct this reward function from a set of metrics that quantify the performance of the system. These metrics, which can be a combination of UI context metrics and network context metrics, must be evaluated against the optimization goals of the system. Based on the optimization goals listed above, the context metrics can be the following: UE energy cost, network energy cost, delay, and accuracy.

[0053] To accommodate different optimization goals, the reward can balance different metrics. For example, the reward could be: R=-α1UE_energy_cost-α2Network_energy_cost-α3Delay+α4Accuracy where each α i is a positive real number that quantifies the importance of each optimization goal.

[0054] The agent's training algorithm trains a decision function. The inputs of this decision function are called the state and are computed from a set of inputs of the local decision module defined above. For example, the state can be a concatenation of all inputs, a subset of those inputs, or any transformation of those inputs or a subset of inputs, which may include normalization, scaling, outlier removal, and / or processing by a neural network. The state can also consist of a concatenation or summary of the decision module's past inputs.

[0055] The output of the decision function is the agent's output, i.e., an action that is the split point of the AI / ML model used for split inference.

[0056] The agent can use any of the off-the-shelf reinforcement learning algorithms such as Q-learning, SARSA, UCB, deep Q-learning, PPO, or RAINBOW.

[0057] The evaluation of the system can be done on the UE side or on the server side.

[0058] When evaluating the system on the UE side, the decision module may look like Figure 5. In this embodiment, the reward is calculated by a separate module called "Evaluate System." This module takes as input the values ​​needed to select the optimization and calculate the reward. These values ​​come from the UE device or the network and are denoted in the figure by the term "context metrics." The output of this module is the reward. This reward is input to the local decision module, where the agent resides, in addition to all the inputs listed above. The output of the decision module is the split point (=action) used by the AI / ML model partitioning module.

[0059] Figure 6 shows a different embodiment. In the case of system evaluation on the UE side, the system evaluation (=reward calculation) can also be located in the local decision module itself. Thus, the input of the module that evaluates the system becomes the input of the decision module, and the reward does not leave the decision module. These two embodiments have different advantages. When the reward module is located outside the decision module, the system is more modular, since these two modules can be developed independently by different companies. When the system evaluation is located within the agent, the system architecture is more versatile, since it can also be used for supervised learning.

[0060] The evaluation of the system can also be located on the network side. Figure 7 shows such a possible embodiment.

[0061] An interest of such a solution may be to generalize the evaluation based on information from several UEs, taking advantage of the larger resources available on the network side compared to the UEs. In addition to moving the evaluation of the system module from the device to the network, there is one difference in the information sent by the UE to the network: it must contain the optimization choices so that it can be used by the environmental evaluation module. Sending network context metrics to the device is optional, but may be useful for training the decision module. For the evaluation of the system on the network side, the decision module looks like Figure 7.

[0062] Offline Supervised Training - Best Split The training inputs and outputs are the same as defined above.

[0063] The goals of the loss function during training are to minimize the expected delay, minimize the energy cost, and maximize the accuracy of the prediction results.

[0064] A formula that represents the best compromise between energy cost / delay / prediction accuracy may be: Let BC be termed the "best compromise": BC = α1UE_energy_cost + α2Network_energy_cost + α3Delay - α4Accuracy however

number

[0065] Parameter α i can be fixed or can be learned.

[0066] Training is done on the server side by a specific module "Module Training".

[0067] This module can achieve training by using data collected by the module "Global Metrics Manager" (see next section).

[0068] Model training can be triggered by various thresholds such as time (model training can be activated at regular intervals), metric size / number (size or number of new collected context metrics that have not yet been integrated into the decision module model), and format changes (updating input data format or updating output data format).

[0069] Once the model is trained, it is communicated to the UE by any suitable means.

[0070] The choice of the moment to transmit the model can be a network operator decision, triggered by some threshold such as the percentage improvement of the new model or the number of updates since the last communication, etc. The model can also be transmitted based on the request of the UE.

[0071] Data collection To train the model efficiently, delay and energy costs are collected from both the UE and the network, and this information can be used to refine the model once it is deployed.

[0072] The collection of this information can be done synchronously or asynchronously.

[0073] For synchronous collection, two messages need to be modified in the exchange between the UE and the network to also carry the "context metrics": the first from the UE to the network, which already carries the segmentation points and intermediate data to be used, and the second from the network to the UE, which carries the inference results.

[0074] In the first message from the UE to the network, these context metrics are the delay in processing the partitioned model M1, the energy used in processing the partitioned model M1, the battery level of the UE, the memory used at the UE level, and the CPU load sampled at the start of processing and periodically (i.e. every ms) during all processing.

[0075] In the second message from the network to the UE, these context metrics are the delay of processing the partition model M2, the energy used to process the partition model M2, the memory used at the network level, or the CPU load sampled at the start of processing and periodically (i.e. every ms) during all processing. In asynchronous collection, the context metrics are the same as described in these two above messages, but this collection relies on a different mechanism.

[0076] In each message exchanged between the UE and the network (a "Split Point + Intermediate Data" message), the context metrics are stored in a local database hosted by the module "Local Metrics Manager".

[0077] At regular intervals, some or all of the metrics are sent to a network-side hosted module called the "Global Metrics Manager."

[0078] At regular intervals or triggered by some threshold (see section above), the Global Metrics Manager has the ability to train models, Local Decision Models.

[0079] The two approaches mentioned above are among the best solutions for building a split point determination module. Nevertheless, other options are possible. Some examples are listed below.

[0080] Offline reinforcement learning techniques are possible: such models can be trained from a database of observations (decision module inputs, split points, and UE / system context metrics) and / or using a simulator, or tested using techniques such as bandit algorithms or off-policy or batch reinforcement learning.

[0081] Online supervised learning approaches are possible to predict the performance of a given partition. Such training algorithms observe data points generated live as the system is used, and train or fine-tune an initial model.

[0082] Supervised learning models that directly predict split points typically cannot be trained live because the optimal split points are unknown. Training such a model is only possible by submitting the model to our server, which generates training data by extensively testing all split points, or by first training a supervised model as described in the previous bullet point, using that model to estimate the best split points, and using that estimate as the label.

[0083] Federated learning (distributed learning across multiple devices) can also be used with any live method, including reinforcement learning.

[0084] message Two types of messages can be used to collect and exchange metrics:

[0085] Synchronization messages based on existing messages between the UE and the server that are responsible for communicating segmentation points and intermediate data ("piggybacking messages"), to which a part containing metrics is added.

[0086] Asynchronous messages: Messages carrying metrics can be exchanged between the UE and the server periodically according to certain rules.

[0087] These two types of messages rely on a new structure that contains metrics.

[0088] Metric Clapper For example, in the structure for all combinations, the sender can be a UE or a server, and the first part of the model M1 may be running on the client or on the server: Struct ContextualMetrics { Int Model_ReflD; #Model identifier Int SplitPointID; #SplitPoint Bit Source; # 0=inference on UE; 1=inference on server Int SourcelD; #UE ID or server ID Bit SplitPart; # 0 = inference for the first subset M1; # 1 = Inference on the second subset M2 Int InferenceLatencyForSubsetModel; #Inference time for a "subset" (i.e., M1 or M2) of the model on the source (i.e., UE or server) Int EstimatedEnergyCostsForThisInference; #Estimated or measured energy consumption for this inference (UE for uplink, server for downlink) Int MemoryAvailableBeforeInference; #UE or server side Int MemoryAvailableDuringlnference; #UE or server side Int CPULoadBeforeInference; #UE or server side Int CPULoadDuringlnference; #UE or server side Int GPULoadBeforeInference; #UE or server side Int GPULoadDuringlnference; #UE or server side Int NumberOfApplicationsRunningBeforelnference; #UE specific Int NumberOfApplicationsRunningAfterlnference; #UE specific Array input_data_characteristics[]; #width, height, nb_bits (8 / 10 / 12), Frame Rate Int optimization choice; #Optimization choice (0: delay, 1: minimize UE energy, 2: minimize network consumption, 3: minimize both UE and network energy consumption, 4: best compromise) }

[0089] Synchronous Messages Existing messages from previous approaches can be used with the added metric information.

[0090] Transporting intermediate data The intermediate data is wrapped in a structure that indicates the origin of the data. The structure is as follows: Struct lntermediateDataWrapper { Int Model_ReflD; #Model identifier bit ChangeOfSplitPoint; # 1 if the split point has changed from the previous message Int SplitPointID; #Split point used by UE Int NextSplitpointID; #Next split point used by UE (optional) Int IntermediateDataLength; #Length of intermediate data Dim IntermediateDataDim; #Intermediate data array dimension byte EncodingMethod; #Indicates whether the data is compressed and by which algorithm int SequenceNumber; #Sequential identifier of input data double TimeStamp; #Timestamp of intermediate data Byte lntermediateData[] #Intermediate data int UEContextualMetricsLength[] #UE contextual metrics length ContextualMetrics UEContextualMetricsData[] #UE contextual metrics }

[0091] Asynchronous Messages These messages are exchanged between the UE module "Local Metrics Manager" module and the server module "Global Metrics Manager" module. The structure is as follows: Struct AsynchronousContextualMetricsMessages { Int UE_id ; #UE identifier Int Server_id; #Server identifier Int metrics_ id ; #metric identifier TimeStamp timestamp ; #metric timestamp (date and time) Contextual Metrics metrics; #metrics }

[0092] Figure 12 shows the call flow based on prior work for synchronous exchange of metrics.

[0093] Figure 12 presents a sequence diagram with the current 3GPP SA4 AI / ML components.

[0094] The steps of the general aspect described here are step 6 completed to integrate the exchange of UE context metrics, step 6 completed to integrate the exchange of optimization selections if reinforcement learning is used, step 7 completed to integrate the exchange of network context metrics, and step 7 completed to integrate the exchange of rewards if reinforcement learning is used.

[0095] Steps 1 and 2 are AI / ML application requirements for dynamic partitioning configuration scope for network services. The scope proposed in the previous section considers that an AI / ML model (e.g., "M") can be dynamically partitioned (e.g., into M0 and M1) according to different partitioning points (e.g., A, B, C, D, E) and thereby partially executed by the UE inference or by a subset of the network inference. For example, here, partitioning points B and C can both be considered. Following steps 1 and 2, the UE inference can dynamically begin processing captured sequence frame 1 up to layer A, B, or C.

[0096] In step 3, the AI / ML application calculates its internal resource and application requirements and initializes the AI / ML UE subset including split points A and B.

[0097] In steps 4 and 5, the inference engine processes the first sequence frame 1 up to segmentation point B.

[0098] In step 6, once frame 1 is executed, the UE delivers the output data to the network inference via an intermediate data forwarding function. The forwarded data includes the data payload and metadata information useful for identifying how to process the received data, such as an indication of at least the current segmentation point.

[0099] In these embodiments, the metadata information also includes contextual metric information, as described in the section on metric clappers.

[0100] In these embodiments, if reinforcement learning is used, the metadata information also includes the optimization choice, as described in the section on metric clappers.

[0101] Step 7: Based on the ongoing results, e.g., a streaming overlay of the current input video, the inference engine sends back the final results of the AI / ML model. In these embodiments, the metadata information also includes network context metric information, as described in the metric clapper in Section 4.6.1.

[0102] In these embodiments, when reinforcement learning is used and when the evaluation of the system is on the network side, the metadata information also includes context metric information, as described in the metric clapper in Section 4.6.1.

[0103] In step 8, the UE takes advantage of its flexibility to process different segmentation points in response to various internal triggers (e.g., reduced processing power) and locally updates the UE inference with a new segmentation point from B to A for sequence frame 2 onwards.

[0104] Steps 9, 10, and 11 follow a similar process to steps 4, 5, and 6, except that the UE delivers the data at segmentation point A instead of segmentation point B.

[0105] In step 12, the network inference engine calculates the split point information and changes the input accordingly to split point B. In another embodiment, the UE may trigger the network inference engine when it receives an update to the AI / ML application.

[0106] Step 13 is similar to step 7. This shows the processing of a seamless partition model.

[0107] Asynchronous Exchange In asynchronous collection, the context metrics are the same as those described in these two messages above, but the collection relies on a different mechanism.

[0108] For each message exchanged between the UE and the network ("Split Point + Intermediate Data" message), the context metrics are stored in a local database hosted by the module "Local Metric Manager". In the case of reinforcement learning, the optimization choices are also stored.

[0109] At regular intervals, some or all of the metrics are sent to a network-side hosted module called the "Global Metrics Manager."

[0110] In the case of machine learning based solutions, a "global metrics manager" has the ability to train models, "local decision models", at regular intervals or triggered by some threshold (see section above).

[0111] In the case of a reinforcement learning-based solution, a "System RL Evaluation" module in the network communicates rewards to the UE agent at regular intervals or triggered by some threshold (see section above). One embodiment of a method 1600 for executing a trained DNN model is shown in FIG. 16. The method begins at start block 1601 and proceeds to block 1610 for executing at least a portion of a deep neural network (DNN) up to a particular layer on a first device. Control proceeds from block 1610 to block 1620 for sending intermediate data from the first device to a second device. Control proceeds from block 1620 to block 1630 for executing the remaining layers of the DNN on the second device. Control proceeds from block 1630 to block 1640 for sending information from the second device to the first device.

[0112] One embodiment of a method 1700 for executing a portion of a DNN model is shown in Figure 17. The method begins at start block 1701 and proceeds to block 1710 for receiving intermediate data and segmentation point information. From block 1710, control proceeds to block 1720 for executing a portion of the DNN model based on the segmentation point information. From block 1720, control proceeds to block 1730 for transmitting the execution results to another device.

[0113] 18 shows one embodiment of an apparatus 1000 for performing any of the methods described above. The apparatus includes a processor 1010, which may be interconnected by at least one port to a memory 1020. Both the processor 1010 and the memory 1020 may also have one or more additional interconnections to external connections.

[0114] The processor 1810 is also configured to insert information into a bitstream or data, or to receive information in a bitstream or data, and to execute at least a portion of a DNN model using the methods described above.

[0115] The embodiments described herein include various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are specifically described and often described in a manner that may appear limiting, at least to illustrate their individual characteristics. However, this is for purposes of clarity of description and does not limit the application or scope of those aspects. Indeed, all of the various aspects can be mixed and matched to produce further aspects. Furthermore, aspects can also be mixed and matched with aspects described in prior applications.

[0116] The aspects described and discussed herein can be implemented in many different ways. While Figures 19, 20, and 21 illustrate some embodiments, other embodiments are contemplated, and the discussion of Figures 19, 20, and 21 is not intended to limit the scope of implementations. At least one of the aspects relates generally to encoding and decoding video, and at least one other aspect relates generally to transmitting generated or encoded bitstreams. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium storing a bitstream generated according to any of the described methods.

[0117] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" or "reconstruction" is used on the decoder side.

[0118] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the method to operate properly, the order and / or use of specific steps and / or actions can be modified or combined. In addition, terms such as “first,” “second,” etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as “first decoding” and “second decoding.” The use of such terms does not imply any ordering of the modified operations unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, but could, for example, occur before, during, or within an overlapping period with the second decoding.

[0119] Various methods and other aspects described herein can be used to modify modules, such as the intra-prediction, entropy coding, and / or decoding modules (160, 360, 145, 330) of the video encoder 100 and decoder 200 shown in Figures 19 and 20. Furthermore, aspects of the present invention are not limited to VVC or HEVC, but can be applied, for example, to other standards and recommendations, existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, aspects described herein can be used individually or in combination.

[0120] Various numerical values ​​are used in this application, and the specific values ​​are for illustrative purposes only and the described aspects are not limited to those specific values.

[0121] 19 shows an encoder 100. Variations of this encoder 100 are possible, but the encoder 100 is described below for the sake of clarity without listing all contemplated variations.

[0122] Before being encoded, the video sequence may be subjected to pre-encoding processing (101), for example, applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or remapping the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and may be added to the bitstream.

[0123] Within encoder 100, a picture is coded by the elements of the encoder as described below. The picture to be coded is partitioned (102) and processed, e.g., by CU. Each unit is coded, e.g., using intra-mode or inter-mode. If the unit is coded in intra-mode, the encoder performs intra-prediction (160). In inter-mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use intra-mode or inter-mode to code the unit and indicates the intra / inter decision, e.g., with a prediction mode flag. A prediction residual is calculated, e.g., by subtracting (110) the predicted block from the original image block.

[0124] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying the transform or quantization processes.

[0125] The encoder decodes the coded block to provide a reference for further prediction. It dequantizes (140) and inverse transforms (150) the quantized transform coefficients to decode the prediction residual. It combines (155) the decoded prediction residual with the predicted block to reconstruct an image block. It applies an in-loop filter (165) to the reconstructed picture, for example, to perform deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. It stores the filtered image in a reference picture buffer (180).

[0126] Figure 20 shows a block diagram of a video decoder 200, where the bitstream is decoded by elements of the decoder as described below. The video decoder 200 generally performs a decoding path that is the inverse of the encoding path described in Figure 19. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0127] Specifically, the decoder input includes a video bitstream, such as might be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coding information. Picture partitioning information indicates how the picture is partitioned. The decoder can then separate the picture according to the decoded picture partitioning information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with a predicted block (255) to reconstruct an image block. The predicted block can result from intra-prediction (260) or motion-compensated prediction (i.e., inter-prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0128] The decoded picture can be further subjected to post-decoding processing (285), such as an inverse color transformation (e.g., YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that reverses the remapping process performed in the pre-encoding processing (101). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0129] FIG. 21 illustrates a block diagram of an example system in which various aspects and embodiments can be implemented. System 1000 can be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. The elements of system 1000 can be embodied singly or in combination within a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0130] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein to implement various aspects described herein, for example. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., volatile and / or nonvolatile memory devices). The system 1000 includes storage 1040, which may include nonvolatile and / or volatile memory including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. The storage 1040 may include, by way of non-limiting example, internal storage, additional storage (including removable and non-removable storage), and / or network-accessible storage.

[0131] System 1000 includes an encoder / decoder module 1030 configured to process data, e.g., to provide encoded video or decoded video, which may include its own processor and memory. Encoder / decoder module 1030 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0132] Program code that is loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in storage 1040 and subsequently loaded onto memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, memory 1020, storage 1040, and encoder / decoder module 1030 may store one or more of a variety of items while performing the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results resulting from the processing of equations, formulas, operations, and arithmetic logic.

[0133] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing unit (e.g., the processing unit may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage 1040, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used, for example, to store the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0134] Input to the elements of system 1000 may be provided by various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) an RF section that receives RF (radio frequency) signals transmitted wirelessly, for example by a broadcaster, (ii) a component (COMP) input (or a set of COMP inputs), (iii) a universal serial bus (USB) input, and / or (iv) a high-definition multimedia interface (HDMI) input. Other examples not shown in FIG. 21 include composite video.

[0135] In various embodiments, the input devices of block 1130 have associated individual input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band-limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In some set-top box embodiments, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of them, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0136] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing, e.g., Reed-Solomon error correction, may be implemented as needed, for example, in a separate input processing IC or in processor 1010. Similarly, aspects of USB or HDMI interface processing may be implemented as needed in a separate interface IC or in processor 1010. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030 operating in combination with memory and storage elements, to process the data stream as needed for presentation on an output device.

[0137] The various elements of system 1000 may be provided within a unitary housing in which the various elements are interconnected and may transmit data therebetween using suitable connection configurations, such as internal buses known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0138] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0139] In various embodiments, data is streamed or provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received over communication channel 1060 and communication interface 1050, which are adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide streamed data to system 1000 using a set-top box that delivers data over the HDMI connection of input block 1130. Still other embodiments provide streamed data to system 1000 using the RF connection of input block 1030. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0140] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripherals 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be for a television, a tablet, a laptop, a cell phone, or another device. Display 1100 can be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). In various example embodiments, other peripherals 1120 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR in both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 1120 to provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0141] In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 1000 via dedicated connections through individual interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 using communication channel 1060 via communication interface 1050. Display 1100 and speakers 1110 may be integrated into a single unit with other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0142] For example, if the RF portion of input 1130 is part of a separate set-top box, display 1100 and speakers 1110 can alternatively be separate from one or more of the other components. In various embodiments in which display 1100 and speakers 1110 are external components, the output signal can be provided by a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0143] The embodiments may be performed by computer software implemented by the processor 1010, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, including, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0144] Various implementations include decoding. As used herein, "decoding" may encompass all or part of the processes performed on a received encoded sequence to produce a final output suitable for, for example, display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or instead include processes performed by the decoders of the various implementations described herein.

[0145] As a further example, in some embodiments, "decoding" refers only to entropy decoding, in other embodiments "decoding" refers only to differential decoding, and in other embodiments "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process generally will be clear based on the context of the specific description and is believed to be well understood by one of ordinary skill in the art.

[0146] Various implementations include encoding. Similar to the discussion above regarding "decoding," "encoding," as used herein, may encompass all or part of the processes performed on an input video sequence, for example, to result in a coded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential coding, transforming, quantizing, and entropy coding. In various embodiments, such processes also or instead include processes performed by the encoders of various implementations described herein.

[0147] As a further example, in some embodiments, "encoding" refers only to entropy encoding, in other embodiments "encoding" refers only to differential encoding, and in other embodiments "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a broader encoding process generally will be clear based on the context of the specific description and is believed to be well understood by one of ordinary skill in the art.

[0148] It should be noted that the syntax elements used herein are descriptive terms, so they do not preclude the use of other syntax element names.

[0149] Where a drawing is shown as a flow diagram, it should be understood that the drawing also provides a block diagram of the corresponding device. Similarly, where a drawing is shown as a block diagram, it should be understood that the drawing also provides a flow diagram of the corresponding method / process.

[0150] Various embodiments may refer to parametric models or rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often given a computational complexity constraint. This may be measured by a rate-distortion optimization (RDO) metric, or by least mean squares (LMS), mean absolute error (MAE), or other such measures. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, these approaches may involve a thorough evaluation of the coding cost and the associated distortion of the reconstructed signal after coding and decoding, and may be based on extensive testing of all coding options, including all modes or coding parameter values ​​considered. Faster approaches can also be used to reduce coding complexity, particularly by calculating approximate distortion based on a predicted or predicted residual signal rather than the reconstructed one. A mixture of these two approaches can also be used, such as by using approximate distortion for only some of the possible coding options and full distortion for other coding options. Other approaches evaluate only a subset of the possible encoding choices. More generally, many approaches use any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and the associated distortion.

[0151] Implementations and aspects described herein may be implemented by, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single type of implementation (e.g., only discussed as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented by, for example, appropriate hardware, software, and firmware. A method may be implemented by a processor, which generally refers to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0152] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variants thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation," as well as any other variants, in various places throughout this application do not necessarily all refer to the same embodiment.

[0153] Additionally, this application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0154] Additionally, this application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, replicating information, calculating information, determining information, predicting information, or estimating information.

[0155] Additionally, this application may refer to "receiving" various pieces of information. Receiving is intended to be broad, similar to "accessing." Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves some form of operation, such as, for example, storing information, processing information, transmitting information, moving information, duplicating information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0156] For example, the use of " / ", "and / or", and "at least one of" in the cases of "A / B", "A and / or B", and "at least one of A and B" should be understood to be intended to encompass selecting only the first listed (A) option, or selecting only the second listed (B) option, or selecting both (A and B) options. As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to encompass selecting only the first listed (A) option, or selecting only the second listed (B) option, or selecting only the third listed (C) option, or selecting only the first and second listed options (A and B), or selecting only the first and third listed options (A and C), or selecting only the second and third listed options (B and C), or selecting all three options (A, B, and C). As one skilled in the art would readily appreciate, this representation can be expanded to include as many items as are listed.

[0157] Furthermore, as used herein, the term "signaling" refers, among other things, to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular one of multiple transforms, coding modes, or flags. In this way, the same transform, parameter, or mode is used at both the encoder and decoder sides in one embodiment. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling), allowing the decoder to use the same particular parameter. Conversely, if the decoder already has that particular parameter along with other parameters, signaling can be used without transmission simply to allow the decoder to know and select that particular parameter (implicit signaling). By avoiding transmitting any actual functionality, bit savings are realized in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. While the above content relates to the verb form of the word "signal," the word "signal" can also be used as a noun herein.

[0158] As will be apparent to those skilled in the art, implementations can result in a wide variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data produced by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a wide variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0159] The preceding section describes several embodiments across various claim categories and types. Features of these embodiments may be provided alone or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in any combination:

[0160] At least one embodiment includes running at least a portion of a deep neural network up to a particular split point that indicates a layer of the DNN.

[0161] At least one embodiment includes transmitting the intermediate data, the DNN model, and the segmentation point information from the first device to the second device.

[0162] At least one embodiment includes receiving intermediate data, a DNN model, and split point information from a first device for execution of a portion of the DNN model on a second device.

[0163] At least one embodiment includes a decision model for determining at least one split point that represents a layer of a DNN model that runs on various devices.

[0164] At least one embodiment includes a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0165] At least one embodiment includes a bitstream or signal including syntax conveying information generated according to any of the described embodiments.

[0166] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0167] At least one embodiment involves parsing the video data or bitstream to determine the operating point of the codec.

[0168] At least one embodiment includes a method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments.

[0169] At least one embodiment involves inserting into a signaling syntax element that allows the decoder to determine the decoding information in a manner corresponding to that used by the encoder.

[0170] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0171] At least one embodiment includes a TV, set-top box, cell phone, tablet, or other electronic device that performs a conversion method according to any of the described embodiments.

[0172] At least one embodiment includes a TV, set-top box, cell phone, tablet, or other electronic device that performs the conversion method determination according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0173] At least one embodiment includes a TV, set-top box, cell phone, tablet, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal containing encoded images according to any of the described embodiments and performs the conversion method.

[0174] At least one embodiment includes a TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing the encoded image wirelessly (e.g., using an antenna) and performs the conversion method.

Claims

1. Executing at least a portion of a deep neural network (DNN) up to a particular layer on a first device; transmitting intermediate data from the first device to a second device; running the remaining layers of the DNN on the second device; and transmitting information from the second device to the first device; A method comprising:

2. Executing at least a portion of a deep neural network (DNN) up to a particular layer on a first device; transmitting intermediate data from the first device to a second device; running the remaining layers of the DNN on the second device; and transmitting information from the second device to the first device; The device is configured to:

3. A determination module for determining at least one split point indicative of a particular layer of a deep neural network model for execution across multiple devices.

4. 1. An apparatus configured to receive intermediate data from another apparatus and execute a portion of a deep neural network model, the portion being indicated by at least one split point input.

5. The method of claim 1 or the apparatus of claim 2, wherein the DNN model is trained before execution.

6. The method of claim 1 or 5, or the apparatus of claim 2 or 5, wherein the DNN model is trained on the fly.

7. 7. The method of claim 1 or any one of claims 5-6, or the apparatus of claim 2 or any one of claims 5-6, wherein learning is deployed within the first device or the second device.

8. 8. The method of claim 1 or any one of claims 5 to 7, or the device of claim 2 or any one of claims 5 to 7, wherein the supervised learning is based on ground truth.

9. 9. A method according to claim 1 or any one of claims 5 to 8, or an apparatus according to claim 2 or any one of claims 5 to 8, wherein the model is operated by an application that supplies data to the model.

10. 10. The method of claim 1 or any one of claims 5 to 9, or the apparatus of claim 2 or any one of claims 5 to 9, wherein metrics are collected synchronously or asynchronously.

11. 11. A method according to claim 1 or any one of claims 5 to 10, or an apparatus according to claim 2 or any one of claims 5 to 10, using reinforcement learning for a decision module involving rewards in one device.

12. 10. A non-transitory computer-readable medium containing data content generated according to the method of claim 1 for playback using a processor.

13. 10. A signal comprising video data produced according to the method of claim 1 for playback using a processor.

14. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 1 or any one of claims 3 to 11.

15. 11. A non-transitory computer readable medium comprising data content including instructions for carrying out the method of any one of claims 1 or 3 and 5 to 10.

Images