Method for training decision module model in break point range architecture

Abstract

A decision module determines segmentation points for a deep neural network model. An apparatus implements a DNN model up to a break point. The first device transmits the intermediate data to a second device that implements a remainder of the DNN model after the break point. The second device sends the further data back to the first device. In one embodiment, the first device and the second device are a user equipment and an edge server. The DNN model can be learned before execution and can also be learned through execution. And asynchronous or synchronous index collection is realized.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European serial number 23315025.9, filed on February 10, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] At least one of the present embodiments generally relates to a method or apparatus for implementing an efficient decision module in a split point range architecture. Background Art

[0004] Splitting computing or distributed inference aims to achieve several goals:

[0005] Computational load is distributed between mobile devices and edge / cloud servers. Energy consumption is shared between the mobile device and remote servers. Privacy is protected by transmitting partially processed data rather than raw data. User-specific tasks can be processed on the mobile device, while general tasks can be completed on the server.

[0006] First, the DNN model is divided into multiple parts according to the current system environment factors, such as network bandwidth, device resources (memory, power, processing unit), and edge server workload.

[0007] The UE device executes the DNN model up to a specific layer and sends the intermediate data to the edge server. The edge server will execute the remaining layers and send the prediction results back to the device. Conversely, the server can execute the first few layers and then send the intermediate data to the mobile device to process the remaining layers. Summary of the Invention

[0008] At least one of the present embodiments generally relates to a method or apparatus in the context of a split point range architecture. Specifically, one object of the embodiments is to enable intelligent and automatic change of split points, thereby enabling intelligent and automatic change of sub-parts running on UE and network parts.

[0009] According to a first aspect, a method is provided. The method includes the following steps: executing at least a portion of a deep neural network (DNN) up to a specific layer on a first device; sending intermediate data from the first device to a second device; executing the remaining layers of the DNN on the second device; and sending information from the second device to the first device.

[0010] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to implement the general aspects by executing any of the described methods.

[0011] According to another aspect, an apparatus is provided that is configured to: execute at least a portion of a deep neural network (DNN) up to a particular layer on a first device; send intermediate data from the first device to a second device; execute remaining layers of the DNN on the second device; and send information from the second device to the first device.

[0012] According to another aspect, a decision module is provided for determining at least one split point indicating a specific layer of a deep neural network model for execution between a plurality of devices.

[0013] According to another aspect, an apparatus is provided that is configured to receive intermediate data from another apparatus and execute a portion of a deep neural network model, the portion indicated by at least one segmentation point input.

[0014] According to another general aspect of at least one embodiment, there is provided an apparatus comprising the device of any of the embodiments; and at least one of: (i) an antenna configured to receive a signal, the signal comprising a video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, or (ii) a display configured to display an output representing the video block.

[0015] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium containing data content generated according to any of the described encoding embodiments or variations is provided.

[0016] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variations.

[0017] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0018] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions that, when the program is executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0019] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings.

[0020] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium containing data content including instructions for performing any of the encoding or decoding methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 AI / ML model split is shown, where the UE runs the first part of the model (left) and the edge / cloud server runs the first part of the model (right).

[0022] Figure 2 A block diagram of the local decision module is shown.

[0023] Figure 3 An example of decision module I / O is shown.

[0024] Figure 4 An overview of reinforcement learning is shown.

[0025] Figure 5 An example of a reinforcement learning based decision module is shown - evaluated on the UE side.

[0026] Figure 6 An example of a variation of the decision module based on reinforcement learning is shown - evaluation is performed on the UE side.

[0027] Figure 7 An example of a reinforcement learning based decision module is shown - evaluated on the server side.

[0028] Figure 8 An example of decision module model training is shown.

[0029] Figure 9 One embodiment of context indicator transmission is shown: UE data source.

[0030] Figure 10 One embodiment of contextual indicator transmission is shown: Network data source.

[0031] Figure 11 One embodiment of context indicator transmission is shown: UE data source is first processed in the network.

[0032] Figure 12 An embodiment of a 3GPP sequence flow chart - synchronization exchange is shown.

[0033] Figure 13 An embodiment of a 3GPP sequence flow chart is shown - asynchronous switching, machine learning solution.

[0034] Figure 14An embodiment of a 3GPP sequence flow chart is shown - asynchronous exchange, reinforcement learning, and UE-side reward solution.

[0035] Figure 15 An embodiment of a 3GPP sequence flow chart is shown - asynchronous exchange, reinforcement learning, and network-side reward solution.

[0036] Figure 16 One embodiment of a method for executing a DNN model using the described embodiments is shown.

[0037] Figure 17 Another embodiment of a method of executing at least a portion of a DNN model at a device using the described embodiments is shown.

[0038] Figure 18 One embodiment of a device that uses the described embodiments to execute at least a portion of a DNN model is shown.

[0039] Figure 19 A standard general video compression scheme is shown.

[0040] Figure 20 A standard general video decompression scheme is shown.

[0041] Figure 21 A processor-based system for encoding / decoding according to the generally described aspects is shown. DETAILED DESCRIPTION

[0042] Distributed reasoning consists in splitting the DNN model into at least two sub-parts. The sub-parts are distributed across network infrastructure nodes, which are UEs, edge servers, and cloud servers, such as Figure 1 shown.

[0043] If the system architecture is defined so that the split point can be changed intelligently and automatically, the sub-parts running on the UE and the network part can be changed intelligently and automatically.

[0044] In addition, messages between the UE and the network to support this change are defined.

[0045] In the defined architecture, the decision to change the split point relies on a module called the "local decision module." However, this "local decision module" is described as a black box: its inputs and outputs are described, but no implementation details are provided, as there are several ways to communicate the split point change between the UE and the network.

[0046] The following examples are directed to, for example, Figure 2 The implementation method of the local decision module of the architecture.

[0047] To achieve intelligent decision-making through the local decision-making module, the following technical problems need to be solved.

[0048] The number of parameters influencing the decision can be very large and fluctuate over time: solutions based on static rules established by experts can be complex to set up and produce inefficient results once the system encounters new circumstances related to UE resources, input data sources, or network resources.

[0049] The number of models to manage can be very large, making the task of defining rules by experts too onerous.

[0050] There are many optimization options: the goal can be to optimize latency, energy consumption on the UE side, energy consumption on the network side, or energy consumption on both ends.

[0051] The local decision module must consider the effective quality of its response and be able to improve over time.

[0052] The local decision module may encounter new models or new parameter values ​​that it has never seen before.

[0053] The solution relies on the following ideas:

[0054] Use real-time learning (reinforcement learning) or offline learning (machine learning) to implement efficient decision modules in the split-point range architecture.

[0055] It proposes the creation of metrics to enable the training of the aforementioned machine learning and reinforcement learning solutions.

[0056] Two ways of exchanging these indicators are proposed: synchronously and asynchronously.

[0057] The decision module accepts input:

[0058] The AI / ML model to be used by the application.

[0059] A set of hypothetical / possible split points for the AI / ML model.

[0060] UE local information, from which the first part of the model will be inferred (device 1, e.g., UE):

[0061] o Available memory

[0062] οCPU / GPU / TPU (processing unit) used

[0063] Edge / network local information, from which the second part of the module will be inferred (device 2, such as edge server):

[0064] o Available memory

[0065] οCPU / GPU / TPU (processing unit) used

[0066] Input data characteristics

[0067] οResolution (width, height)

[0068] o Size (8-bit, 16-bit, 32-bit, ...)

[0069] οFrame rate (25FPS, 30FPS...)

[0070] Optimize selection:

[0071] o Minimize UE energy cost

[0072] οMinimize network energy costs

[0073] ο Minimize total energy costs

[0074] Minimize latency

[0075] οMaximize accuracy

[0076] οOptimal tradeoff between energy cost / delay / forecast accuracy

[0077] The output is the optimal split point and / or the expected performance for each split point:

[0078] Expected delays

[0079] UE expected energy cost

[0080] Expected energy cost of the network

[0081] Total expected energy cost

[0082] Accuracy of AI / ML model prediction results

[0083] This is a problem that can be solved by a machine learning algorithm. The machine learning algorithm for this problem needs to consider two important characteristics: 1) when the learning is done, real-time learning or offline learning; and 2) the type of learning used, supervised learning or reinforcement learning.

[0084] Real-time or offline learning

[0085] Offline learning and real-time learning are not mutually exclusive; they can depend on the application provider or the network operator.

[0086] In the first case, it is an AI / ML model that is pre-learned and deployed, and in the other case it is an algorithm that learns by experiencing the environment through trial and error.

[0087] In the first case, it is an algorithm that gives relevant results immediately; in the other case, it is an algorithm that takes time to learn.

[0088] In the first case, it’s an algorithm that can’t adapt without updates; in the other case, it’s an algorithm that continuously learns. Even for real-time learning, the initial model is usually trained offline, then deployed, and finally trained in real time.

[0089] Reinforcement learning or supervised learning

[0090] Reinforcement learning and supervised learning are two machine learning techniques contemplated by this disclosure. Both can be trained in real time or offline.

[0091] Therefore, unsupervised learning is not within or beyond the scope of this disclosure, and in fact, the goal of the decision module is not to cluster the data or find any data categories, but to make decisions based on a set of input variables (described below).

[0092] In both cases, the learning algorithm can be deployed on the UE or on the network.

[0093] Supervised learning is based on a ground truth. The learning phase involves mapping the input sample data to this ground truth. The result of this learning phase is a model. This model can be based on a neural network architecture or a more classic machine learning algorithm, such as a decision tree or random forest. Once deployed on a device (e.g., a UE), the model can be operated via an application. The application feeds the inferred sample data to the AI / ML model, which then delivers a prediction. Therefore, a decision module embedded in such a model can deliver a predicted split point, i.e., the split point that best matches the input sample data.

[0094] Furthermore, deep reinforcement learning is well-suited to personalized split point selection. The choice of split point can vary from user to user, location to location, period to period, and so on. If the action at a split point results in a good delay (for a given state), a reward can be applied. This mechanism can be repeated for energy.

[0095] Rewards will depend on application:

[0096] Online gaming will require very low latency.

[0097] Applications that rely on file transfers will require high bandwidth.

[0098] Some services require constant bandwidth even if the bandwidth is low (ie, no jitter).

[0099] Real-time reinforcement learning variants

[0100] The principle of reinforcement learning can be Figure 4shown.

[0101] One difference compared to other systems is that reinforcement learning evaluates the system's state at each action and calculates a reward that reflects the quality of the current situation relative to the ultimate goal. This reward is used by the agent's training algorithm: the agent attempts to maximize future rewards. Therefore, a quantitative assessment of the system's performance based on the agent's choices is required to calculate this reward. How this reward is calculated based on a performance metric is a design decision. Several possibilities are described in the aspects described.

[0102] In the context of this disclosure, the agent corresponds to the main function of the local decision-making module. In addition, this reward must also be based on the user's preferences to reflect their intentions.

[0103] The reward is tied to the system's objectives, as it is a function of the agent's effort maximization. Therefore, it's logical to construct this reward function through a set of metrics that quantify system performance. These metrics can be a combination of UI contextual metrics and network contextual metrics, and must be evaluated against the system's optimization objective. Based on the optimization objectives listed above, the contextual metrics could be: UE energy cost, network energy cost, latency, and accuracy.

[0104] To account for different optimization goals, rewards can balance different metrics. For example, rewards can be:

[0105] R = -α1UE_energy_cost – α2network_energy_cost – α3delay + α4accuracy

[0106] Among them, each α i is 0 or a positive real number, and quantifies the importance of each optimization objective.

[0107] The agent's training algorithm trains a decision function. The input to this decision function is called the state, and it is computed from the set of inputs to the local decision module defined above. For example, the state can be the concatenation of all inputs, a subset of these inputs, or any transformation of these inputs or input subsets, which can include normalization, scaling, outlier removal, and / or neural network processing. The state can also consist of the concatenation or aggregation of past inputs to the decision module.

[0108] The output of the decision function is the agent’s output: the action, which is the split point for the AI / ML model used for segmented inference.

[0109] The agent can use any existing reinforcement learning algorithm, such as Q-learning, SARSA, UCB, Deep Q-learning, PPO, or RAINBOW.

[0110] The evaluation of the system can be hosted on the UE side or the server side.

[0111] When evaluating the system at the UE end, the decision module can be as follows: Figure 5 As shown in the figure. In this embodiment, the reward is calculated by a separate module called "System Evaluation". It takes as input the values ​​required to optimize the selection and calculate the reward. These values ​​come from the UE device or the network and are represented by the term "contextual indicators" in the figure. Its output is the reward. In addition to all the inputs listed above, this reward is input to the local decision module where the agent resides. The output of the decision module is the split point (=action), which is used by the AI / ML model partitioning module.

[0112] Figure 6 Different embodiments are shown. If the system evaluation is performed on the UE side, the system evaluation (= reward calculation) can also be located in the local decision module itself. Therefore, the input of the module that evaluates the system is now also the input of the decision module, and the reward does not leave the decision module. These two embodiments have different advantages. When the reward module is located outside the decision module, the two modules can be developed independently by different companies, making the system more modular. When the system evaluation is located within the agent, the system architecture can also be used for supervised learning, making the system architecture more versatile.

[0113] The evaluation of the system can also be located on the network side. Figure 7 Such a possible embodiment is shown.

[0114] The advantage of this solution is that it can benefit from the large amount of resources available on the network side (compared to the UE), thereby generalizing the evaluation based on information from multiple UEs. In addition to moving the evaluation of the system module from the device to the network, there is another difference in the information sent by the UE to the network: it must contain optimization options so that it can be used by the environment evaluation module. Sending network environment indicators to the device is optional, but can be useful for training the decision module. In the case of a systematic evaluation on the network side, the decision module, such as Figure 7 shown.

[0115] Offline supervised training - optimal segmentation

[0116] The input and output of training are the same as defined above.

[0117] The objectives of the loss function during training are to minimize expected latency, minimize energy cost, and maximize prediction accuracy.

[0118] The formula for expressing the best trade-off between energy cost / delay / prediction accuracy can be:

[0119] We call BC the "Best Compromise":

[0120] BC = α1UE_energy_cost + α2network_energy_cost + α3delay – α4accuracy,

[0121] in

[0122] Parameter α i It can be fixed or learned.

[0123] Training is performed on the server side by a dedicated module "Model Training".

[0124] This module can be trained using data collected by the module "Global Metrics Manager" (see next item).

[0125] Model training can be triggered by different thresholds, such as time (model training can be activated periodically), metric size / number (the size or number of newly collected contextual metrics that have not yet been integrated into the decision module model), and format changes (updates to the input data format or output data format).

[0126] Once the model training is completed, it will be communicated to the UE through appropriate means.

[0127] The timing of sending to communicate the model can be a network operator decision, triggered by certain thresholds, such as the percentage of improvement of the new model or the number of updates since the last communication. The model can also be communicated on UE request.

[0128] Data Collection

[0129] To train the model efficiently, latency and energy costs are collected from both UE and network ends, which can be used to refine the model once it is deployed.

[0130] The collection of this information can be done synchronously or asynchronously.

[0131] To synchronize the collection, two messages exchanged between the UE and the network need to be modified to also carry “context indicators.” First, a message from the UE to the network already carries the used split point and intermediate data, and second, a message from the network to the UE carries the inference result.

[0132] On the first message (UE to network), these contextual metrics are: processing delay of segmentation model M1, energy used for processing of segmentation model M1, battery level of the UE, memory used at UE level, CPU load at the start of processing and sampled periodically (i.e. every millisecond) throughout the processing.

[0133] For the second message (network to UE), these context metrics are: processing latency of segmentation model M2, energy used for processing segmentation model M2, memory used at the network level, or CPU load sampled at the start of processing and periodically (i.e., every millisecond) throughout the processing. For asynchronous collection, the context metrics are the same as those described in the two messages above, but this collection relies on a different mechanism.

[0134] In each message exchanged between the UE and the network (message "Split Point + Intermediate Data"), context indicators are stored in a local database, which is hosted by the module "Local Indicator Manager".

[0135] At regular intervals, some or all indicators are sent to the module "Global Indicators Manager" hosted in the web side.

[0136] At regular intervals or triggered by a threshold (see above), the "Global Metrics Manager" can train a model "Local Decision Model".

[0137] The two methods above are among the best solutions for building a split point decision module. Of course, other options are possible. Some examples are as follows.

[0138] Offline reinforcement learning approaches are possible. Such models can be trained using a database of observations (decision module inputs, split points, and UE / system context metrics), or trained using a simulator or testing, using methods such as bandit algorithms or offline policies or batch reinforcement learning.

[0139] It is possible to use an online supervised learning approach to predict the performance of a given segmentation. This training algorithm will observe data points generated in real time as the system is used and will train or fine-tune the initial model.

[0140] Supervised learning models that directly predict split points cannot be trained in real time, as the optimal split point is typically unknown. Training such models is only possible by sending the model to our servers, which generate training data by extensively testing all split points, or by first training the supervised model described in the previous bullet and using that model to estimate the optimal split point and using that estimate as the label.

[0141] Federated learning (distributed learning across multiple devices) can also be used in conjunction with any real-time method, including reinforcement learning.

[0142] information

[0143] Two types of messages can be used to collect and exchange metrics:

[0144] Synchronous messages based on existing messages between the UE and the server are responsible for communicating the split point and intermediate data ("piggybaking messages"). A section containing indicators is added to these existing messages.

[0145] Asynchronous messaging: Messages carrying metrics can be exchanged between the UE and the server regularly following specific rules.

[0146] Both types of messages rely on new structures that contain indicators.

[0147] Indicator wrapper

[0148] For example, for all combined structures: the sender can be either the UE or the server, and the first part of the model M1 can be run on either the client or the server:

[0149] Struct ContextualMetrics{

[0150] Int Model_RefID;#Model Identifier

[0151] Int SplitPointID;#Split point

[0152] Bit Source; #0=inference on UE; 1=inference on server

[0153] Int SourceID; #ID of UE or ID of Server

[0154] Bit SplitPart;#0=inference on first subset M1;

[0155] #1=inference on second subset M2

[0156] Int InferenceLatencyForSubsetModel; #Inference time of the “splitpart” of the model (ie M1 or M2) on the source (ie UE or server)

[0157] Int EstimatedEnergyCostsForThisInference;#Estimated or measuredenergy consumption for the inference(UE if uplink,Server if downlink)

[0158] Int MemoryAvailableBeforeInference;#UE or server side

[0159] Int MemoryAvailableDuringInference;#UE or server side

[0160] Int CPULoadBeforeInference;#UE or server side

[0161] Int CPULoadDuringInference;#UE or server side

[0162] Int GPULoadBeforeInference;#UE or server side

[0163] Int GPULoadDuringInference;#UE or server side

[0164] Int NumberOfApplicationsRunningBeforeInference;#UE specific

[0165] Int NumberOfApplicationsRunningAfterInference;#UE specific

[0166] Array input_data_characteristics[];#width,height,nb_bits(8 / 10 / 12),frame rate

[0167] Int optimization choice; #optimization choice(0:latency,1:minimize UEenergy,2:minimize network consumption,3:minimize both UE and network energyconsumption,4:best compromise

[0168] }

[0169] Synchronous Messages

[0170] Existing messages from previous methods can be used with the added metrics information.

[0171] Intermediate data transmission

[0172] Intermediate data is encapsulated in structures that indicate the origin of the data.

[0173] The structure is as follows:

[0174] Struct IntermediateDataWrapper{

[0175] Int Model_RefID;#Model Identifier

[0176] bit ChangeOfSplitPoint; #1if the splitpoint is changed from the previous message

[0177] Int SplitPointID; #Split point used by the UE

[0178] Int NextSplitpointID; #Next split point used by the UE(optional)

[0179] Int IntermediateDataLength; #Length of Intermediate Data

[0180] Dim IntermediateDataDim;#Array dimension of intermediate data

[0181] byte EncodingMethod; #Indicate if data are compressed and by which algorithm

[0182] int SequenceNumber;#Sequential identifier of input data

[0183] double TimeStamp;#Timestamp of the intermediate data

[0184] Byte IntermediateData[]#Intermediate Data

[0185] int UEContextualMetricsLength[]#Length of UE contextual Metrics

[0186] ContextualMetrics UEContextualMetricsData[]#UE contextual Metrics

[0187] }

[0188] Asynchronous messaging

[0189] These messages are exchanged between the UE module "Local Metrics Manager" and the server module "Global Metrics Manager" module.

[0190] The structure is as follows:

[0191] Struct AsynchronousContextualMetricsMessages{

[0192] Int UE_id; #UE identifier

[0193] Int Server_id;#Server identifier

[0194] Int metrics_id;#Metrics identifier

[0195] TimeStamp timestamp;#Timestamp of the metrics(date and time)

[0196] ContextualMetrics metrics; #metrics

[0197] }

[0198] Figure 12 The call flow for synchronous indicator exchange based on previous research is shown:

[0199] Figure 12 A sequence diagram according to current 3GPP SA4 AI / ML components is presented.

[0200] The steps under the general aspects described herein are: step 6, which is performed to integrate the exchange of UE context indicators; step 6, which is performed to integrate the exchange of optimization options when using reinforcement learning; step 7, which is performed to integrate the exchange of network context indicators; and step 7, which is performed to integrate the exchange of rewards when using reinforcement learning.

[0201] Steps 1 and 2 involve the AI / ML application requesting a dynamic segmentation configuration scope from the network service. This scope, as described in the previous section, allows for the AI / ML model (e.g., "M") to be dynamically segmented (e.g., in M0 and M1) based on different segmentation points (e.g., A, B, C, D, E), thereby operating partially within the UE inference or network inference subset. For example, segmentation points B and C can be considered at both ends. After steps 1 and 2, the UE inference can dynamically begin processing the captured sequence frame 1 up to layer A, B, or C.

[0202] For step 3, the AI / ML application calculates its internal resources and application requirements, and initializes the AI / ML UE subset containing split points A and B.

[0203] For steps 4 and 5 , the inference engine processes the first sequence of frames 1 up to segmentation point B.

[0204] In step 6, when executing frame 1, the UE delivers the output data to the network inference through the intermediate data transmission function. The transmitted data includes a data payload and metadata information that can be used to identify how to process the received data, such as an indication of at least the current segmentation point.

[0205] For these embodiments, the metadata information will also include contextual metric information, as described in the clause regarding metric packaging.

[0206] For these embodiments, when reinforcement learning is used, the metadata information will also include optimization choices, as described in the clause regarding metric packaging.

[0207] Step 7: After continuous results (e.g., streaming overlay of the current input video), the inference engine returns the final result of the AI / ML model. For these embodiments, the metadata information will also include network context metric information, as described in clause 4.6.1 Metric Encapsulation.

[0208] For these embodiments, when reinforcement learning is used and when the evaluation of the system is performed on the network side, the metadata information will also contain contextual metric information, as described in clause 4.6.1 Metric Encapsulation.

[0209] In step 8, according to different internal triggers (eg, reduced processing capability), the UE utilizes flexibility to handle different segmentation points, and starting from sequence frame 2, locally updates the new segmentation point from B to A for the UE to reason about.

[0210] Steps 9, 10, and 11 follow a similar process to steps 4, 5, and 6, except that the UE delivers data at split point A instead of B.

[0211] In step 12, the network inference engine calculates the split point information and accordingly changes the input to split point B. In another embodiment, the UE may trigger the network inference engine upon receiving an AI / ML application update.

[0212] Step 13 is similar to step 7. This illustrates the seamless segmentation model process.

[0213] Asynchronous Exchange

[0214] For asynchronous collection, the context metrics are the same as those described in the two messages above, but this collection relies on a different mechanism.

[0215] In every message exchanged between the UE and the network (messages "split point + intermediate data"), contextual indicators are stored in a local database managed by the module "Local Indicator Manager". In the case of reinforcement learning, the optimization choices are also stored.

[0216] At regular intervals, some or all indicators are sent to the module "Global Indicators Manager" hosted in the web side.

[0217] In the case of machine learning based solutions, at regular intervals or triggered by a certain threshold (see the above clause), a “global metrics manager” is able to train a model “local decision model”.

[0218] In case of reinforcement learning based solutions, at regular intervals or triggered by a certain threshold (see above clause), the “RL Evaluation of the System” module at the network will communicate rewards to the UE agent. Figure 16One embodiment of a method 1600 for executing a learned DNN model is shown in FIG. The method begins at a startup block 1601 and then proceeds to block 1610 to execute at least a portion of a deep neural network (DNN) on a first device, up to a particular layer. Control proceeds from block 1610 to block 1620 to send intermediate data from the first device to a second device. Control proceeds from block 1620 to block 1630 to execute the remaining layers of the DNN on the second device. Control proceeds from block 1630 to block 1640 to send information from the second device to the first device.

[0219] Figure 17 One embodiment of a method 1700 for executing a portion of a DNN model is shown in FIG. The method begins at a startup block 1701 and proceeds to block 1710 to receive intermediate data and split point information. Control proceeds from block 1710 to block 1720 to execute a portion of the DNN model based on the split point information. Control proceeds from block 1720 to block 1730 to send the execution results to another device.

[0220] Figure 18 One embodiment of a device 1000 for performing any of the above methods is shown. The device includes a processor 1010 and may be interconnected to a memory 1020 via at least one port. Both the processor 1010 and the memory 1020 may also have one or more additional interconnects to connect to the outside world.

[0221] Processor 1810 is also configured to insert or receive information in a bitstream or data and execute at least a portion of a DNN model using the above-described method.

[0222] The embodiments described herein include various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to show each feature, are usually described in a manner that may sound restrictive. However, this is for the purpose of describing clarity and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with the aspects described in previous documents.

[0223] The aspects described and contemplated in this application can be implemented in many different forms. Figure 19 、 Figure 20 and Figure 21 Some embodiments are provided, but other embodiments are contemplated and are not intended to be construed as Figure 19 、 Figure 20 and Figure 21The discussion does not limit the breadth of implementations. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the described methods.

[0224] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" or "reconstruction" is used on the decoder side.

[0225] Various methods are described herein, and each method in these methods includes one or more steps or actions for realizing the method.Unless the correct operation of the method requires the specific order of step or action, otherwise the order and / or use of specific steps and / or action can be modified or combined.Additionally, in various embodiments, terms such as "first", "second" can be used to modify elements, parts, steps, operations, etc., such as, for example, "first decoding" and "second decoding".Unless specifically required, otherwise using such terms does not imply that the modified operation is sorted.Therefore, in this example, the first decoding does not need to be performed before the second decoding, but can occur in the time period overlapping with the second decoding, for example, before, during, or in a second decoding.

[0226] The various methods and other aspects described in this application can be used to modify e.g. Figure 19 and Figure 20 Modules of the video encoder 100 and decoder 200 shown, for example, intra prediction, entropy coding and / or decoding modules (160, 360, 145, 330). Furthermore, the present aspects are not limited to VVC or HEVC and can be applied, for example, to other standards and recommendations (whether previously existing or developed in the future), as well as extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0227] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0228] Figure 19 An encoder 100 is shown. Variations of this encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations.

[0229] Before being encoded, a video sequence may undergo pre-encoding processing (101), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0230] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When encoding a unit in intra mode, intra prediction (160) is performed. In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which mode to use for encoding the unit, intra or inter mode, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0231] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0232] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0233] Figure 20 1 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same operations as described above. Figure 19 The encoding process shown is the reverse of the decoding process. The encoder 100 typically also performs video decoding as part of encoding the video data.

[0234] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0235] The decoded picture may also undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0236] Figure 21 A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be embodied as a device including the various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 1000 can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0237] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein to implement, for example, various aspects described in this document. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0238] System 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video, for example, and may include its own processor and memory. Encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both encoding and decoding modules. Alternatively, encoder / decoder module 1030 may be implemented as a separate element of system 1000, or may be incorporated into processor 1010 as a combination of hardware and software known to those skilled in the art.

[0239] Program code to be loaded onto the processor 1010 or encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic processing.

[0240] In some embodiments, memory internal to processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, i.e., a new standard developed by JVET (Joint Video Experts Group)).

[0241] Input to the elements of system 1000 may be provided through various input devices, as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 21 Other examples not shown include composite video.

[0242] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as are known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band, which in some embodiments may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to desired frequency band again and perform frequency selection by filtering, down-conversion.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements and / or add other elements that perform similar or different functions.Adding element can comprise and insert element between existing element, for example, such as inserting amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0243] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It will be appreciated that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 1010, as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 1010, as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030 operating in conjunction with memory and storage elements, to process the data streams as desired for presentation on an output device.

[0244] The various components of system 1000 may be disposed within an integrated housing in which the various components may be interconnected and data may be transmitted between them using suitable connection means (e.g., an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and a printed circuit board).

[0245] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data through communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0246] In various embodiments, a wireless network (such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to the system 1000. The Wi-Fi signal of these embodiments is received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other top communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which delivers data via an HDMI connection of the input block 1130. Still other embodiments use an RF connection of the input block 1130 to provide streaming data to the system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use a wireless network other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0247] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a laptop computer, a cell phone (mobile phone), or another device. The display 1100 can also be integrated with other components (for example, as in a smartphone), or be separate (for example, an external monitor for a laptop computer). In various examples of embodiments, the other peripheral devices 1120 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000.

[0248] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling (such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention). Output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. The display 1100 and speaker 1110 can be integrated into a single unit with other components of the system 1000 in an electronic device (e.g., such as a television). In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (TCon) chip, for example.

[0249] For example, if the RF input portion 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0250] The embodiments may be executed by computer software implemented by the processor 1010 or by hardware or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0251] Various implementations involve decoding. As used herein, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include or alternatively include processes performed by an encoder of the various implementations described herein.

[0252] As another example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the particular description and is believed to be well understood by those skilled in the art.

[0253] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used herein, may include, for example, all or part of the processing performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processing includes one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processing also includes or alternatively includes processes performed by a decoder of the various implementations described herein.

[0254] As another example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of entropy encoding and differential encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0255] It should be noted that the syntactic elements used herein are descriptive terms. Therefore, they do not exclude the use of other syntactic element names.

[0256] When a diagram is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / device.

[0257] Various embodiments may involve parametric models and rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is often considered, often in light of computational complexity constraints. This can be measured using a rate-distortion optimization (RDO) metric or by least mean squares (LMS), mean absolute error (MAE), or other such measures. Rate-distortion optimization is often formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options, including all considered modes or coding parameter values, where their coding costs and associated distortions in the reconstructed signal are fully evaluated after encoding and decoding. Faster approaches can also be used to save coding complexity, particularly by computing approximate distortion based on prediction or prediction residual signals rather than on the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some possible coding options and full distortion for others. Other approaches evaluate only a subset of possible coding options. More generally, many methods employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a comprehensive assessment of the coding cost and associated distortion.

[0258] The implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., a device or program). The device can be implemented, for example, with appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, for example, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0259] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0260] Additionally, this application may refer to “determining” various information. Determining information may include one or more of the following: estimating information, calculating information, predicting information, or retrieving information from a memory.

[0261] Furthermore, this application may refer to "accessing" various information. Accessing information may include one or more of: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0262] Additionally, the present application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include one or more of the following: for example, accessing information or retrieving information (e.g., from a memory device). Furthermore, during an operation, "receiving" is often referred to in one manner or another, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0263] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," any of the following uses of " / ," "and / or," and "at least one of" are intended to include selecting only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selecting only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first and second-listed options (A and B), or only the first and third-listed options (A and C), or only the second and third-listed options (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to any number of items listed.

[0264] Furthermore, as used herein, the word "signaling" specifically refers to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a specific one of multiple transforms, coding modes, or flags. In this way, in embodiments, the same transform, parameters, or mode are used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntactic elements, flags, etc. are used to send information to the corresponding decoder. Although the verb form of the word "signaling" is mentioned above, the word "signal" can also be used as a noun in this article.

[0265] It will be apparent to one of ordinary skill in the art that implementations may generate a variety of signals formatted to carry information that can be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, signals may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0266] The preceding section describes numerous embodiments across various claim categories and types. Features of these embodiments may be provided individually or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim categories and types:

[0267] At least one embodiment includes executing at least a portion of a deep neural network until a specific split point of a DNN layer is indicated.

[0268] At least one embodiment includes sending intermediate data, a DNN model, and segmentation point information from a first device to a second device.

[0269] At least one embodiment includes receiving intermediate data, a DNN model, and split point information from a first device to execute a portion of the DNN model in a second device.

[0270] At least one embodiment includes a decision model for determining at least one split point indicating a layer of a DNN model to be executed on respective devices.

[0271] At least one embodiment includes a bitstream or signal including one or more of the described syntax elements or variations thereof.

[0272] At least one embodiment includes a bitstream or signal including syntax conveying information generated according to any of the described embodiments.

[0273] At least one embodiment comprises creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0274] At least one embodiment includes parsing video data or a bitstream to determine an operating point of a codec.

[0275] At least one embodiment includes a method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0276] At least one embodiment includes inserting a syntax element in the signaling that enables a decoder to determine decoding information in a manner corresponding to that used by an encoder.

[0277] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding a bitstream or signal including one or more of the described syntax elements or variations thereof.

[0278] At least one embodiment includes a TV, set-top box, mobile phone, tablet or other electronic device that performs a conversion method according to any of the described embodiments.

[0279] At least one embodiment includes a TV, set-top box, phone, tablet, or other electronic device that performs a transformation method determination according to any of the described embodiments and displays (eg, using a monitor, screen, or other type of display) the resulting image.

[0280] At least one embodiment includes a TV, set-top box, mobile phone, tablet or other electronic device that selects, band-limits or tunes a channel (e.g., using a tuner) to receive a signal including an encoded image and performs a conversion method according to any of the described embodiments.

[0281] At least one embodiment includes a TV, set-top box, cell phone, tablet, or other electronic device that receives over the air (eg, using an antenna) a signal including an encoded image and performs the transformation method.

Claims

1. A method comprising: executing at least a portion of a deep neural network (DNN) on the first device, up to a specific layer; sending intermediate data from the first device to a second device; executing the remaining layers of the DNN on the second device; as well as, Information is sent from the second device to the first device.

2. A device configured to perform: executing at least a portion of a deep neural network (DNN) on the first device, up to a specific layer; sending intermediate data from the first device to a second device; executing the remaining layers of the DNN on the second device; as well as, Information is sent from the second device to the first device.

3. A decision module for determining at least one split point, the split point indicating a specific layer of a deep neural network model to be executed between multiple devices.

4. A device configured to receive intermediate data from another device and execute a portion of a deep neural network model, the portion being indicated by at least one split point input.

5. The method according to claim 1 or the apparatus according to claim 2, wherein: The DNN model is learned before execution.

6. The method according to any one of claims 1 or 5 or the apparatus according to any one of claims 2 or 5, wherein The DNN model is learned at execution time.

7. A method according to any one of claims 1 or 5 to 6 or an apparatus according to any one of claims 2 or 5 to 6, wherein Learning is deployed in either the first device or the second device.

8. A method according to any one of claims 1 or 5 to 7 or an apparatus according to any one of claims 2 or 5 to 7, wherein Supervised learning is based on ground truth.

9. A method according to any one of claims 1 or 5 to 8 or an apparatus according to any one of claims 2 or 5 to 8, wherein The model is operated via an application that feeds data to the model.

10. The method according to any one of claims 1 or 5 to 9 or the apparatus according to any one of claims 2 or 5 to 9, wherein the metrics are collected synchronously or asynchronously.

11. The method according to any one of claims 1 or 5 to 10 or the apparatus according to any one of claims 2 or 5 to 10, using reinforcement learning for the decision module at one device and with rewards.

12. A non-transitory computer-readable medium comprising data content generated by the method of claim 1 for playback using a processor.

13. A signal comprising video data generated by the method of claim 1 for playback using a processor.

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to claim 1 or any one of claims 3 to 11.

15. A non-transitory computer-readable medium containing data content, the data content comprising instructions for executing the method according to any one of claims 1 or 3, and 5 to 10.

Images