Entropy-based pre-filtering using neural networks for streaming applications

Abstract

The present disclosure relates to entropy-based pre-filtering using neural networks for streaming applications. In various examples, a deep neural network (DNN)-based pre-filter for content streaming applications is used to dynamically adjust scene entropy (e.g., complexity) in response to changing network or system conditions for end-user devices. For example, where network and / or system performance issues or degradations are identified, the DNN can be implemented as a frame pre-filter to reduce the complexity or entropy of frames prior to streaming, allowing the frames to be streamed at a reduced bitrate without requiring changes to resolution. The DNN-based pre-filter can be adjusted to preserve image detail along object, boundary, and / or surface edges so that scene navigation - e.g., by a user participating in an application instance - can be easier, more natural for the user.

Description

Background Technology

[0001] Traditional video game streaming systems employ dynamic resolution changes (e.g., from 1080p to 720p) to cope with changing network conditions. For example, these systems might reduce frame resolution as network bandwidth decreases and increase it as bandwidth increases. To address this, these traditional systems might introduce coded intraframes (“I-frames”) or other intraframe types into the video stream to achieve resolution conversion. However, I-frames only undergo spatial compression, not temporal compression, thus requiring higher bit rates for transmission. Therefore, since dynamic resolution changes may be triggered in response to already strained network conditions, these traditional systems might transmit high-bit-rate I-frames during periods of reduced channel capacity. Consequently, end-user equipment might receive incomplete I-frames (e.g., due to packet loss) and / or complete or incomplete I-frames with higher latency. This can result in degraded image quality of streamed frames and frames being received with higher latency, leading to noticeable lag and impacting the user experience, particularly in high-performance applications such as cloud gaming, virtual reality, augmented reality, and / or mixed reality applications. For example, streams received from these legacy systems may display degraded video streams, where image details are reduced due to packet loss, without regard to the preservation of important scene information. Furthermore, these legacy systems may require reinitializing client-side decoders to support new resolutions, consuming additional resources on the end-user side and further contributing to a degraded quality of the displayed video—for example, through "hanging," "crackling," "failure," or "jittering," where the video may display at a reduced frame rate or with an interrupted frame rhythm. Summary of the Invention

[0002] Embodiments of this disclosure relate to neural network-based pre-filters for content streaming applications. Systems and methods are disclosed that dynamically adapt scene entropy (e.g., complexity) in response to changing network or system conditions to manage streaming bitrates. For example, contrary to conventional systems, network and / or system conditions at the end-user device can be monitored to determine whether frame entropy should be adjusted to reduce the bitrate of streaming video. Under optimal network and / or system conditions, reduction may not be necessary, and frames with full complexity (maximum entropy) can be streamed to the end-user device. However, if network and / or system performance problems or degradation are identified, a deep neural network (DNN) can be implemented as a frame pre-filter to reduce the complexity or entropy of frames before streaming, allowing frames to be streamed at a reduced bitrate without changing the resolution. Therefore, since no change in frame resolution is required, it may not be necessary to insert I-frames or other intra-frame types into the reduced-capacity network channel, thus also avoiding decoder reinitialization at the end-user device. To account for the loss of detail in the pre-filter, the DNN-based pre-filter can be tuned to preserve image details along objects, boundaries, and / or surface edges (which may be visually / perceptually important), so scene navigation (e.g., by users participating in an application instance) may be easier and more natural for the user.

[0003] Furthermore, in the embodiments, different DNNs can be trained for different entropy values ​​to adapt to different network and / or system conditions. For example, as network conditions improve, a DNN can be selected that pre-filters fewer frames than when network conditions are compromised (e.g., resulting in higher entropy pre-filtered frames). In this way, rather than adjusting the frame resolution—a drawback of conventional systems—this system can use intelligently selected DNNs to adjust the filtering amount, which can adjust the bit rate to meet the current network and / or system conditions of the end-user device. Attached Figure Description

[0004] The present disclosure's system and method for a neural network-based pre-filter for content streaming applications are described in detail below with reference to the accompanying drawings, wherein:

[0005] Figures 1A to 1B This is a data flow diagram illustrating an example process for training a deep neural network (DNN) to pre-filter frames, according to some embodiments of the present disclosure;

[0006] Figure 1C This is an example DNN architecture for pre-filtered frames according to some embodiments of this disclosure;

[0007] Figure 2A This is a data flow diagram illustrating an example process for generating an edge map according to some embodiments of the present disclosure;

[0008] Figure 2B The data flow diagram illustrates an example process for temporal filtering of an edge map according to some embodiments of this disclosure;

[0009] Figure 2C This is an example visualization of a framework based on some embodiments of the present disclosure;

[0010] Figure 2D This is a use of some embodiments according to the present disclosure. Figure 2C Example visualization of the depth map generated by the framework;

[0011] Figure 2E This is a use of some embodiments according to the present disclosure. Figure 2C An example visualization of the surface normal map generated by the framework;

[0012] Figure 2F This is a use of some embodiments according to the present disclosure. Figure 2C An example visualization of the edge graph generated by the framework;

[0013] Figure 3 The flowchart illustrates a method for training a DNN to pre-filter frames according to some embodiments of the present disclosure;

[0014] Figure 4 The present disclosure illustrates a data flow diagram of an example process for pre-filtering frames using a DNN, according to some embodiments thereof.

[0015] Figures 5A to 5D This is an example visualization of pre-filtered frames at different entropy levels according to some embodiments of this disclosure;

[0016] Figure 6 The flowchart illustrates a method for pre-filtering frames using a DNN according to some embodiments of this disclosure;

[0017] Figure 7 This is a block diagram of an example content streaming system suitable for implementing some embodiments of the present disclosure;

[0018] Figure 8 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0019] Figure 9 This is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0020] Systems and methods related to neural network-based pre-filters for content streaming applications are disclosed. For example, the systems and methods disclosed herein can be implemented for any application in which frame pre-filtering can be implemented to reduce the bit rate or complexity of frames—e.g., in analog applications, virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) applications, content editing applications, social media applications, remote desktop applications, content streaming applications, game streaming applications, video conferencing applications, and / or similar applications. Therefore, the pre-filters described herein can be used to adjust the frame entropy level of images or videos generated from any source and within any number of different application types.

[0021] The disclosed systems and methods may use at least one deep neural network (DNN) to pre-filter frames (e.g., before compression and transmission to the end-user device). For example, an entropy control parameter (e.g., λ) for filtering may be determined based on the current system and / or network conditions of the end-user device. In some embodiments, the entropy control parameter may be inversely proportional to the entropy level, complexity, or detail required for a given time scenario. Therefore, when system and / or network performance is low, a lower entropy level may be required, and thus the entropy control parameter may be higher, leading to the selection of a DNN-based pre-filter to more thoroughly filter or blur the frames, thereby reducing the bit rate of the stream. Thus, in embodiments, any number of different DNNs may be trained as pre-filters corresponding to the respective entropy control parameters. Network and / or system conditions may be determined, compared with a lookup table, or inserted into an algorithm, and the entropy control parameter may be computed. Once computed, the entropy control parameter can be used to select the appropriate DNN-based pre-filter, and frames may be filtered to generate pre-filtered frames. As network and / or system performance changes during the stream, the DNN-based pre-filter may be modified to account for the changing entropy control parameter. In some embodiments, in addition to having separate DNNs for different entropy control parameter values, or as an alternative to having separate DNNs for different entropy control parameter values, a single DNN can be trained to perform different levels of filtering based on currently determined entropy control parameter values. In such embodiments, the value of the entropy control parameter can be provided to the DNN as a separate input, and this value can be used by the DNN to perform the relevant filtering level.

[0022] During training, one or more DNNs can be trained through an unsupervised learning process involving one or more loss functions. For example, in each iteration, the DNN may receive the current frame and edge maps (such as saliency maps, binary maps, or image and / or similar maps) as input, which are encoded using values ​​indicating the locations of surfaces, objects, and / or boundary edges depicted in the frame. In some examples, depth information and / or surface normal information may be used to compute the edge map for each frame. For example, depth information and / or surface normal information may be maintained by the currently executing application, and this information can be used to generate the edge map or saliency map. The depth information or representation and / or surface normal information or representation can then be used to generate an edge map indicating the edges depicted in the frame. For example, the depth representation and / or normal representation may be scaled and / or normalized, filtered (e.g., using morphological closure, including dilation and erosion), and / or may undergo edge detection operations (e.g., by using an edge detection algorithm, including the Sobel operator). The generated edge map may include a saliency map, a binary map, or an image and / or other edge map representation. In such an example, edge pixels can be encoded with a first value (e.g., 1), and non-edge pixels can be encoded with a second value (e.g., 0) to indicate the position of the edge within the frame. The edge map corresponding to the frame and the frame can then be used as input to the DNN.

[0023] A DNN can compute a filtered image as output and can use one or more loss functions to compare the filtered image (in an unsupervised manner in this embodiment) with frames and / or edge maps. For example, an edge loss function can be used to train a DNN-based pre-filter to maintain edge details between frames of a scene and their corresponding pre-filtered frames, so that pixel values ​​along and / or near identified edges in the frame are maintained, or closely maintained, for clearer navigation through the scene. In this embodiment, the edge loss function can compare pixel values ​​in a frame with pixel values ​​in the pre-filtered frame at pixel locations determined to correspond to edges in the edge map. That is, for a given pixel from the edge map that indicates it corresponds to an edge, the pixel value at that location in the frame and the pixel value at that location in the pre-filtered frame can be compared in such a way that differences are penalized—e.g., a higher loss occurs when the pixel value difference is greater. As another example, an entropy loss function corresponding to reducing or satisfying the frame entropy corresponding to a currently desired entropy control parameter value can be used (e.g., the higher the value of the entropy control parameter, the lower the entropy of the pre-filtered frame, and vice versa). For example, an entropy loss function can measure pixel gradients within a portion of a pre-filtered frame—excluding the portion corresponding to edges identified using an edge map—to reduce gradients between neighboring pixels. In such an example, a higher entropy control parameter value results in more gradient penalty from the entropy loss function. Therefore, when using a high value for the entropy control parameter (e.g., indicating lower frame entropy), gradients between neighboring or surrounding pixels can be reduced, minimizing the difference in pixel values ​​between neighboring or surrounding pixels. Similarly, for a lower entropy control parameter value (e.g., indicating higher frame entropy), gradients between neighboring or surrounding pixels can be reduced to allow for greater pixel differences (but not as much as in a full-entropy frame). If multiple loss functions are used, they may be weighted during training. For example, entropy control parameter values ​​can be used to weight loss functions so that higher entropy control parameter values ​​more strictly enforce consistency between pixel values ​​of neighboring or surrounding pixels, and vice versa.

[0024] The DNN used can include any DNN type—such as a convolutional neural network (CNN)—and can include any architecture type (such as an autoencoder architecture). After training, the selected DNN-based pre-filter can be used to generate pre-filtered frames using the input frames and corresponding edge maps. The computed pre-filtered frames can then be encoded and compressed at a lower bit rate than the input frames (expected to be less than full entropy), and the encoded frames can be transmitted to the end-user device. Therefore, the bit rate of the stream can be reduced, and latency may not be introduced with changes in frame resolution—for example, because the frame resolution remains consistent and only the level of detail within the frame changes.

[0025] refer to Figures 1A to 1B , Figures 1A to 1BThese are example data flow diagrams for frame pre-filtering processes 100A and 100B according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or instead of the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or combined with other components, and in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Processes 100A and 100B can use... Figure 7 Example content streaming system 700, Figure 8 Example computing devices 800 and / or Figure 9 The example data center 900 is implemented with similar features, functions and / or components.

[0026] Figure 1A The process 100A includes an application engine 102, an edge map generator 104, an edge map 106, frame data 108, a DNN 110, pre-filtered frames 112, and a training engine 114. In some embodiments, the application engine 102 may be a streaming application configured to provide image frames, I-frames, code, files, and other data necessary to visualize the application. By way of non-limiting examples, the application engine 102 may execute simulation applications, virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) applications, content editing applications, social media applications, remote desktop applications, content streaming applications, game streaming applications, video conferencing applications, and / or similar applications.

[0027] In operation, process 100A can be implemented to train the DNN 110 through an unsupervised learning process involving one or more loss functions. Initially, at each interval, the application engine 102 can provide frames to the edge map generator 104—for example… Figure 1B The input frame 120 can depict frames from videos, games, applications, etc. The edge map generator 104 can use the depth information and / or surface normal information of the frame to compute the edge map 106. In some embodiments, such as when the depth information and / or surface normal information is not available to the application, the edge map generator 104 can use frame data 108 to generate the edge map 106, for example, by using edge detection on the frame data 108. For example, a sharp contrast between the pixel values ​​of adjacent pixels represented by the frame data can represent an edge. The edge map 106 can include a saliency map, a binary map or image, a grayscale map or image, and / or another edge map representation. Figure 1BAs shown in the example edge diagram 122, edge pixels (e.g., displayed as white pixels) can be encoded with a first value (e.g., 1), while non-edge pixels (e.g., displayed as black pixels) can be encoded with a second value (e.g., 0) to indicate the position of the edge in the frame.

[0028] Depth information and / or surface normal information can be maintained by the application engine 102, and this information can be used to generate the edge map 106. (Briefly transition to...) Figure 2C-2F , Figure 2C-2F Frames 220C / D / E / F of the scene are demonstrated, each depicting a different example visualization of frame information. In some embodiments, frame 220C may depict a full-detail frame and may be processed by edge map generator 104 to determine, extract, and / or identify depth information of frame 220C to generate frame 220D. Frame 220D may describe depth information that may be encoded into frame 220C by application engine 102 and / or by another application or process. The depth information in frame 220D can then be used to generate edge map 220F. For example, edge map generator 104 may use disparity in the depth information (e.g., the difference in depth values ​​between adjacent pixels exceeding a threshold) to identify edges of frame 220C to generate edge map 220F. In some embodiments, frame 220C may be processed by edge map generator 104 to determine, extract, and / or identify surface normal information (e.g., lines, rays, and / or vectors perpendicular to a given surface) from frame 220C to generate frame 220E, which may depict the surface normal information, for example, the surface normal value at each pixel may be encoded into the pixel. The surface normal information can then be used to generate edge map 220F. For example, since it can be assumed that the edges of an object can be formed by the intersection of two or more planes or half-planes, edge map generator 104 may utilize surface normal vector variations exceeding a threshold (e.g., the angle between adjacent vectors) to identify edges of frame 220C to generate edge map 220F.

[0029] In some embodiments, one or more of frames 220D and 220E—for example, corresponding to a depth map and a surface normal map, respectively—can be used to generate the edge map of frame 220F. In other embodiments, frame 220C—without first determining depth and / or surface normal information—can be used to generate the edge map of frame 220F. For example, a change in pixel value exceeding a threshold amount may indicate an edge location, and this information can be used to determine the edges of the edge map. However, using pixel values ​​alone without a depth map and / or surface normal map may result in less accurate results than using a depth map and / or surface normal map.

[0030] In a further embodiment of generating the edge map, for example in Figure 2AAs shown in process 200, depth information and / or normal information can be scaled and / or normalized, filtered (e.g., using morphological closure, including dilation and erosion), and / or can undergo edge detection operations (e.g., by using an edge detection algorithm, including the Sobel operator). Alternatively, or optionally, it can be based on... Figure 2B Process 200B generates a time-filtered edge map. For example, in process 200B, a frame motion vector (e.g., corresponding to the movement of a virtual camera in a virtual environment from time t-1 to time t) can be used to warp or compensate for the motion and generate a warped edge map at t-1. In such an example, the edge map corresponding to time t-1 can be converted to the coordinate system of time t, so that the warped edge map at time t-1 and the edge map at time t can be blended together. In an embodiment, the warped edge map can then be blended with the current edge map according to the following equation (1) to generate a time-stable edge map:

[0031] α*EMAP(t)+(1-α)*warpedEMAP(t-1) (1)

[0032] Where 'a' is a weighting value (e.g., between 0.0 and 1.0, or 0.5 in a non-limiting embodiment), which can be determined empirically to produce the most accurate time-stabilized results.

[0033] Back Figure 1A In some embodiments, edge maps 106 corresponding to frames provided by the application engine and frame data 108 of frames provided by the application engine can be applied to DNN 110 as input. DNN 110 can include any DNN type, such as convolutional neural networks (CNNs), and can include any architecture type, such as autoencoders, encoder / decoders, and / or other architectures. Although described as DNN 110, DNN 110 can include, for example, but not limited to, any type of machine learning model, such as machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long / short-term memory / LSTMs, Hopfield, Boltzmann, deep beliefs, deconvolution, generative adversarial, liquid machines, etc.), computer vision algorithms, and / or other types of machine learning models. Once trained by training engine 114, DNN 110 can be used to generate pre-filtered frames 112 using frame data 108 and edge maps 106.

[0034] According to some embodiments of this disclosure, a brief reversal Figure 1C , Figure 1CThis is an example of a DNN architecture 110A for pre-filtering input frame 120. The DNN may include any number of layers 130. One or more layers 130 may include an input layer. The input layer may hold values ​​associated with input frame 120 (e.g., before or after post-processing). One or more layers 130 may include convolutional layers. The convolutional layer may compute the output of neurons connected to local regions in the input layer, with each neuron computing the dot product of its weights with the small region connected in the input volume. One or more layers 130 may include rectified linear unit (ReLU) layers. The ReLU layers may apply an element-wise activation function, such as max(0, x), for example, thresholding at zero. The resulting ReLU layer may have the same volume as the input volume of the ReLU layer. One or more layers 130 may include pooling layers. The pooling layer may perform downsampling along spatial dimensions (e.g., height and width), which may result in a volume smaller than the input of the pooling layer. One or more layers 130 may include one or more fully connected layers. Each neuron in a fully connected layer may be connected to each neuron in the previous volume. Fully connected layers can compute class scores, and the resulting volume can be 1 x 1 x the number of classes. In some examples, a CNN may include one or more fully connected layers, such that the output of one or more layers 130 of the CNN can be fed as input to the fully connected layers of the CNN. In some examples, one or more convolutional streams may be implemented by a DNN, and some or all of the convolutional streams may include corresponding fully connected layers. In some non-limiting embodiments, the DNN may include a series of convolutional layers and max-pooling layers to facilitate image feature extraction, followed by multi-scale dilated convolutional layers and upsampling layers to facilitate global contextual feature extraction.

[0035] Although the input layer, convolutional layer, pooling layer, ReLU layer, and fully connected layer are discussed here in relation to DNN 110, this is not intended to be limiting. For example, additional or alternative layers, such as normalization layers, SoftMax layers, and / or other layer types, can be used in DNN 110. In embodiments of DNN 110 that include CNNs, different orders and numbers of CNN layers can be used depending on the embodiment. In other words, the order and number of layers 130 of DNN 110 are not limited to any particular architecture.

[0036] Furthermore, some layers 130 may include parameters (e.g., weights and / or biases), such as convolutional and fully connected layers, while others may not, such as ReLU layers and pooling layers. In some examples, the DNN 110 may learn the parameters during training. Further, some layers 130 may include additional hyperparameters (e.g., learning rate, stride, epoch, etc.), such as convolutional, fully connected, and pooling layers, while others may not, such as ReLU layers. Parameters and hyperparameters are not limited and may vary depending on the embodiment.

[0037] In some embodiments, to train the DNN 110, the training engine 114 may employ one or more loss functions, such as... Figure 1B The functions used for unsupervised training block 124. Training engine 114 can also receive and / or access various training datasets from training data repository 126 for training DNN 110. Regarding loss functions, for example, the edge loss function of training engine 114 can be used to train DNN 110 to preserve edge details between the frame output by application engine 102 and pre-filtered frame 112, such that the identified edges in the edge map 106 of the frame and / or the pixel values ​​near the identified edges are maintained, or at least closely maintained, making navigation in the scene depicted by the application corresponding to application engine 102 clearer. For example, comparing input frame 120 with pre-filtered frame 112, input frame 120 includes substantial levels of detail not visible in pre-filtered frame 112. In particular, ground 130A of input frame 120 includes details not visible in ground 130B of pre-filtered frame 112. In an embodiment, the edge loss function can compare the pixel values ​​of frame data 108 with those of pre-filtered frame 112 at a pixel location determined to correspond to edge map 106. That is, for a given pixel from edge map 106 that is indicated to correspond to an edge, the pixel value at that location in frame data 108 and the pixel value at that location from pre-filtered frame 112 can be compared in such a way that the difference is penalized.

[0038] In some embodiments, the edge loss function can be calculated according to the following equation (2):

[0039]

[0040] The pixel positions (i, j) in the edge mask (e.g., 122) are used to compare the corresponding pixel positions (i, j) in the original frame (F) (e.g., frame 120) with the corresponding pixel positions (i, j) in the pre-filtered frame (PF) (128). This process can be repeated for each pixel determined to correspond to an edge pixel in the edge map.

[0041] As another embodiment, the entropy loss function of the training engine 114 can be used, corresponding to reducing or satisfying the frame entropy corresponding to the currently desired entropy control parameter value (e.g., the higher the value of the entropy control parameter, the lower the entropy of the pre-filtered frame, and vice versa). For example, the entropy loss function can measure the pixel gradient within a portion of the pre-filtered frame 112—excluding the portion corresponding to the edges identified using edge map 106—to reduce the gradient between adjacent pixels. In such an example, the higher the value of the entropy control parameter, the more higher gradients are penalized by the entropy loss function. Therefore, when a high value of the entropy control parameter is used (e.g., indicating a lower frame entropy), the gradient between adjacent or surrounding pixels can be reduced, minimizing the difference in pixel values ​​between adjacent or surrounding pixels. Similarly, for a lower value of the entropy control parameter (e.g., indicating a higher frame entropy), the gradient between adjacent or surrounding pixels can be reduced less to allow for greater differences in pixels (but not as much as in a full-entropy frame).

[0042] In some embodiments, the entropy loss function can be calculated according to the following equation (3):

[0043]

[0044] Where (i, j) corresponds to the (x, y) pixel position that is not an edge pixel, m is the pixel distance in the x direction, and n is the pixel distance in the y direction. For example, n can include values ​​of +1 and -1, m can include values ​​of +1 and -1, and one or more combinations of these values ​​are used to identify pixels for comparison with pixel (i, j).

[0045] If more than one loss function is used, the loss functions may be weighted during training. For example, entropy control parameter values ​​can be used to weight the loss functions so that higher entropy control parameter values ​​more strictly enforce consistency between pixel values ​​of adjacent or surrounding pixels, and vice versa. For example, the total loss function can be calculated according to the following equation (4):

[0046]

[0047] Here, λ corresponds to the entropy control parameter. Therefore, in examples with low entropy, the higher the value of λ, the greater the weight of the entropy loss function compared to the marginal loss function. Conversely, in examples with high entropy, the lower the value of λ, the smaller the weight of the entropy loss function compared to the marginal loss function.

[0048] See now Figure 3Each block of method 300 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, different functions can be implemented by a processor executing instructions stored in memory. Method 300 can also be embodied as computer-usable instructions stored on a computer storage medium. To name just a few examples, method 300 can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services) or a plug-in to another product. Furthermore, method 400 is described by way of example with respect to the content item identification system of Figure 1. However, this method 300 can be additionally or alternatively performed by any system or any combination of systems, including but not limited to those described herein.

[0049] Figure 3 This is a flowchart illustrating a method 300 for training a DNN to pre-filter frames, according to some embodiments of the present disclosure. Method 300, at block B302, includes generating a saliency map indicating edges depicted in the frame using at least one surface normal or depth value corresponding to the frame. For example, depth information and / or surface normal information may be maintained by a currently executing application, and this information can be used to generate the edge map or saliency map. In some embodiments, pixel values ​​from frame data may be used in addition to or as an alternative to using surface normal and / or depth values.

[0050] Method 300, at box B304, includes computing a pre-filtered frame using a DNN and at least in part based on data representing the saliency map and the frame. For example, DNN 110 can be used to generate a pre-filtered frame using frame data and an edge map.

[0051] The method 300, at block B306, includes calculating a first loss value using a first loss function based at least in part on comparing a first pixel value of a pre-filtered frame at a pixel location corresponding to an edge determined from the saliency map with a second pixel value of that frame. For example, the edge loss function could compare the pixel value of the frame at a pixel location determined to correspond to an edge in the edge map with the pixel value of the pre-filtered frame. That is, for a given pixel indicated by the edge map to correspond to an edge, the pixel value at that location in the frame and the pixel value at that location in the pre-filtered frame could be compared in such a way that a difference is penalized—for example, a higher loss when the difference in pixel values ​​is greater.

[0052] Method 300, at box B308, includes calculating a second loss value using a second loss function based at least in part on comparing pixel values ​​at approximate locations within a pre-filtered frame. For example, an entropy loss function can measure pixel gradients within a portion of the pre-filtered frame—excluding portions corresponding to edges identified using an edge map—to reduce gradients between adjacent pixels. For instance, for a DNN 110 corresponding to a higher entropy control parameter (and therefore a lower frame entropy), differences between adjacent pixels may be penalized more. As another example, for a DNN 110 corresponding to a lower entropy control parameter (and therefore a higher frame entropy), differences between adjacent pixels may be penalized less—but still penalized to reduce frame entropy from the original frame's frame entropy.

[0053] Method 300, at box B310, includes updating one or more parameters of the DNN based on at least a first loss value and a second loss value. For example, the parameters of the DNN 110 (e.g., weights and / or biases) can be updated using a training engine 114 until an acceptable level of accuracy is achieved. As described herein, in some implementations, the first loss value and the second loss value can be used together—and weighted—to produce a final loss value.

[0054] Turning Figure 4 , Figure 4 This is a data flow diagram according to some embodiments of the present disclosure, illustrating an example process 400 for pre-filtering frames using a DNN. Figure 4 It includes a video game streaming system 422, an application engine 102, an edge graph generator 104, an edge graph 106, frame data 108, a selected DNN 418, a pre-filtered frame 112, an encoder 402, an encoded frame 404, an end-user device 420, a channel condition monitor 414, channel condition data, a DNN selector 412, and a DNN 430.

[0055] End-user device 420 may include smartphones, laptops, tablets, desktop computers, wearable devices, game consoles, virtual reality (VR) or augmented reality (AR) systems (e.g., headphones, computers, game consoles, remote controls, controllers and / or other components), content streaming devices (e.g., NVIDIA SHIELD), smart home devices that may include intelligent personal assistants, and / or another device capable of supporting application streaming.

[0056] End-user equipment 420 may include decoder 406, display 408, and application 410. Although only a few components and / or features of end-user equipment 420 are present... Figure 4As shown, but not for limitation. For example, end-user equipment 420 may include additional or alternative components, such as those described below relative to... Figure 8 The computing device 800 described herein. Application 410 can be any application that can implement frame pre-filtering to reduce the bit rate or complexity of frames—for example, in analog applications, virtual reality (VR), augmented reality (AR) and / or mixed reality (MR) applications, content editing applications, social media applications, remote desktop applications, content streaming applications, game streaming applications, video conferencing applications, and / or similar applications.

[0057] Display 408 may include any type of display capable of displaying application 410 (e.g., light-emitting diode display (LED), organic LED display (OLED), liquid crystal display (LCD), active-matrix OLED display (AMOLED), quantum dot display (QDD), plasma display, LED / LCD display, and / or other types of display). In some examples, display 408 may include more than one display (e.g., a dual-monitor display for computer gaming, a first display for configuring the game, and a virtual reality display for playing the game, etc.). In some examples, display 408 is a touchscreen display, such as the touchscreen of a smartphone, tablet, laptop, etc.

[0058] In operation, end-user equipment 420 can transmit channel conditions (e.g., bandwidth, channel capacity, bit rate, signal-to-noise ratio (SINR), spectral efficiency, and / or additional state information) to a channel condition monitor of video game streaming system 422. Channel condition monitor 414 can process and / or format the transmitted channel conditions to generate channel condition data 416, which can be provided to DNN selector 412. DNN selector 412 can compare channel condition data 416 with channel condition data 416 in a lookup table or insertion algorithm, and entropy control parameters can be calculated by DNN selector 412. Once calculated, DNN selector 412 can access DNN 430 and use the entropy control parameters to select the DNN corresponding to the control parameters associated with the channel condition data 416. It should be noted that DNN 430 and the selected DNN 418 may correspond to... Figures 1A to 1C Instances of trained or deployed DNN 110.

[0059] DNN 430 can store multiple DNNs, each corresponding to a different filtering level—for example, corresponding to different entropy control parameters. Depending on the filtering level, the pre-filtered frame can include more or less visual detail when displayed on the display 408 of the end-user device 420. (Briefly transition to...) Figures 5A to 5DFor example, each of frames 500A / B / C / D corresponds to a different entropy control parameter (e.g., 0.1, 0.5, 1.0, and 3.0, respectively), corresponding to high entropy, medium entropy, low entropy, and very low entropy. Each of frames 500A / B / C / D includes arches 502A / B / C / D. It can be seen that arch 502A includes a significant level of detail, with well-defined and visible bricks. Compared to 502A, arch 502B contains less detail. Arch 502B includes some degree of texture to indicate the bricks, but they are not visually well-defined. Compared to 502B, arch 502C contains less detail. Arch 502C includes some degree of detail, but no bricks are visible. Finally, compared to arch 502C, arch 502D contains even less detail. Arch 502D has no texture and no visible bricks. However, the edges of archway 502D are retained, which allows users to navigate the game corresponding to frames 500A / B / C / D.

[0060] Return to Figure 4 The DNN selector 412 can use entropy control parameters to select a DNN corresponding to the control parameters, for example, the selected DNN 418. The video game streaming system 422 can then provide—as per [the relevant information]—to the selected DNN 418. Figures 1A to 1C The described frame data 108 and edge map 106 are used to generate a pre-filtered frame 112. The pre-filtered frame 112 can then be encoded and compressed by encoder 402 to generate an encoded frame 404. The encoded frame 404 may include a reduced bit rate compared to the input frame 404—where a selected DNN 418 is trained to reduce entropy—and the encoded frame 404 can then be transmitted to application 410 of end-user device 420. Application 410 can use decoder 406 to decode the encoded frame 404 and generate the pre-filtered frame 112 for display via display 408.

[0061] In some embodiments, in addition to having different DNNs 430 for different entropy control parameters or as an alternative to having different DNNs 430 for different entropy control parameters—for example, generating pre-filtered frames 112 with different entropy levels—a single DNN 430 can be trained to use entropy control parameters as input and, in addition to frame data 108 and edge map 106, use entropy control parameters to compute pre-filtered frames 112 according to the desired entropy value.

[0062] See now Figure 6Each block of the method 600 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, different functions can be implemented by a processor executing instructions stored in memory. The method 600 can also be embodied as computer-usable instructions stored on a computer storage medium. To name just a few examples, the method 600 can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services) or a plug-in to another product. Furthermore, by way of example, for... Figure 4 Method 600 is described using a system. However, this method may be additionally or alternatively performed by any system or any combination of systems, including but not limited to those described herein.

[0063] Figure 6 This is a flowchart illustrating a method 600 for pre-filtering frames using a DNN according to some embodiments of the present disclosure. Method 600, at block B602, includes: selecting a DNN from a plurality of deep neural networks (DNNs) based on at least one of network or system conditions that at least partially correspond to an end-user device. For example, when system and / or network performance is low, lower entropy may be required, and therefore the entropy control parameter may be higher; thus, a DNN-based pre-filter is selected to more thoroughly filter or blur the frames to reduce the bit rate of the stream.

[0064] The method 600, at block B604, includes: generating a saliency map indicating edges depicted in the frame using at least one surface normal or depth value corresponding to the frame. For example, depth information and / or surface normal information may be maintained by a currently executing application and may be used by edge map generator 104 to generate an edge map or saliency map. In embodiments, in addition to using depth information and / or surface normal information, or as an alternative to using depth information and / or surface normal information, frame data may be used to compare adjacent pixel values ​​and determine edges from the stark contrast of adjacent pixel values.

[0065] The method 600, at box B606, includes: computing pre-filtered frames using a selected DNN and based at least in part on data representing saliency maps and frames. For example, once trained by the training engine, the selected DNN 418 can be used to generate pre-filtered frames 112 using frame data 108 and edge maps 106.

[0066] Method 600, at block B608, includes transmitting data representing a pre-filtered frame to an end-user equipment. For example, the calculated pre-filtered frame 112 can then be encoded and compressed at a lower bit rate than the input frame (wherein it needs to be less than full entropy), and the encoded frame 404 can be transmitted to the end-user equipment 420. As a result, the entropy of the frame can be adjusted so that the pre-filtered frame can be transmitted at the same frame resolution without having to switch image resolution due to channel conditions. Thus, since the difference between pixel values ​​between frames can be reduced as entropy decreases, the amount of data required to transmit the pre-filtered frame at the frame resolution is reduced compared to the original frame data 108, resulting in reduced latency.

[0067] Example content streaming system

[0068] Now for reference Figure 7 , Figure 7 This is an example system diagram of a content streaming system 700 according to some embodiments of the present disclosure. Figure 7 Including application server 702 (which may include and Figure 8 Example computing device 800 (similar components, features, and / or functions), client device 704 (which may include components, features, and / or functions similar to those of the example computing device 800), and client device 704 (which may include components, features, and / or functions similar to those of the example computing device 800). Figure 8 The example computing device 800 may have similar components, features, and / or functions, and the network 706 may be similar to the network described herein. In some embodiments of this disclosure, system 700 may be implemented. Application sessions may correspond to game streaming applications (e.g., NVIDIA GeForce Now), remote desktop applications, simulation applications (e.g., autonomous or semi-autonomous vehicle simulations), computer-aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types.

[0069] In system 700, for an application session, client device 704 can receive input data only in response to input from an input device, transmit the input data to application server 702, receive encoded display data from application server 702, and display the display data on display 724. Therefore, computationally intensive computation and processing are offloaded to application server 702 (e.g., rendering—especially ray or path tracing—for the graphics output of the application session, is performed by the GPU of the game server 702). In other words, the application session is streamed from application server 702 to client device 704, thereby reducing the graphics processing and rendering requirements of client device 704.

[0070] For example, regarding the instantiation of an application session, client device 704 can display frames of the application session on display 724 based on display data received from application server 702. Client device 704 can receive input from one of the input devices and generate input data in response. Client device 704 can send the input data to application server 702 via communication interface 720 and network 706 (e.g., the Internet), and application server 702 can receive the input data via communication interface 718. CPU can receive the input data, process the input data, and transfer the data to GPU, which enables GPU to generate renderings of the application session. For example, the input data can represent a user character's movement, firing weapons, reloading, passing the ball, turning a vehicle, etc., within a game session of a game application. Rendering component 712 can render the application session (e.g., the result representing the input data), and rendering capture component 714 can capture the rendering of the application session as display data (e.g., image data capturing the rendered frames of the application session). The rendering of an application session may include ray or path tracing lighting and / or shadow effects computed using one or more parallel processing units (e.g., GPUs), which may further utilize one or more dedicated hardware accelerators or processing cores to compute and execute the ray or path tracing techniques performed by application server 702. In some embodiments, one or more virtual machines (VMs)—e.g., including one or more virtual components such as vGPUs, vCPUs, etc.—may be used by application server 702 to support the application session. Encoder 716 can then encode the display data to generate encoded display data, and can send the encoded display data to client device 704 via communication interface 718 through network 706. Client device 704 can receive the encoded display data via communication interface 720, and decoder 722 can decode the encoded display data to generate display data. Client device 704 can then display the display data via display 724.

[0071] Example computing device

[0072] Figure 8This is a block diagram of an example computing device 800 suitable for implementing some embodiments of the present disclosure. The computing device 800 may include an interconnect system 802 directly or indirectly coupled to: a memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply 816, one or more presentation components 818 (e.g., one or more displays), and one or more logic units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 808 may include one or more vGPUs, one or more CPUs 806 may include one or more vCPUs, and / or one or more logic units 820 may include one or more virtual logic units. Accordingly, (one or more) computing devices 800 may include discrete components (e.g., a full GPU dedicated to computing device 800), virtual components (e.g., a portion of the GPU dedicated to computing device 800), or a combination thereof.

[0073] although Figure 8 The various blocks are shown as being connected to lines via interconnect system 802, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 818 (such as a display device) may be considered I / O component 814 (e.g., if the display is a touchscreen). As another example, CPU 806 and / or GPU 808 may include memory (e.g., memory 804 may represent a storage device in addition to the memory of GPU 808, CPU 806, and / or other components). In other words, Figure 8 The computing devices described are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are conceived in… Figure 8 Within the scope of computing devices.

[0074] Interconnect system 802 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 802 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 806 may be directly connected to memory 804. Further, CPU 806 may be directly connected to GPU 808. In cases where there is a direct connection or point-to-point connection between components, interconnect system 802 may include a PCIe link to perform that connection. In these examples, a PCI bus is not required in computing device 800.

[0075] The memory 804 may include any of a variety of computer-readable media. The computer-readable media may be any available medium accessible by the computing device 800. The computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

[0076] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 800. As used herein, computer storage media does not include the signal itself.

[0077] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals (such as carrier waves or other transmission mechanisms) and include any information transmission medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the foregoing should also be included within the scope of computer-readable media.

[0078] CPU 806 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. Each CPU 806 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. CPU 806 may include any type of processor and may include different types of processors depending on the type of computing device 800 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 800, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplemental coprocessors such as math coprocessors, computing device 800 may also include one or more CPUs 806.

[0079] In addition to or replacing CPU 806, one or more GPUs 808 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. One or more GPUs 808 may be integrated GPUs (e.g., having one or more CPUs 806) and / or one or more GPUs 808 may be discrete GPUs. In embodiments, one or more GPUs 808 may be coprocessors of one or more CPUs 806. GPUs 808 may be used by computing device 800 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU 808 may be used for general-purpose computing on a GPU (GPGPU). GPU 808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 808 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 806 via a host interface). GPU 808 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 804. GPU 808 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 808 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0080] In addition to or replacing CPU 806 and / or GPU 808, one or more logic units 820 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 806, one or more GPUs 808, and / or one or more logic units 820 may perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 820 may be part of and / or integrated into one or more CPUs 806 and / or GPUs 808, and / or one or more logic units 820 may be discrete components or otherwise external to CPUs 806 and / or GPUs 808. In embodiments, one or more logic units 820 may be coprocessors of one or more CPUs 806 and / or one or more GPUs 808.

[0081] Examples of logic unit 820 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0082] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 810 may include components and functions that enable communication over any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wireless bandwidth), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet). In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 802 to one or more GPUs 808 (e.g., the memory of one or more GPUs 808).

[0083] I / O port 812 enables computing device 800 to be logically coupled to other devices including I / O component 814, presentation component 818, and / or other components, some of which may be built into (e.g., integrated into) computing device 800. Illustrative I / O component 814 includes microphones, mice, keyboards, joysticks, gamepads, game controllers, disc-type satellite antennas, scanners, printers, wireless devices, etc. I / O component 814 provides a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some instances, the input may be passed to appropriate network elements for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 800 (as described in more detail below). The computing device 800 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for attitude detection and recognition. Additionally, the computing device 800 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 800 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0084] Power supply 816 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 816 may provide power to computing device 800 so that the components of computing device 800 can operate.

[0085] One or more presentation components 818 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 818 may receive data from other components (e.g., GPU 808, CPU 806, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0086] Example Data Center

[0087] Figure 9 An example data center 900 that can be used in at least one embodiment of this disclosure is shown. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and / or an application layer 940.

[0088] like Figure 9As shown, the data center infrastructure layer 910 may include a resource coordinator 912, grouped computing resources 914, and node computing resources (“nodes CRs”) 916(1)-916(N), where “N” represents any integer, a positive integer. In at least one embodiment, nodes CRs 916(1)-916(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and / or cooling modules, etc. In some embodiments, one or more nodes CRs 916(1)-916(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 916(1)-9161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs 916(1)-916(N) may correspond to virtual machines (VMs).

[0089] In at least one embodiment, the grouped computing resources 914 may include separate groups of node CRs 916 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The separate groups of node CRs 916 within the grouped computing resources 914 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of node CRs 916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0090] Resource coordinator 912 may be configured or otherwise control one or more nodes CRs 916(1)-916(N) and / or grouped computing resources 914. In at least one embodiment, resource coordinator 912 may include a Software Design Infrastructure (“SDI”) management entity for data center 900. Resource coordinator 912 may include hardware, software, or some combination thereof.

[0091] In at least one embodiment, such as Figure 9As shown, framework layer 920 may include a job scheduler 932, a configuration manager 934, a resource manager 936, and / or a distributed file system 938. Framework layer 920 may include a framework of software 932 supporting software layer 930 and / or one or more applications 942 of application layer 940. Software 932 or application 942 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 920 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 938 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM (Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 932 may include Spark drivers to facilitate the scheduling of workloads supported by various layers of the data center 900. The configuration manager 934 may be able to configure different layers, such as the software layer 930 and the framework layer 920, which includes Spark and a distributed file system 938 for supporting large-scale data processing. The resource manager 936 may be able to manage cluster or group computing resources mapped to or allocated to support the distributed file system 938 and the job scheduler 932. In at least one embodiment, the cluster or group computing resources may include group computing resources 914 at the data center infrastructure layer 910. The resource manager 936 may coordinate with the resource coordinator 912 to manage these mapped or allocated computing resources.

[0092] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a plurality of portions of nodes CRs916(1)-916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0093] In at least one embodiment, the application 942 included in the application layer 940 may include one or more types of applications used by at least a plurality of portions of nodes CRs916(1)-916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments).

[0094] In at least one embodiment, any of the configuration manager 934, resource manager 936, and resource coordinator 912 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can protect the data center operator of data center 900 from making potentially poor configuration decisions and may prevent underutilized and / or poorly performing portions of the data center.

[0095] According to one or more embodiments described herein, data center 900 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 900. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 900 by using weight parameters calculated through one or more training techniques (such as, but not limited to, those described herein).

[0096] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured to allow users to train or execute information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0097] Example network environment

[0098] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network attached storage (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 8 The implementation is carried out on one or more instances of computing device 800, for example, each device may include similar components, features and / or functions of computing device 800. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 900, an example of which is relative to the data center 900 described herein. Figure 9 To describe in more detail.

[0099] Components of a network environment can communicate with each other via one or more networks, which may be wired, wireless, or both. A network can include multiple networks or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0100] A compatible network environment may include one or more peer-to-peer network environments—in which case the network environment may not include a server—and one or more client-server network environments—in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to (one or more) servers can be implemented on any number of client devices.

[0101] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for software supporting the software layer and / or one or more applications supporting the application layer. The software or application may respectively include web-based service software or applications. In embodiments, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, free and open-source software web application frameworks, such as those used for large-scale data processing (e.g., "big data") using distributed file systems.

[0102] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions may be distributed across multiple locations from a central or core server (e.g., across one or more data centers distributed across states, regions, countries, globally, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0103] (One or more) client devices may be included in this discussion. Figure 8 At least some of the components, features, and functions of the (one or more) example computing device 800 described. By way of example and not limitation, the client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0104] This disclosure can be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network.

[0105] As used herein, the phrase “and / or” relating to two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0106] The subject matter of this disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in combination with other current or future techniques to include combinations of different steps or steps similar to those described in this document. Furthermore, although the terms “step” and / or “box” may be used herein to refer to different elements of the method employed, such terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless and only if the order of individual steps is explicitly described.

Claims

1. At least one processor, comprising: Processing circuitry, used for: The value of the entropy control parameter is selected based on at least one or more network or system conditions associated with the end-user equipment, wherein the corresponding value of the entropy control parameter corresponds to the corresponding entropy level; A saliency map indicating the edges depicted in the frame is generated using at least one of surface normal information corresponding to the frame or depth information corresponding to the frame. Based at least on the selection of the value of the entropy control parameter, the frame is filtered using a deep neural network (DNN) and the saliency map, and using the entropy level corresponding to the value in the respective entropy levels, to produce a pre-filtered frame. as well as The data representing the pre-filtered frame is transmitted to the end user equipment.

2. The at least one processor of claim 1, wherein the saliency map comprises pixel values ​​representing one of a binary image or a grayscale image, a first size of the pixel values ​​corresponding to pixels at edges being distinguishable from a second size of the pixel values ​​not corresponding to pixels at edges, and the DNN filters the frame based at least on the first size and the second size.

3. The at least one processor according to claim 1, further comprising: A processing circuit is configured to encode the pre-filtered frame to generate an encoded frame of the video stream, wherein the data representing the pre-filtered frame is the data representing the encoded frame, and the one or more network or system conditions correspond to the bit rate of the video stream.

4. The at least one processor according to claim 1, further comprising: Processing circuitry, used for: Monitor the one or more network or system conditions associated with the end-user equipment; as well as The DNN is selected from a plurality of DNNs based at least on the value, wherein each DNN corresponds to a corresponding value of the entropy control parameter.

5. The at least one processor according to claim 1, wherein the depth information corresponds to a depth map, the surface normal information corresponds to a surface normal map, and generating the saliency map comprises: At least one of the depth map or the surface normal map is scaled or normalized to generate one or more first intermediate maps; Morphological closure of the one or more first intermediate images to generate one or more second intermediate images; and An edge detection algorithm is performed on one or more second intermediate graphs to generate the saliency graph.

6. The at least one processor according to claim 1, wherein the DNN is used to adjust the filter to the corresponding entropy level based at least on the corresponding value of the entropy control parameter.

7. The at least one processor according to claim 1, wherein the DNN is trained using a first loss function corresponding to maintaining edge details and a second loss function corresponding to the respective entropy level.

8. The at least one processor of claim 1, wherein the DNN is used to filter the pixels using the saliency map at least based on the proximity of the pixels of the frame to the edges.

9. The at least one processor of claim 1, wherein the generation of the saliency map includes performing edge detection on the frame to identify the edge, and encoding a representation of the edge into the saliency map based on the edge detection, wherein the DNN filters the frame based at least on the representation of the edge.

10. The at least one processor according to claim 1, wherein the processor comprises at least one of the following: A system used to perform simulation operations; A system used to perform deep learning operations; Systems implemented using edge devices; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

11. A system for streaming applications, comprising: One or more processing units are used to perform operations including the following: The value of the entropy control parameter is determined based on the analysis of at least one network or system condition associated with an end-user device. A DNN is selected from a plurality of deep neural networks (DNNs) based at least on the following: the DNN includes a corresponding value corresponding to the entropy control parameter; A saliency map indicating the edges depicted in the frame is generated using at least one of the surface normal or depth values ​​corresponding to the frame. The pre-filtered frame is computed using the selected DNN and based at least on data representing the saliency map and the frame; as well as The data representing the pre-filtered frame is transmitted to the end user equipment.

12. The system of claim 11, wherein each of the plurality of DNNs corresponds to a corresponding value of the entropy control parameter.

13. The system of claim 11, wherein the saliency map comprises a binary image, the binary image comprising a first pixel having a first value and a second pixel having a second value, wherein the first pixel corresponds to the edge.

14. The system of claim 11, wherein the operation further comprises: The pre-filtered frame is encoded to generate an encoded frame, wherein the data representing the pre-filtered frame is the same as the data representing the encoded frame.

15. The system of claim 11, wherein each of the plurality of DNNs includes a corresponding value of the entropy control parameter.

16. The system of claim 11, wherein the depth values ​​are encoded in a depth map, the surface normals are encoded in a surface normal map, and generating the saliency map comprises: At least one of the depth map or the surface normal map is scaled or normalized to generate one or more first intermediate maps; Perform morphological closure on the one or more first intermediate graphs to generate one or more second intermediate graphs; and An edge detection algorithm is performed on one or more second intermediate graphs to generate the saliency graph.

17. The system of claim 16, wherein the edge detection algorithm includes the Sobel operator.

18. The system of claim 11, wherein the system comprises at least one of: A system used to perform simulation operations; A system used to perform deep learning operations; Systems implemented using edge devices; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

19. A method for a streaming application, comprising: The value of the entropy control parameter is selected based on at least one or more network or system conditions associated with the end-user equipment, wherein the corresponding value of the entropy control parameter corresponds to the corresponding entropy level; A saliency map indicating the edges depicted in the frame is generated using at least one of surface normal information corresponding to the frame or depth information corresponding to the frame. Based at least on the selection of the value of the entropy control parameter, a deep neural network (DNN) and the saliency map are used, and the frames are filtered using the entropy level corresponding to the value in the respective entropy levels to produce pre-filtered frames; as well as The data representing the pre-filtered frame is transmitted to the end user equipment.

20. The method of claim 19, wherein the saliency map comprises one of a binary image or a grayscale image, and wherein the pixel values ​​of the saliency map are distinguishable between the pixel values ​​of pixels corresponding to edges and the pixel values ​​of pixels not corresponding to edges.

21. The method of claim 19, further comprising encoding the pre-filtered frame to generate an encoded frame, wherein the data representing the pre-filtered frame is data representing the encoded frame.

22. The method of claim 19, further comprising: Monitor one or more network or system conditions associated with the end-user equipment; as well as The DNN is selected from a plurality of DNNs based at least on the value, wherein each DNN corresponds to a corresponding value of the entropy control parameter.

23. The method of claim 19, wherein the depth information corresponds to a depth map, the surface normal information corresponds to a surface normal map, and the generation of the saliency map comprises: At least one of the depth map or the surface normal map is scaled or normalized to generate one or more first intermediate maps; Perform a morphological closing operation on the one or more first intermediate images to generate one or more second intermediate images; and An edge detection algorithm is performed on one or more second intermediate maps to generate the saliency map.

Images